Mellanox -ib驱动安装手册linux
Mellanox CS7510智能交换机说明书
©2018 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.Mellanox provides the world’s first smart switch, enabling in-network computing through the Co-Design Scalable Hierarchical Aggregation and Reduction Protocol (SHARP)™ technology. The CS7510 system provides the highest performing fabric solution in a 16U form factor by delivering 64Tb/s of full bi-directional bandwidth with 400ns port latency.SCALING-OUT DATA CENTERS WITH EXTENDED DATA RATE (EDR) INFINIBANDFaster servers based on PCIe 3.0, combined with high-performance storage and applications that use increasingly complex computations, are causing data bandwidth requirements to spiral upward. As servers are deployed with next generation processors, High-Performance Computing (HPC) environments and Enterprise Data Centers (EDC) will need every last bit of bandwidth delivered with Mellanox’s next generation of EDR InfiniBand high-speed smart switches.SUSTAINED NETWORK PERFORMANCEBuilt with Mellanox’s latest Switch-IB™ InfiniBand switch devices, the CS7510 provides up to 324 100Gb/s full bi-directional bandwidth per port. The CS7510 modular chassis switch provide an excellent price-performance ratio for medium to extremely large size clusters, along with the reliability and manageability expected from a director-class switch.CS7510 is the world’s first smart network switch, designed to enable in-network computing through the Co-Design Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) technology. The Co-Design architecture enables the usage of all active data center devices to accelerate the communications frameworks, resulting in order of magnitude applications performance improvements.WORLD-CLASS DESIGNCS7510 is an elegant director switch designed for performance, serviceability, energy savings and high-availability. The CS7510 comes with highly efficient, 80 gold+ and energy star certified AC power supplies.The leaf, spine blades and management modules, as well as the power supplies and fan units, are all hot-swappable to help eliminate down time.COLLECTIVE COMMUNICATION ACCELERATIONCollective is a term used to describe communication patterns in which all members of a group of communication endpoints participate.324-Port EDR 100Gb/s InfiniBand Smart Director SwitchCS7510 InfiniBand SwitchPRODUCT BRIEFSWITCH SYSTEM†350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: © Copyright 2018. Mellanox Technologies. All rights reserved.Mellanox, Mellanox logo and MLNX-OS are registered trademarks of Mellanox Technologies, Ltd. Mellanox Scalable Hierarchical Aggregation and Reduction Protocol (SHARP), Switch-IB, UFM, and Unified Fabric Manager are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.Mellanox CS7520 InfiniBand Switchpage 2Collectives have implications on overall application performance and scale. CS7510 introduces the Co-Design SHARP technology, which enables the switch to manage collective communications using embedded hardware. Switch-IB 2 improves the performance of selected collective operations by processing the data as it traverses the network, eliminating the need to send data multiple times between endpoints. This decreases the amount of data traversing the network and frees up CPU resources for computation rather than using them to process communication.MANAGEMENTThe CS7510, dual-core x86 CPU, comes with an onboard subnetmanager, enabling simple, out-of-the-box fabric bring-up for up to 2048 nodes. CS7510 switch runs the same MLNX-OS ® software package as Mellanox FDR products to deliver complete chassis management, to manage the firmware, power supplies, fans and ports.Mellanox CS7510–16U modular chassis–36 QSFP28 EDR 100Gb/s InfiniBand ports per dual IC leaf bladeSwitch Specifications–Compliant with IBTA 1.21 and 1.3 –9 virtual lanes:8 data + 1 management –256 to 4Kbyte MTU–4x48K entry linear forwarding databaseManagement Ports–DHCP–Familiar Industry Standard CLI–Management over IPv6 –Management IP –SNMP v1,v2,v3 –Web UIFabric Management–On-board Subnet Manager supporting fabrics of up to 2048 nodes–Unified Fabric Manager™ (UFM™) AgentConnectors and Cabling–QSFP28 connectors–Passive copper or active fiber cables –Optical modulesIndicators–Per port status LED Link, Activity –System status LEDs: System, fans, power supplies –Port Error LED –Unit ID LEDPhysical Characteristics–Dimensions:28’’H x 17.64’’W x 30.3’’D –Weight:Fully populated 275kg (606lb)Power Supply–Hot swappable with N+N redundancy–Input range: 180-265VAC –Frequency:47-63Hz, single phase ACCooling–Hot-swappable fan trays –Front-to-rear air flow –Auto-heat sensing fansPower Consumption–Typical power consumption (fully populated):–Passive cable: 4939W –Active cable: 6543WFEATURESSafety–CB –cTUVus –CE –CUEMC (Emissions)–CE –FCC –VCCI –ICES –RCMOperating Conditions–Operating 0ºC to 40ºC–Non-Operating -40ºC to 70ºC–Humidity: Operating 10% to 85%, non-condensing–Altitude: Operating -60 to 3200mAcoustic–ISO 7779 –ETS 300 753Others–RoHS compliant –1-year warrantyCOMPLIANCETable 1 - CS7510 Series Part Numbers and Descriptions53363PB Rev 2.0。
IB卡使用说明
Mellanox infiniband 卡使用说明：非SR-IOV安装驱动后使用service openibd start来启动IB卡，使用chkconfig openibd on来加入启动。
请不要删除驱动安装包，中间还有几个应用程序需要使用。
IB卡默认支持IPoIB模式，直接在/etc/sysconfig/netwok-scripts/ifcfg-ibx里修改即可(注意#掉MAC地址，IB的MAC地址太长无法识别)，这是redhat系统。
Ubuntu中直接在/etc/netwok/interfaces里输入ib信息即可，配置等同网卡。
connetcx.port_config 改接口类型（未SR-IOV之前）cat /etc/infiniband/openib.conf中的SET_IPOIB_CM=auto 改成connetced才能对插（无IB交换机）互联（单边改就可以），改完service openibd restart，默认值（auto）是datagram IB卡的sr-iov确认bios里打开了VT-X VT-D 及SR-IOV（一般都在CPU设置里，几个选项在一起，有时SR-IOV会在PCI选项里）Vi /boot/grub/grub.conf在kernel后面加上intel_iommu=on重启后生效，可以cat /proc/cmdline确认是否已经生效输入mst start或service mst start打开mst，使用chkconfig mst on将mst加入启动，然后用mst status来检查状态：一般情况下提供了两种方式来打开IB卡对SR-IOV的支持，我们选择最简单的方式，使用驱动程序中带的mlnx_fw_updater.pl来操作，具体为./mlnx_fw_updater.pl –enable-sriov即可。
使用mstflint –d PCI号 dc来检查是否已经打开sr-iov检查[HCA]项目。
mellanox infiniband技术原理
Mellanox Infiniband是一种高速计算机网络技术，主要用于数据中心的互连。
其设计目的是为了提高数据传输的速度和效率，降低延迟，并提供更高的带宽。
Infiniband技术使用点对点通信，可以在服务器、存储设备和网络设备之间提供高带宽、低延迟的数据传输。
Infiniband技术的工作原理如下：
1. 基于通道化：Infiniband使用通道化技术，将数据传输划分为多个通道，每个通道可以独立进行数据传输。
这种方式可以提高带宽利用率，降低冲突和延迟。
2. 基于信用机制：Infiniband使用信用机制来管理数据传输。
发送方在发送数据之前，先发送一个请求，接收方收到请求后，返回一个信用。
发送方在收到信用后，才开始发送数据。
这种方式可以确保数据传输的顺序性和可靠性。
3. 具有流量控制：Infiniband技术具有流量控制功能，可以根据网络状况和接收方的能力来调整数据传输的速度和大小，避免网络拥塞和数据丢失。
4. 支持虚拟化：Infiniband支持虚拟化技术，可以在不同的虚拟机和应用程序之间提供高效的数据传输。
总之，Mellanox Infiniband技术通过通道化、信用机制、流量控制和支持虚拟化等技术，提供了高性能、高可靠性和高带宽的网络连接，适用于数据中心和云计算环境。
Mellanox SwitchX Firmware Release Notes说明书
Mellanox SwitchX® FirmwareRelease NotesRev 9.4.2000Last Modified: November 29, 20169.4.2000Table of ContentsTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Chapter 1 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.1 Supported Systems in 9.4.2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.2 Supported Cables and Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.3 Firmware Upgrade. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4 PRM Revision Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Chapter 2 Changes and Major New Features . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1 Changes in Rev 9.4.2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.2 Changes in Rev 9.4.1000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 Changes in Rev 9.3.8000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.4 Changes in Rev 9.3.6000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.5 Changes in Rev 9.3.4000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.6 Changes in Rev 9.3.2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.7 Changes in Rev 9.3.1200 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.8 Changes in Rev 9.3.0000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.9 Changes in Rev 9.2.8000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.10 Changes in Rev 9.2.6100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.11 Changes in Rev 9.2.4002 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.12 Changes in Rev 9.2.3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.13 Changes in Rev 9.2.0000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.14 Changes in Rev 9.1.7000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.15 Changes in Rev 9.1.5000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.16 Changes in Rev 9.1.3000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.17 Changes in Rev 9.1.2000 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.18 New Features and Changes in Rev 9.1.0000 . . . . . . . . . . . . . . . . . . . . . . 8 Chapter 3 Known Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Chapter 4 Bug Fixes History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101OverviewThese are the release notes for the SwitchX® and SwitchX-2 firmware, 9.4.2000. This firmware complements the SwitchX silicon architecture with a set of advanced features, allowing easy and remote management of the switch.1.1Supported Systems in 9.4.2000This firmware supports the devices and protocols listed in Table 1. For the most updated list of switches supported, visit the firmware download pages via .1.2Supported Cables and ModulesPlease refer to the interconnect overview page on the Mellanox website:/page/cables?mtag=cable_overview1.3Firmware UpgradeFirmware upgrade may be performed directly from any previous version to this version. To upgrade firmware, please refer to the Mellanox Firmware Tools (MFT) package at:/page/management_toolsFirmware 9.4.2000 is compatible with MFT tools v2.7.1 and above. Using older ver-sions of MFT tools causes failure during image creation.Table 1 - Supported PSIDs1.4PRM Revision CompatibilityFirmware 9.4.2000 complies with the following programmer’s reference manual:•Switches Programmer's Reference Manual (PRM), Rev 1.40 or later2Changes and Major New Features2.1Changes in Rev 9.4.2000•N/A2.2Changes in Rev 9.4.1000•Added system MKey support2.3Changes in Rev 9.3.8000•Support FORE EMAD query operation according to section 2.17.5 of the SwitchX® PRM rev 1.05•Added support to read UCD sensor and V in sensing according to section 2.8.5 of the SwitchX® PRM rev 1.05•Added AccessRegisterGMP support for MSPS and fix MSPS firmware configuration2.4Changes in Rev 9.3.6000•Added support for bandwidth monitoring according to Performance histogram in Vendor Specific MAD spec2.5Changes in Rev 9.3.4000•Added support for fan-out-of-range event (FORE) according to PRM section “FORE – Fan Out of Range Event”•Added LLR max retransmission rate as specified in Vendor Specific MAD V1.1, Table 110 – PortLLRStatistics MAD Description2.6Changes in Rev 9.3.2000•Lowered the externally managed switches default fan speed from 80% to 60%•Added congestion control attribute “PORT_VL_XMIT_TIME_CONG” according to3.23 CounterGroupInfo in Vendor Specific MADs version 1.1•Added support for adaptive routing per L4 transport layer2.7Changes in Rev 9.3.1200•Added GA support for MFA1A00-Exxx EDR 100Gb/s AOCs over SwitchX® based systems2.8Changes in Rev 9.3.0000•Added support for SX6710 and SX1710 switch systems•Added support for timestamp in mirrored packages•Added support for MCP1600-E001/MCP1600-E002 EDR cables•Added support for FDR 2.5m (MC2207128-0A2)•Added power optimizations when ports are disabled•Added TuneX support•Bug fixes2.10Changes in Rev 9.2.6100•Added LLR capability indication as part of extended port info MAD according to Ven-dor Specific MADs document version 1.1•3rd party FDR cables are allowed to raise link at FDR (56Gb/s)•Added support for fan and power supply monitor MAD according to section 2.12 in Vendor Specific MADs document version 1.1 and section 2.17 in Mellanox PRM ver-sion 1.0•Improved FDR systems link-up time after reboot2.11Changes in Rev 9.2.4002•Added support for 12-port SX6005 switch system•Added ability to Set Node Description according to errata proposal #9109•Updated SMP and GMP CapabilityMask as appear in VS Mad Specification document version 0.56•Added support for temperature sensor MAD according to section 2.12 in Vendor Spe-cific MADs document version 1.1 and section 2.17 in Mellanox PRM version 1.0•Added support for voltage sensor MAD•Added a port disable power-saving feature•Added support for reading VPD from flash (supported with systems coming out of fac-tory with FW 9.2.4002 or higher only)•Added support for alternate source of the following fiber cable P/Ns: MC2207310-XXX-F-A2, MC2210310-XXX-F-A2, MC2206310-XXX-F-A2 (See PCN MLNX-15-838)•Added support for the MC2207312-XXX 56Gb/s active fiber cables.•Added support for MC2207310-XXX-F-A2, MC2207310-XXX-F-A2, MC2206310-XXX-F-A2 active fiber cables•Added support for MC2207411-SR4 56Gb/s active optical module2.12Changes in Rev 9.2.3000•Improved FDR and FDR10 link stability•Port mirroring bug fixes•Added best-effort mirroring capability•Fixing FDR10 switch link issue discovered in 9.2.0000•Added support for SwitchX®-2 SX6015•Added support for Congestion Control Class•Added support for InfiniBand multi-swid•Enabled MultiCastFDBTop=0xBFFF to discard MC traffic•Added a new INI field for the NodeInfo and DeviceID parameters2.14Changes in Rev 9.1.7000•Enabled LLR by default•Added Mellanox Proprietary Link negotiation (MLPN) support•Added support for MC2207312-XXX FDR optical cables up to 300m•Added support for SwitchX®-2 based FDR and FDR10 edge switch systems•Improved Eye Opener Machine2.15Changes in Rev 9.1.5000•Added InfiniBand Port Mirroring support•Reduced port PLLs jitter•Enabled Gradient Decent (GDS) in MSX6025T and MSX6025F systems•Added optimize SL2VL support•Improved link negotiation process2.16Changes in Rev 9.1.3000•Link negotiation protocols improvements2.17Changes in Rev 9.1.2000•Added PKEY enforcement on external ports2.18New Features and Changes in Rev 9.1.0000•InfiniBand switch supported speed:•FDR v1.3•FDR10 - Mellanox Technologies propriety•QDR/DDR/SDR v1.2.1•36 FDR (56Gb/s) ports in a 1U switch•FDR10 supports 20% more bandwidth over QDR using the same cables/connectors3Known IssuesTable2 describes known issues in this firmware release and possible workarounds.Table 2 - Known Issues4Bug Fixes HistoryTable3 lists the bugs fixed in this release.Table 3 - Bug Fixes History。
Mellanox 红帽企业级 Linux（RHEL）7.5 驱动用户手册说明书
Red Hat Enterprise Linux (RHEL) 7.5 DriverUser ManualMellanox Technologies350 Oakmead Parkway Suite 100Sunnyvale, CA 94085U.S.A.Tel: (408) 970-3400Fax: (408) 970-3403© Copyright 2016. Mellanox Technologies Ltd. All Rights Reserved.Mellanox®, Mellanox logo, Accelio®, BridgeX®, CloudX logo, CompustorX®, Connect-IB®, ConnectX®,CoolBox®, CORE-Direct®, EZchip®, EZchip logo, EZappliance®, EZdesign®, EZdriver®, EZsystem®,GPUDirect®, InfiniHost®, InfiniBridge®, InfiniScale®, Kotura®, Kotura logo, Mellanox CloudRack®, MellanoxCloudXMellanox®, Mellanox Federal Systems®, Mellanox HostDirect®, Mellanox Multi-Host®, Mellanox OpenEthernet®, Mellanox OpenCloud®, Mellanox OpenCloud Logo®, Mellanox PeerDirect®, Mellanox ScalableHPC®,Mellanox StorageX®, Mellanox TuneX®, Mellanox Connect Accelerate Outperform logo, Mellanox Virtual ModularSwitch®, MetroDX®, MetroX®, MLNX-OS®, NP-1c®, NP-2®, NP-3®, Open Ethernet logo, PhyX®, PlatformX®,PSIPHY®, SiPhy®, StoreX®, SwitchX®, Tilera®, Tilera logo, TestX®, TuneX®, The Generation of Open Ethernetlogo, UFM®, Unbreakable Link®, Virtual Protocol Interconnect®, Voltaire® and Voltaire logo are registeredtrademarks of Mellanox Technologies, Ltd.All other trademarks are property of their respective owners.For the most updated list of Mellanox trademarks, visit /page/trademarksNOTE:THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT (S)”) AND ITS RELATEDDOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES “AS -IS” WITH ALL FAULTS OF ANYKIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USETHE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUSTOMER'S MANUFACTURING TEST ENVIRONMENTHAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT(S)AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CANNOT AND DOES NOTGUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY. ANYEXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OFMERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. INNO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT,INDIRECT, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING, BUT NOTLIMITED TO, PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISINGIN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OFTHE POSSIBILITY OF SUCH DAMAGE.Table of ContentsDocument Revision History (5)1Firmware Burning (6)2Port Type Management (7)3Modules Loading and Unloading (8)4Important Packages and Their Installation (9)5SR-IOV Configuration (10)5.1Setting up SR-IOV in ConnectX-3/ConnectX-3 Pro (10)6Default RoCE Mode Setting (12)7PXE over InfiniBand Installation (13)Table 1: Document Revision History (5)Table 1: Document Revision History1 Firmware Burning1.Check the device’s PCI address.lspci | grep MellanoxExample:00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family[ConnectX-3 Pro]2.Identify the adapter card's PSID.# mstflint -d 81:00.0 qImage type: FS2FW Version: 2.36.5000FW Release Date: 26.1.2016Rom Info: type=PXE version=3.4.718 devid=4103Device ID: 4103Description: Node Port1 Port2Sys imageGUIDs: e41d2d0300b3f590 e41d2d0300b3f591 e41d2d0300b3f592e41d2d0300b3f593MACs: e41d2db3f591 e41d2db3f592VSD:PSID: MT_10901110193.Download the firmware BIN file from the Mellanox website that matches your card'sPSID: → Support/Education→Support Downloader4.Burn the firmware.# mstflint -d <lspci-device-id> -i <image-file> b5.Reboot your machine after the firmware burning is completed.2 Port Type ManagementConnectX®-3/ConnectX®-3 Pro/ConnectX®-4/ConnectX®-4 Lx/ConnectX®-5/ConnectX®-5 Ex ports can be individually configured to work as InfiniBand or Ethernetports. By default both ConnectX®-4 VPI ports are initialized as InfiniBand ports. If youwish to change the port type use the mstconfig after the driver is loaded.1.Install mstflint tools.yum install mstflint2.Check the device’s PCI address.lspci | grep MellanoxExample:00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family[ConnectX-3 Pro]e mstconfig to change the link type as desired IB – for InfiniBand, ETH – for Ethernet.mstconfig –d <device pci> s LINK_TYPE_P1/2=<ETH|IB|VPI>Example:# mstconfig -d 00:06.0 s LINK_TYPE_P1=ETHDevice #1:----------Device type: ConnectX3ProPCI device: 00:06.0Configurations: Current NewLINK_TYPE_P1 IB(1) ETH(2)Apply new Configuration? ? (y/n) [n] : yApplying... Done!-I- Please reboot machine to load new configurations.4.Reboot your machine.3 Modules Loading and UnloadingMellanox modules for ConnectX®-2/ConnectX®-3/ConnectX®-3 Pro are:∙mlx4_en, mlx4_core, mlx4_ibMellanox modules for ConnectX®-4/ConnectX®-4 Lx/ ConnectX®-5/ ConnectX®-5 Exare:∙mlx5_core, mlx5_ibIn order to unload the driver, you need to first unload mlx*_en/ mlx*_ib and then themlx*_core module.To load and unload the modules, use the commands below:∙Loading the driver: modprobe <module name># modprobe mlx5_ib∙Unloading the driver: modprobe –r <module name># modprobe –r mlx5_ib4 Important Packages and Their Installationrdma-corerdma-core RDMA core userspace libraries and daemonslibibmad: Low layer InfiniBand diagnostic and management programslibibmad OpenFabrics Alliance InfiniBand MAD libraryopensm: InfiniBand Subnet Manageropensm-libs Libraries used by OpenSM and included utilitiesopensm OpenIB InfiniBand Subnet Manager and management utilitiesIbutils: OpenIB Mellanox InfiniBand Diagnostic Toolsibutils-libs Shared libraries used by ibutils binariesibutils OpenIB Mellanox InfiniBand Diagnostic Toolsinfiniband-diags: OpenFabrics Alliance InfiniBand Diagnostic Toolsinfiniband-diags OpenFabrics Alliance InfiniBand Diagnostic Toolsperftest: IB Performance testsperftest IB Performance Testsmstflint: Mellanox Firmware Burning and Diagnostics Toolsmstflint Mellanox firmware burning toolTo install the packages above run:# yum install libibverbs librdmacm libibcm libibmad libibumad libmlx4libmlx5 opensm ibutils infiniband-diags srptools perftest mstflint rdmacm-utils ibverbs-utils librdmacm-utils -y5 SR-IOV Configuration5.1 Setting up SR-IOV in ConnectX-3/ConnectX-3 Pro1.Install the mstflint tools.# yum install mstflint2.Check the device’s PCI.# lspci | grep MellanoxExample:00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family[ConnectX-3 Pro]3.Check if SR-IOV is enabled in the firmware.mstconfig -d <device pci> qExample:# mstconfig -d 00:06.0 qDevice #1:----------Device type: ConnectX3ProPCI device: 00:06.0Configurations: CurrentSRIOV_EN True(1)NUM_OF_VFS 8LINK_TYPE_P1 ETH(2)LINK_TYPE_P2 IB(1)LOG_BAR_SIZE 3BOOT_PKEY_P1 0BOOT_PKEY_P2 0BOOT_OPTION_ROM_EN_P1 True(1)BOOT_VLAN_EN_P1 False(0)BOOT_RETRY_CNT_P1 0LEGACY_BOOT_PROTOCOL_P1 PXE(1)BOOT_VLAN_P1 1BOOT_OPTION_ROM_EN_P2 True(1)BOOT_VLAN_EN_P2 False(0)BOOT_RETRY_CNT_P2 0LEGACY_BOOT_PROTOCOL_P2 PXE(1)BOOT_VLAN_P2 1IP_VER_P1 IPv4(0)IP_VER_P2 IPv4(04.Enable SR-IOV:mstconfig –d <device pci> s SRIOV_EN=<False|True>5.Configure the needed number of VFsmstconfig –d <device pci> s NUM_OF_VFS=<NUM>NOTE: This file will be generated only if IOMMU is set in the grub.conf file (byadding “intel_iommu=on” to /boo t/grub/grub.conf file).6.[mlx4 devices only] Create/Edit the file /etc/modprobe.d/mlx4.conf:options mlx4_core num_vfs=[needed num of VFs] port_type_array=[1/2 for IB/ETH],[ 1/2 for IB/ETH]Example:options mlx4_core num_vfs=8 port_type_array=1,17.[mlx5 devices only] Write to the sysfs file the number of needed VFs.echo [num_vfs] > sys/class/net/ib2/device/sriov_numvfsExample:# echo 8 > /sys/class/net/ib2/device/sriov_numvfs8.Reboot the driver.9.Load the driver and verify that the VFs were created.# lspci | grep mellanoxExample:00:06.0 Network controller: Mellanox Technologies MT27520 Family[ConnectX-3 Pro]00:06.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]00:06.0 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]For further information, refer to section Setting Up SR-IOV MLNX_OFED User Manual.6 Default RoCE Mode Setting1.Mount the configfs file.# mount -t configfs none /sys/kernel/config2.Create a directory for the mlx4/mlx5 device.# mkdir -p /sys/kernel/config/rdma_cm/mlx4_0/3.Validate what is the used RoCE mode in the default_roce_mode configfs file.# cat /sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_modeIB/RoCE v14.Change the default RoCE mode,∙For RoCE v1: IB/RoCE v1∙For RoCE v2: RoCE v2# echo "RoCE v2" >/sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_mode# cat /sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_modeRoCE v2# echo "IB/RoCE v1" >/sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_mode# cat /sys/kernel/config/rdma_cm/mlx4_0/ports/1/default_roce_modeIB/RoCE v17 PXE over InfiniBand InstallationPXE over InfiniBand infrastructure has additional parameter in the Boot Loader file forloading the necessary modules and interfaces and for allowing sufficient time to get the link.To install RHEL from PXE using the IPoIB interfaces, add the following parameters to theBoot Loader file, located in the var/lib/tftpboot/pxelinux.cfg directory, at thePXE server:bootdev=ib0 ksdevice=ib0 net.ifnames=0 biosdevname=0 rd.neednet=1rd.bootif=0 rd.driver.pre=mlx5_ib,mlx4_ib,ib_ipoib ip=ib0:dhcp.dhcp.retry=10 .timeout.iflink=60 .timeout.ifup=80.timeout.carrier=80Example:default RH7.5prompt 1timeout 600label RH7.5kernelappend bootdev=ib0 ksdevice=ib0 net.ifnames=0 biosdevname=0 rd.neednet=1rd.bootif=0 rd.driver.pre=mlx5_ib,mlx4_ib,ib_ipoib ip=ib0:dhcp.dhcp.retry=10 .timeout.iflink=60 .timeout.ifup=80.timeout.carrier=80。
Mellanox 光纤网卡驱动Driver for Linux的安装
\
英文原厂文档
/related-docs/prod_software/Mellanox_EN_for_Linux_Release_Notes_v3_4-1_0_0_3.pdf
Mellanox ConnectX®-4
Mellanox ConnectX®-4 Lx
Mellanox ConnectX®-3/ConnectX®-3 Pro
Mellanox ConnectX®-2
Mellanox PCI Express 2.0
Mellanox PCI Express 3.0
2
2.1下载驱动的地址比较难找
Mellanox EN Driver for Linux的安装
编者按：Redhat默认支持博科和intel的网卡驱动，对于其他厂商的光纤网卡，或者统后，网卡不能识别工作，安装厂家驱动后正常工作的情况
1
Mellanox EN Driver for Linux支持的网络光纤卡系列有大致以下几种：
2.2Redhat 7系列安装比较麻烦
3
1、在redhat/centos/Oracle linux 6等其他linux上安装驱动
2、在redhat/centos/Oracle linux等其他linux 7上安装驱动
(原程序不支持redhat7，需要重新编译添加支持)
3、驱动下载地址
第一部分
redhat 6.X(在6.6测试过)
yum install kernel-devel-3.10.0-229.el7.x86_64 redhat-rpm-config
---安装网卡驱动
[root@localhost /itsm/mlnx-en-3.4-1.0.0.3-ol7.1-x86_64 ]#./install --add-kernel-support
Mellanox 直接连接铜（DAC）被动电缆组件说明书
Mellanox® direct attach copper (DAC) passive cable assemblies are high speed, cost-effective alternativesto fiber optics in InfiniBand EDR 100Gb/s applications.Based on the QSFP28 electrical connector, 100G EDR DAC cables are configured as 4 channels of 25G NRZ.Each QSFP28 connector includes an EEPROM providing product information which can be read by the hostsystem.Mellanox’s unique quality, passive, copper cable solutions provide power-efficient connectivity for shortdistance interconnects. They enable higher port bandwidth, density and configurability at a low cost andreduced power requirement in the data centers.Rigorous cable production testing ensures best out-of-the-box installation experience, performance anddurability.T able 1 - Absolute Maximum RatingsT able 2 - Operational SpecificationsT able 3 - Electrical SpecificationsMCP1600-ExxxINTERCONNECTPRODUCT BRIEF†100Gb/s QSFP28 Direct Attach Copper Cable©2018 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.Table 4 - Cable Mechanical SpecificationsTFigure 1.Cable Length Definition©2018 Mellanox Technologies. All rights reserved.350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: © Copyright 2018. Mellanox Technologies. All rights reserved.Mellanox and Mellanox logo are registered trademarks of Mellanox Technologies, Ltd.LinkX is a trademark of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.Warranty InformationMellanox LinkX direct attach copper cables include a 1-year limited hardware warranty, which covers parts repair or replacement.Mechanical Schematics53311PBRev 1.72。
Mellanox Quantum QM8700 InfiniBand交换机产品说明说明书
©2020 Mellanox Technologies. All rights reserved.†For illustration only. Actual products may vary.© Copyright 2020. Mellanox, Mellanox logo, Connect-X, MLNX-OS, and UFM are registered trademarks of Mellanox Technologies, Ltd. Mellanox Quantum and Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) are trademarks of Mellanox Technologies, Ltd. All other trademarks are property of their respective owners.page 2Mellanox QM8700 InfiniBand Switch 350 Oakmead Parkway, Suite 100, Sunnyvale, CA 94085Tel: 408-970-3400 • Fax: Table 1 - Part Numbers and Descriptions53779PB Rev 2.2Note : All tall-bracket adapters are shipped with the tall bracket mounted and a short bracket as accessory.Support : For information about our support packages, please contact your Mellanox Technologies sales representative or visit our Support Index page .HDR100QM8700 together with the Mellanox ConnectX ®-6 adapter card support HDR100. By utilizing two pairs of two lanes per port, theQM8700 can support up to 80 ports of 100G to create the densest TOR switch available in the market. This is a perfect solution for double dense racks with more than 40 servers per rack and also helps small-medium deployments with the need to scale to 3-level fat-tree, to lower power, latency and space.MANAGEMENTThe QM8700’s x86 ComEx Broadwell CPU comes with an on-board subnet manager, enabling simple, out-of-the-box bring-up for up to 2K nodes in the fabric. Running the MLNX-OS ® software package, it delivers full chassis management through CLI, WebUI, SNMP or JSON interfaces.QM8700 also incorporates Mellanox’s Unified Fabric Manager (UFM ®) software for managing scale-out, InfiniBand, computing environments to enable efficient provisioning, health indications and monitoring of the cluster. UFM ® ensures that the fabric is up and running at maximum performance at all times.Safety –CB –cTUVus –CE –CUEMC (Emissions) –CE –FCC –VCCI –ICES –RCMOperating Conditions –Temperature:• Operating 0ºC to 40ºC• Non-operating -40ºC to 70ºC –Humidity:• Operating 10% to 85% non-condensing• Non-operating 10% to 90% non-condensing–Altitude: Up to 3200m Acoustic –ISO 7779 –ETS 300 753Others–RoHS compliant –Rack-mountable, 1U –1-year warrantyCOMPLIANCEMellanox QM8700–19’’ rack mountable 1U chassis –40 QSFP56 non-blocking ports with aggregate data throughput up to 16Tb/s (HDR)Switch Specifications–Compliant with IBTA 1.21 and 1.3 –9 virtual lanes: 8 data + 1 management–256 to 4Kbyte MTU –Adaptive Routing –Congestion control –Port Mirroring–VL2VL mapping–4X48K entry linear forwarding databaseManagement Ports–100/1000 RJ45 Ethernet port –RS232 console port –USB port –DHCP–Industry standard CLI –Management over IPv6 –Management IP –SNMP v1,v2,v3 –WebUIFabric Management–On-board Subnet Manager (SM) supporting fabrics of up to 2K nodes –Unified Fabric Manager (UFM) agentConnectors and Cabling–QSFP56 connectors–Passive copper or active fiber cables–Optical modulesIndicators–Per port status LED Link, Activity–System LEDs: System, fans, powersupplies–Unit ID LEDPower Supply–Dual redundant slots –Hot plug operation –Input range:100-127VAC, 200-240VAC –Frequency: 50-60Hz, single phase AC, 4.5A, 2.9A Cooling –Front-to-rear or rear-to-front cooling option–Hot-swappable fan unitPower Consumption –Contact Mellanox Sales FEATURES。
- 1、下载文档前请自行甄别文档内容的完整性，平台不提供额外的编辑、内容补充、找答案等附加服务。
- 2、"仅部分预览"的文档,不可在线预览部分如存在完整性等问题,可反馈申请退款(可完整预览的文档不适用该条件!)。
- 3、如文档侵犯您的权益，请联系客服反馈,我们会尽快为您处理(人工客服工作时间：9:00-18:30)。
Mellanox OFED for Linux Installation GuideRev 1.5.1Mellanox Technologies350 Oakmead Parkway, Suite 100Sunnyvale, CA 94085U.S.A.Tel: (408) 970-3400Fax: (408) 970-3403Mellanox Technologies, Ltd.PO Box 586 Hermon Building Yokneam 20692Israel Tel: +972-4-909-7200Fax: +972-4-959-3245© Copyright 2010. Mellanox Technologies, Inc. All Rights Reserved.Mellanox®, BridgeX®, ConnectX®, InfiniBlast®, InfiniBridge®, InfiniHost®, InfiniPCI®, InfiniRISC®, InfiniScale®, andVirtual Protocol Interconnect are registered trademarks of Mellanox Technologies, Ltd.CORE-Direct , FabricIT, and PhyX are trademarks of Mellanox Technologies, Ltd.All other marks and names mentioned herein may be trademarks of their respective companies.Document Number: 2914Rev 1.5.1Mellanox Technologies 2NOTE:THIS HARDWARE, SOFTWARE OR TEST SUITE PRODUCT (“PRODUCT(S)”) AND ITS RELATED DOCUMENTATION ARE PRO-VIDED BY MELLANOX TECHNOLOGIES “AS-IS” WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS. THE CUS-TOMER'S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO(S) AND/OR THE SYSTEM USING IT. THEREFORE, MELLANOX TECHNOLOGIES CAN-NOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY . ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING , BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MER-CHANTABILITY , FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED. IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT, INDIRECT, SPECIAL, EXEM-PLARY , OR CONSEQUENTIAL DAMAGES OF ANY KIND (INCLUDING , BUT NOT LIMITED TO, PAYMENT FOR PROCURE-MENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY , WHETHER IN CONTRACT, STRICT LIABILITY , OR TORT(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY FROM THE USE OF THE PRODUCT(S) AND RELATED DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.Mellanox OFED for Linux User’s Manual Rev 1.5.1 InstallationThis chapter describes how to install and test the Mellanox OFED for Linux package on a singlehost machine with Mellanox InfiniBand and/or Ethernet adapter hardware installed. The chapterincludes the following sections:•“Hardware and Software Requirements” (page 3)•“Downloading Mellanox OFED” (page 4)•“Installing Mellanox OFED” (page 4)•“Uninstalling Mellanox OFED” (page 15)Hardware and Software RequirementsHardware RequirementsPlatforms• A server platform with an adapter card based on one of the following Mellanox Technologies’InfiniBand HCA devices:-MT25408 ConnectX®-2 (VPI, IB, EN, FCoE) (firmware: fw-ConnectX2)-MT25408 ConnectX® (VPI, IB, EN, FCoE) (firmware: fw-25408)-MT25208 InfiniHost® III Ex (firmware: fw-25218 for Mem-Free cards, and fw-25208 for cards withmemory)-MT25204 InfiniHost® III Lx (firmware: fw-25204)-MT23108 InfiniHost® (firmware: fw-23108)Note For the list of supported architecture platforms, please refer to the Mellanox OFEDRelease Notes file.Required Disk Space for Installation•400MBSoftware RequirementsOperating System•Linux operating systemNote For the list of supported operating system distributions and kernels, please refer to the Mellanox OFED Release Notes file.Installer Privileges•The installation requires administrator privileges on the target machineMellanox Technologies3Rev 1.5.1Mellanox Technologies 4Downloading Mellanox OFEDStep 1.Verify that the system has a Mellanox network adapter (HCA/NIC) installed by ensuringthat you can see ConnectX or InfiniHost entries in the display.The following example shows a system with an installed Mellanox HCA:host1# lspci -v | grep Mellanox02:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (reva0)Step 2.Download the ISO image to your host.The image’s name has the format MLNX_OFED_LINUX-<ver>-<OS label>.iso. You candownload it from > Products > IB SW/Drivers.Step e the md5sum utility to confirm the file integrity of your ISO image. Run the followingcommand and compare the result to the value provided on the download page.host1$ md5sum MLNX_OFED_LINUX-<ver>-<OS label>.isoInstalling Mellanox OFEDThe installation script, mlnxofedinstall , performs the following:•Discovers the currently installed kernel•Uninstalls any software stacks that are part of the standard operating system distribution oranother vendor's commercial stack•Installs the MLNX_OFED_LINUX binary RPMs (if they are available for the current kernel)•Identifies the currently installed InfiniBand and Ethernet network adapters and automatically 1upgrades the firmwarePre-installation Notes•The installation script removes all previously installed Mellanox OFED packages and re-installsfrom scratch. You will be prompted to acknowledge the deletion of the old packages.Note Pre-existing configuration files will be saved with the extension “.conf.saverpm”.•If you need to install Mellanox OFED on an entire (homogeneous) cluster, a common strategy isto mount the ISO image on one of the cluster nodes and then copy it to a shared file system suchas NFS. To install on all the cluster nodes, use cluster-aware tools (such as pdsh).•If your kernel version does not match with any of the offered pre-built RPMs, you can add yourkernel version by using the “mlnx_add_kernel_support.sh ” script located under the docs/directory.Usage:mlnx_add_kernel_support.sh -i|--iso <mlnx iso>[-t|--tmpdir <local work dir>][-v|--verbose]1.The firmware will not be updated if you run the install script with the ‘--without-fw-update’ option.Mellanox OFED for Linux User’s Manual Rev 1.5.1 ExampleThe following command will create a MLNX_OFED_LINUX ISO image for RedHat 5.4 underthe/tmp directory.MLNX_OFED_LINUX-1.5.1-rhel5.4/docs/mlnx_add_kernel_support.sh -i/mnt/MLNX_OFED_LINUX-1.5.1-rhel5.4.isoAll Mellanox, OEM, OFED, or Distribution IB packages will be removed.Do you want to continue?[y/N]:yRemoving OFED RPMs...Running mkisofs...Created /tmp/MLNX_OFED_LINUX-1.5.1-rhel5.4.isoInstallation ScriptMellanox OFED includes an installation script called mlnxofedinstall. Its usage is describedbelow. You will use it during the installation procedure described in Section , “Installation Proce-dure,” on page 6.Usage./mlnxofedinstall [OPTIONS]Note If no options are provided to the script, then all available RPMs are installed.Options-c|--config <packages config_file>Example of the configuration file can be found under docs -n|--net <network config file>Example of the network configuration file can be foundunder docs-p|--print-available Print available packages for the current platform and cre-ate a corresponding ofed.conf file. The installation scriptexits after creating ofed.conf.--with-fc Install FCoE support — Available on RHEL5.2 ONLY--with-32bit Install 32-bit libraries (default). This is relevant forx86_64 and ppc64 platforms.--without-32bit Skip 32-bit libraries installation--without-ib-bonding Skip ib-bonding RPM installation--without-depcheck Skip Distro's libraries check--without-fw-update Skip firmware update--force-fw-update Force firmware update--force Force installation (without querying the user)--all Install all kernel modules, libibverbs, libibumad, librd-macm, mft, mstflint, diagnostic tools, OpenSM, ib-bonding,MVAPICH, Open MPI, MPI tests, MPI selector, perftest, sdp-netstat and libsdp srptools, rds-tools, static and dynamiclibrariesMellanox Technologies5Rev 1.5.1Mellanox Technologies6--hpc Install all kernel modules, libibverbs, libibumad, librd-macm, mft, mstflint, diagnostic tools, OpenSM , ib-bonding,MVAPICH, Open MPI, MPI tests, MPI selector, dynamic librar-ies--basic Install all kernel modules, libibverbs, libibumad, mft,mstflint, dynamic libraries--msmInstall all kernel modules, libibverbs, libibumad, mft,mstflint, diagnostic tools, OpenSM , ib-bonding, dynamiclibrariesNOTE: With --msm flag, the OpenSM daemon is configured torun upon boot.-v|-vv|-vvvSet verbosity level -q Set quiet - no messages will be printed mlnxofedinstall Return CodesTable 1 lists the mlnxofedinstall script return codes and their meanings.Installation ProcedureStep 1.Login to the installation machine as root.Step 2.Mount the ISO image on your machine host1# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label>.iso /mntNote After mounting the ISO image, /mnt will be a Read Only folder.Step 3.Run the installation scripthost1# /mnt/mlnxofedinstallThis program will install the MLNX_OFED_LINUX package on your machine.Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.Do you want to continue?[y/N]:yUninstalling the previous version of OFEDTable 1 - mlnxofedinstall Return CodesReturn CodeMeaning 0The Installation ended successfully 1The installation failed 2No firmware was found for the adapter device 3Failed to start the mst driverMellanox OFED for Linux User’s Manual Rev 1.5.1 Starting MLNX_OFED_LINUX-1.5.1-rc6 installation ...Installing kernel-ib RPMPreparing... ########################################### [100%]1:kernel-ib ########################################### [100%]Installing kernel-ib-devel RPMPreparing... ########################################### [100%]1:kernel-ib-devel ########################################### [100%]Installing mft RPMPreparing... ########################################### [100%]1:mft ########################################### [100%]Installing mpi-selector RPMPreparing... ########################################### [100%]1:mpi-selector ########################################### [100%]Install user level RPMs:Preparing... ########################################### [100%]1:libibumad ########################################### [100%]Preparing... ########################################### [100%]1:libibumad ########################################### [100%]Preparing... ########################################### [100%]1:libibmad ########################################### [100%]Preparing... ########################################### [100%]1:libibmad ########################################### [100%]Preparing... ########################################### [100%]1:libibumad-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibumad-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibmad-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibmad-devel ########################################### [100%]Preparing... ########################################### [100%]1:ofed-scripts ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-devel-static########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-devel-static########################################### [100%]Preparing... ########################################### [100%]1:libibverbs-utils ########################################### [100%]Preparing... ########################################### [100%]1:libmthca ########################################### [100%]Mellanox Technologies7Rev 1.5.1Mellanox Technologies8Preparing... ########################################### [100%]1:libmthca ########################################### [100%]Preparing... ########################################### [100%]1:libmthca-devel-static ########################################### [100%]Preparing... ########################################### [100%]1:libmthca-devel-static ########################################### [100%]Preparing... ########################################### [100%]1:libmlx4 ########################################### [100%]Preparing... ########################################### [100%]1:libmlx4 ########################################### [100%]Preparing... ########################################### [100%]1:libmlx4-devel ########################################### [100%]Preparing... ########################################### [100%]1:libmlx4-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibcm ########################################### [100%]Preparing... ########################################### [100%]1:libibcm ########################################### [100%]Preparing... ########################################### [100%]1:libibcm-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibcm-devel ########################################### [100%]Preparing... ########################################### [100%]1:libibumad-static ########################################### [100%]Preparing... ########################################### [100%]1:libibumad-static ########################################### [100%]Preparing... ########################################### [100%]1:libibmad-static ########################################### [100%]Preparing... ########################################### [100%]1:libibmad-static ########################################### [100%]Preparing... ########################################### [100%]1:ibsim ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm-utils ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm-devel ########################################### [100%]Preparing... ########################################### [100%]1:librdmacm-devel ########################################### [100%]Preparing... ########################################### [100%]1:libsdp ########################################### [100%]Preparing... ########################################### [100%]1:libsdp ########################################### [100%]Preparing... ########################################### [100%]1:libsdp-devel ########################################### [100%]Preparing... ########################################### [100%]1:libsdp-devel ########################################### [100%]Mellanox OFED for Linux User’s Manual Rev 1.5.1Preparing... ########################################### [100%]1:opensm-libs ########################################### [100%]Preparing... ########################################### [100%]1:opensm-libs ########################################### [100%]Preparing... ########################################### [100%]1:opensm ########################################### [100%]opensmd 0:off 1:off 2:off 3:off 4:off 5:off 6:offPreparing... ########################################### [100%]1:opensm-devel ########################################### [100%]Preparing... ########################################### [100%]1:opensm-devel ########################################### [100%]Preparing... ########################################### [100%]1:opensm-static ########################################### [100%]Preparing... ########################################### [100%]1:opensm-static ########################################### [100%]Preparing... ########################################### [100%]1:compat-dapl ########################################### [100%]Preparing... ########################################### [100%]1:compat-dapl ########################################### [100%]Preparing... ########################################### [100%]1:compat-dapl-devel ########################################### [100%]Preparing... ########################################### [100%]1:compat-dapl-devel ########################################### [100%]Preparing... ########################################### [100%]1:dapl ########################################### [100%]Preparing... ########################################### [100%]1:dapl ########################################### [100%]Preparing... ########################################### [100%]1:dapl-devel ########################################### [100%]Preparing... ########################################### [100%]1:dapl-devel ########################################### [100%]Preparing... ########################################### [100%]1:dapl-devel-static ########################################### [100%]Preparing... ########################################### [100%]1:dapl-devel-static ########################################### [100%]Preparing... ########################################### [100%]1:dapl-utils ########################################### [100%]Preparing... ########################################### [100%]1:perftest ########################################### [100%]Preparing... ########################################### [100%]1:mstflint ########################################### [100%]Preparing... ########################################### [100%]1:sdpnetstat ########################################### [100%]Preparing... ########################################### [100%]1:srptools ########################################### [100%]Preparing... ########################################### [100%]1:rds-tools ########################################### [100%]Preparing... ########################################### [100%]1:rnfs-utils ########################################### [100%]Preparing... ########################################### [100%]Mellanox Technologies9Rev 1.5.1Mellanox Technologies 10 1:ibutils ########################################### [100%]Preparing... ########################################### [100%]1:ibutils2 ########################################### [100%]Preparing... ########################################### [100%]1:ibdump ########################################### [100%]Preparing... ########################################### [100%]1:infiniband-diags ########################################### [100%]Preparing... ########################################### [100%]1:qperf ########################################### [100%]Preparing... ########################################### [100%]1:mlnxofed-docs ########################################### [100%]Preparing... ########################################### [100%]1:mvapich_gcc ########################################### [100%]Preparing... ########################################### [100%]1:mvapich_pgi ########################################### [100%]Preparing... ########################################### [100%]1:mvapich_intel ########################################### [100%]Preparing... ########################################### [100%]1:openmpi_gcc ########################################### [100%]Preparing... ########################################### [100%]1:openmpi_pgi ########################################### [100%]Preparing... ########################################### [100%]1:openmpi_intel ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_mvapich_gcc ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_mvapich_pgi ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_mvapich_intel ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_openmpi_gcc ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_openmpi_pgi ########################################### [100%]Preparing... ########################################### [100%]1:mpitests_openmpi_intel ########################################### [100%]Device (15b3:634a):02:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0)Link Width: 8xLink Speed: 2.5Gb/sInstallation finished successfully.Programming HCA firmware for /dev/mst/mt25418_pci_cr0 deviceRunning: mlxburn -d /dev/mst/mt25418_pci_cr0 -fw /tmp/MLNX_OFED_LINUX-1.5.1/MLNX_OFED_LINUX-1.5.1-sles11/firmware/fw-25408/2_7_000/fw-25408-rel.mlx -dev_type 25408 -no-I- Querying device ...-I- Using auto detected configuration file: /tmp/MLNX_OFED_LINUX-1.5.1/MLNX_OFED_LINUX-1.5.1-sles11/firmware/fw-25408/2_7_000/MHGH28-XTC_A4-A7.ini (PSID = MT_04A0140005)-I- Generating image ...Current FW version on flash: 2.6.0New FW version: 2.7.0Burning FW image without signatures - OKRestoring signature - OK-I- Image burn completed successfully.Please reboot your system for the changes to take effect.warning: /etc/infiniband/openib.conf saved as /etc/infiniband/openib.conf.rpmsaveNote In case your machine has the latest firmware, no firmware update will occur and the installation script will print at the end of installation a message similar to the following:...Installation finished successfully.The firmware version 2.7.000 is up to date.Note: To force firmware update use '--force-fw-update' flag.Note In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. Please contact your hardwarevendor for help on firmware updates.Error message:-I- Querying device ...-E- Can't auto detect fw configuration file: ...Step 4.In case the installation script performed firmware updates to your network adapter hard-ware, it will ask you to reboot your machine.Step 5.The script adds the following lines to /etc/security/limits.conf for the userspace components such as MPI:* soft memlock unlimited* hard memlock unlimitedThese settings unlimit the amount of memory that can be pinned by a user space application.If desired, tune the value unlimited to a specific amount of RAM.Step 6.For your machine to be part of the InfiniBand/VPI fabric, a Subnet Manager must be run-ning on one of the fabric nodes. At this point, Mellanox OFED for Linux has alreadyinstalled the OpenSM Subnet Manager on your machine. For details on starting OpenSM,see Chapter 12, “OpenSM – Subnet Manager”.Step 7.(InfiniBand only) Run the hca_self_test.ofed utility to verify whether or not the InfiniBand link is up. The utility also checks for and displays additional information suchas•HCA firmware version•Kernel architecture•Driver version•Number of active HCA ports along with their states•Node GUIDNote For more details on hca_self_test.ofed, see the file hca_self_test.readme under docs/.host1# /usr/bin/hca_self_test.ofed---- Performing InfiniBand HCA Self Test ----Number of HCAs Detected (1)PCI Device Check ....................... PASSKernel Arch ............................ x86_64Host Driver Version .................... MLNX_OFED_LINUX-1.5.1 (OFED-1.5.1-mlnx9) 1.5.1-2.6.9_89.ELlargesmpHost Driver RPM Check .................. PASSHCA Firmware on HCA #0 ................. 2.7.000HCA Firmware Check on HCA #0 ........... PASSHost Driver Initialization ............. PASSNumber of HCA Ports Active 0Port State of Port #0 on HCA #0 ........ INITPort State of Port #0 on HCA #0 ........ DOWNError Counter Check on HCA #0 .......... PASSKernel Syslog Check .................... PASSNode GUID on HCA #0 .................... 00:02:c9:03:00:00:10:e0------------------ DONE ---------------------Note After the installer completes, information about the Mellanox OFED installation such as prefix, kernel version, and installation parameters can be retrieved by running the com-mand /etc/infiniband/info.Installation ResultsSoftware•The OFED and MFT packages are installed under the /usr directory.•The kernel modules are installed under:-InfiniBand subsystem:/lib/modules/`uname -r`/updates/kernel/drivers/infiniband/-mlx4 driver:Under /lib/modules/`uname -r`/updates/kernel/drivers/net/mlx4you will find mlx4_core.ko, mlx4_en.ko, mlx4_ib.ko (and mlx4_fc ifyou ran the installation script with --with-fc)-RDS:/lib/modules/`uname -r`/updates/kernel/net/rds/rds.ko-Bonding module:/lib/modules/`uname -r`/updates/kernel/drivers/net/bonding/bonding.ko •The package kernel-ib-devel include files are placed under /usr/src/ofa_kernel/include/. These include files should be used when building kernel modules that use the stack. (Note that the include files, if needed, are “backported” to your kernel.)•The raw package (un-backported) source files are placed under/usr/src/ofa_kernel-<ver>•The script openibd is installed under /etc/init.d/. This script can be used to load and unload the software stack.•The script connectx_port_config is installed under /sbin. This script can be used to con-figure the ports of ConnectX network adapter cards to Ethernet and/or InfiniBand. For details on this script, please see Section 3.1, “Port Type Management”.•The directory /etc/infiniband is created with the files info and openib.conf and con-nectx.conf. The info script can be used to retrieve Mellanox OFED installation information. The openib.conf file contains the list of modules that are loaded when the openibd script is used. The connectx.conf file saves the ConnectX adapter card’s ports configuration to Ether-net and/or InfiniBand. This file is used at driver start/restart (/etc/init.d/openibd start) .•The file 90-ib.rules is installed under /etc/udev/rules.d/•If OpenSM is installed, the daemon opensmd is installed under /etc/init.d/ and opensm.conf is installed under /etc.•If IPoIB configuration files are included, ifcfg-ib<n> files will be installed under:-/etc/sysconfig/network-scripts/ on a RedHat machine-/etc/sysconfig/network/ on a SuSE machine•The installation process unlimits the amount of memory that can be pinned by a user space application. See Step 5.•Man pages will be installed under /usr/share/man/Firmware•The firmware of existing network adapter devices will be updated if the following two condi-tions are fullfilled:1. You run the installation script in default mode; that is, without the option‘--without-fw-update’.2. The firmware version of the adapter device is older than the firmware version includedwiththe Mellanox OFED ISO imageNote If an adapter’s Flash was originially programmed with an Expansion ROM image, the automatic firmware update will also burn an Expansion ROM image.•In case your machine has an unsupported network adapter device, no firmware update will occur and the error message below will be printed. Please contact your hardware vendor for help on firmware updates.Error message:。