Turpinot lietot mājas lapu cloudhosting.lv, Jūs piekrītat sīkdatņu izmantošanas noteikumiem. Uzzināt vairāk. Piekrītu
ESXi 7.0 un Mellanox ConnectX 2 - atbalsta fiksēto plāksteri
Veiksmīgi jaunināju vCenter uz 7. versiju, bet neizdevās, kad vajadzēja atjaunināt savus resursdatorus no 6.7 uz 7.
Es saņēmu brīdinājumu, norādot, ka PCI ierīces nav saderīgas, bet tomēr mēģināju. Izrādās, ka tas neizdevās, mans Mellanox ConnectX 2 netika rādīts kā pieejams fiziskais NIC.
Sākumā bija nepieciešams VID / DID ierīces kods MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT / s].
Novecojušas ierīces, kuras atbalsta VMKlinux draiveri
Ierīces, kuras tikai 6.7 vai jaunākās versijās atbalstīja VMKlinux iesūtnes draiveris. Šīs ierīces vairs netiek atbalstītas, jo 7.0 versijā ir pilnībā noņemts viss atbalsts VMKlinux draiveriem un to ierīcēm.
Kā to novērst? Es noskaņoju nelielu skriptu ESXi7-enable-nmlx4_co.v00.sh, lai DARĪTU. Piezīmes:
cp /bootbank/nmlx4_co.v00 /vmfs/volumes/ISO/nmlx4_co.v00.orig cp /bootbank/nmlx4_co.v00 /vmfs/volumes/ISO/n.tar cd /vmfs/volumes/ISO/ vmtar -x n.tar -o output.tar rm -f n.tar mkdir tmp-network mv output.tar tmp-network/output.tar cd tmp-network tar xf output.tar rm output.tar echo '' >> /vmfs/volumes/ISO/tmp-network/etc/vmware/default.map.d/nmlx4_core.map echo 'regtype=native,bus=pci,id=15b36750..............,driver=nmlx4_core' >> /vmfs/volumes/ISO/tmp-network/etc/vmware/default.map.d/nmlx4_core.map cat /vmfs/volumes/ISO/tmp-network/etc/vmware/default.map.d/nmlx4_core.map echo ' 6750 Mellanox ConnectX-2 Dual Port 10GbE ' >> /vmfs/volumes/ISO/tmp-network/usr/share/hwdata/default.pciids.d/nmlx4_core.ids cat /vmfs/volumes/ISO/tmp-network/usr/share/hwdata/default.pciids.d/nmlx4_core.ids tar -cf /vmfs/volumes/ISO/FILE.tar * cd /vmfs/volumes/ISO/ vmtar -c FILE.tar -o output.vtar gzip output.vtar mv output.vtar.gz nmlx4_co.v00 rm FILE.tar cp /vmfs/volumes/ISO/nmlx4_co.v00 /bootbank/nmlx4_co.v00
Skripti pievieno HW ID atbalstu failā nmlx4_core.map:
********************************************************************* /vmfs/volumes/ISO/tmp-network/etc/vmware/default.map.d/nmlx4_core.map ********************************************************************* regtype=native,bus=pci,id=15b301f6..............,driver=nmlx4_core regtype=native,bus=pci,id=15b301f8..............,driver=nmlx4_core regtype=native,bus=pci,id=15b31003..............,driver=nmlx4_core regtype=native,bus=pci,id=15b31004..............,driver=nmlx4_core regtype=native,bus=pci,id=15b31007..............,driver=nmlx4_core regtype=native,bus=pci,id=15b3100715b30003......,driver=nmlx4_core regtype=native,bus=pci,id=15b3100715b30006......,driver=nmlx4_core regtype=native,bus=pci,id=15b3100715b30007......,driver=nmlx4_core regtype=native,bus=pci,id=15b3100715b30008......,driver=nmlx4_core regtype=native,bus=pci,id=15b3100715b3000c......,driver=nmlx4_core regtype=native,bus=pci,id=15b3100715b3000d......,driver=nmlx4_core regtype=native,bus=pci,id=15b36750..............,driver=nmlx4_core ------------------------->Last Line is FIX ________________________________________
Un pievienojiet HW ID atbalstu failā nmlx4_core.ids:
************************************************************************************** /vmfs/volumes/FreeNAS/ISO/tmp-network/usr/share/hwdata/default.pciids.d/nmlx4_core.ids ************************************************************************************** # # This file is mechanically generated. Any changes you make # manually will be lost at the next build. # # Please edit_devices.py file for permanent changes. # # Vendors, devices and subsystems. # # Syntax (initial indentation must be done with TAB characters): # # vendor vendor_name # device device_name <-- single TAB # subvendor subdevice subsystem_name <-- two TABs 15b3 Mellanox Technologies 01f6 MT27500 [ConnectX-3 Flash Recovery] 01f8 MT27520 [ConnectX-3 Pro Flash Recovery] 1003 MT27500 Family [ConnectX-3] 1004 MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 1007 MT27520 Family [ConnectX-3 Pro] 15b3 0003 ConnectX-3 Pro VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE (MCX354A-FCC) 15b3 0006 ConnectX-3 Pro EN network interface card 40/56GbE dual-port QSFP(MCX314A-BCCT ) 15b3 0007 ConnectX-3 Pro EN NIC; 40GigE; dual-port QSFP (MCX314A-BCC) 15b3 0008 ConnectX-3 Pro VPI adapter card; single-port QSFP; FDR IB (56Gb/s) and 40GigE (MCX353A-FCC) 15b3 000c ConnectX-3 Pro EN NIC; 10GigE; dual-port SFP+ (MCX312B-XCC) 15b3 000d ConnectX-3 Pro EN network interface card; 10GigE; single-port SFP+ (MCX311A-XCC) 6750 Mellanox ConnectX-2 Dual Port 10GbE -------->Last Line is FIX
Pēc pārstartēšanas es redzēju atbalstu MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT / s].
Tikai ALERT: Neizdevās pārbaudīt šīs (-o) vibrācijas (-u) parakstus: [nmlx4-core].
2020-XX-XXTXX:XX:44.473Z cpu0:2097509)ALERT: Failed to verify signatures of the following vib(s): [nmlx4-core]. All tardisks validated 2020-XX-XXTXX:XX:47.909Z cpu1:2097754)Loading module nmlx4_core ... 2020-XX-XXTXX:XX:47.912Z cpu1:2097754)Elf: 2052: module nmlx4_core has license BSD 2020-XX-XXTXX:XX:47.921Z cpu1:2097754)nmlx4_core: init_module called 2020-XX-XXTXX:XX:47.921Z cpu1:2097754)Device: 194: Registered driver 'nmlx4_core' from 42 2020-XX-XXTXX:XX:47.921Z cpu1:2097754)Mod: 4845: Initialization of nmlx4_core succeeded with module ID 42. 2020-XX-XXTXX:XX:47.921Z cpu1:2097754)nmlx4_core loaded successfully. 2020-XX-XXTXX:XX:47.951Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_core_Attach - (nmlx4_core_main.c:2476) running 2020-XX-XXTXX:XX:47.951Z cpu1:2097754)DMA: 688: DMA Engine 'nmlx4_core' created using mapper 'DMANull'. 2020-XX-XXTXX:XX:47.951Z cpu1:2097754)DMA: 688: DMA Engine 'nmlx4_core' created using mapper 'DMANull'. 2020-XX-XXTXX:XX:47.951Z cpu1:2097754)DMA: 688: DMA Engine 'nmlx4_core' created using mapper 'DMANull'. 2020-XX-XXTXX:XX:49.724Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_ChooseRoceMode - (nmlx4_core_main.c:382) Requested RoCE mode RoCEv1 2020-XX-XXTXX:XX:49.724Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_ChooseRoceMode - (nmlx4_core_main.c:422) Requested RoCE mode is supported - choosing RoCEv1 2020-XX-XXTXX:XX:49.934Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_CmdInitHca - (nmlx4_core_fw.c:1408) Initializing device with B0 steering support 2020-XX-XXTXX:XX:50.561Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_InterruptsAlloc - (nmlx4_core_main.c:1744) Granted 38 MSIX vectors 2020-XX-XXTXX:XX:50.561Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_InterruptsAlloc - (nmlx4_core_main.c:1766) Using MSIX 2020-XX-XXTXX:XX:50.781Z cpu1:2097754)Device: 330: Found driver nmlx4_core for device 0xxxxxxxxxxxxxxxxxxxxxxx
Aptuveni 10 Gb / s regulēšanas pārbaude izskatās lieliski, starp 2x ESXi 7.0 un 2x MT2644:
[ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-120.00 sec 131 GBytes 9380 Mbits/sec 0 sender [ 4] 0.00-120.00 sec 131 GBytes 9380 Mbits/sec receiver
RoCEv1 tiek atbalstīts tikai tāpēc, ka:
[root@esxi~] esxcli system module parameters list -m nmlx4_core Name Type Value Description ---------------------- ---- ----- ----------- enable_64b_cqe_eqe int Enable 64 byte CQEs/EQEs when the the FW supports this enable_dmfs int Enable Device Managed Flow Steering enable_qos int Enable Quality of Service support in the HCA enable_rocev2 int Enable RoCEv2 mode for all devices enable_vxlan_offloads int Enable VXLAN offloads when supported by NIC log_mtts_per_seg int Log2 number of MTT entries per segment log_num_mgm_entry_size int Log2 MGM entry size, that defines the number of QPs per MCG, for example: value 10 results in 248 QP per MGM entry msi_x int Enable MSI-X mst_recovery int Enable recovery mode(only NMST module is loaded) rocev2_udp_port int Destination port for RoCEv2
Izmantojiet to tikai savā HomeLAB.