3.235.228.219

ESXi 7.0 un Mellanox ConnectX 2 - atbalsta fiksēto plāksteri

ESXi 7.0 un Mellanox ConnectX 2 - atbalsta fiksēto plāksteri

Veiksmīgi jaunināju vCenter uz 7. versiju, bet neizdevās, kad vajadzēja atjaunināt savus resursdatorus no 6.7 uz 7.

Es saņēmu brīdinājumu, norādot, ka PCI ierīces nav saderīgas, bet tomēr mēģināju. Izrādās, ka tas neizdevās, mans Mellanox ConnectX 2 netika rādīts kā pieejams fiziskais NIC.

Sākumā bija nepieciešams VID / DID ierīces kods MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT / s].

Novecojušas ierīces, kuras atbalsta VMKlinux draiveri

Ierīces, kuras tikai 6.7 vai jaunākās versijās atbalstīja VMKlinux iesūtnes draiveris. Šīs ierīces vairs netiek atbalstītas, jo 7.0 versijā ir pilnībā noņemts viss atbalsts VMKlinux draiveriem un to ierīcēm.

Kā to novērst? Es noskaņoju nelielu skriptu ESXi7-enable-nmlx4_co.v00.sh, lai DARĪTU. Piezīmes:

  •  rediģēt plāksteri savā Datastore piemērā ir / vmfs / volume / ISO
  •  nmlx4_co.v00.orig ir oriģinālā nmlx4_co.v00 dublējums
  •  Jauns VIB ir bez parakstiem - pārstartēšanas laikā ALERT ziņojums būs reģistrēts:
  •  BRĪDINĀJUMS: Neizdevās pārbaudīt šīs vib parakstu
  •  Jauna draivera ielādei ir nepieciešama ESXi atsāknēšana
  • cp /bootbank/nmlx4_co.v00 /vmfs/volumes/ISO/nmlx4_co.v00.orig
    cp /bootbank/nmlx4_co.v00 /vmfs/volumes/ISO/n.tar
    cd /vmfs/volumes/ISO/
    vmtar -x n.tar -o output.tar
    rm -f n.tar
    mkdir tmp-network
    mv output.tar tmp-network/output.tar
    cd tmp-network
    tar xf output.tar
    rm output.tar
    echo '' >> /vmfs/volumes/ISO/tmp-network/etc/vmware/default.map.d/nmlx4_core.map
    echo 'regtype=native,bus=pci,id=15b36750..............,driver=nmlx4_core' >> /vmfs/volumes/ISO/tmp-network/etc/vmware/default.map.d/nmlx4_core.map
    cat /vmfs/volumes/ISO/tmp-network/etc/vmware/default.map.d/nmlx4_core.map
    echo '        6750  Mellanox ConnectX-2 Dual Port 10GbE '                 >> /vmfs/volumes/ISO/tmp-network/usr/share/hwdata/default.pciids.d/nmlx4_core.ids 
    cat /vmfs/volumes/ISO/tmp-network/usr/share/hwdata/default.pciids.d/nmlx4_core.ids
    tar -cf /vmfs/volumes/ISO/FILE.tar *
    cd /vmfs/volumes/ISO/
    vmtar -c FILE.tar -o output.vtar
    gzip output.vtar
    mv output.vtar.gz nmlx4_co.v00
    rm FILE.tar
    cp /vmfs/volumes/ISO/nmlx4_co.v00 /bootbank/nmlx4_co.v00

    Skripti pievieno HW ID atbalstu failā nmlx4_core.map:

    *********************************************************************
    /vmfs/volumes/ISO/tmp-network/etc/vmware/default.map.d/nmlx4_core.map
    *********************************************************************
    regtype=native,bus=pci,id=15b301f6..............,driver=nmlx4_core
    regtype=native,bus=pci,id=15b301f8..............,driver=nmlx4_core
    regtype=native,bus=pci,id=15b31003..............,driver=nmlx4_core
    regtype=native,bus=pci,id=15b31004..............,driver=nmlx4_core
    regtype=native,bus=pci,id=15b31007..............,driver=nmlx4_core
    regtype=native,bus=pci,id=15b3100715b30003......,driver=nmlx4_core
    regtype=native,bus=pci,id=15b3100715b30006......,driver=nmlx4_core
    regtype=native,bus=pci,id=15b3100715b30007......,driver=nmlx4_core
    regtype=native,bus=pci,id=15b3100715b30008......,driver=nmlx4_core
    regtype=native,bus=pci,id=15b3100715b3000c......,driver=nmlx4_core
    regtype=native,bus=pci,id=15b3100715b3000d......,driver=nmlx4_core
    regtype=native,bus=pci,id=15b36750..............,driver=nmlx4_core
    ------------------------->Last Line is FIX
    ________________________________________

    Un pievienojiet HW ID atbalstu failā nmlx4_core.ids:

    **************************************************************************************
    /vmfs/volumes/FreeNAS/ISO/tmp-network/usr/share/hwdata/default.pciids.d/nmlx4_core.ids 
    **************************************************************************************
    #
    # This file is mechanically generated.  Any changes you make
    # manually will be lost at the next build.
    #
    # Please edit _devices.py file for permanent changes.
    #
    # Vendors, devices and subsystems.
    #
    # Syntax (initial indentation must be done with TAB characters):
    #
    # vendor  vendor_name
    #       device  device_name                            <-- single TAB
    #               subvendor subdevice  subsystem_name    <-- two TABs
    
    15b3  Mellanox Technologies
            01f6  MT27500 [ConnectX-3 Flash Recovery]
            01f8  MT27520 [ConnectX-3 Pro Flash Recovery]
            1003  MT27500 Family [ConnectX-3]
            1004  MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function]
            1007  MT27520 Family [ConnectX-3 Pro]
                    15b3 0003  ConnectX-3 Pro VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE (MCX354A-FCC)
                    15b3 0006  ConnectX-3 Pro EN network interface card 40/56GbE dual-port QSFP(MCX314A-BCCT )
                    15b3 0007  ConnectX-3 Pro EN NIC; 40GigE; dual-port QSFP (MCX314A-BCC)
                    15b3 0008  ConnectX-3 Pro VPI adapter card; single-port QSFP; FDR IB (56Gb/s) and 40GigE (MCX353A-FCC)
                    15b3 000c  ConnectX-3 Pro EN NIC; 10GigE; dual-port SFP+ (MCX312B-XCC)
                    15b3 000d  ConnectX-3 Pro EN network interface card; 10GigE; single-port SFP+ (MCX311A-XCC)
            6750  Mellanox ConnectX-2 Dual Port 10GbE
    -------->Last Line is FIX

    Pēc pārstartēšanas es redzēju atbalstu MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT / s].

    Tikai ALERT: Neizdevās pārbaudīt šīs (-o) vibrācijas (-u) parakstus: [nmlx4-core].

    2020-XX-XXTXX:XX:44.473Z cpu0:2097509)ALERT: Failed to verify signatures of the following vib(s): [nmlx4-core]. All tardisks validated
    2020-XX-XXTXX:XX:47.909Z cpu1:2097754)Loading module nmlx4_core ...
    2020-XX-XXTXX:XX:47.912Z cpu1:2097754)Elf: 2052: module nmlx4_core has license BSD
    2020-XX-XXTXX:XX:47.921Z cpu1:2097754) nmlx4_core: init_module called
    2020-XX-XXTXX:XX:47.921Z cpu1:2097754)Device: 194: Registered driver 'nmlx4_core' from 42
    2020-XX-XXTXX:XX:47.921Z cpu1:2097754)Mod: 4845: Initialization of nmlx4_core succeeded with module ID 42.
    2020-XX-XXTXX:XX:47.921Z cpu1:2097754)nmlx4_core loaded successfully.
    2020-XX-XXTXX:XX:47.951Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_core_Attach - (nmlx4_core_main.c:2476) running
    2020-XX-XXTXX:XX:47.951Z cpu1:2097754)DMA: 688: DMA Engine 'nmlx4_core' created using mapper 'DMANull'.
    2020-XX-XXTXX:XX:47.951Z cpu1:2097754)DMA: 688: DMA Engine 'nmlx4_core' created using mapper 'DMANull'.
    2020-XX-XXTXX:XX:47.951Z cpu1:2097754)DMA: 688: DMA Engine 'nmlx4_core' created using mapper 'DMANull'.
    2020-XX-XXTXX:XX:49.724Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_ChooseRoceMode - (nmlx4_core_main.c:382) Requested RoCE mode RoCEv1
    2020-XX-XXTXX:XX:49.724Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_ChooseRoceMode - (nmlx4_core_main.c:422) Requested RoCE mode is supported - choosing RoCEv1
    2020-XX-XXTXX:XX:49.934Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_CmdInitHca - (nmlx4_core_fw.c:1408) Initializing device with B0 steering support
    2020-XX-XXTXX:XX:50.561Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_InterruptsAlloc - (nmlx4_core_main.c:1744) Granted 38 MSIX vectors
    2020-XX-XXTXX:XX:50.561Z cpu1:2097754) nmlx4_core: 0000:05:00.0: nmlx4_InterruptsAlloc - (nmlx4_core_main.c:1766) Using MSIX
    2020-XX-XXTXX:XX:50.781Z cpu1:2097754)Device: 330: Found driver nmlx4_core for device 0xxxxxxxxxxxxxxxxxxxxxxx

    Aptuveni 10 Gb / s regulēšanas pārbaude izskatās lieliski, starp 2x ESXi 7.0 un 2x MT2644:

    [ ID] Interval           Transfer     Bandwidth       Retr
    [  4]   0.00-120.00 sec   131 GBytes  9380 Mbits/sec    0             sender
    [  4]   0.00-120.00 sec   131 GBytes  9380 Mbits/sec                  receiver

    RoCEv1 tiek atbalstīts tikai tāpēc, ka:

    • RoCEv2 atbalsts ir virs kartes - Mellanox ConnectX-3 Pro
    • Mēs varam redzēt RoCEv2 opcijas nmlx2_core draiverī, bet, kad es iespējoju enable_rocev2, tas nedarbojas
    [root@esxi~] esxcli system module parameters list -m nmlx4_core
    Name                    Type  Value  Description
    ----------------------  ----  -----  -----------
    enable_64b_cqe_eqe      int          Enable 64 byte CQEs/EQEs when the the FW supports this
    enable_dmfs             int          Enable Device Managed Flow Steering
    enable_qos              int          Enable Quality of Service support in the HCA
    enable_rocev2           int          Enable RoCEv2 mode for all devices
    enable_vxlan_offloads   int          Enable VXLAN offloads when supported by NIC
    log_mtts_per_seg        int          Log2 number of MTT entries per segment
    log_num_mgm_entry_size  int          Log2 MGM entry size, that defines the number of QPs per MCG, for example: value 10 results in 248 QP per MGM entry
    msi_x                   int          Enable MSI-X
    mst_recovery            int          Enable recovery mode(only NMST module is loaded)
    rocev2_udp_port         int          Destination port for RoCEv2

    Izmantojiet to tikai savā HomeLAB.