ESXI 6 and intel i211 nic

patrtech

I'm new to hypervisors, so I installed esxi 6 on a shuttle ds57u device and was trying to follow the esxi guide. The DS57U has two nics, one a i218-LM and the other a i211. Only the i218 nic is showing up as a available adapter, the i211 isn't available as a physical nic, but it does show up as a pass through device (under advanced settings). The hardware compatibility guide shows both of these nics as having drivers in esxi 6, so I'm trying to understand what the problem is and if there is a solution for it? Has anyone any experience with the i211 intel nic in esxi?

I211 compatibility:

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=37601&deviceCategory=io&details=1&releases=273&keyword=i211&deviceTypes=6&VID=8086&DID=1539&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc

I218-LM compatibility:

http://www.vmware.com/resources/compatibility/detail.php?deviceCategory=io&productid=37653&deviceCategory=io&details=1&keyword=i21&page=1&display_interval=10&sortColumn=Partner&sortOrder=Asc

gjaltemba

Slightly off topic in a pfsense forum but I'll bite.

Have you checked that the device id is 8036:1539?

ssh into esxi

lspci -v

patrtech

I confirmed the device id is correct and you'll see below that it actually says its vmnic1, but yet not accessible. I booted up ubuntu server and it detects both nics. pfsense bare metal also detects both nics as fine and working.

lspci -v | grep Network -A1
0000:00:19.0 Ethernet controller Network controller: Intel Corporation Ethernet Connection (3) I218-LM [vmnic0]
Class 0200: 8086:15a2 (e1000e version 2.5.4-5vmw)
–
0000:02:00.0 Ethernet controller Network controller: Intel Corporation I211 Gigabit Network Connection [vmnic1]
Class 0200: 8086:1539 (igb version 5.0.5.1.1-4vmw_
–
0000:03:00.0 Network controller Network controller:
Class 0280: 10ec:8179

gjaltemba

Passthrough is disabled for vmnic1 in Advanced Settings and it is also not listed in Network Adapters?

patrtech

@gjaltemba:

Passthrough is disabled for vmnic1 in Advanced Settings and it is also not listed in Network Adapters?

Yes, passthrough is disabled and it is not listed in network adapters. I went through these settings numerous times, I reinstalled esxi 6 and even 5.5 several times and the same issue consistently. I checked the bios, not much in there other than enable and disable network adapters - which i tried disabling just for the sake of it. I did a restore to defaults also, but no change at all. Watching esxi load, I see modules for e1000e and igb (which is supposedly the i211) but in network adapters only the i218 is listed.

I don't know what it is about the i211, hyper-v 2012r2 similarly shows only the i218. I had to manually install the drivers but only i218 shows up. The i211 nic is not bad because I tested with ubuntu live cd and pfsense bare metal.

gjaltemba

What is the output for

vmkchdev -l | grep vmnic1

patrtech

@gjaltemba:

What is the output for

vmkchdev -l | grep vmnic1

Just ran it:

[root@localhost:~] vmkchdev -l | grep vmnic1
0000:02:00.0 8086:1539 1297:4028 vmkernel vmnic1

gjaltemba

Yikes!

So vmnic1 is not shown with

esxcfg-nics -l

All I can suggest is a refresh to be sure

vim-cmd hostsvc/net/refresh

patrtech

No change even with the refresh command above. :(

Thanks for your efforts, I will keep searching and maybe try a different hypervisor or just eventually put pfsense on bare metal.

gjaltemba

The following article seems to describe the same symptoms and a resolution

http://atherbeg.com/2014/01/27/missing-vmnics-from-esxi-after-installation/

So basically

Enter maintenance mode
Keep only vmnic0 remove all other vmnics
Shutdown computer
Disable I211 in bios
Boot computer
Shutdown computer
Enable I211 in bios
Boot computer

patrtech

I tried the above and unfortunately no change. Not sure if the listing order on the esx.conf file is weird? :

/device/00000:002:00.0/vmkname = "vmnic1"
/device/00000:000:31.2/vmkname = "vmhba0"
/device/00000:000:25.0/vmkname = "vmnic0"

and then checking to see installed nics, it shows i211 isn't even installed (i blanked out the MAC for the one that is installed)

[root@localhost:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:00:19.0 e1000e Up 1000Mbps Full xxxx:xx:xx:xx 1500 Intel Corporation <class>Ethernet controller</class>

gjaltemba

Not so much the order but vmnic0 seems to be at different locations. That cannot be correct.

/device/00000:000:25.0/vmkname = "vmnic0"

vmnic0 0000:00:19.0

patrtech

Maybe I should reformat and install esxi again but with i211 disabled? I lost count how many times I installed this now, but I haven't tried in this method.

gjaltemba

I am curious which is correct for vmnic0. 00000:000:25.0 or 0000:00:19.0

What does lspci show?

Before a reinstall I would try a Restore System Configuration (factory reset) on the Direct Console (DCUI) first.

patrtech

I believe I tried a factory reset yesterday in one of the iterations of using 2 drives to troubleshoot, but course I can do it again. Here's the lspci output:

[root@localhost:~] lspci
0000:00:00.0 Bridge:
0000:00:02.0 Display controller:
0000:00:03.0 Multimedia controller:
0000:00:14.0 Serial bus controller:
0000:00:16.0 Communication controller:
0000:00:19.0 Network controller: [vmnic0]
0000:00:1b.0 Multimedia controller:
0000:00:1c.0 Bridge: [PCIe RP[0000:00:1c.0]]
0000:00:1c.2 Bridge: [PCIe RP[0000:00:1c.2]]
0000:00:1c.3 Bridge: [PCIe RP[0000:00:1c.3]]
0000:00:1d.0 Serial bus controller:
0000:00:1f.0 Bridge:
0000:00:1f.2 Mass storage controller: [vmhba0]
0000:00:1f.3 Serial bus controller:
0000:02:00.0 Network controller: Intel Corporation I211 Gigabit Network Connection [vmnic1]
0000:03:00.0 Network controller:

gjaltemba

Then in esx.conf
/device/00000:000:25.0/vmkname = "vmnic0"
should be
/device/00000:000:19.0/vmkname = "vmnic0"

I wonder what is at 0000:03:00.0 Network controller:

patrtech

Should i edit esx.conf to /device/00000:000:19.0/vmkname = "vmnic0"?

0000:03:00.0 Network controller <–-----that probably is a wireless nic, which I can't disable in the bios. Maybe I should physicall just remove that?

gjaltemba

Yeah. It is the wifi card. You could try and edit esx.conf but it is not supported.

Personally, I would try re-installing with VT-d off in the bios but it is just a shot in the dark.

patrtech

So I tried again today, formatted the drive, disabled VT-d, disabled nic and exactly the same results as before. I tried editing the esx.conf file from /device/00000:000:25.0/vmkname = "vmnic0" to /device/00000:000:19.0/vmkname = "vmnic0" but after rebooting it went back to what it was before. That second nic just won't show up, I wish they hadn't put two different nics into this unit. I'm going to try installing some igb driver i found igb-5.2.7-1331820-2157967.zip, not sure if this is will do anything but it can't make it worse.

gjaltemba

If I211 nic works in passthru why not use it for the wan port in pfsense? You will still have I218 for a vswitch to assign to pfsense lan port and new VMs.

patrtech

I'm totally new to esxi and pfsense….as trivial as it might sound, are there any downsides to your suggestion? in terms of security, performance etc. Basically I was following this guide https://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5 and trying to implement it that way. If nothing much is being affected, maybe this is the way to go? I'll have to reformat one last time as the vib driver I just installed didn't make a difference either.

gjaltemba

A passthru nic is bare metal to the pfsense vm and cannot be used as a vnic for other VMs. If you configure the I211 as passthru and assign it to the wan port in pfsense then it can used to provide internet for all the VMs on the lan (vmnic0).

Configure vswitch0 to use vmnic0 and add a VMKernel port group for management and Virtual Machine port group for pfsense lan port and other VMs.

https://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5

patrtech

I'll give this a shot tomorrow. Thanks for all your help with this issue! It's given me some hope from where I was a couple days back.

washer

patrtech, did you get any further with this?

I just got the same board and believe I'm running into the same exact issue, all three NICs (including the wifi card, I might try to take that out. Don't think it will help but I doubt that would hurt and I don't need/want it now) show up in lspci:

0000:00:19.0 Ethernet controller Network controller: Intel Corporation Ethernet Connection (3) I218-LM [vmnic0]
	 Class 0200: 8086:15a2

0000:02:00.0 Ethernet controller Network controller: Intel Corporation I211 Gigabit Network Connection [vmnic1]
	 Class 0200: 8086:1539

0000:03:00.0 Network controller Network controller:
	 Class 0280: 10ec:8179

But only vmnic0, the I218-LM, is showing up as a Network Adapter. I also see the I211 as an option for pass-through, so I'm going to try and start setting up with that as my WAN port like gjaltemba suggested. It sure would be nice to know why it only shows up for pass-through and not as a valid network adaptor though, assuming it really is supported like we think…

preid

@patrtech:

I'm totally new to esxi and pfsense….as trivial as it might sound, are there any downsides to your suggestion? in terms of security, performance etc. Basically I was following this guide https://doc.pfsense.org/index.php/PfSense_2_on_VMware_ESXi_5 and trying to implement it that way. If nothing much is being affected, maybe this is the way to go? I'll have to reformat one last time as the vib driver I just installed didn't make a difference either.

There is one gotcha with pass-through devices - you will no longer be able to move the VM to another host via vMotion or High Availability - so if you're running a VMWare cluster, those features will not be available for that virtual machine. Aside from that, and the NIC no longer being available for other VMs, it shouldn't give you any issues.

VFrontDe

I investigated this issue and found that the igb driver is refusing to initialize lots of i211 adapters, because their NVM checksums do not match the expected value.
To work around the issue I modified the driver to ignore invalid NVM checksums. The modified driver resolved the issue in at least two cases.

More details are in this blog post:
http://www.v-front.de/2015/08/a-fix-for-intel-i211-and-i350-adapters.html

miloman

@VFrontDe:

To work around the issue I modified the driver to ignore invalid NVM checksums.

you make it sound so easy… also, the stuff on v-front.de is amazing. thank you for all your work.