No carrier with Intel X520-SR2 on VLAN interface (ix1)

[krollebol]

Hi,

We have 2 identical Supermicro servers with each an 10GbE Intel X520-SR2 card (also 2 onboard GbE, and Intel I350 with 2 x GbE). Since we are using the X520 we are having issues with the second interface ix1.

After every pfSense upgrade ix1 stays down:

• First time (from pfsense 2.2.2. to 2.2.3) we set "hw.ix.unsupported_sfp=1" in /boot/loader.conf.local. That solved it.
• after upgrade from 2.2.3 to 2.2.4 (and later to 2.2.5) ix1 remained down again. Reboot did not help. Only a cold boot (remove powerplug) solved it.
• since upgrade to 2.2.6 ix1 is down, and no way to get it working again (powercyle, reboot, etc does not help)

We have this issue on both servers. So it's not a bad fiber or switch.

During boot the interface is up on the switch. Only when pfsense configures the interfaces at boot, the interface goes down on the switch.
ifconfig ix1 shows:
ix1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
options=8500bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwfilter,vlan_hwtso>ether 90:e2:ba:68:85:51
inet6 fe80::92e2:baff:fe68:8551%ix1 prefixlen 64 scopeid 0x6
nd6 options=21 <performnud,auto_linklocal>media: <unknown type="">(autoselect)
status: no carrier

I've tried disabling vlanhwtagging, but that doesn't solve the issue.
Also changing settings in system : advanced: Networking doesn't work.
Settings are:

• Device polling : disabled (not checked)
• Hardware Checksum Offloading: enabled (not checked)
• Hardware TCP Segmentation Offloading: disable (checked)
• Hardware Large Receive Offloading: disable (checked)
• ARP Handling: not checked

Our setup (same on each server)

• WAN: ix0  (UP)
• PFSYNC: igb1  (UP)
• 5 interface on ix1 with VLAN tagging  (all DOWN)

ix1 only has 5 VLANs, no native VLAN. It's always ix1 that doesn't work. So maybe it's related to the VLAN tagging?

Does anyone have the same issue? And how can I solve this.

Once the interface is up and running, it works without issues. But on the next reboot (after upgrade) it will not come up.
We have a master/slave setup with CARP and pfsync. Currently the slave is down because of this issue, so we need a sollution….

PS: also booted the server with Ubuntu Live CD. Then the interface is working correctly...

Thanks for your help and input !

Kind regards
Bruno</unknown></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwfilter,vlan_hwtso></up,broadcast,running,promisc,simplex,multicast>

[bl0815]

Hi,
I have the same troubles, as I wrote some time ago: https://forum.pfsense.org/index.php?topic=96233.0

The last time I did the upgrade from 2.2.4 to 2.2.5 and I had to reboot my pfsense boxes one or two times to get ix0 (in my case the interface with tagged vlans only) working again.
I also tried to setup a dummy IP-Adress on the default untagged vlan on ix0, but it didn't help.

Also ca. once a week the link on ix0 on my master-PFsense goes down for 1sec. and up again.
The PFsense box is connected to a HP ProCurve 8212ZL, perhaps the combination of this switch and the intel 520 is the cause. In a few months we will replace the switch with a newer model, I hope it will get better then.
On the other side, only the active PFsense-Box has problems with link stability… As far as I can remember, increasing the mbuf size also increased the intervall between two link outages.
In my use case the short outages don't bother me, so I didn't look deeper.

[krollebol]

Hi,

I found a workaround for the problem with ix1.

I created a LAGG interface (failover type) with ix1 as member. Then I recreated the VLANs, and assigned them to the corresponding interface.
After a reboot the LAGG interface (and thus ix1) came up, and the VLANs are working again. I rebooted several times to make sure it continues working.

However, when I created a second LAGG with ix0 (used for WAN, without VLAN tagging), that LAGG will not come up after reboot. (also no carrier)
So to me it looks like the driver for X520 has some issues in pfsense…

For now both our servers are working with this setup:
ix0 - directly assigned to WAN
ix1 - member of LAGG, used with VLANs and assigned to 5 LANs

Hopefully it will still work after a next pfsense update .

[jasonlitka]

Similar issue over here:

https://forum.pfsense.org/index.php?topic=100293

I had to remove all the vLAN interfaces and recreate them after upgrading to 2.2.x.  If I left the ones from 2.1.x then the upgrade left me with no working 10Gbe ports.

[pgomersbach]

Hi,

We have the same problem on a different intel nic, see: https://forum.pfsense.org/index.php?topic=106109.msg593874#msg593874

I will try the workaround using the LAGG interface