WAN flapping on 2.4.5-p1

larold42

@Tzvia i'll give the dumb switch idea a shot. Didnt even think of that. So old hardware has intel 82583v.
New hardware is 1x intel 219-LM, 1x intel 211-AT, 4x intel 350-AM4. I may try and figure out which port is which model and then make the WAN a new port to try an troubleshoot.

larold42

@Tzvia soooooo i put a dumb switch in front of my pfsense.... and i i havent had an issue. How is this a solution. This isnt even an additional hop. I'm thinking the route cause is the NIC driver not being compatable with freebsd 11.2/3. Someone had a similarish experience https://redmine.pfsense.org/issues/9414, https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=235147. Different intel ethernet controller than what i'm rocking, but i may do some troubleshooting and see if i can kick this over to freeBSD bug report.

Tzvia

@larold42 Yea I know it's nuts. Hope to get time this weekend to test out a few ideas. I had played a bit with the hardware offload settings, didn't seem to make a difference. Setting it to gig did on the ancient HP server pull nic I had. But when I put in the 350t4 nothing I did would stop the flapping until I put in the dumb switch- I had read about someone else doing that earlier and figured it couldn't hurt to try. Guessing it's something between that Hitron modem nic and my nic, the autosense not stable on the Hitron with the new nic on the router. PFSense sees pings failing, and restarts that interface, and we see it as flapping. Bet if I could get into the Hitron settings and set it to gig full duplex that would fix it. But it's a Spectrum 'free' modem and I don't have a way into the settings...

larold42

@Tzvia just for tracking NIC hardware and Driver information so we can compare. I'm half tempted to put in a bug report for freeBSD 11.3.

igb0@pci0:1:0:0:        class=0x020000 card=0x0000ffff chip=0x15218086 rev=0x01 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xdf160000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0xe060, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xdf18c000, size 16384, enabled
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 10 messages, enabled
                 Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 003018ffff0f0d21
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 8 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0004
                     VF Device ID 0x1520
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 0017[1a0] = TPH Requester 1
    ecap 0018[1c0] = LTR 1
    ecap 000d[1d0] = ACS 1

Driver info

dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.%parent:

Tzvia

@larold42
Can't figure the upload here, but here is my NIC setup:

dev.igb.3.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.2.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.1.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k
dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k

And the NICs are:

igb3@pci0:1:0:3: class=0x020000 card=0x03091dcf chip=0x15218086 rev=0x01 hdr=0x00
vendor = 'Intel Corporation'
device = 'I350 Gigabit Network Connection'
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfe580000, size 524288, enabled
bar [18] = type I/O Port, range 32, base 0xe000, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfe900000, size 16384, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 1 message, 64 bit, vector masks
cap 11[70] = MSI-X supports 10 messages, enabled
Table in map 0x1c[0x0], PBA in map 0x1c[0x2000]
cap 10[a0] = PCI-Express 2 endpoint max data 512(512) FLR NS
link x4(x4) speed 5.0(5.0) ASPM disabled(L0s/L1)
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 80615fffff08059c
ecap 000e[150] = ARI 1
ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
0 VFs configured out of 8 supported
First VF RID Offset 0x0180, VF RID Stride 0x0004
VF Device ID 0x1520
Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
ecap 0017[1a0] = TPH Requester 1
ecap 000d[1d0] = ACS 1

Working from home means I have to keep the network UP all week so troubleshooting is limited till the weekend, but I did try setting the WAN to auto again now that it's connected to a dumb switch, and the problem flapping began again. The Hitron modem remained connected to the switch. Without the switch, I could see the link light going out every 8~10 seconds on the modem as well as on the PFSense WAN link. I unplugged the Hitron modem leaving the switch in place, and the PFSense WAN link kept flapping so I don't now think that anything about that modem is a part of this in my setup. It's strictly PFSense. I also tried setting the MTU manually along with MSS clamping to 1460 and that made no change.

larold42

@Tzvia yea so the short answer is we need to patch, https://downloadcenter.intel.com/download/15815/Intel-Network-Adapter-Driver-for-82575-6-and-82580-Based-Gigabit-Network-Connections-under-FreeBSD-?product=46827. Or upgrade to 2.5, which "i believe" has support for these newer drivers because its based on freebsd12. Still have to see if these drivers are in there.

bmeeks

@larold42 said in WAN flapping on 2.4.5-p1:

@Tzvia yea so the short answer is we need to patch, https://downloadcenter.intel.com/download/15815/Intel-Network-Adapter-Driver-for-82575-6-and-82580-Based-Gigabit-Network-Connections-under-FreeBSD-?product=46827. Or upgrade to 2.5, which "i believe" has support for these newer drivers because its based on freebsd12. Still have to see if these drivers are in there.

The drivers you download from Intel will usually be for FreeBSD-11 and earlier. As I mentioned in another thread similar to this one, FreeBSD-12 and newer uses the iflib API wrapper for NIC drivers. So that is a completely different type of driver software, and in the case of Intel even has a different version numbering scheme. So Intel and others have a situation now where they sort of have to maintain two different driver familes: one for FreeBSD-11 and another for iflib and FreeBSD-12 and newer. That sets up a situation where things may get fixed in one family but not necessarily get backported to the other (or at least not at the same time).

bfeitell

Please check the MTU on the flapping interface. This sounds familiar:

https://forum.netgate.com/topic/136089/solved-and-revised-2-4-4-release-arpresolve-can-t-allocate-llinfo-for-gateway-on-interface0-dhcp-mtu-576?_=1604207774054

Cheers,
Bennett

Tzvia

@bfeitell Interesting read. I had played a bit with it this weekend. I had played with traffic shaping about a year ago, thought I had removed all the bits I had configured, but found a WAN interface limit that was well below the current speed tier I have. Removed that. I then found that I had set the LAN interface speed/duplex to 1000f at some point in the distant past, so set that to 'Default, no preference' along with the WAN. Connected to the switch it comes up 1000f stable. If I connect direct to the modem, it no longer flaps but it comes up 100f with the modem at 1000f so it still doesn't work. But it doesn't flap... So I set "supersede interface-mtu 0" per that link you found, and am just letting that alone for a bit to verify it hasn't introduced anything negative. In a while I will try connecting it to the switch...

Tzvia

Just had to try a few more things because I am stubborn. Character flaw, but I think at this point I am now done. There is no fix for this NIC/Hitron modem combo. I grabbed another SSD I had around, installed the 2.5 devel, did basic setup only on it- WAN set dhcp, lan set static ipv4. Nothing else. Connected the WAN to the Hitron modem and it flapped maybe twice, then no lights on modem or PFSense NIC. They just don't talk to each other. Connect the WAN to that little Netgear dumb switch and it's fine. So it stays like that till I can think of something else. BTW I also booted that SSD on an old test computer I have that has an old 2 port intel PRO 1000 NIC that also uses the IGB driver and it flapped then lights out as well.