vlans not working in 20200613.0050

msm

My vlans stopped working as of 2.5.0.a.20200613.0050. Stupidly, I saw that 20200613.0650 was available and did not save my /boot/kernel.old before upgrading again.

My non-vlan interfaces work fine.

Is there a way to downgrade to the 20200612-1850 release?

msm

I switched the VLANs from a LAGG interface to non-LAGG interface and now they work.

msm

well for a while anyway. then they stop working and need reboot. still looking into what is happening...

msm

Now getting crashes, not sure if related...

CRASH 1

Filename: /var/crash/info.0
Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 98304
  Blocksize: 512
  Compression: none
  Dumptime: Sat Jun 13 15:00:00 2020
  Hostname: curb.reade.tribeca.com
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE 06bf794ac10(devel-12) pfSense
  Panic String: page fault
  Dump Parity: 1127926089
  Bounds: 0
  Dump Status: good

Filename: /var/crash/info.1
Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 79872
  Blocksize: 512
  Compression: none
  Dumptime: Sat Jun 13 17:30:33 2020
  Hostname: curb.reade.tribeca.com
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE 06bf794ac10(devel-12) pfSense
  Panic String: page fault
  Dump Parity: 173918537
  Bounds: 1
  Dump Status: good

Filename: /var/crash/info.2
Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 98816
  Blocksize: 512
  Compression: none
  Dumptime: Sat Jun 13 18:27:20 2020
  Hostname: curb.reade.tribeca.com
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE 06bf794ac10(devel-12) pfSense
  Panic String: page fault
  Dump Parity: 3142240585
  Bounds: 2
  Dump Status: good

textdump.tar.0.gz

textdump.tar.1.gz

textdump.tar.2.gz

msm

CRASH 2

Filename: /var/crash/info.0
Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 112640
  Blocksize: 512
  Compression: none
  Dumptime: Sat Jun 13 22:50:52 2020
  Hostname: curb.reade.tribeca.com
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE 06bf794ac10(devel-12) pfSense
  Panic String: page fault
  Dump Parity: 4287154505
  Bounds: 0
  Dump Status: good

Filename: /var/crash/info.1
Dump header from device: /dev/ada0p3
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 75776
  Blocksize: 512
  Compression: none
  Dumptime: Sat Jun 13 23:10:56 2020
  Hostname: curb.reade.tribeca.com
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.1-STABLE 06bf794ac10(devel-12) pfSense
  Panic String: page fault
  Dump Parity: 864930121
  Bounds: 1
  Dump Status: good

textdump.tar.0.gz

textdump.tar.1.gz

msm

Crashes may occur just during shutdown.

Toggling hardware offloads in advanced/networking gets the VLAN interfaces into a working state. Seems like something went wrong with interfaces in 20200613.0050

Quarkz

Did you ever figure out what was causing your VLAN issue?

I am having the same issue on every kernel past June 10th.

msm

Nope. For now I just toggle hardware offloads and restart DHCP for ipv6 and seems to get me into a working state.

no1warr1or

Im as well having an issue with VLANs. Will try the hardware offloading trick when I get home.

msm

Just to be a little more specific...

If after upgrade/reboot, you tail -f /var/log/dhcpd.log and you see lots of DHCPREQUEST and DHCPOFFER, but no corresponding DHCPACK, then VLANs are probably not working.

I've found that going to System/Advanced/Networking and toggling any of the options like "Disable hardware checksum offload" and saving that will reset. But then DHCP is left in a weird state (can't read config and any dhcpv6 assignments on Track6 interfaces will be lost). So I stop dhcpd and unbound, then restart unbound.

no1warr1or

After checking the disable hardware options under Advanced Networking my VLANs started working correctly again. I'm going to play around with different combinations to narrow it down and then see what else I can find.

Quarkz

I found you just have to ifconfig down and up the parent interface.
So I created a I script to run with shellcmd at boot up to do that until the developers get a chance to look at things.

I think it may be related to https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=240818

msm

@Quarkz it's ironic that it was working until the freebsd updates were included in the commits that day. would you share your script and let me know how you're running it? I haven't been able to time it successfully.

Quarkz

Sure.

I created the following very simple script in /usr/local/etc/rc.d/emfix.sh make sure to chmod +x it. It will be run via the shellcmd package if put there automatically.

#!/bin/sh
ifconfig em1 down
ifconfig em1 up

Per the bug report you want to make sure to down/up the parent interface (em1 in this instance) and not the vlan interface.

msm

Thanks.

I'm finding that this often does work, but I frequently need to down/up the interfaces more than once, but that's probably just a timing issue with the restart of services. I'm using a LAGG interface as the VLAN parent, so I have to cycle more than one physical interface.

My host has an Intel Atom CPU E3845 with Intel PRO/1000 4-port network interfaces.

Did anyone create a ticket for this?

Quarkz

I have not created a redmine ticket for it as I wasnt sure what additional details may be required by the Netgate team. I assume it may only be relevant to em interfaces?

no1warr1or

Wanted to add an additional note that this is still an issue in the latest dev release. As well that I switched from having all my vlans run over 1 interface (em1) to assigning a separate interface to each of my 3 vlans. (em1,em2,em3). After each restart it has the same behavior where the 2 created vlans will not do anything yet the default lan/vlan still works without issue. If anyone that is experienced with logs wants to point me in the direction of which ones to post I can certainly try to get some more technical data here.

msm

This is resolved for me with latest builds based on FreeBSD 12.2-STABLE