New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14

peter-v

@stephenw10 said in New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14:

It does look like there have been some driver updates in FreeBSD that might apply to this.
If you're able to test FreeBSD 11-stable or 12 that would be useful.

Steve

I can help, but don't know how to upgrade to a newer version of FreeBSD in PFSense. The only way I know how to upgrade FreeBSD, freebsd-update, does not exist - so a brief instruction would be helpful.

stephenw10

You would need to install FreeBSD instead of pfSense really. The changes look significant, I don't think it would load into 11.2 from 11-stable. Our current dev snapshots are still built on 11.2.

Steve

peter-v

I've found that if I disable a physical interface (the parent so to speak) and only have tagged vlans on that interface, the error does not show. Also, if the error does occur, it can provoke a kernel panic immediately or at shutdown. There is definitely something fishy going on.

Is there a way to log the commands that are applied after a save in the web interface so I can further debug which command is causing the problem?

stephenw10

Not really. The best you can probably do is boot in verbose mode to log more debug info. Other then recompiling the driver with debugging enabled but that will probably give you far too much detail.

Add to /boot/loader.conf.local:
boot_verbose="YES"

Steve

peter-v

@stephenw10 Adding boot_verbose did not give more info, other than the occasional printing of "vlanx: bpf attached" in between the errors.

Found from the Intel source code that "-14" means "invalid argument".

It is also not consistent. In one test I moved all the VLAN's from ixl0 (which was throwing the error) to ixl1 (which did not) and then back - error gone. The resulting config.xml did not show any difference from before the moving the vlans back-and-forth.

After a reboot the errors where back. To provoke the error reliably: press save on an interface detail page (no change needed, just press save on e.g. the WAN page), then "Apply Changes" will throw the error.

stephenw10

Really I think the only way to do this is to try to replicate it in FreeBSD. First in 11.2 and then in 12.

Unfortunately I don't have access to any ixl NICs to try that.

Steve

peter-v

@stephenw10 I've ifconfig-ed myself a finger hernia but I cannot get the error that way.
My feeling is that the error occurs in the calls done to pfSense.so; specifically when interfaces are un-configured before being reconfigured. Since it does not seem to occur when loading the inital config but does when reconfiguring.

Also tried with the driver in debug mode (compiled with -DIXL_DEBUG, you need to add two functions to the header that Intel seem to have missed) but that did not render any useful output. So I am out of my witz.

I can give you remote access to a box with IXL interfaces if you like, PM me for that.

stephenw10

I'm not the right guy to be doing that. You'd be better off offering in the bug report. Or just updating that with everything you have found to help developers investigating.

https://redmine.pfsense.org/issues/9123

Steve

Juve

Hi,

I am facing the same issue:

A system with two XL710 dual port, using a lagg over xl0 and xl2, no traffic is passing inbound

vlan0: changing name to 'lagg0.50'
ixl0: aq_add_macvlan err -53, aq_error 14
ixl0: aq_add_macvlan err -53, aq_error 14
ixl0: aq_add_macvlan err -53, aq_error 14
ixl0: aq_add_macvlan err -53, aq_error 14
vlan1: changing name to 'lagg0.51'
ixl0: aq_add_macvlan err -53, aq_error 14

Can't find any solution, I am still investigating right now.
If anyone has a clue ?

Thank you all

Juve

Looks like the error is coming from the way the php module is configuring the vlan on the interface.

stephenw10

Data on the bug report suggests this is a FreeBSD 11.2 issue. So try a 2.5 snapshot if you can.

Steve

xciter327

Hi,

I too have a X710 based system that I'm testing with. I have had some success with the following tweaks to network adapter under pfSense 2.4.3:

ifconfig ixl0 -vlanhwfilter -vlanhwtso -tso
ifconfig ixl1 -vlanhwfilter -vlanhwtso -tso
ifconfig ixl2 -vlanhwfilter -vlanhwtso -tso
ifconfig ixl3 -vlanhwfilter -vlanhwtso -tso

To re-iterate, the error was still being thrown, but the system continued to process packets.

Juve

So, I did lot of testing and tried the lastest driver compiled for FreeBSD 11.2. My conclusion is that problems are related to LACP lagg and not the driver itself.

If you don't use LACP lagg or use a Failover Lagg there are no issues.
If you use LACP mode you will suffer "queue hanging" problems under traffic.
If you use a newer driver, the kernel error message "aq_add_macvlan err -53, aq_error 14" isn't present anymore.

For the moment I am running stock 2.4.4-P3 with embeded driver (1.9.9.k) with a failover lagg and I am not seeing any issue. The error message logged at configure time (aq_add_macvlan err -53, aq_error 14) seems to be harmless. I stressed the system with iperf (14 threads) during 30 minutes without any packet drop or kernel message about queue hanging.

I'll try to see if there are any performance enhancement with the latest driver.

stephenw10

Please add that info to the bug report if you have confirmed it.
https://redmine.pfsense.org/issues/9123

Steve

xciter327

@Juve

I have got to say in my case the firewall did freeze eventually after seeing those "mcvlan" errors. I've disabled WOL in BIOS which apparently controls like 10 different power saving options. It can also be disabled by an Intel utility without reboot. Since then I've put a firewall under testing on a 10g link and have not been able to crash/freeze it.

Juve

Interesting. The hardware I'm working with is a HPE Proliant DL360. The bios tuning is
maximal performance, no virtualization, no C-States, no hyper-threading, no boot on lan

looks like IXL and LACP don't play well together at least on 11.x : https://lists.freebsd.org/pipermail/freebsd-net/2016-April/045091.html

@stephenw10 , I will do so when I will have tested the system for enougth time to tell it is working as expected. We need to be sure.

xciter327

@Juve said in New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14:

Interesting. The hardware I'm working with is a HPE Proliant DL360. The bios tuning is
maximal performance, no virtualization, no C-States, no hyper-threading, no boot on lan

looks like IXL and LACP don't play well together at least on 11.x : https://lists.freebsd.org/pipermail/freebsd-net/2016-April/045091.html

@stephenw10 , I will do so when I will have tested the system for enougth time to tell it is working as expected. We need to be sure.

I'm under the same impression. Check out the advisory document from Intel regarding bugs with that adapter. Many many thing are broken.

Juve

On day of testing my setup:

2xHPE DL360 with HPE NC562 SFP+ (Intel X710 DA2), 6core @3,4Ghz, 48 GB of RAM.

I stressed the boxes with iperf3 because that was all I got at hand.

Common tuning:
- earlyshellcmd doing "ifconfig ixl -vlanhwtso"
- tunable hw.intr_storm_threshold set at 0
- IXL cards are configured in two LAGGS configured in failover mode
- each interface is a vlan tagged on one of those laggs
- /boot/loader.conf.local with
net.pf.source_nodes_hashsize="1048576"
net.pf.states_hashsize="67108864"

testing the box with 1.9.9-k driver (stock FreeSD 11.2):
- iperf3 5 thread during 900s with pf enable: 6,13Gbit/s, peak at 4,25MPPS
- iperf3 5 thread during 900s with pf disabled: 9,45Gbit/s
- one "queue hung" on slave node at boot in dmesg but nothing after
- no problem with CARP failover, going from master to slave and failing back as expected in a timely manner

testing the box with 1.11.9 driver (lastest on Intel WebSite):
- iperf3 5 thread during 900s with pf enable: 6,55Gbit/s, peak at 4,58MPPS
- iperf3 5 thread during 900s with pf disabled: 9,45Gbit/s
- some "queue hung" on slave node (mainly at boot)
- problem with CARP failover, slave node stay Master randomly on different interfaces (IPv4 or IPV6) during few seconds to 10 minutes. A tcpdump on the slave node shows it can't see advertisements packet from master. A tcpdump on master shows both packet from master and slave !

to sumup, newest driver seems to improve a litlle the performance using pf.
disabling TSO on vlans seems to have solved a lot of problem for me and I also gained arround 900mbit/s of throughput and 300KPPS
I decided to stay on stock driver 1.9.9-k without LACP (see LLDP agent problem in Intel document) and with vlanhwtso disabled. I will continue stress testing that setup soon.

Side note : for those struggling with IPV6 CARP not willing to configure (the file exists issue). The rule is to no use CAP in hexadecimal numbers and to remove leading zeroes. Ok, but that is not a always wining rule. if you have only one useless 0 in the adress, you should keep that zero. You will see in the shell that even you configured the adress using "::" in the UI, FreeBSD impose the 0. If you have more than one 0 you are good.

Juve

Some followup because it will help others for sure.

After a lot more testing I can confirm that the only working solution for me (= 10 days uptime without any issue) is:

stock driver 1.9.9k
failover LAGG
no IPV6 at all

As soon as you will be using IPV6 you will get hanging queues.

xciter327

Thanks for the update. I'm somewhat convinced we are hitting a similar issue, but in different ways.

Do You have "Allow IPv6" in the Advanced/Networking enabled or disabled? We don't actually have IPv6 on any of the locations where we have the issue, but in our office, where we do have IPv6, there isn't an issue. Also we are not using LAGG.