New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14
-
On day of testing my setup:
- 2xHPE DL360 with HPE NC562 SFP+ (Intel X710 DA2), 6core @3,4Ghz, 48 GB of RAM.
I stressed the boxes with iperf3 because that was all I got at hand.
Common tuning:
- earlyshellcmd doing "ifconfig ixl -vlanhwtso"
- tunable hw.intr_storm_threshold set at 0
- IXL cards are configured in two LAGGS configured in failover mode
- each interface is a vlan tagged on one of those laggs
- /boot/loader.conf.local with
net.pf.source_nodes_hashsize="1048576"
net.pf.states_hashsize="67108864"testing the box with 1.9.9-k driver (stock FreeSD 11.2):
- iperf3 5 thread during 900s with pf enable: 6,13Gbit/s, peak at 4,25MPPS
- iperf3 5 thread during 900s with pf disabled: 9,45Gbit/s
- one "queue hung" on slave node at boot in dmesg but nothing after
- no problem with CARP failover, going from master to slave and failing back as expected in a timely mannertesting the box with 1.11.9 driver (lastest on Intel WebSite):
- iperf3 5 thread during 900s with pf enable: 6,55Gbit/s, peak at 4,58MPPS
- iperf3 5 thread during 900s with pf disabled: 9,45Gbit/s
- some "queue hung" on slave node (mainly at boot)
- problem with CARP failover, slave node stay Master randomly on different interfaces (IPv4 or IPV6) during few seconds to 10 minutes. A tcpdump on the slave node shows it can't see advertisements packet from master. A tcpdump on master shows both packet from master and slave !to sumup, newest driver seems to improve a litlle the performance using pf.
disabling TSO on vlans seems to have solved a lot of problem for me and I also gained arround 900mbit/s of throughput and 300KPPS
I decided to stay on stock driver 1.9.9-k without LACP (see LLDP agent problem in Intel document) and with vlanhwtso disabled. I will continue stress testing that setup soon.Side note : for those struggling with IPV6 CARP not willing to configure (the file exists issue). The rule is to no use CAP in hexadecimal numbers and to remove leading zeroes. Ok, but that is not a always wining rule. if you have only one useless 0 in the adress, you should keep that zero. You will see in the shell that even you configured the adress using "::" in the UI, FreeBSD impose the 0. If you have more than one 0 you are good.
-
Some followup because it will help others for sure.
After a lot more testing I can confirm that the only working solution for me (= 10 days uptime without any issue) is:
- stock driver 1.9.9k
- failover LAGG
- no IPV6 at all
As soon as you will be using IPV6 you will get hanging queues.
-
Thanks for the update. I'm somewhat convinced we are hitting a similar issue, but in different ways.
Do You have "Allow IPv6" in the Advanced/Networking enabled or disabled? We don't actually have IPv6 on any of the locations where we have the issue, but in our office, where we do have IPv6, there isn't an issue. Also we are not using LAGG.
-
Yes we do have thave "Allow IPv6" enabled.
Remember that I have a shellscript ran at boot that does a "ifconfig -vlanhwtso ixl0..1..2..3"
Without disabling vlanhwtso we quicky have hung queues.We are still stable after 12 days of uptime. I'll get you posted if we encounter any issue.
-
@Juve said in New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14:
Yes we do have thave "Allow IPv6" enabled.
Remember that I have a shellscript ran at boot that does a "ifconfig -vlanhwtso ixl0..1..2..3"
Without disabling vlanhwtso we quicky have hung queues.We are still stable after 12 days of uptime. I'll get you posted if we encounter any issue.
Do You actually use VLANs on the interfaces?
-
Yes we do.
There are dozens of vlan used. -
We experienced a hard reboot during VLAN configuration on the slave node.
Looks like problem is only occuring when reconfiguring interfaces.I'll keep you updated
-
Regarding the crash he message was:
Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x20:0xffffffff80e38d40
stack pointer = 0x28:0xfffffe0b9ceba130
frame pointer = 0x28:0xfffffe0b9ceba170
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (swi4: clock (0))the trace is :
Tracing pid 12 tid 100034 td 0xfffff8000a331620
carp_master_down_locked() at carp_master_down_locked+0xf0/frame 0xfffffe0b9ceba170
carp_master_down() at carp_master_down+0x21/frame 0xfffffe0b9ceba190
softclock_call_cc() at softclock_call_cc+0x13a/frame 0xfffffe0b9ceba240
softclock() at softclock+0x79/frame 0xfffffe0b9ceba260
intr_event_execute_handlers() at intr_event_execute_handlers+0xe9/frame 0xfffffe0b9ceba2a0
ithread_loop() at ithread_loop+0xe7/frame 0xfffffe0b9ceba2f0
fork_exit() at fork_exit+0x83/frame 0xfffffe0b9ceba330
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0b9ceba330
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---Regarding IXL driver, I noticed that a 1.11.20 version was released in september but but download page wasn't displaying the good driver (IAVF instead of IXL).
So I emailed Intel today and they fixed the issue and we can now download the 1.11.20 version.
We will try it tomorrow. -
So here we are after a lot of trial:
-
we know and are sure that reconfiguring interface capabilities (TSO) while transmitting trafic is what is causing queue hang (like explained in the FreeBSD ticket)
-
we know adding vlan with 1.9.9k produce a queue hang and a mac vlan error
-
we know IPV6 was causing hang, but we did not test it again in the last setup (see below)
-
the hard reboot is caused by an issue not directly related to the interface problem but a result of a lock when a queue hang
-
we did compile and install version 1.11.20 of the IXL driver and we did remove the execution of our script that was removing HWVLANTSO at boot because when it was executed the NIC was already transmitting trafic and we had one or two queue hung at boot. This would cause issues later on.
-
we did test adding vlan this morning and nothing bad happened, no error message, no queue hang,
So, we start a new phase of monitoring of how it is going.
-
-
A quick FollowUP running 1.11.20 :
- 9 days uptime
- no error so far.
- throuhput is good, no issue
- we added 6 new vlan in the last 9 Days and everything works as expected for the moment.
- we did test multiple master/slave failover/failback, no issues
I'll continue to keep you informed.
-
@stephenw10 said in New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14:
Please add that info to the bug report if you have confirmed it.
https://redmine.pfsense.org/issues/9123Steve
@Juve could you add those intel to the ticket, Stephen mentioned?
It would provide additional help/info.also @stephenw10 if that driver release seems to solve the problem, how are chances that it be included in a 2.4.5 release? As we're currently having information from two customers they are seeing similar problems (one with "no response - dead" pfsense when adding VLANs, one having a reboot with queue and VLAN errors shown), that would be a great thing.
best regards
-
If it's in 11 stable it would likely make it into a 2.4.X release. The drivers in 12 look to be significantly different, I'm not sure that could be brought back.
Steve
-
I did update the ticket :-)
-
Quick update:
50 days uptime, no error so far.
Looks like the driver was the issue. -
@Juve said in New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14:
Quick update:
50 days uptime, no error so far.
Looks like the driver was the issue.What were the latest changes you were running? Latest driver as you wrote (1.11.20) and HWVLANTSO removal?
If so, any chance @stephenw10 to bring that driver version into the mix for the upcoming 2.4.5?
-
It is still super stable.
The only two things we did at the end are:
- use the latest driver
- use failover LAGG and not LACP lagg (but we did not test LACP with latest driver so we can't confirm is is not working with it)
no VLANTSO removal etc.
-
Is anyone still seeing this with the 1.11.9 driver in 2.4.5?
Steve
-
@stephenw10
with 2.4.5 no error (LACP LAGG, 15 VLANs) -
Ah, good. We were trying to replicate it in 2.4.5 and have not done so.
If anyone is seeing it though please let us know the steps required.
Steve
-
Hi, I think I can reproduce this problem with pfSense version 2.4.5-RELEASE-p1 (amd64) and native driver 1.11.9-k.
We have ~45 vlans on one of the interfaces, adding a new vlan on this interface leads to impossibility to ping / reach the system on any of the sub-interfaces related.
Console prompts "queue appears hung" messages, reboot seems to be the only solution to get the system working again (new vlan included).
I did set net.inet.tcp.tso = 0 via Advanced / System tunables : the messages don't show anymore, but still the same problem (system unreachable when adding vlan, reboot needed).Tested on X710 and X722 Intel NICs, Supermicro server.
I will test upgrading the Intel driver and/or VLANTSO removal, keep you informed.
Don't hesitate to ask if I can provide more info.
Have a nice day,
Fabien
