Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14

    Problems Installing or Upgrading pfSense Software
    11
    59
    7.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      peter-v @stephenw10
      last edited by

      @stephenw10 Adding boot_verbose did not give more info, other than the occasional printing of "vlanx: bpf attached" in between the errors.

      Found from the Intel source code that "-14" means "invalid argument".

      It is also not consistent. In one test I moved all the VLAN's from ixl0 (which was throwing the error) to ixl1 (which did not) and then back - error gone. The resulting config.xml did not show any difference from before the moving the vlans back-and-forth.

      After a reboot the errors where back. To provoke the error reliably: press save on an interface detail page (no change needed, just press save on e.g. the WAN page), then "Apply Changes" will throw the error.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Really I think the only way to do this is to try to replicate it in FreeBSD. First in 11.2 and then in 12.

        Unfortunately I don't have access to any ixl NICs to try that. ☹

        Steve

        P 1 Reply Last reply Reply Quote 0
        • P
          peter-v @stephenw10
          last edited by

          @stephenw10 I've ifconfig-ed myself a finger hernia but I cannot get the error that way.
          My feeling is that the error occurs in the calls done to pfSense.so; specifically when interfaces are un-configured before being reconfigured. Since it does not seem to occur when loading the inital config but does when reconfiguring.

          Also tried with the driver in debug mode (compiled with -DIXL_DEBUG, you need to add two functions to the header that Intel seem to have missed) but that did not render any useful output. So I am out of my witz.

          I can give you remote access to a box with IXL interfaces if you like, PM me for that.

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            I'm not the right guy to be doing that. You'd be better off offering in the bug report. Or just updating that with everything you have found to help developers investigating.

            https://redmine.pfsense.org/issues/9123

            Steve

            1 Reply Last reply Reply Quote 0
            • J
              Juve
              last edited by

              Hi,

              I am facing the same issue:

              A system with two XL710 dual port, using a lagg over xl0 and xl2, no traffic is passing inbound

              vlan0: changing name to 'lagg0.50'
              ixl0: aq_add_macvlan err -53, aq_error 14
              ixl0: aq_add_macvlan err -53, aq_error 14
              ixl0: aq_add_macvlan err -53, aq_error 14
              ixl0: aq_add_macvlan err -53, aq_error 14
              vlan1: changing name to 'lagg0.51'
              ixl0: aq_add_macvlan err -53, aq_error 14

              Can't find any solution, I am still investigating right now.
              If anyone has a clue ?

              Thank you all

              1 Reply Last reply Reply Quote 0
              • J
                Juve
                last edited by

                Looks like the error is coming from the way the php module is configuring the vlan on the interface.

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  Data on the bug report suggests this is a FreeBSD 11.2 issue. So try a 2.5 snapshot if you can.

                  Steve

                  1 Reply Last reply Reply Quote 0
                  • X
                    xciter327
                    last edited by xciter327

                    Hi,

                    I too have a X710 based system that I'm testing with. I have had some success with the following tweaks to network adapter under pfSense 2.4.3:

                    ifconfig ixl0 -vlanhwfilter -vlanhwtso -tso
                    ifconfig ixl1 -vlanhwfilter -vlanhwtso -tso
                    ifconfig ixl2 -vlanhwfilter -vlanhwtso -tso
                    ifconfig ixl3 -vlanhwfilter -vlanhwtso -tso

                    To re-iterate, the error was still being thrown, but the system continued to process packets.

                    1 Reply Last reply Reply Quote 0
                    • J
                      Juve
                      last edited by

                      So, I did lot of testing and tried the lastest driver compiled for FreeBSD 11.2. My conclusion is that problems are related to LACP lagg and not the driver itself.

                      If you don't use LACP lagg or use a Failover Lagg there are no issues.
                      If you use LACP mode you will suffer "queue hanging" problems under traffic.
                      If you use a newer driver, the kernel error message "aq_add_macvlan err -53, aq_error 14" isn't present anymore.

                      For the moment I am running stock 2.4.4-P3 with embeded driver (1.9.9.k) with a failover lagg and I am not seeing any issue. The error message logged at configure time (aq_add_macvlan err -53, aq_error 14) seems to be harmless. I stressed the system with iperf (14 threads) during 30 minutes without any packet drop or kernel message about queue hanging.

                      I'll try to see if there are any performance enhancement with the latest driver.

                      X 1 Reply Last reply Reply Quote 1
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        Please add that info to the bug report if you have confirmed it.
                        https://redmine.pfsense.org/issues/9123

                        Steve

                        JeGrJ 1 Reply Last reply Reply Quote 0
                        • X
                          xciter327 @Juve
                          last edited by xciter327

                          @Juve

                          I have got to say in my case the firewall did freeze eventually after seeing those "mcvlan" errors. I've disabled WOL in BIOS which apparently controls like 10 different power saving options. It can also be disabled by an Intel utility without reboot. Since then I've put a firewall under testing on a 10g link and have not been able to crash/freeze it.

                          1 Reply Last reply Reply Quote 1
                          • J
                            Juve
                            last edited by Juve

                            Interesting. The hardware I'm working with is a HPE Proliant DL360. The bios tuning is
                            maximal performance, no virtualization, no C-States, no hyper-threading, no boot on lan

                            looks like IXL and LACP don't play well together at least on 11.x : https://lists.freebsd.org/pipermail/freebsd-net/2016-April/045091.html

                            @stephenw10 , I will do so when I will have tested the system for enougth time to tell it is working as expected. We need to be sure.

                            X 1 Reply Last reply Reply Quote 0
                            • X
                              xciter327 @Juve
                              last edited by

                              @Juve said in New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14:

                              Interesting. The hardware I'm working with is a HPE Proliant DL360. The bios tuning is
                              maximal performance, no virtualization, no C-States, no hyper-threading, no boot on lan

                              looks like IXL and LACP don't play well together at least on 11.x : https://lists.freebsd.org/pipermail/freebsd-net/2016-April/045091.html

                              @stephenw10 , I will do so when I will have tested the system for enougth time to tell it is working as expected. We need to be sure.

                              I'm under the same impression. Check out the advisory document from Intel regarding bugs with that adapter. Many many thing are broken.

                              1 Reply Last reply Reply Quote 0
                              • J
                                Juve
                                last edited by

                                On day of testing my setup:

                                • 2xHPE DL360 with HPE NC562 SFP+ (Intel X710 DA2), 6core @3,4Ghz, 48 GB of RAM.

                                I stressed the boxes with iperf3 because that was all I got at hand.

                                Common tuning:
                                - earlyshellcmd doing "ifconfig ixl -vlanhwtso"
                                - tunable hw.intr_storm_threshold set at 0
                                - IXL cards are configured in two LAGGS configured in failover mode
                                - each interface is a vlan tagged on one of those laggs
                                - /boot/loader.conf.local with
                                net.pf.source_nodes_hashsize="1048576"
                                net.pf.states_hashsize="67108864"

                                testing the box with 1.9.9-k driver (stock FreeSD 11.2):
                                - iperf3 5 thread during 900s with pf enable: 6,13Gbit/s, peak at 4,25MPPS
                                - iperf3 5 thread during 900s with pf disabled: 9,45Gbit/s
                                - one "queue hung" on slave node at boot in dmesg but nothing after
                                - no problem with CARP failover, going from master to slave and failing back as expected in a timely manner

                                testing the box with 1.11.9 driver (lastest on Intel WebSite):
                                - iperf3 5 thread during 900s with pf enable: 6,55Gbit/s, peak at 4,58MPPS
                                - iperf3 5 thread during 900s with pf disabled: 9,45Gbit/s
                                - some "queue hung" on slave node (mainly at boot)
                                - problem with CARP failover, slave node stay Master randomly on different interfaces (IPv4 or IPV6) during few seconds to 10 minutes. A tcpdump on the slave node shows it can't see advertisements packet from master. A tcpdump on master shows both packet from master and slave !

                                to sumup, newest driver seems to improve a litlle the performance using pf.
                                disabling TSO on vlans seems to have solved a lot of problem for me and I also gained arround 900mbit/s of throughput and 300KPPS
                                I decided to stay on stock driver 1.9.9-k without LACP (see LLDP agent problem in Intel document) and with vlanhwtso disabled. I will continue stress testing that setup soon.

                                Side note : for those struggling with IPV6 CARP not willing to configure (the file exists issue). The rule is to no use CAP in hexadecimal numbers and to remove leading zeroes. Ok, but that is not a always wining rule. if you have only one useless 0 in the adress, you should keep that zero. You will see in the shell that even you configured the adress using "::" in the UI, FreeBSD impose the 0. If you have more than one 0 you are good.

                                1 Reply Last reply Reply Quote 0
                                • J
                                  Juve
                                  last edited by

                                  Some followup because it will help others for sure.

                                  After a lot more testing I can confirm that the only working solution for me (= 10 days uptime without any issue) is:

                                  • stock driver 1.9.9k
                                  • failover LAGG
                                  • no IPV6 at all

                                  As soon as you will be using IPV6 you will get hanging queues.

                                  1 Reply Last reply Reply Quote 0
                                  • X
                                    xciter327
                                    last edited by

                                    Thanks for the update. I'm somewhat convinced we are hitting a similar issue, but in different ways.

                                    Do You have "Allow IPv6" in the Advanced/Networking enabled or disabled? We don't actually have IPv6 on any of the locations where we have the issue, but in our office, where we do have IPv6, there isn't an issue. Also we are not using LAGG.

                                    1 Reply Last reply Reply Quote 0
                                    • J
                                      Juve
                                      last edited by

                                      Yes we do have thave "Allow IPv6" enabled.

                                      Remember that I have a shellscript ran at boot that does a "ifconfig -vlanhwtso ixl0..1..2..3"
                                      Without disabling vlanhwtso we quicky have hung queues.

                                      We are still stable after 12 days of uptime. I'll get you posted if we encounter any issue.

                                      X 1 Reply Last reply Reply Quote 1
                                      • X
                                        xciter327 @Juve
                                        last edited by xciter327

                                        @Juve said in New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14:

                                        Yes we do have thave "Allow IPv6" enabled.

                                        Remember that I have a shellscript ran at boot that does a "ifconfig -vlanhwtso ixl0..1..2..3"
                                        Without disabling vlanhwtso we quicky have hung queues.

                                        We are still stable after 12 days of uptime. I'll get you posted if we encounter any issue.

                                        Do You actually use VLANs on the interfaces?

                                        1 Reply Last reply Reply Quote 0
                                        • J
                                          Juve
                                          last edited by

                                          Yes we do.
                                          There are dozens of vlan used.

                                          1 Reply Last reply Reply Quote 0
                                          • J
                                            Juve
                                            last edited by

                                            We experienced a hard reboot during VLAN configuration on the slave node.
                                            Looks like problem is only occuring when reconfiguring interfaces.

                                            I'll keep you updated

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post