Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14

    Scheduled Pinned Locked Moved Problems Installing or Upgrading pfSense Software
    59 Posts 11 Posters 13.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      Hmm, there does seem to be something that has snuck in here.

      The previous reports of this suggested that VLANs already added to the interface in 2.4.3 were not affected and continued to function. That implies it's something in the actual addition process that is triggering the error.
      It would be interesting to try manually editing the config to add a new VLAN with an ixl parent and see if that works.

      If that was the case though you would think that simply rebooting after adding those new VLANs would bring them up correctly.

      It does seem to be VLAN hardware offloading failing.

      Steve

      1 Reply Last reply Reply Quote 0
      • JeGrJ
        JeGr LAYER 8 Moderator
        last edited by

        Does is have something to do with the older "error" in this thread that mentioned the problem would be gone with a further driver update? Could this be related to a newer driver or driver changes to ixl on FreeBSD 11.2 perhaps?

        https://communities.intel.com/thread/103549

        Otherwise the VLANs came up alright, what I did see was CARP on those VLAN interfaces somewhat "jittery". If you refresh CARP status on both nodes, you could see the them switching master roles very very shortly but noticable for a second. After witnessing this, I rebooted both nodes. After a bit of research this weekend I found this thread and tried salvaging the reboot log from those boxes:

        Nov 2 15:48:06	kernel		done.
Nov 2 15:48:05	php-cgi		rc.bootup: Configuring CARP settings finalize...
Nov 2 15:48:05	php-cgi		rc.bootup: pfsync done in 0 seconds.
Nov 2 15:48:05	php-fpm	334	/rc.carpbackup: HA cluster member "(192.168.91.4@ixl0.91): (V091_PHONE)" has resumed CARP state "BACKUP" for vhid 4
Nov 2 15:48:05	php-fpm	334	/rc.carpbackup: HA cluster member "(192.168.82.4@ixl0.82): (V082_BZD)" has resumed CARP state "BACKUP" for vhid 4
Nov 2 15:48:05	php-fpm	334	/rc.carpbackup: HA cluster member "(192.168.80.4@ixl0.80): (V080_HNR)" has resumed CARP state "BACKUP" for vhid 4
Nov 2 15:48:05	php-cgi		rc.bootup: waiting for pfsync...
Nov 2 15:48:05	php-fpm	335	/rc.carpbackup: HA cluster member "(192.168.95.4@ixl0.95): (V095_ADMIN)" has resumed CARP state "BACKUP" for vhid 4
Nov 2 15:48:05	php-fpm	334	/rc.carpbackup: HA cluster member "(10.0.0.4@ixl0.10): (V010_VERWA)" has resumed CARP state "BACKUP" for vhid 4
Nov 2 15:48:04	kernel		carp: 4@ixl0.91: INIT -> BACKUP (initialization complete)
Nov 2 15:48:04	kernel		ixl0.91: promiscuous mode enabled
Nov 2 15:48:04	check_reload_status		Carp backup event
Nov 2 15:48:04	kernel		carp: 4@ixl0.82: INIT -> BACKUP (initialization complete)
Nov 2 15:48:04	kernel		ixl0.82: promiscuous mode enabled
Nov 2 15:48:04	check_reload_status		Carp backup event
Nov 2 15:48:04	kernel		carp: 4@ixl0.80: INIT -> BACKUP (initialization complete)
Nov 2 15:48:04	kernel		ixl0.80: promiscuous mode enabled
Nov 2 15:48:04	check_reload_status		Carp backup event
Nov 2 15:48:04	kernel		carp: 4@ixl0.10: INIT -> BACKUP (initialization complete)
Nov 2 15:48:04	kernel		ixl0.10: promiscuous mode enabled
Nov 2 15:48:04	kernel		carp: demoted by 240 to 720 (interface down)
Nov 2 15:48:04	kernel		igb4: promiscuous mode enabled
Nov 2 15:48:04	kernel		carp: demoted by 240 to 480 (interface down)
Nov 2 15:48:04	kernel		igb0: promiscuous mode enabled
Nov 2 15:48:04	check_reload_status		Carp backup event
Nov 2 15:48:04	kernel		carp: 4@ixl0.95: INIT -> BACKUP (initialization complete)
Nov 2 15:48:04	kernel		ixl0.95: promiscuous mode enabled
Nov 2 15:48:04	kernel		ixl0: promiscuous mode enabled
Nov 2 15:48:04	kernel		carp: demoted by 240 to 240 (interface down)
Nov 2 15:48:04	kernel		igb1: promiscuous mode enabled
Nov 2 15:48:04	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:04	check_reload_status		Carp backup event
Nov 2 15:48:04	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:04	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:04	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:04	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:04	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:04	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:04	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:04	kernel		done.
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	kernel		done.
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	kernel		vlan4: changing name to 'ixl0.95'
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	syslogd		Logging subprocess 11242 (exec /usr/local/sbin/sshguard) exited due to signal 15.
Nov 2 15:48:03	sshd	10982	Server listening on 0.0.0.0 port 22.
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	sshd	10982	Server listening on :: port 22.
Nov 2 15:48:03	kernel		vlan3: changing name to 'ixl0.91'
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:03	kernel		vlan2: changing name to 'ixl0.82'
Nov 2 15:48:03	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:02	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:02	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:02	kernel		vlan1: changing name to 'ixl0.80'
Nov 2 15:48:02	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:02	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:02	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:02	kernel		ixl0: aq_add_macvlan err -53, aq_error 14
Nov 2 15:48:02	kernel		vlan0: changing name to 'ixl0.10'
Nov 2 15:48:02	kernel		device_attach: est3 attach returned 6
Nov 2 15:48:02	kernel		est: cpu_vendor GenuineIntel, msr 211200002200
Nov 2 15:48:02	kernel		est: CPU supports Enhanced Speedstep, but is not recognized.
Nov 2 15:48:02	kernel		est3: <Enhanced SpeedStep Frequency Control> on cpu3
Nov 2 15:48:02	kernel		coretemp3: <CPU On-Die Thermal Sensors> on cpu3

        (log is newest on top)
        Seems to me that assigning and renaming the VLANs somehow triggers that error, too.

        Don't forget to upvote 👍 those who kindly offered their time and brainpower to help you!

        If you're interested, I'm available to discuss details of German-speaking paid support (for companies) if needed.

        1 Reply Last reply Reply Quote 1
        • S
          sde
          last edited by

          I reported a bug for this case.
          https://redmine.pfsense.org/issues/9123

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            It does look like there have been some driver updates in FreeBSD that might apply to this.
            If you're able to test FreeBSD 11-stable or 12 that would be useful.

            Steve

            P 1 Reply Last reply Reply Quote 0
            • JeGrJ
              JeGr LAYER 8 Moderator
              last edited by

              I'd like to help but as those are remote installations from a customer, I'm not at freedom to drive there, pull out the standby one and throw FreeBSD on it. ;) I'd like to (at least to help sort things out), but unfortunately that will be a hard one.

              Don't forget to upvote 👍 those who kindly offered their time and brainpower to help you!

              If you're interested, I'm available to discuss details of German-speaking paid support (for companies) if needed.

              1 Reply Last reply Reply Quote 0
              • P
                peter-v @stephenw10
                last edited by peter-v

                @stephenw10 said in New Version 2.4.4 - Interface Error --> aq_add_macvlan err -53, aq_error 14:

                It does look like there have been some driver updates in FreeBSD that might apply to this.
                If you're able to test FreeBSD 11-stable or 12 that would be useful.

                Steve

                I can help, but don't know how to upgrade to a newer version of FreeBSD in PFSense. The only way I know how to upgrade FreeBSD, freebsd-update, does not exist - so a brief instruction would be helpful.

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by

                  You would need to install FreeBSD instead of pfSense really. The changes look significant, I don't think it would load into 11.2 from 11-stable. Our current dev snapshots are still built on 11.2.

                  Steve

                  1 Reply Last reply Reply Quote 0
                  • P
                    peter-v
                    last edited by peter-v

                    I've found that if I disable a physical interface (the parent so to speak) and only have tagged vlans on that interface, the error does not show. Also, if the error does occur, it can provoke a kernel panic immediately or at shutdown. There is definitely something fishy going on.

                    Is there a way to log the commands that are applied after a save in the web interface so I can further debug which command is causing the problem?

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Not really. The best you can probably do is boot in verbose mode to log more debug info. Other then recompiling the driver with debugging enabled but that will probably give you far too much detail.

                      Add to /boot/loader.conf.local:
                      boot_verbose="YES"

                      Steve

                      P 1 Reply Last reply Reply Quote 0
                      • P
                        peter-v @stephenw10
                        last edited by

                        @stephenw10 Adding boot_verbose did not give more info, other than the occasional printing of "vlanx: bpf attached" in between the errors.

                        Found from the Intel source code that "-14" means "invalid argument".

                        It is also not consistent. In one test I moved all the VLAN's from ixl0 (which was throwing the error) to ixl1 (which did not) and then back - error gone. The resulting config.xml did not show any difference from before the moving the vlans back-and-forth.

                        After a reboot the errors where back. To provoke the error reliably: press save on an interface detail page (no change needed, just press save on e.g. the WAN page), then "Apply Changes" will throw the error.

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Really I think the only way to do this is to try to replicate it in FreeBSD. First in 11.2 and then in 12.

                          Unfortunately I don't have access to any ixl NICs to try that. ☹

                          Steve

                          P 1 Reply Last reply Reply Quote 0
                          • P
                            peter-v @stephenw10
                            last edited by

                            @stephenw10 I've ifconfig-ed myself a finger hernia but I cannot get the error that way.
                            My feeling is that the error occurs in the calls done to pfSense.so; specifically when interfaces are un-configured before being reconfigured. Since it does not seem to occur when loading the inital config but does when reconfiguring.

                            Also tried with the driver in debug mode (compiled with -DIXL_DEBUG, you need to add two functions to the header that Intel seem to have missed) but that did not render any useful output. So I am out of my witz.

                            I can give you remote access to a box with IXL interfaces if you like, PM me for that.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              I'm not the right guy to be doing that. You'd be better off offering in the bug report. Or just updating that with everything you have found to help developers investigating.

                              https://redmine.pfsense.org/issues/9123

                              Steve

                              1 Reply Last reply Reply Quote 0
                              • J
                                Juve
                                last edited by

                                Hi,

                                I am facing the same issue:

                                A system with two XL710 dual port, using a lagg over xl0 and xl2, no traffic is passing inbound

                                vlan0: changing name to 'lagg0.50'
                                ixl0: aq_add_macvlan err -53, aq_error 14
                                ixl0: aq_add_macvlan err -53, aq_error 14
                                ixl0: aq_add_macvlan err -53, aq_error 14
                                ixl0: aq_add_macvlan err -53, aq_error 14
                                vlan1: changing name to 'lagg0.51'
                                ixl0: aq_add_macvlan err -53, aq_error 14

                                Can't find any solution, I am still investigating right now.
                                If anyone has a clue ?

                                Thank you all

                                1 Reply Last reply Reply Quote 0
                                • J
                                  Juve
                                  last edited by

                                  Looks like the error is coming from the way the php module is configuring the vlan on the interface.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Data on the bug report suggests this is a FreeBSD 11.2 issue. So try a 2.5 snapshot if you can.

                                    Steve

                                    1 Reply Last reply Reply Quote 0
                                    • X
                                      xciter327
                                      last edited by xciter327

                                      Hi,

                                      I too have a X710 based system that I'm testing with. I have had some success with the following tweaks to network adapter under pfSense 2.4.3:

                                      ifconfig ixl0 -vlanhwfilter -vlanhwtso -tso
                                      ifconfig ixl1 -vlanhwfilter -vlanhwtso -tso
                                      ifconfig ixl2 -vlanhwfilter -vlanhwtso -tso
                                      ifconfig ixl3 -vlanhwfilter -vlanhwtso -tso

                                      To re-iterate, the error was still being thrown, but the system continued to process packets.

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        Juve
                                        last edited by

                                        So, I did lot of testing and tried the lastest driver compiled for FreeBSD 11.2. My conclusion is that problems are related to LACP lagg and not the driver itself.

                                        If you don't use LACP lagg or use a Failover Lagg there are no issues.
                                        If you use LACP mode you will suffer "queue hanging" problems under traffic.
                                        If you use a newer driver, the kernel error message "aq_add_macvlan err -53, aq_error 14" isn't present anymore.

                                        For the moment I am running stock 2.4.4-P3 with embeded driver (1.9.9.k) with a failover lagg and I am not seeing any issue. The error message logged at configure time (aq_add_macvlan err -53, aq_error 14) seems to be harmless. I stressed the system with iperf (14 threads) during 30 minutes without any packet drop or kernel message about queue hanging.

                                        I'll try to see if there are any performance enhancement with the latest driver.

                                        X 1 Reply Last reply Reply Quote 1
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Please add that info to the bug report if you have confirmed it.
                                          https://redmine.pfsense.org/issues/9123

                                          Steve

                                          JeGrJ 1 Reply Last reply Reply Quote 0
                                          • X
                                            xciter327 @Juve
                                            last edited by xciter327

                                            @Juve

                                            I have got to say in my case the firewall did freeze eventually after seeing those "mcvlan" errors. I've disabled WOL in BIOS which apparently controls like 10 different power saving options. It can also be disabled by an Intel utility without reboot. Since then I've put a firewall under testing on a 10g link and have not been able to crash/freeze it.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.