• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Weird Behavior with x710-da2 in 2.5.x

Scheduled Pinned Locked Moved Hardware
26 Posts 5 Posters 4.1k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    stephenw10 Netgate Administrator
    last edited by Jan 15, 2022, 1:58 PM

    Hmm, something has to be changing but it's hard to see what that could be just by down/up-ing the interface.
    You could start checking the sysctl stats for ixl0 but there's a lot to wade through:

    sysctl dev.ixl.0
    

    You might be able to spot some key difference between the two states.

    Steve

    1 Reply Last reply Reply Quote 0
    • T
      tman222
      last edited by tman222 Jan 15, 2022, 7:15 PM Jan 15, 2022, 7:14 PM

      Hi @ashtonianagain - looking at the logs above:

      1. If the card is PCI Express 3.0, I'd expect the bus speed to be higher (e.g. 8.0GT/s), but that will depend on which the slot the card is sitting in on the motherboard it - could be that the bandwidth is shared or the slot is only 2.0/2.1 capable.
      2. I also saw that your system defaulted to using MSI vs. MSI-X. Are you passing the card through to pfSense or going fully virtual? A couple links to check out that may help:

      https://forum.netgate.com/topic/158860/pfsense-latency-spikes-in-esxi
      https://forum.netgate.com/topic/157688/remove-vmware-msi-x-from-the-pci-blacklist/

      In particular, this setting might help with some of the performance issues you are seeing:

      hw.pci.honor_msi_blacklist=0
      

      Hope this helps.

      1 Reply Last reply Reply Quote 0
      • S
        stephenw10 Netgate Administrator
        last edited by Jan 15, 2022, 7:31 PM

        That syctl usually only affects vmxnet, which are on the MSI blacklist by default. But there's no harm setting it.

        T 1 Reply Last reply Jan 15, 2022, 7:34 PM Reply Quote 0
        • T
          tman222 @stephenw10
          last edited by Jan 15, 2022, 7:34 PM

          @stephenw10 said in Weird Behavior with x710-da2 in 2.5.x:

          That syctl usually only affects vmxnet, which are on the MSI blacklist by default. But there's no harm setting it.

          That's a good point - I was just going by the last post in this thread, thinking it might be worth a shot:

          https://forum.netgate.com/topic/157688/remove-vmware-msi-x-from-the-pci-blacklist/5

          1 Reply Last reply Reply Quote 0
          • A
            ashtonianagain
            last edited by Jan 16, 2022, 1:48 AM

            @tman222 said in Weird Behavior with x710-da2 in 2.5.x:

            hw.pci.honor_msi_blacklist=0

            THIS WORKED! thank you guys for your help. After setting this I've been able to get normal pings and expected bandwidth performance.

            Now Im not sure if I should enable or disable all the of the offloading for performance - what do you guys think?

            1 Reply Last reply Reply Quote 0
            • S
              stephenw10 Netgate Administrator
              last edited by stephenw10 Jan 16, 2022, 2:43 PM Jan 16, 2022, 2:40 PM

              Nice. That's interesting. I wonder what is on that list that affects pass-through NICs.

              Good catch @tman222 👍

              The hardware off-load settings rarely make much difference. I would certainly not enable anything that's disabled by default. So that means only checksum off-load which is still enabled by default.

              Steve

              T 1 Reply Last reply Jan 16, 2022, 6:51 PM Reply Quote 0
              • T
                tman222 @stephenw10
                last edited by Jan 16, 2022, 6:51 PM

                @stephenw10 said in Weird Behavior with x710-da2 in 2.5.x:

                Nice. That's interesting. I wonder what is on that list that affects pass-through NICs.

                Good catch @tman222 👍

                The hardware off-load settings rarely make much difference. I would certainly not enable anything that's disabled by default. So that means only checksum off-load which is still enabled by default.

                Steve

                Hi @stephenw10 - looking Jim's comment here and the bug report (towards the end):

                https://forum.netgate.com/topic/157688/remove-vmware-msi-x-from-the-pci-blacklist/2
                https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=203874

                If I understand right, this fix never made into FreeBSD 12. Would it be worth raising an issue on Redmine? At the very least, if the fix is missing, maybe add the sysctl tunable?

                Thanks in advance!

                1 Reply Last reply Reply Quote 0
                • S
                  stephenw10 Netgate Administrator
                  last edited by Jan 16, 2022, 8:19 PM

                  I had thought this only applied to the vmxnet NICs but if you look at the diff on the patch it actually applied to the VMWare PCI bridge so I guess this could still be in play.
                  It is odd though, it still comes up with 6 queues in the above example, just using MSI not MSIX.

                  D 1 Reply Last reply Feb 4, 2022, 12:59 AM Reply Quote 0
                  • D
                    deridiot @stephenw10
                    last edited by Feb 4, 2022, 12:59 AM

                    @stephenw10

                    I am the other individual mentioned. I have a Quad SFP+ PCI-e 3x card on a ESXi 6.7U3 server. It's an OEM? card (Branded- Silicom PE310G4I71LB-XR) which runs an older FW. I have passed through 2 ports to pfSense and it had the same latency issues mentioned but is working perfectly fine in Windows 10, Server 2022 and Sophos UTM (bleh). I will attempt the above fix this weekend when I perform a switchover and report back.

                    D 1 Reply Last reply Feb 9, 2022, 9:47 AM Reply Quote 0
                    • D
                      deridiot @deridiot
                      last edited by Feb 9, 2022, 9:47 AM

                      @deridiot

                      Issue still present on below build, resolved with the same change mentioned above.

                      2.7.0-DEVELOPMENT (amd64)
                      built on Fri Feb 04 19:41:27 UTC 2022
                      FreeBSD 12.3-STABLE

                      ? 1 Reply Last reply Mar 19, 2022, 7:52 AM Reply Quote 0
                      • ?
                        A Former User @deridiot
                        last edited by Mar 19, 2022, 7:52 AM

                        @deridiot said in Weird Behavior with x710-da2 in 2.5.x:

                        Issue still present on below build, resolved with the same change mentioned above.
                        2.7.0-DEVELOPMENT (amd64)
                        built on Fri Feb 04 19:41:27 UTC 2022
                        FreeBSD 12.3-STABLE

                        Installing 2.6 and update to 21.05.2 was solving that problem for a X710-T2 adapter, so perhaps it could work for you too.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                          [[user:consent.lead]]
                          [[user:consent.not_received]]