• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Chelsio T520 not working as WAN interface

General pfSense Questions
5
25
4.1k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    mikenemat @mikenemat
    last edited by Oct 6, 2018, 5:10 PM

    Just tried pfsense 2.4.3 which has a slightly older cxgbe driver and firmware. No change :( Also tried forcing PCIe Gen3 link in the BIOS which fixed a vaguely similar issue for someone else, no luck. I did confirm the card was already linking at PCIe 3.0 x8 to begin with.

    1 Reply Last reply Reply Quote 0
    • M
      mikenemat
      last edited by mikenemat Oct 6, 2018, 9:37 PM Oct 6, 2018, 6:15 PM

      Popped in a Myricom 10G card and it works fine. Seems clear that there is a problem with the FreeBSD Chelsio drivers when used with PPPoE. Chelsio card works fine in Ubuntu, Myricom card works fine in FreeBSD.

      Any thoughts on next steps?

      G 1 Reply Last reply Oct 6, 2018, 9:38 PM Reply Quote 0
      • G
        Grimson Banned @mikenemat
        last edited by Oct 6, 2018, 9:38 PM

        @mikenemat said in Chelsio T520 not working as WAN interface:

        Any thoughts on next steps?

        Contact the FreeBSD developers.

        M 1 Reply Last reply Oct 6, 2018, 9:55 PM Reply Quote 0
        • M
          mikenemat @Grimson
          last edited by mikenemat Oct 6, 2018, 9:56 PM Oct 6, 2018, 9:55 PM

          @grimson said in Chelsio T520 not working as WAN interface:

          @mikenemat said in Chelsio T520 not working as WAN interface:

          Any thoughts on next steps?

          Contact the FreeBSD developers.

          Hoping to find some obscure explanation and workaround for the problem. Really struggling to accept that this is a driver bug, despite all signs pointing to it. I can't find anyone else who's run into this problem. This is the Netgate recommended 10GB NIC for Pfsense so you'd think if it is a problem with the driver, it would be widespread. PPPoE is common with many FTTH providers.

          G 1 Reply Last reply Oct 6, 2018, 10:01 PM Reply Quote 0
          • G
            Grimson Banned @mikenemat
            last edited by Oct 6, 2018, 10:01 PM

            @mikenemat said in Chelsio T520 not working as WAN interface:

            PPPoE is common with many FTTH providers.

            Maybe in your area, but not on a global scale. And certainly not with 10GB.

            1 Reply Last reply Reply Quote 0
            • M
              mikenemat
              last edited by Oct 10, 2018, 10:24 PM

              Replaced the T520 with an Intel X520-DA2. Worked perfectly the first time. Chelsio = garbage.

              1 Reply Last reply Reply Quote 0
              • S
                stephenw10 Netgate Administrator
                last edited by Oct 11, 2018, 1:04 PM

                It's certainly very unusual to be using a 10G card of any type for PPPoE. Do you really have a >1Gbps connection using that?

                You could well be the only person using it in which case any bugs that might exist for that combination simply may not have been discovered, until now.

                The next step here would be to test it with FreeBSD 11.2 and if it fails there report it upstream.

                Steve

                M 1 Reply Last reply Oct 11, 2018, 3:00 PM Reply Quote 0
                • M
                  mikenemat @stephenw10
                  last edited by mikenemat Oct 11, 2018, 3:05 PM Oct 11, 2018, 3:00 PM

                  @stephenw10

                  Yep. I really am :) Bell Canada's new 1.5gbps FTTH service works over PPPoE. Apparently this is very common in Japan as well. Their supplied equipment links a SFP GPON module at 2.5gbps. However, their supplied equipment is junk and aside from many other issues, the only way to reach 1.5gbps is to use the wired 1gbps LAN interface and the wireless interface simultaneously, since it is not a 10G appliance. This is not ideal.

                  So in order to move past this, I'm using a ubiquiti ES-16-XG to link the SFP GPON module. The ES-16-XG is one of the rare pieces of hardware to support linking a SFP module at 2.5gbps. I've configured it to strip the VLAN tag (35), and pass it to pfsense over a proper 10G link. At that point, I'm able to establish a PPPoE session and utilize the full 1.5gbps over a 10G LAN interface.

                  Forgive my frustration - this issue has been a nightmare to troubleshoot and my appetite for taking down my internet connection for hours at a time (which I use for business as well) is dwindling. I'm going to resell the Chelsio card to someone who hopefully has a more conventional use case. The Intel X520-DA2 has been working flawlessly for over 24 hours now. I think I prefer the Intel card anyways, much less firmware-driven magic going on.

                  login-to-view

                  1 Reply Last reply Reply Quote 0
                  • S
                    stephenw10 Netgate Administrator
                    last edited by Oct 11, 2018, 5:58 PM

                    I understand. Interesting use case.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • P
                      posto587
                      last edited by posto587 Feb 7, 2019, 3:16 PM Feb 7, 2019, 3:11 PM

                      We are seeing a similar issue but not related with pppoe.

                      Hardware:
                      supermicro h2sdi-4c-ln4f
                      chelsio T520-SO-CF

                      WAN on the chelsio seems to work ok in our case via DAC on 10G.
                      If we dial in via ipsec the ipsec connection is established but partly blocked.
                      So we can ping the pfsense lan interface and other devices on the LAN but not reach the webinterfaces etc.

                      Switching the WAN from the chelsio to the onboard intel RJ45 solves the problem.

                      pfsense 2.4.4p2

                      Same issue on two identical hardware setups.

                      After pfctl -d access was possible but ipsec allow any rule was present.

                      1 Reply Last reply Reply Quote 0
                      • S
                        stephenw10 Netgate Administrator
                        last edited by Feb 8, 2019, 1:19 AM

                        Hmm, if you're coming over IPSec then it's not a TCP off-loading issue. Perhaps an MTU issue though. Try setting enabling mss clamping on IPSec.

                        Steve

                        P 1 Reply Last reply Feb 8, 2019, 7:48 AM Reply Quote 0
                        • P
                          posto587 @stephenw10
                          last edited by posto587 Feb 8, 2019, 8:05 AM Feb 8, 2019, 7:48 AM

                          @stephenw10 said in Chelsio T520 not working as WAN interface:

                          Perhaps an MTU issue though. Try setting enabling mss clamping on IPSec.
                          Steve

                          We are not onsite today but I will check MTU settings.
                          The strange thing is, that disabling pf resolved the issue with blocking access. This shouldn't change MTU settings?
                          Are you guessing it is MTU related because ping works and probably bigger packets are blocked/dropped?

                          edit: MTU on cxl (chelsio 10G) and ix (intel 1G) are on 1500 but enc0 is on 1536.

                          1 Reply Last reply Reply Quote 0
                          • S
                            stephenw10 Netgate Administrator
                            last edited by Feb 8, 2019, 2:51 PM

                            Yes large packets would be my first thought. Disabling pf wouldn't affect that though.
                            However it does disable pf-scrub. Perhaps you have packet fragments that are not being assembled correctly. Maybe the Chelsio card is doing something with them, it has all sorts of offloading hardware that we don't use.

                            You can disable pfscrub separately in System > Advanced > Firewall & NAT.

                            Steve

                            1 Reply Last reply Reply Quote 0
                            • S
                              ssjucrono
                              last edited by Sep 17, 2019, 8:57 PM

                              Hello I am having a very similar issue! I have a chelsio t404-bt and I am using Centurylink. Which WAN tags vlan201 and pppoe. This setup worked fine with 2 intel nics. Now with the chelsio card I get an WAN IP and I can traceroute from pfsense out but I have no internet on my devices.

                              Any solution to this?

                              1 Reply Last reply Reply Quote 0
                              • S
                                stephenw10 Netgate Administrator
                                last edited by Sep 17, 2019, 10:27 PM

                                Do you have other connectivity from pfSense itself? Can it check for updates or install packages for example?

                                If so it's a different issue. Probably no NAT.

                                Steve

                                S 1 Reply Last reply Sep 17, 2019, 10:31 PM Reply Quote 0
                                • S
                                  ssjucrono @stephenw10
                                  last edited by ssjucrono Sep 17, 2019, 10:33 PM Sep 17, 2019, 10:31 PM

                                  @stephenw10 no I cannot get a list of packages either on pfsense

                                  see odd graph and wan ip

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    stephenw10 Netgate Administrator
                                    last edited by Sep 17, 2019, 10:40 PM

                                    Ok so you can ping out though?

                                    Try pinging out with large packets:
                                    ping -s 1000 -c 3 1.1.1.1

                                    Try different sized packets to see if you really are seeing an MTU issue.

                                    Steve

                                    1 Reply Last reply Reply Quote 0
                                    • M mehiyok680 referenced this topic on Jun 29, 2022, 2:44 PM
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.