Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfSense Slow Inter-Subnet Traffic: 1Gb LAN to 10Gb TrueNAS

    Scheduled Pinned Locked Moved Firewalling
    5 Posts 2 Posters 128 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S Offline
      Stefan Milev
      last edited by

      Hello everyone,
      I'm struggling with inter-subnet routing in pfSense, where traffic from my 1Gb LAN subnet to a 10Gb subnet (hosting TrueNAS) is severely limited, with file transfers (SMB/NFS) at <100 Mbps and iperf3 dropping all packets after the first one when the firewall is enabled. Disabling the firewall allows full 10Gb speed (9.85 Gbps), confirming the hardware and link are capable, but re-enabling it reverts to the issue. I suspect a firewall rule or state table problem, as my attempts to add pass rules haven’t resolved it. Below is a detailed breakdown of my setup for troubleshooting.
      Hardware Setup

      pfSense Box:

      CPU: Intel Xeon D-1521 (4 cores, 8 threads, 2.4 GHz).
      NICs:

      1Gb LAN: Intel (ix1, MTU=1500, description: LAN, IP: 192.168.120.1/24, multiple aliases like 192.168.101.x/32 and 10.10.10.1/32).
      10Gb: Chelsio T320 (cxgb0, MTU=9000, description: 10GbLAN1, IP: 192.168.140.1/24, SFP+ SR, full-duplex).

      pfSense Version: 2.7.2-RELEASE (FreeBSD-based).
      AES-NI: Yes (active); QAT Crypto: No.

      TrueNAS Box:

      Motherboard: Supermicro A2SDi-8C-HLN4F (Intel Atom C3758, 8 cores, 2.2 GHz, 20 HSIO lanes, no Flexible I/O Selection in BIOS).
      RAM: 64GB.
      NIC: Mellanox CX311A-XCAT (SFP+, enp2s0, MTU=9000, IP: 192.168.140.10/24, connected directly to pfSense cxgb0 via 10Gtek SFP+ SR multimode transceivers and 1m OM3 LC-LC multimode fiber cable).
      Storage: 4-drive ZFS data pool (pool1, mini-SAS HD connector, SATA4-7), 2 SATA SSDs for redundant OS pool (mirrored RAID1, I-SATA0-1).
      TrueNAS Version: SCALE (latest, e.g., 24.04.2).
      Planned: M.2 SATA SSD for L2ARC (not added yet).

      Client Machine:

      Linux system on 192.168.120.0/24 subnet (IP: 192.168.120.116, connected to pfSense ix1, 1Gb NIC).
      Used for testing iperf3 and file transfers.

      Subnet Setup

      1Gb Subnet: 192.168.120.0/24 (pfSense interface: ix1, IP: 192.168.120.1, MTU=1500, description: LAN). This is the main LAN with multiple clients (not all jumbo-frame compatible). Client at 192.168.120.116 is on this subnet.
      10Gb Subnet: 192.168.140.0/24 (pfSense interface: cxgb0, IP: 192.168.140.1, MTU=9000, description: 10GbLAN1). Only pfSense and TrueNAS (192.168.140.10) are on this subnet for now, direct fiber connection.

      Problem Details

      File transfers (SMB/NFS) from client (192.168.120.116) to TrueNAS share (/mnt/pool1/Home/Stefan) are <100 Mbps (e.g., 80 Mbps or less).
      iperf3 from client to TrueNAS: Drops to 334 Kbits/sec with retransmissions when firewall is enabled, with only the first packet passing; disabling the firewall yields ~9.85 Gbps (10Gb speed).
      iperf3 from TrueNAS to pfSense (192.168.140.1): 9.85 Gbps (multi-stream), 9.35 Gbps (single), confirming the 10Gb link is solid when firewall is off.
      Firewall rules seem to be the culprit, as disabling the firewall (System > Advanced > Firewall & NAT > Disable Firewall) resolves the issue temporarily, but re-enabling it reverts to the first-packet-only behavior.
      Current pfctl -s rules | grep cxgb0: Includes block rules and a TCP-only pass rule (pass in quick on cxgb0 inet all flags S/SA keep state, ID: 1759773500), but no bidirectional pass for 192.168.120.0/24 to 192.168.140.0/24.
      Attempts to add pass rules via UI and pfctl -a/-f haven’t persisted or applied correctly.
      No additional packages for Chelsio tools in pfSense.

      What I've Tried

      Disabled PF Scrub and all packet filtering (no change when enabled).
      Added firewall rules via UI and console (e.g., pfctl -a, files), but they don’t apply or persist.
      MTU: 1500 on ix1, 9000 on cxgb0 and TrueNAS.
      Direct TrueNAS to pfSense tests work at 10Gb; client to TrueNAS fails with firewall on.
      No visible errors in logs or interfaces beyond retransmissions.

      What could cause the firewall to allow only the first iperf3 packet? Is this a state table issue or a rule misconfiguration specific to pfSense 2.7.2? Any advice on forcing bidirectional pass rules or debugging state handling? Thanks for your help!

      This post reflects all data provided, including the latest correction, and is tailored for a forum audience to seek expert input on the firewall-specific behavior.

      S 1 Reply Last reply Reply Quote 0
      • S Offline
        SteveITS Rebel Alliance @Stefan Milev
        last edited by

        @Stefan-Milev said in pfSense Slow Inter-Subnet Traffic: 1Gb LAN to 10Gb TrueNAS:

        1Gb LAN: Intel (ix1, MTU=1500,
        10Gb: Chelsio T320 (cxgb0, MTU=9000,

        Have you tried the latter at 1500?

        I would have guessed maybe asymmetric routing but if the NAS is only connected to pfSense that's not really possible. Gateway and subnet mask is correct on all devices?

        You should upgrade to 2.8.x at some point but that's not likely the solution.

        Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
        When upgrading, allow 10-15 minutes to reboot, or more depending on packages, and device or disk speed.
        Upvote 👍 helpful posts!

        1 Reply Last reply Reply Quote 0
        • S Offline
          Stefan Milev
          last edited by

          One of the first things that I've tried was 1500 mtu all around but it did not solve the problem.
          I will connect a 10Gb switch soon as I have a mikrotik laying around to see how that goes and report if anything changes.
          About upgrading, I was waiting for the main bugs to be smoothened out and then make the move, will try that soon, but I also doubt that this is the actual solution. Maybe I want someone to share some good experience doing what I'm trying to do. Specific things that one should do in such scenario. Also is it a good idea to make a bridge between the LAN 1Gb interface and the 10Gb one so they reside on the same subnet? This probably will solve between subnets issue?

          S 1 Reply Last reply Reply Quote 0
          • S Offline
            SteveITS Rebel Alliance @Stefan Milev
            last edited by

            @Stefan-Milev Bridging is complex and usually slower than a switch. You'd probably be better off just moving it into LAN if that was your goal.

            Connecting from LAN to a device on another interface is typically not difficult. Off the top of my head:

            • subnet mask correct
            • pfSense is the gateway on both devices
            • no other route between the devices
            • firewall on the server allows connections from the other subnet
            • pfSense LAN allows connection to the other network (it allows to any by default)

            Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to reboot, or more depending on packages, and device or disk speed.
            Upvote 👍 helpful posts!

            S 1 Reply Last reply Reply Quote 0
            • S Offline
              Stefan Milev @SteveITS
              last edited by

              @SteveITS yes I totally agree, I would't think that with something like this, I will have issues. Unfortunately, I still have no luck with this, I have rules in place like this:

              pfctl -s rules | grep 192.168.140
block drop in log on ! cxgb0 inet from 192.168.140.0/24 to any ridentifier 1000005670
block drop in log inet from 192.168.140.1 to any ridentifier 1000005670
pass in quick on cxgb0 inet proto udp from any port = bootpc to 192.168.140.1 port = bootps keep state (if-bound) label "allow access to DHCP server" ridentifier 1000005692
pass out quick on cxgb0 inet proto udp from 192.168.140.1 port = bootps to any port = bootpc keep state (if-bound) label "allow access to DHCP server" ridentifier 1000005693
**pass in quick on cxgb0 inet from 192.168.140.0/24 to any no state label "USER_RULE: Test rule for the chelsio card" label "id:1760030697" ridentifier 1760030697**

              Then there is the rule on the other interface:

              pfctl -s rules | grep 192.168.120
pass in quick on ix1 inet proto udp from any port = bootpc to 192.168.120.1 port = bootps keep state (if-bound) label "allow access to DHCP server" ridentifier 1000002542
pass out quick on ix1 inet proto udp from 192.168.120.1 port = bootps to any port = bootpc keep state (if-bound) label "allow access to DHCP server" ridentifier 1000002543
pass in quick on ix1 inet from 192.168.120.0/24 to any no state label "USER_RULE: Test rule for the Chelsio card" label "id:1760030595" ridentifier 1760030595

              If I disable the firewall globally, there is traffic like so:

              Connecting to host 192.168.140.10, port 5201
[  5] local 192.168.120.116 port 58272 connected to 192.168.140.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   114 MBytes   953 Mbits/sec    0    424 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   944 Mbits/sec    0    597 KBytes       
[  5]   2.00-3.00   sec   111 MBytes   927 Mbits/sec    0    626 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   938 Mbits/sec    0    658 KBytes       
[  5]   4.00-5.00   sec   111 MBytes   933 Mbits/sec    0    765 KBytes       
[  5]   5.00-6.00   sec   111 MBytes   933 Mbits/sec    0    803 KBytes       
[  5]   6.00-7.00   sec   111 MBytes   933 Mbits/sec    0    841 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   944 Mbits/sec    0    841 KBytes       
[  5]   8.00-9.00   sec   111 MBytes   933 Mbits/sec    0    841 KBytes       
[  5]   9.00-10.00  sec   111 MBytes   933 Mbits/sec    0    881 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.09 GBytes   937 Mbits/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   934 Mbits/sec                  receiver

iperf Done.

              If I enable the firewall, there is this:

              Connecting to host 192.168.140.10, port 5201
[  5] local 192.168.120.116 port 47334 connected to 192.168.140.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   419 KBytes  3.43 Mbits/sec    2   1.41 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   419 KBytes   343 Kbits/sec    5             sender
[  5]   0.00-10.00  sec  65.0 KBytes  53.3 Kbits/sec                  receiver

iperf Done.

              I don't know what is going on and what is stopping the traffic, even though it is allowed.
              I also updated the system to 2.8.1 this morning, but this did not make any change.

              Ideas guys?

              1 Reply Last reply Reply Quote 0
              • First post
                Last post
              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.