pfSense Slow Inter-Subnet Traffic: 1Gb LAN to 10Gb TrueNAS

[Stefan Milev]

Hello everyone,
I'm struggling with inter-subnet routing in pfSense, where traffic from my 1Gb LAN subnet to a 10Gb subnet (hosting TrueNAS) is severely limited, with file transfers (SMB/NFS) at <100 Mbps and iperf3 dropping all packets after the first one when the firewall is enabled. Disabling the firewall allows full 10Gb speed (9.85 Gbps), confirming the hardware and link are capable, but re-enabling it reverts to the issue. I suspect a firewall rule or state table problem, as my attempts to add pass rules haven’t resolved it. Below is a detailed breakdown of my setup for troubleshooting.
Hardware Setup

pfSense Box:

CPU: Intel Xeon D-1521 (4 cores, 8 threads, 2.4 GHz).
NICs:

1Gb LAN: Intel (ix1, MTU=1500, description: LAN, IP: 192.168.120.1/24, multiple aliases like 192.168.101.x/32 and 10.10.10.1/32).
10Gb: Chelsio T320 (cxgb0, MTU=9000, description: 10GbLAN1, IP: 192.168.140.1/24, SFP+ SR, full-duplex).

pfSense Version: 2.7.2-RELEASE (FreeBSD-based).
AES-NI: Yes (active); QAT Crypto: No.

TrueNAS Box:

Motherboard: Supermicro A2SDi-8C-HLN4F (Intel Atom C3758, 8 cores, 2.2 GHz, 20 HSIO lanes, no Flexible I/O Selection in BIOS).
RAM: 64GB.
NIC: Mellanox CX311A-XCAT (SFP+, enp2s0, MTU=9000, IP: 192.168.140.10/24, connected directly to pfSense cxgb0 via 10Gtek SFP+ SR multimode transceivers and 1m OM3 LC-LC multimode fiber cable).
Storage: 4-drive ZFS data pool (pool1, mini-SAS HD connector, SATA4-7), 2 SATA SSDs for redundant OS pool (mirrored RAID1, I-SATA0-1).
TrueNAS Version: SCALE (latest, e.g., 24.04.2).
Planned: M.2 SATA SSD for L2ARC (not added yet).

Client Machine:

Linux system on 192.168.120.0/24 subnet (IP: 192.168.120.116, connected to pfSense ix1, 1Gb NIC).
Used for testing iperf3 and file transfers.

Subnet Setup

1Gb Subnet: 192.168.120.0/24 (pfSense interface: ix1, IP: 192.168.120.1, MTU=1500, description: LAN). This is the main LAN with multiple clients (not all jumbo-frame compatible). Client at 192.168.120.116 is on this subnet.
10Gb Subnet: 192.168.140.0/24 (pfSense interface: cxgb0, IP: 192.168.140.1, MTU=9000, description: 10GbLAN1). Only pfSense and TrueNAS (192.168.140.10) are on this subnet for now, direct fiber connection.

Problem Details

File transfers (SMB/NFS) from client (192.168.120.116) to TrueNAS share (/mnt/pool1/Home/Stefan) are <100 Mbps (e.g., 80 Mbps or less).
iperf3 from client to TrueNAS: Drops to 334 Kbits/sec with retransmissions when firewall is enabled, with only the first packet passing; disabling the firewall yields ~9.85 Gbps (10Gb speed).
iperf3 from TrueNAS to pfSense (192.168.140.1): 9.85 Gbps (multi-stream), 9.35 Gbps (single), confirming the 10Gb link is solid when firewall is off.
Firewall rules seem to be the culprit, as disabling the firewall (System > Advanced > Firewall & NAT > Disable Firewall) resolves the issue temporarily, but re-enabling it reverts to the first-packet-only behavior.
Current pfctl -s rules | grep cxgb0: Includes block rules and a TCP-only pass rule (pass in quick on cxgb0 inet all flags S/SA keep state, ID: 1759773500), but no bidirectional pass for 192.168.120.0/24 to 192.168.140.0/24.
Attempts to add pass rules via UI and pfctl -a/-f haven’t persisted or applied correctly.
No additional packages for Chelsio tools in pfSense.

What I've Tried

Disabled PF Scrub and all packet filtering (no change when enabled).
Added firewall rules via UI and console (e.g., pfctl -a, files), but they don’t apply or persist.
MTU: 1500 on ix1, 9000 on cxgb0 and TrueNAS.
Direct TrueNAS to pfSense tests work at 10Gb; client to TrueNAS fails with firewall on.
No visible errors in logs or interfaces beyond retransmissions.

What could cause the firewall to allow only the first iperf3 packet? Is this a state table issue or a rule misconfiguration specific to pfSense 2.7.2? Any advice on forcing bidirectional pass rules or debugging state handling? Thanks for your help!

This post reflects all data provided, including the latest correction, and is tailored for a forum audience to seek expert input on the firewall-specific behavior.

[SteveITS]

@Stefan-Milev said in pfSense Slow Inter-Subnet Traffic: 1Gb LAN to 10Gb TrueNAS:

1Gb LAN: Intel (ix1, MTU=1500,
10Gb: Chelsio T320 (cxgb0, MTU=9000,

Have you tried the latter at 1500?

I would have guessed maybe asymmetric routing but if the NAS is only connected to pfSense that's not really possible. Gateway and subnet mask is correct on all devices?

You should upgrade to 2.8.x at some point but that's not likely the solution.

[Stefan Milev]

One of the first things that I've tried was 1500 mtu all around but it did not solve the problem.
I will connect a 10Gb switch soon as I have a mikrotik laying around to see how that goes and report if anything changes.
About upgrading, I was waiting for the main bugs to be smoothened out and then make the move, will try that soon, but I also doubt that this is the actual solution. Maybe I want someone to share some good experience doing what I'm trying to do. Specific things that one should do in such scenario. Also is it a good idea to make a bridge between the LAN 1Gb interface and the 10Gb one so they reside on the same subnet? This probably will solve between subnets issue?

[SteveITS]

@Stefan-Milev Bridging is complex and usually slower than a switch. You'd probably be better off just moving it into LAN if that was your goal.

Connecting from LAN to a device on another interface is typically not difficult. Off the top of my head:

• subnet mask correct
• pfSense is the gateway on both devices
• no other route between the devices
• firewall on the server allows connections from the other subnet
• pfSense LAN allows connection to the other network (it allows to any by default)