I just found out it must have something to do with VLANs.
My setup:
4core 2ghz / 4gb / 4x I211-AT NIC
2x WAN (1x 250MBit / 1x GBit)
2x LAN
2x Netgear GS724T switches
Case1; with vlans (issues)
-> Server1 connected to LAN1 (no vlan)
-> Server2 connected to LAN2 (with tagged vlan behind the 2 Netgear switches)
Doing iperf3 between the two servers is bringing down the WAN1 interface! )(strange bacause there should be no traffic going over that interface)
Doing iperf3 from server2 over WAN2 to another iperf3 server brings down WAN1 (note; traffic is going over WAN2, but WAN1 latency spikes and gateway goes down)
Latency spikes above 1000ms and then brings down the WAN.
Case2; no vlans (no issues)
-> Server1 connected to LAN1 (no vlan)
-> Server2 connected to LAN2 (NO VLAN behind the 2 Netgear switches)
Doing iperf3 between the two servers is giving me GBit.
Doing iperf3 from server2 over WAN2 is giving me GBit WAN.
Latency doesn't spike above 10ms.
So, my conclusion is that the VLANs are causing a lot of problems with high throughput. I couldn't find many tweaks to do with VLANs in pfSense, so I hope someone has an idea about this? Thanks!
EDIT: It's not WAN2 that goes down in Case1 but WAN1. So, it's always WAN1 that's going down. Is there something about the default gateway that's used with all VLANs or something?