The curl command is not working correctly

[G_Costa]

@stephenw10 On each pf wan I allways have more than 500/500 and i executed an iperf of the vm behind the pf to the local pf and got around 3gbps

[stephenw10]

What latency do you have over the tunnel?

Try an iperf test between the two pfSense instances directly. Try to determine where the throttling is actually happening.

[G_Costa]

@stephenw10
Iperf between both pf's without going through the tunnel:

Local Pf logs (This pf is on a vm inside the proxmox)

SpeedTest on a VM with the tunnel working:

[stephenw10]

Do you see the same results in both directions?

That's a lot of variation in the result, even outside the tunnel.

How much traffic is running through that local pfSense? How much RAM does it have?

You can increase the state table size in Sys > Adv > Firewall+NAT but exhausting it usually implies some very high use. You may need to reduce the state timeouts so the table is pruned more frequently.

[G_Costa]

@stephenw10

There's the iperf of the other direction:

The only traffic is from speedtest, im not running anything else and the pf has 8GB Ram and 8 Cores

Pflocal:

Pfremote:

[stephenw10]

Are those showing bits or bytes there?

How are you testing across the tunnel? Also with iperf?

[G_Costa]

@stephenw10 said in The curl command is not working correctly:

How are you testing across the tunnel? Also with iperf?

Reply

It's Bytes

This is on the tunnel and the ips are:
10.0.8.1 -> OpenVPN remote Tunnel
10.0.8.2 -> OpenVPN local Tunnel

[stephenw10]

Hmm, how is the tunnel configured? Is it using UDP? There are a lot of retries there, it could be an MTU issue.

Sometime the openvpn interface does not behave as expected when used directly or services like that. Try using an internal IP as source if you can. Though in a bridge it shouldn't really matter.

[G_Costa]

@stephenw10 Yes UDP, there's all the configurations:

--

[stephenw10]

You should set AES-GCM and enable UDP Fast I/O for better performance there.

However that isn't going to get you to the full rate there.

You are seeing ~15ms across the tunnel?

Did you bump the state table size?

[G_Costa]

@stephenw10

Remote pf:

Local pf:

[stephenw10]

Those images are too small to read I think.

[G_Costa]

@stephenw10 Im trying to send them as image instead of attachment but they are too large, do you mind if i send them with imgur?
https://imgur.com/a/7CqmzkO

[stephenw10]

Mmm, OK so no significant difference to throughput. I assume neither side shows any CPU cores at 100%?

I would try setting a lower MSS value and see if that makes any difference. If it does try to fins the actual tunnel MTU with some large pings.
Packet fragmentation across the tunnel can cause significant throttling.

[G_Costa]

@stephenw10

While downloading:

While uploading:

MSS -> 576 -> OpenVPN interface and bridge

MSS -> 1152

MSS -> 2304

MSS -> 4608

About the MTU i cant change on the interfaces because it says "This interface is a bridge member, its MTU is controlled by its parent bridge interface."

[stephenw10]

Hmm, Ok so it looks you are hitting a CPU limit on the upload with a single core at 100%.

Try MSS values at, say, 1400 and 1300. However with bridging in play normal fixes like that can fail since there's no routing....

[G_Costa]

@stephenw10
MSS 1300 Downloading:

MSS 1300 Uploading:

MSS 1400 Downloading:

MSS 1400 Uploading:

While uploading some cores go to 100% but the speed is good but when downloading the cores dont go to 100% and the speed is low

[stephenw10]

Hmm, well I'd try a packet capture on the tunnel and see if the download is being fragmented or there are retransmissions etc.