IPSEC, UPLOAD = 40MB/s, DOWNLOAD = 500KB/s
-
Here is an extract from wireshark:
https://i.ibb.co/RH6QK88/Screenshot-42.png -
After several hours of searching and testing, we realized that the packets were fragmented into 536 bytes.
We don't know exactly why and it is for this misunderstanding that we need your help. -
@yazur Can you provide a packet capture between the system that is sending the file and the nearby pfsense, starting with the establishing of the TCP session and including ICMP traffic that's send to the sending system.
Kind regards,
Mathias -
Here is more information:
We received our Freebox pro a few weeks ago and we are experiencing IPSEC throughput problems.
To schematize our infrastructure, we have :
Virtual infrastructure :
- Infra OVH 3Gb/s symmetrical
Physical infrastructure :
- Internet 1 - Freebox pro at the head office (shared fiber) with a speed of 1Gb/s and 700Mb/s
- Internet 2 - Fingerprint at the head office (dedicated fiber) with a speed of 100Mb/s symmetrical
We use the Pfsense solution to connect our head office with the infrastructure hosted by OVH through IPSEC.
It is important to know that we are currently working with an IPSEC tunnel instantiated through our internet connection N°2 and everything is working perfectly.
Here are the speed tests observed when the tunnel is instanciated through the freebox pro:HEADQUARTERS --> OVH = 40MB/s (Normal)
OVH --> SIEGE = 500KB/s (Inconsistent)We noticed that the packets were fragmented with a length of 536 bytes.
We then looked at the MTU/MSS and spent a whole afternoon changing the values without ever seeing a difference with Wireshark.
Today we did another test, this one to disconnect the fiber and go through the 4g backup.
The test was good, we have the full 4g throughput available in IPSEC.
However, the packets were also fragmented in length of 536 bytes.
(As a comparison, with the dedicated fiber Fingerprint we have packets of 1436 bytes).So I would like to know if Free or Jaguar Network is throttling the IPSEC throughput on port 500/4500 to the Freebox.
If so, this pro offer is completely unsuitable for professional use. -
@yazur This is not what I have asked for.
What I can see, is that 192.168.98.1 sends datagrams with a length of 576 bytes (20 byte IP header, 20 byte TCP header, 536 byte TCP segment).
What I don't see, is why it sends these small datagrams.I assume that path MTU discovery is broken, but to be sure about this I wanted to see the datagrams that establish the TCP connection until 192.168.98.1 sends the 576-byte-datagrams and the ICMP-traffic. These packets should be captured as near as possible to 192.168.98.1, at least between this system and the nearest pfSense.
I guess that 192.168.98.1 tries at least once to send a bigger datagram and doesn't get an ACK for this and also doesn't get an ICMP error message type 3 code 4 (Destination Unreachable Fragmentation Needed, DF set) which would state the usable MTU.
Some TCP-Stacks then fall back to datagrams of length 576 byte.
You may want to look at RFC1122 Section 3.3.3 for details.Kind regards,
Mathias -
Thank you for your very complete answer.
However, I don't see ICMP packets on Wireshark with the "icmp" filter.
On the other hand I have this information:https://nsa40.casimages.com/img/2021/06/14/21061405520636118.png
-
This is not what I asked for.
I wrote:
I wanted to see the datagrams that establish the TCP connection until 192.168.98.1 sends the 576-byte-datagrams and the ICMP-traffic.
These packets should be captured as near as possible to 192.168.98.1, at least between this system and the nearest pfSense.What you have shown is how the TCP connection was established but apparently the capture was taken near the system with address 192.168.99.2.
So it's not from the side which sends the 576-byte datagrams and it doesn't show the first big datagram.I can't help you with that.
-
Here is the file closest to the source "directly on the client machine that sends the data" :
https://www.partage-temporaire.fr/2021/06/14/wireshark/?download&st=1623687072
-
@yazur
I assume that this capture was taken on the Windows VM itself, because it shows really big datagrams that normally don't get send on the wire.On the other hand, when both the Windows VM and the pfSense VM run in the same VMware environment it may be, that they could exchange such big datagrams.
Do you know that you can take a capture from outside the VM using the pktcap-UW utility?
But this is off topic here, you should talk to your VMware admin.While the PCAP file shows that there are really big datagrams sent, the ACK-datagrams are in 1152-byte-steps which is just two times 576 bytes.
So I assume there is some kind of TCP-offloading going on here and what we see in the capture is not what is sent on the wire or between the VMs.I can see one TCP connection and several UDP connections but no ICMP in the PCAP file.
On the other hand the TCP connection is missing the datagrams of the 3-step-handshake that established the connection, so it is still incomplete.Assuming that there were no ICMP error messages regarding this connection, I would check why the pfSense did not sent any such message when the big datagrams were coming from the Windows VM.
Is it possible that these got filtered by the VMware environment?
You can check this by taking a capture on the pfSense VM and look if you can find it there.
If yes, look in the VMware environment.
If no, look at the pfSense configuration.Kind regards,
Mathias -
Hello,
Thank you for all this very relevant information.
If you look at the second screenshot I mentioned in my second post, you can see that the fragmentation is done at the arrival on the LAN interface of the Pfsense OVH.
The Wireshark capture is actually coming from the "OVH" client machine, that's why there is no fragmentation.What I can add to our problem is that we did a second test.
Our ISP offers a backup 4G connection if the fiber optic connection fails.
We then disconnected the fiber optic connection and switched to 4g.
I then reconfigured the IPSEC tunnel to go through this new public IP address and the throughput was as expected.
Moreover, the 576 bytes packet fragmentation was still in place.Could this fragmentation cause huge problems at a certain throughput?
Because in 4G the throughput was 5MB/s in the direction that works at 500KB/s with fiber.I will attach a Wireshark capture of the OVH Pfsense LAN in a few minutes.
Thanks for your help.
-
Capture wireshark realized on the Lan pfsense OVH :
https://www.partage-temporaire.fr/2021/06/15/wiresharklanpfsenseovh/
In this capture you can see me browsing through some folders and then starting the transfer of a file named 1GB.bin at line 937.
But the navigation is already very slow so I think what you are looking for is before the first communication between the two clients so it will be more in the first lines with the MTU size exchange.