Slow IPSEC - Strange behavior

Itou 0

Hi,
I have a very strange behavior.
First, let me describe the setup.
I have a small site with one PfSense running on version 2.6.0. It's install on a ESX VM.
The hardware config of the VM is :
2 vCPU Intel E32220 @ 3.1GHz
4GB of RAM
The CPU support AES-NI crypto (and it's activate).
Internet access is a Fibre 200Mbps download/50Mbps upload

On the main site, I have a PfSense (also version 2.6.0) running on a APU4C4.
The hardware config is :
CPU AMD GX-412TC, 4 cores at 1GHz (turbo at 1.4GHz) with AES-NI crypto activated.
4GB of RAM.
Because it's the main site there's a secondary PfSense, but this one is running on a VM.
1 vCPU E5640 @2.66GHz.
2GB of RAM.
Internet access is fiber 500Mbps symetrical.

There's a IPSEC between the two sites.
Few days ago I realize the traffic start to be very slow on the tunnel. Usually the traffic on the tunnel can goes up to 100Mbps.
Since couple of day, the traffic can go over 4 Mbps.

After checking and trying many thing, I try something.
If I shutdown the master PfSense, the backup is able to have 150Mbps throughput over the tunnel. Even after a reboot, the master is not able to go over 4Mbps.
I've try to reboot it, shut it down and let it rest for couple of hour but unable to have more than 4Mbps on the tunnel.

The strange thing about all that is nothing has changed, no configuration modification. All PfSense was running smoothly for more than 300 days.
And suddenly, the speed drop for no reason.

Someone could help ?

Regards.

SteveITS

@itou-0 only IPSec not other traffic? As in, someone throttled the VM?

Itou 0

Hi,

Yes, only the IPSEC traffic get stuck at 4Mbps.

I don't think it's an issue with VM.
The only thing I was able to notice is if the main site run on the APU4C4, the traffic is slow. If it run on the VM, the traffic is good.

But, as I mention, the APU4C4 was able to get 100Mbps on the IPSEC before.
But before what ?????

NOCling

Is there any Limitet Setup on the APU?

Itou 0

No limit, no traffic shaping.

Itou 0

Hi,

I "solve" my issue.
But I cannot really tell you what is the root cause.

I did not mention, the APU4C4 as a 16GB mSATA SSD where PfSense is install.
So, at some point, I think about reinstall PfSense of the APU4C4.
Before doing that I did a backup of the SSD by clone it to an onther SSD.
To validate my backup, I decide to try the second SSD and I boot on it.

MAGIC !!! I get backup the performance !!!!!

Then, I decide to test the initial SSD to check if it is defect or if I reach the max write on it.
Test result, it work fine, very fine. Over time, it reach only 15% of the write commit.

So, at the end I don't know what was the real problem but I manage to solve it.

Thanks to all that try to help me solve the issue.

Regards.

SteveITS

@itou-0 glad to hear it but that doesn’t make much sense. The drive isn’t involved in routing unless the firewall rule is logging every packet. (Shrug). Maybe the drive was dying?