Poor Network Performance from 2.3

[randyruiz]

This post was originally named "Poor network performance with 2.3". I ran out of options troubleshooting why my pfsense 2.3 vm could only push 150Mb with iperf/scp so I decided to get a sanity check by going back to bare metal. I was shocked to see that at bare metal I would max out at 550Mb. When I install centos on this same machine I get 980Mb off the same interface that under pfsense would only push 500. I saw this same behavior when I had pfsense virtualized where a centos 7 vm siting side by side with pfsense vm would push 950Mb. This looks like it has to be a software config issue but I cant see where. Has anyone seen this before? This is pretty standard hardware for pfsense as I understand it.

Hardware Setup
Intel C2578 SOC chip
SUPERMICRO MBD-A1SRi-2758F-O
16 GB RAM

Forgot to mention no services are enabled this is a fresh install of 2.3 with only pfsense added.

[johnpoz]

centos is not bsd.. Are you testing through pfsense or to pfsense?

From just a common sense perspective you have router firewall distro - its purpose is to PUSH packets through itself, not answer traffic to itself..  So you would hope that that the pfsense teams goal is bandwidth through pfsense vs when talking to it.

But your still going to want to do apples to apples for your comparison - test with install of freebsd 10.3 if your want to see what its performance is compared to pfsense.

[randyruiz]

I am using pfsense as the iperf server and I am copying through it with scp. In either case the max bandwidth is the same.

[johnpoz]

well install a copy of just freebsd 10.3 which is what pfsense is using, and compare those apples.  As I mentioned centos is not bsd.. not really a fair comparison.. Have to assume different drivers for the nic, etc..  You would for sure hope freebsd is fast - but even if its not atleast then we have the same apples to work with.

[randyruiz]

I was able to push 1 gig with pfsense when it was at 2.6 on much lessor hardware (celeron n2980). I know that freebsd and pfsense is capable of routing line speed, that is not the question. The question is why is pfsense performing so slowly on hardware that is relatively  common to pfsense and is overpowered for the task.

[johnpoz]

2.6 – so your from the future? ;)

[Harvy66]

@randyruiz:

I am using pfsense as the iperf server and I am copying through it with scp. In either case the max bandwidth is the same.

This could be part of the issue, even if not too likely. When I run iperf on PFSense, I only get around 930Mb/s and 25% cpu usage. When I run iperf through PFSense, ie client outside the NAT and client in the LAN, I get 3.9Gb/s and 5% cpu usage. I'm not sure why there is a such a difference, but I don't really care, it's fastest at being a firewall, not an app server.

[randyruiz]

Got me johnpoz I meant 2.2

Harvy66
I also have scp'ed through the firewall and still ended up with the same number.

[Harvy66]

What does System Activity say for your top CPU percentages when running SCP through?

[randyruiz]

Here is a screenshot of an scp session.


[Guest]

I was shocked to see that at bare metal I would max out at 550Mb.

Me too, but more pending on what you are doing and how do you it are doing.
In a VM you haven´t only one CPU core or SoC core for the WAN part, its multi core threated!
And on a bare metal installation it is so, that the modem is put at the WAN port and there fore
and on top of this together with PPPoE you are now single CPU threated!!!! please don´t forget this.

When I install centos on this same machine I get 980Mb off the same interface that under pfsense would only push 500.

Where the hell CentOS is doing in any kind of direction NAT / SPI and performing firewall rules?
It don´t do that all!!! And there fore on top of this it is multi CPU core usage and pfSense together with
PPPoE single treated again. So it can´t be really the true doing that.

Its like I have 2 cars one is a Mercedes 600S AMG and the other is a Fiat500, but both have a motor
4 wheels and burning benzine.

Forgot to mention no services are enabled this is a fresh install of 2.3 with only pfsense added.

This will be really good but only one step of two that should be done!

• do a fresh install of version 2.3 64Bit
• configure WAN and LAN part
• connect the modem and the WAN port to a smaller switch and on top of this a PC or Laptop as iPerf server
• connect now to the LAN port a switch that is connecting too a PC or Laptop and then acting as a iPerf client.

Now do some speed or throughput tests! This would be showing you the real throughput of your pfSense box
and nothing more or less. SCP and other programs should not be in usage! Please try out iPerf or NetIO.

If the results are not matching you should be tune the NICs, by high up the mbufs size to 1000000.
The SG-4860 is similar to the A1SRi-2558 and is able to push 500+ MBit/s over IPSec together with
AES-GCM. And your board is able to push nearly 1 GBit/s over the WAN Port as I see it right but perhaps
you should understand that a test is not likes other tests. If you own a router with an integrated modem
and you will be put your pfsense behind of that it would be more using the static IP instead of the PPPoE
and then you will be getting more throughput or speed.