Very poor NAT performance



  • Hi all,

    I have a pfsense box running in an ESXi 5.5 with two vmxnet3 adapters. I have IPSec + OpenVPN service on the WAN adapter activated. Accessing a service through IPSec produces around 10MB/s speed whereas accessing the same service via NAT (port forward) produces 200KB/s speed.

    There's nothing exciting about the setup, just a simple port forward. I checked the pf ruleset and indeed nothing extra is generated: (vmx0 being the WAN and vmx1 being the LAN interface)

    no nat proto carp all
    nat-anchor "natearly/" all
    nat-anchor "natrules/
    " all
    nat on vmx0 inet from 127.0.0.0/8 to any port = isakmp -> 172.16.0.2 static-port
    nat on vmx0 inet from 172.16.7.0/24 to any port = isakmp -> 172.16.0.2 static-port
    nat on vmx0 inet from 10.0.0.0/24 to any port = isakmp -> 172.16.0.2 static-port
    nat on vmx0 inet from 127.0.0.0/8 to any -> 172.16.0.2 port 1024:65535
    nat on vmx0 inet from 172.16.7.0/24 to any -> 172.16.0.2 port 1024:65535
    nat on vmx0 inet from 10.0.0.0/24 to any -> 172.16.0.2 port 1024:65535
    no rdr proto carp all
    rdr-anchor "relayd/" all
    rdr-anchor "tftp-proxy/
    " all
    rdr on vmx0 inet proto tcp from any to 172.16.0.2 port = https -> 172.16.7.1
    rdr-anchor "miniupnpd" all

    This performance is amazingly bad and couldn't figure out why. Tried "fetch" from the pfSense box as well resulting in 92MB/s inwards and 10MB/s outwards of the box, indicating that the network is fine.

    Anyone have any idea why the performance is soon bad?

    VM HW: 4CPUs, 1GB RAM, 2 vmxnet3

    Any help is appreciated.



  • Do you have 5.5 U2? FreeBSD10 is only supported on 5.5U2 and 6.



  • @EMWEE:

    Do you have 5.5 U2? FreeBSD10 is only supported on 5.5U2 and 6.

    i've got a 2.2.X running on esxi4.1 running the legacy e1000 nics, just fine … hitting 1gbit/s wire speed without too much trouble.

    there is something else going on here.
    did you accidently install the official vmware-tools ? if yes --> reinstall and don't do it again  ;)



  • I've got the latest ESXi as the update manager keeps all the hosts updated.
    As for the vmware-tools, I had it installed way back when my pfsense was 2.1 or 2.0 not sure. Back then the FreeBSD kernel did not support the vmxnet3 out of the box. Before I have upgraded the pfsense (using the autoupdated) I have uninstalled properly the vmware-tools.

    However that could be a candidate. One thing which leaves some doubt, if I change the adapters to e1000, the performance still the same. But only for the forwarded ports. When I connect to any VPN provided on the WAN interface and reach the LAN like that, all's good. Only the port forwards are extremely bad. I think I going to have to reinstall maybe, but if I do I'd like to understand why? It just doesn't make much sense to me at the moment.



  • Well, the bad news is that I have reinstalled and the issue remains. I used the latest stable 2.2.2 amd64 release.


  • Banned

    I can easily get wirespeed on the 2.2.2 release using NAT.

    I use the E1000 NIC's. FreeBSD support VMXnet3 out of the box and it could easily be shitty drivers.



  • I mentioned in one of the posts that I tried to change the NIC to e1000 and did not help. But let me try that again.


  • Banned

    It could be the fact you NAT a very large subnet to a smaller one, but still belonging to the same overall subnet.

    Pretty weird rules tbh.



  • What do you mean? It's a simple port forward. Are you looking at the rdr rules (which is the problem) or the nat (which is outgoing NAT). The outgoing NAT couldn't be more standard….



  • what does the cpu graph show on the vsphere client? (while pushing traffic)

    how fast can you fetch a file from the pfSense console ? (to find out if its only while forwarding, or a general connection issue)



  • CPU is around 0-1 percent both on pfSense and ESXi side. Virtually not utilised at all.

    The fetch is near 100MB/sec with once again near zero CPU utilisation. When I connect to OpenVPN or IPSec on WAN I can reach the LAN with full speed. The network drivers are absolutely fine in my view.



  • Confirming the very same issue



  • I'm seeing the same type of behaviour. When the gateway is the CARP Vip my throughput out of WAN is ~3mbps max as soon as I switch to the real router LAN interface I have connection speeds of 50mbps (which is normal). No raise in CPU or memory usage either.

    ESXi 6.0
    4gb Ram
    5 CPUs


Log in to reply