Planning for 3rd internet line

[Paladinemishakal]

Hi All,

Currently I am using the following for my office pfsense servicing 500 ~ 600 users locally and with IPSec and OpenVPN configured for regional offices and roaming users. I am located in Singapore.

Server Hardware
Physical Server: IBM System x3650 M2
CPU: 2 x Intel Xeon E5520 @ 2.27 Ghz (16 CPU)
RAM: 16 GB
HDD: SAS HDD use for ESXi usage

VM server: ESXi 5.1.0 Build 799733
pfSense Version: 2.15-release (amd64)
CPU: 4 vCPU (2 virtual socket with 2 cores per socket)
RAM: 8 GB
HDD: NFS mounted folder with 10G backbone
Network adapter 1: E1000 (Singtel)  <– Broadcom NetXtreme II BCM5709, Managed by ESXi
Network adapter 2: E1000 (Starhub) <-- Broadcom NetXtreme II BCM5709, Managed by ESXi
Network adapter 3: E1000 (DMZ)      <-- Broadcom NetXtreme II BCM5709, Managed by ESXi
Network adapter 4: (LAN)                  <--Broadcom NetXtreme II BCM5709, Configured as pass-through so as to achieve higher throughtput

I have 2 Fiber line from the ISP - Singtel and Starhub. This is the statistics of the interface from 01 Dec 2014 to 23 Mar 2015.

Singtel Line : 30 Mbps – Fiber,
  Incoming Traffic - Avg: 5.47 Mbps, Min: 1.16 Kbps, Max: 60.97 Mbps
  Outgoing Traffic - Avg: 3.32 Mbps, Min: 2.77 Kbps, Max: 60.47 Mbps
Starhub Line : 20 Mbps – Fiber
  Incoming Traffic - Avg: 6.69 Mbps, Min: 24.76 Kbps, Max: 20.49 Mbps
  Outgoing Traffic - Avg: 1.02 Mbps, Min: 17.49 Kbps, Max: 17.12 Mbps
LAN : 1Gbps - Copper
  Incoming Traffic - Avg:  4.92 Mbps, Min: 108.06 Kbps, Max: 830.43 Mbps
  Outgoing Traffic - Avg: 12.95 Mbps, Min: 128.09 Kbps, Max: 170.71 Mbps
CPU Utilization is about 15 ~ 20%
Memory usage is about 5.6 Gb out of 7.98 Gb

On the Firewall, I have also use the Traffic Shaping feature and Squid to throttle the network access to popular sites and streaming media.

Bandwidth Throttling by Squid Proxy Server
Rule:
redirect_children 3;
acl dom_restricted dstdomain .googlevideo.com .youtube.com .phobos.apple.com .skype.com .digitalrivercontent.net;
acl net_nolimit src 10.25.1.0/24 10.25.69.0/24;
delay_class 2 3;
delay_parameters 2 262144/15728640 262144/7864320 262144/262144;
delay_access 2 allow dom_restricted;
delay_access 2 deny net_nolimit;
delay_access 2 deny all;
redirect_program /usr/pbi/squidguard-amd64/bin/squidGuard -c /usr/pbi/squidguard-amd64/etc/squidGuard/squidGuard.conf;redirector_bypass off;
url_rewrite_children 5
Traffic Shaping is configured:
By Queue

• qInternet
    - WANStarHub, bandwidth: 19000 Kb
    - All except Singtel Line, bandwidth: 19456 Kb
• qACK – (All except Singtel Line, bandwidth: 19%)
• qDefault – (Starhub Line, bandwidth: 25%)
• qLink – (All except Singtel/Starhub line, bandwidth: 20%)
  By Limiter
• By source and by destination
• Download Limit is set to 2048 Kbit/s
• Limiter-10mb is set to 10000 Kbit/s
  By Layer7
• Block bittorrent protocol
• ThrottleHttp video
    - httpvideo,httpaudio,flash,100bao limited by limiter-10mb

IPSec Tunnel

• 1 tunnel to another location within the country
• 10 tunnels to oversea location

OpenVPN Server

• 3 server configured for about 30 ~ 40 users for roaming users.

The Singtel Line is use for the server whereas the Starhub Line is use for general net browsing by the users and the usage for the Starhub Line is always maxing out at 20Mbps.

Looking the above usage, I am looking at adding another 1GBps fiber internet line from a local ISP to this setup.
Questions that I would l like to ask is:
1. Is the currently allocated server resources (eg. CPU, RAM, network interfaces), is it suitable?
2. If no, what specs should I be looking at?
3. If the new fiber line has to be terminated at another location with the office building, should I split the server?
4. If splitting the server, what is the specs that I should be looking at?
5. What is the optimal setup in this case?

Appreciate is someone can advise me.

Thanks & Regards.

[heper]

you should check your vsphere cpu statistics … they greatly differ from the one you get inside the pfSense VM.

the legacy software em(x) drivers use a lot of cpu when pushing a lot of bandwidth. (generally 1-2gbit/s shouldn't be an issue on your virtual machine).
upgrading to esxi 5.5u2 or newer and using the vmxnet3 drivers could help a bit in that department. ( you'd need to update to 2.2.1-release to make use of vmxnet3 out of the box)
----> there are some open bugs in the traffic-shaping/L7 department on 2.2.x, so check them out if they are relevant in your setup.

What might be another cpu hog is your tunnels ... not sure how much extra bandwidth you are expecting there but encryption is cpu intensive if you cannot offload it to a crypto-card or eas-ni.

[edwardwong]

I saw that pfSense by default disabling NIC's TCP offload engine, is this the reason for high CPU load?

[Paladinemishakal]

Hi,

I have attached 3 graphs as follows:
1. 2 graphs captured by ESXi and it show that the usage is almost same as what is captured by Zabbix. The item of interest is sgfw01-new.
2. 1 graph captured by Zabbix.

Looking at the graphs, should I be increasing the CPU and RAM to accommodate the increase in network capacity? or should I stay with the current setup?

As of now, I have tested on pfsense 2.2.1 and it seem the VPN portion is not that stable so will not be upgrading to that version.

Can you point me to the traffic shaping bugs as mentioned by you for the pfsense 2.2.x?

Thanks & Regards.


[Paladinemishakal]

Also to note is that the 1Gbps line will be servicing just the internet browsing so will most likely be using the Traffic Shaping and Squid Proxy Throttling.

[heper]

https://redmine.pfsense.org/issues/4276

https://redmine.pfsense.org/issues/4326
https://redmine.pfsense.org/issues/4405

https://redmine.pfsense.org/issues/4524
https://redmine.pfsense.org/issues/4529

most of those won't be an issue but 4276 will if you plan to use L7 on 2.2.x