PFSense in-front of high traffic web-servers
-
Hi everyone,
I have been using PFSense for several years but now I am planning to deploy a beefy PFSense box (10GB SFP+) in-front of our web-servers and using NAT to assign multiple public IPs to internal IPs.
My concern is that we can peak at over 5,000 connections per second from unique IP addresses, that's around 300,000 unique IP connections per minute.
Is their anything I need to be mindful of when building the router (hardware) and any features I should ideally turn off to ensure it can handle the load with ease?
-
I'm not sure that many people here have experience with that kind of load. You might be best served by logging a support request with Netgate. I'm sure they've dealt with your scenario before.
-
That is going to make a very large state table. You will probably need to tune the state timeouts to keep it rational.
https://docs.netgate.com/pfsense/en/latest/book/config/advanced-firewall-nat.html#firewall-adaptive-timeoutsSteve
-
Is pfSense really appropriate for this? High performance routers, aka layer 3 switch, have much of the function done in hardware, not software. Also, NAT is a real performance killer. You'd be better off not using it and going with separate addresses instead.
-
@g7cloud why not use HAProxy? What point to use NAT?
HAProxy will give you:- centralized view of all yours frontends/backends
- possibility to use multiply backends for round-robin with high availability with static sessions (by client IP or cookies value, etc.)
- health checking of backends (from simple port knocking and up to Layer 7 OPTION/GET /page). Brilliant addition is Email notification for system administrators if backend is down
- possibility to give 503 or other error HTTP codes (you can create custom simple http pages for that) to clients when yours backends is down or busy
- possibility to prevent DDoS by adding limits for requests/second for frontends/backends and etc.
- possibility to add some redirects, headers.
- possibility to use pfSense (or even pfBlocker NG devel package) created Alias IP lists to work with them as a conditions for rules. I see in this really cool future to:
- have reject action if IP not from allowed lists
- choose specific backend based on IPs condition
There can go list based on:
- ASNs
- Countrys
- Domain resolving
- Static lists of IP or IP subnets
- centralized SSL certificate management with possibility use ACME and easy get HTTP/2 and OCSP support
- centralized logging of requests
- and much-much more...
Only two remarks:
- use haproxy-devel (haproxy-1.8.x) package [ not haproxy (haproxy-1.7.x) one ] because 1.7 is too old and devoid of many goodies
- better install Syslog-ng package, and configure HAproxy to write logs in Syslog-ng. Not use build-in Syslog of pfSense for HAProxy, because it not have logrotate and can't move logs to archives. All that build-in Syslog can do is bulk write to one log with overwrite all that does not fit in file due to log size limit
-
Yes, if you can use routed IPs instead of NAT I would definitely do that.
Steve
-
Thanks for all your feedback.
I currently route public IPs directly to each server as opposed to using NAT however this doesn't satisfy my requirements.
The servers are not in a cluster, they are not used for load balancing. Each server contains its own set of websites running in complete isolation to the other servers. HAProxy with load balancing isn't what I require for this.
The reason I want to switch from routed IPs to NAT is that with NAT I can easily divert traffic to another server via PFSense for maintenance with no downtime. I recently had to restart one of our beefiest servers that deals with huge traffic spikes. I was in literally sweating and praying it will come back online with no issues, although I had an identical spare ready to go in the event of a disaster I rather have the ability to do this with zero downtime.
To avoid this I am planning to use NAT and when I need to switch off or restart a server for maintenance I can setup an identical secondary server and via PFSense just switch the NAT internal IP to the secondary server whilst I do maintenance on the primary server resulting in no time.
I tried this in the past with routed IPs but there was about a 5-minute downtime until the switch realises the device for a given IP has now moved to a different mac address.
The PFSense spec I am thinking of is
12 cores E5-2620 V2 processors
32GB Ram
256GB SSD
Dual redundant power supplies
An identical back up server will always be on stand by. -
@g7cloud
NAT is real evil in any network. Specially for Web-apps. This my point of view, but I think it correct.
With HAproxy you doesn't forced to use load balancing. You can configure one IP per backend.
In case it goes down in any of issues client will have nice 503 and you will have alert on your Email. This in 100 times better then have: No response from server which will happen with NATed not working web-server.
You can force HTTPS 301 redirect for all sites, add strict-transport-security, referrer-policy, x-xss-protection, x-content-type and x-frame-options headers on pfSense Haproxy WebGUI and not worry about missed back-end configuration. -
@dragoangel can HAProxy pass SSL termination? I need SSL termination to take place at the server-level not at HAProxy. Additionally, Can HAProxy forward any port I require i.e. 22, 21, 3307 etc. If so I will explore this option for sure.
-
@g7cloud yes for all: http://cbonte.github.io/haproxy-dconv/
- HAproxy can work with SSL offloading or forward SSL without termination end even log requests with this case. But in the case or forward SSL without termination, HAproxy will have incomplete capabilities: it will only see who was connected, what link it went to and which backend responsible for that request. Health checks work for SSL websites OK even on Layer 7 with SNI if you add Host header to backend.
- HAproxy can be proxy for any TCP protocol. It can even be used for load balancing LDAP or MySQL. For example I use it to load-balancing GIT over SSH. Part of functionality only available for HTTP type, but many of functions like reject connection rules, logging, health checks will work even with TCP.
- Now only UDP proxy are not supported by HAproxy. But it already in HAproxy 2.1 development. =)
-
Hey @dragoangel thanks for all your comments.
I have used HAProxy before for load balancing a single website therefore slightly familiar with it. However I am not sure its suitable for my specific requirements.
As I currently have 26 public IPs I would need to assign all of them to a single virtual machine running HAProxy which means 26 virtual Ethernet adapters. Furthermore I need all ports forwarded for mail, DNS, MySQL etc. I still think mapping out a public IP to a local IP via NAT might be the simplest and best solution. With the spec I posted earlier it should be good for 30 or so million states and with slight tuning to close TCP connections early it should be up for the task at hand. I could quite easily upgrade to 128GB or even 384GB of ram if ever needed.
If anyone strongly opposes my PFSense + NAT plan please do speak up, I will take everyone's comments on board.
-
@g7cloud DNS can't be proxied due it mostly UDP. Mail and SQL over proxy have more control over simple nat, but if you have no time on confirmation and tuning then nat is ok. P.s. be careful when nat sql: use only with ssl
-
Some good read here:
https://forum.netgate.com/topic/7226/how-far-have-you-scaled-your-pfs-box/14
just a bit dated. Pictures missing from forum conversion. -
@jahonix what did you try looking at post with 8 year old? This posts not about this internet that we have now and not about this pfsense os and hardware too... It not comparable I think
-
Some nostalgia from 11 years ago. Same problem then, just scaled.