IP-Based Failover with AWS Marketplace pfSense App

Erez Buchnik

Hi,

I'm using the pfSense AWS appliance for letting remote clients access our AWS servers through OpenVPN. I'd like to apply IP-based active-active failover with two servers sharing the same subnet with the pfSense.
The load-balancer service is working well, but is not usable for me because my servers accept only UDP traffic, and to my understanding UDP cannot be load-balanced by pfSense.
Is there a way of implementing failover like this with pfSense? I don't necessarily need synchronization between the servers, I just need pfSense to expose a virtual IP that maps to either one of the servers, and redirect traffic to either one of them so that if one server fails, the other server can pick up the traffic. Round-robin is fine as a strategy.

Thanks,
Erez

netblues

@erez-buchnik I doubt pfsense can do openvpn active active out of the box since this is not directly supported by openvpn itself.
And pfsense on carp is active - standby in nature as far as redundancy is concerned.
Having said that, if you have two instances of openvpn in active standby and
have three "public ip's on the same wan subnet (as per typical carp requirements) then you don't need anything special fon openvpn.
When one instance fails, the other will take over.

Alternatively you can always deploy two openvpn instances
and allow the client to select on which it connects randomly.
This works well too.
But you need a program to dynamically manipulate openvpn routing tables (via its api) and adjust routing accordingly.
This is something I have done in the past and works well, without pfsense. I suspect it could also happen on pfsense openvpn implementation too.
However since routing is adjusted in polling mode, it might take up to 120seconds for the network to converge.

Another option would be to deploy nginx which supports udp load balancing.
Still, this would be another single point of failure and probably not suitable for what you are asking for.

Regards

Erez Buchnik

@netblues thanks for your reply,

Sorry if I wasn't completely clear - my pfSense appliance is the termination point for the OpenVPN tunnel. From there it is plaintext into the AWS subnet. pfSense is responsible for routing the UDP traffic, after decrypting it, to the servers in the back, which are located on the same AWS subnet as the pfSense instance. I just want pfSense to route the UDP traffic in a way that would allow the traffic to reach either one of these servers, and if one of the servers goes down, the UDP packets would reach the other server that is up. The clients would only need to know the virtual IP that pfSense exposes.
I've done this successfully with the pfSense load-balancer service, but only for TCP. Although not clearly stated in the pfSense documentation, I found out through Google that the pfSense load-balancing service does not support UDP.
We need UDP for keeping the latency as low as possible - this is an IoT application which requires real-time responsiveness to sensor readings.

Thanks,
Erez

netblues

@erez-buchnik Have a look at nginx udp load balancer
If you introduce another box internaly that runs nginx and pfsense sends the unencrypted traffic there, nginx will do what you are looking for.

You can always try to use the built in pfsense nginx binaries
Have a look here
https://forum.netgate.com/topic/98298/pfsense-2-3-nginx-questions/8

You don't need to install anything else, just don't try to modify the GUI's nginx >instance. Setup your own nginx.conf, add a shell script /usr/local/etc/rc.d/my-nginx.sh containing something like:

#!/bin/sh
/usr/local/sbin/nginx -c /path/to/your/nginx.conf

Keeping things separated might be better on the long run though.

Regards

Erez Buchnik

@netblues Thanks!

I'll give it a try now. Actually, load-balancing on-board the pfSense instance would be ideal for me, because it would save money, latency and points-of-failure.
Strange that this feature is not supported by pfSense out-of-the-box. Wouldn't it be useful for e.g. load-balancing SIP servers?

Thanks again,
Erez

netblues

I haven't tried the nginx udp load balancer to be honest. I guess the difficult part would be to determine when the receiving end of the udp stream has issues, mainly due to the connectionless orientation of udp.
A mechanism must exist to know if the server is still alive.
Ping would be an option, but if the application crashes, ping wouldn't know it.
And then it becomes tricky at least.
As for sip, if one needs to load balance most probably has also an sbc in place and then we are talking about front end sip aware load balancers which understands the sip protocol and can adjust rtp streams (which is essentially the load.)
Do let us know how well it worked (or not)

Regards

Erez Buchnik

@netblues thanks for all your help, your suggestion works for me!

I've reconfigured the nginx service, rather than using a separate service as was suggested in the thread you shared above (required some googling to get it right).
Now the challenge, as you said, is to construct a reliable healthcheck. I understood from the nginx documentation that an ICMP response when the port is unreachable can be used for that. When I get it working, I'll share the configuration - hope it will help others.
10x,
Erez

Erez Buchnik

@netblues Hi, following up on this, below is a small and crude (sorry...) script for setting up a basic UDP LB with Nginx on-board pfSense. This script assumes that the directory /root/NGINX exists, and you have your custom nginx.conf file in it.

#!/bin/sh

if [ -f /usr/local/etc/rc.d/nginx ]
then
        echo "Backup and rename nginx service"
        cp /usr/local/etc/rc.d/nginx /root/NGINX/nginx-dist
        mv /usr/local/etc/rc.d/nginx /usr/local/etc/rc.d/nginx.sh
        cp /usr/local/etc/nginx/nginx.conf-dist /root/NGINX/nginx.conf-dist
        echo 'nginx_enable="YES"' >> /etc/rc.conf.local
fi

echo "Update nginx config"
cp /root/NGINX/nginx.conf /usr/local/etc/nginx/nginx.conf
echo "Restart nginx"
service nginx.sh restart

...and this is the diff between the default nginx.conf and my custom one, which balances two AWS instances (addresses intentionally changed):

[2.4.4-RELEASE][ec2-user@MY-pfSense.localdomain]/home/ec2-user: diff /usr/local/etc/nginx/nginx.conf-dist /usr/local/etc/nginx/nginx.conf
0a1
> load_module /usr/local/libexec/nginx/ngx_stream_module.so;
15a17
> user root wheel;
122a125,142
>
> stream {
>
>     upstream lb_instances {
>         server 1.1.1.17:1234;
>         server 1.1.1.147:1234;
>         server 1.1.1.140:1234;
>     }
>
>     server {
>         listen 2.2.2.1:5678  udp;
>         proxy_pass lb_instances;
>         proxy_bind $remote_addr:$remote_port transparent;
>         proxy_responses 0;
>     }
> }
>

It seems that the failover feature is an Nginx+ feature, which requires a paid subscription.

Thanks a lot for your help!
Erez