IP-Based Failover with AWS Marketplace pfSense App

  • Hi,

    I'm using the pfSense AWS appliance for letting remote clients access our AWS servers through OpenVPN. I'd like to apply IP-based active-active failover with two servers sharing the same subnet with the pfSense.
    The load-balancer service is working well, but is not usable for me because my servers accept only UDP traffic, and to my understanding UDP cannot be load-balanced by pfSense.
    Is there a way of implementing failover like this with pfSense? I don't necessarily need synchronization between the servers, I just need pfSense to expose a virtual IP that maps to either one of the servers, and redirect traffic to either one of them so that if one server fails, the other server can pick up the traffic. Round-robin is fine as a strategy.


  • @erez-buchnik I doubt pfsense can do openvpn active active out of the box since this is not directly supported by openvpn itself.
    And pfsense on carp is active - standby in nature as far as redundancy is concerned.
    Having said that, if you have two instances of openvpn in active standby and
    have three "public ip's on the same wan subnet (as per typical carp requirements) then you don't need anything special fon openvpn.
    When one instance fails, the other will take over.

    Alternatively you can always deploy two openvpn instances
    and allow the client to select on which it connects randomly.
    This works well too.
    But you need a program to dynamically manipulate openvpn routing tables (via its api) and adjust routing accordingly.
    This is something I have done in the past and works well, without pfsense. I suspect it could also happen on pfsense openvpn implementation too.
    However since routing is adjusted in polling mode, it might take up to 120seconds for the network to converge.

    Another option would be to deploy nginx which supports udp load balancing.
    Still, this would be another single point of failure and probably not suitable for what you are asking for.


  • @netblues thanks for your reply,

    Sorry if I wasn't completely clear - my pfSense appliance is the termination point for the OpenVPN tunnel. From there it is plaintext into the AWS subnet. pfSense is responsible for routing the UDP traffic, after decrypting it, to the servers in the back, which are located on the same AWS subnet as the pfSense instance. I just want pfSense to route the UDP traffic in a way that would allow the traffic to reach either one of these servers, and if one of the servers goes down, the UDP packets would reach the other server that is up. The clients would only need to know the virtual IP that pfSense exposes.
    I've done this successfully with the pfSense load-balancer service, but only for TCP. Although not clearly stated in the pfSense documentation, I found out through Google that the pfSense load-balancing service does not support UDP.
    We need UDP for keeping the latency as low as possible - this is an IoT application which requires real-time responsiveness to sensor readings.


  • @erez-buchnik Have a look at nginx udp load balancer
    If you introduce another box internaly that runs nginx and pfsense sends the unencrypted traffic there, nginx will do what you are looking for.

    You can always try to use the built in pfsense nginx binaries
    Have a look here

    You don't need to install anything else, just don't try to modify the GUI's nginx >instance. Setup your own nginx.conf, add a shell script /usr/local/etc/rc.d/my-nginx.sh containing something like:

    /usr/local/sbin/nginx -c /path/to/your/nginx.conf

    Keeping things separated might be better on the long run though.


  • @netblues Thanks!

    I'll give it a try now. Actually, load-balancing on-board the pfSense instance would be ideal for me, because it would save money, latency and points-of-failure.
    Strange that this feature is not supported by pfSense out-of-the-box. Wouldn't it be useful for e.g. load-balancing SIP servers?

    Thanks again,

  • I haven't tried the nginx udp load balancer to be honest. I guess the difficult part would be to determine when the receiving end of the udp stream has issues, mainly due to the connectionless orientation of udp.
    A mechanism must exist to know if the server is still alive.
    Ping would be an option, but if the application crashes, ping wouldn't know it.
    And then it becomes tricky at least.
    As for sip, if one needs to load balance most probably has also an sbc in place and then we are talking about front end sip aware load balancers which understands the sip protocol and can adjust rtp streams (which is essentially the load.)
    Do let us know how well it worked (or not)


  • @netblues thanks for all your help, your suggestion works for me!

    I've reconfigured the nginx service, rather than using a separate service as was suggested in the thread you shared above (required some googling to get it right).
    Now the challenge, as you said, is to construct a reliable healthcheck. I understood from the nginx documentation that an ICMP response when the port is unreachable can be used for that. When I get it working, I'll share the configuration - hope it will help others.

  • @netblues Hi, following up on this, below is a small and crude (sorry...) script for setting up a basic UDP LB with Nginx on-board pfSense. This script assumes that the directory /root/NGINX exists, and you have your custom nginx.conf file in it.

    if [ -f /usr/local/etc/rc.d/nginx ]
            echo "Backup and rename nginx service"
            cp /usr/local/etc/rc.d/nginx /root/NGINX/nginx-dist
            mv /usr/local/etc/rc.d/nginx /usr/local/etc/rc.d/nginx.sh
            cp /usr/local/etc/nginx/nginx.conf-dist /root/NGINX/nginx.conf-dist
            echo 'nginx_enable="YES"' >> /etc/rc.conf.local
    echo "Update nginx config"
    cp /root/NGINX/nginx.conf /usr/local/etc/nginx/nginx.conf
    echo "Restart nginx"
    service nginx.sh restart

    ...and this is the diff between the default nginx.conf and my custom one, which balances two AWS instances (addresses intentionally changed):

    [2.4.4-RELEASE][ec2-user@MY-pfSense.localdomain]/home/ec2-user: diff /usr/local/etc/nginx/nginx.conf-dist /usr/local/etc/nginx/nginx.conf
    > load_module /usr/local/libexec/nginx/ngx_stream_module.so;
    > user root wheel;
    > stream {
    >     upstream lb_instances {
    >         server;
    >         server;
    >         server;
    >     }
    >     server {
    >         listen  udp;
    >         proxy_pass lb_instances;
    >         proxy_bind $remote_addr:$remote_port transparent;
    >         proxy_responses 0;
    >     }
    > }

    It seems that the failover feature is an Nginx+ feature, which requires a paid subscription.

    Thanks a lot for your help!

Log in to reply