Load balancing dns



  • I have a pfsense box configured for incoming load balancing with two servers behind it running powerdns and apache. The load balancing to the http port works fine. But my dns requests always timeout. I have configured a firewall rule for port 53 udp/tcp but my requests are not answered. I have also tried changing the dns timeout to 10 seconds but nothing seems to work i always get timeouts.



  • Make sure your monitoring for the pool works. If it is setup incorrectly your pool might be assumed empty and no traffic will be forwarded.



  • Okay i will give it a try tommorrow. I will start of from scratch as you mentioned in my other posting. As i suspected something has gone wrong because i cannot set a monitor ip for my load balancer pools.



  • @wizard:

    Okay i will give it a try tommorrow. I will start of from scratch as you mentioned in my other posting. As i suspected something has gone wrong because i cannot set a monitor ip for my load balancer pools.

    On the off-chance you don't see my other reply.  Server pools don't need monitor IPs - they were never used for server pools and it was a bug to allow you to enter them.  Also, load balancing only works with TCP protocols.  I'll work on UDP for post 1.0 (slb, the load balance daemon we use doesn't currently support UDP).

    –Bill



  • Thats a great shame as i was hoping to use pfsense to load balance four name servers running powerdns. I have been running various tests with pfsense during the last few weeks and it works a charm so i would really like to use it. How long do you think it will take to support load balancing UDP? Would it help if i ask my boss to raise some money for this feature. Because otherwise i will have to go back to using LVS unless you can think of a better solution.



  • Raising money ALWAYS helps. If you check out the bounty section you will notice that a lot of features that were put up there have been included already or are worked on. Also some of the features not mentioned there were implemented because of donations.



  • I can understand your argument money always helps :) What would the minimal bounty be for pfsense to support load balancing UDP. I need a rough amount so i can talk to my boss about the matter because it's his decision. How long do you think it would take is it a matter of days or months? Because we need a working solution fairly quickley and i told him pfsense would meet our requirements. Before finding out that load balancing UDP doesn't work i didn't realise this restriction.



  • Billm ist best suitable to pick that task as he did nearly all the work on the balancer. I'll leave it up to him to answer the questions  ;)



  • Okay then i will stay put and wait for an answer before installing gentoo with keepalived or open bsd with carp. Which seem to be the alternatives to pfsense i would still prefer pfsense.



  • @wizard:

    I can understand your argument money always helps :) What would the minimal bounty be for pfsense to support load balancing UDP. I need a rough amount so i can talk to my boss about the matter because it's his decision. How long do you think it would take is it a matter of days or months? Because we need a working solution fairly quickley and i told him pfsense would meet our requirements. Before finding out that load balancing UDP doesn't work i didn't realise this restriction.

    Your biggest issue will be that 1.0 is frozen.  This won't/can't be imported.  I'd be willing to work on this outside of pfSense and provide/support a patch for 1.0 until we have a feature release which officially supports it.  I can give you a version which will do UDP load balancing (no failover - but the great thing about DNS is that it'll try again) in short order, but actual availability checking (which won't be optimal anyway - it's udp) will take more thought and time.

    My gut feel is that to get you straight up UDP load balancing will take me about 2 hours (including the generating a custom firmware update for 1.0).  As this would have to be supported by myself until at least 1.0.1 (most likely 1.1 as I don't see us adding this to RELENG_1 even after 1.0 is out the door), I'd want to make sure that I'm moderately compensated for it.  Let's call it $300 for the above.  Failover detection will be considerably more complex and I'll have to think about the best way to make that happen - feel free to contact me privately if you want me to come up with a reasonable estimate for that.

    –Bill



  • Thanks alot for your answer, as i stated earlier it's not my decision my boss seems to think the solution will do us no good without failover detection. Because we need to load balance four dns servers (two primary and two slave name servers) with two load balancers in an active active setup. So for the moment it looks like i will have to turn to openbsd with carp and configure the two load balancers manually. I would still like to return to pfsense once load balancing UDP is fully functional because i find your project very promising. I am sure i will be able to persuade my boss to donate some money at a later date when load balancing UDP is fully functional in the stable tree. I will keep posting and stay patient waiting for newer versions of pfsense  ;D THX

    Wizard



  • @wizard:

    Thanks alot for your answer, as i stated earlier it's not my decision my boss seems to think the solution will do us no good without failover detection. Because we need to load balance four dns servers (two primary and two slave name servers) with two load balancers in an active active setup. So for the moment it looks like i will have to turn to openbsd with carp and configure the two load balancers manually. I would still like to return to pfsense once load balancing UDP is fully functional because i find your project very promising. I am sure i will be able to persuade my boss to donate some money at a later date when load balancing UDP is fully functional in the stable tree. I will keep posting and stay patient waiting for newer versions of pfsense  ;D THX

    Uhhh, carp isn't a load balancer, nor does it do protocol failure detection.  slbd is the only daemon on openbsd that will do failure detection of load balanced ports and it doesn't do UDP at all currently (hence the problems in pfSense).  HAProxy would be good, but it too only does TCP.  I'm interested in your final setup as I know of no BSD based load balance solutions that will not only balance UDP, but do failure detection (unless you were planning on writing scripts for that part - heh, come to think of it, that approach could technically be done in pfsense too, ugly hack, but it'd likely work).

    –Bill



  • hm perhaps i need to do some re thinking. I thought i would be able to use something like this http://www.openbsd.org/faq/pf/pools.html for my load balancing solution. I wasn't sure that slbd was the only deamon under BSD which can handle load balancing because i could hardly find any information on it. But if you say so i am sure you are right so i will have to go back to LVS with keepalived under Linux which supports UDP load balancing. With the price of losing the flexibility of CARP which i was beginning to like. I will keep you posted on my project wish me luck. If you have any other ideas please tell me.



  • @wizard:

    hm perhaps i need to do some re thinking. I thought i would be able to use something like this http://www.openbsd.org/faq/pf/pools.html for my load balancing solution. I wasn't sure that slbd was the only deamon under BSD which can handle load balancing because i could hardly find any information on it. But if you say so i am sure you are right so i will have to go back to LVS with keepalived under Linux which supports UDP load balancing. With the price of losing the flexibility of CARP which i was beginning to like. I will keep you posted on my project wish me luck. If you have any other ideas please tell me.

    PF itself does no availability checking.  That's what we use slbd for - it's responsibility is to insert rules into an anchor (slightly different than the pools, but same concept) based on what's actually up.  Again, load balancing is easy - availability checking is considerably more difficult and not usually terribly conclusive.

    Consider this.  UDP is a stateless protocol, it's not required to reply to anything it doesn't understand (TCP at least sends resets!).  The way port scanners detect an "open" UDP port is by the lack of an ICMP port unreachable reply.  Guess what happens if the box is down?  Oh yeah, ALL ports will refuse to return ICMP port unreachable.  OK, so now we have to tie in some other means of checking - let's say ICMP.  So, now we get if the box is pingable and I'm not getting an ICMP port unreach answer, the daemon on that port must be good right?  Bzzt…what if it's just b0rked but still listening (never happen you say?  heh, I've had djbdns ick zombify on me and refuse to die - still listening on port 53).

    FWIW, even our commercial F5 BIGIP (LTMs now) at work don't load balance (and do availability checking of) UDP - it can't be done reliably.  Specific protocols are doable, but UDP in general isn't (consider syslog...you can't send a valid syslog packet to a syslog daemon that will make it reply to you...how do you know it's not b0rked?  you don't)

    The way I'd design your setup is the following (and it's free advice, so take it for what it's worth)

    Two firewalls in an active passive pair with two CARP virtual IPs.
    Four PowerDNS servers with one CARP VIP each, active for one, passive for the other three at different skews - this will cover any box failure that might occur.
    If daemon failure is a serious concern, then write a dig script on the firewall to dig all four CARP VIPs and check the result, if they're answering, update your DNS server table in PF with the addresses.  Alternately, on the machines themselves, you can use ifstated to do essentially the same thing - check to see if it's resolving, if not, set the CARP address to backup and let the other machines duke it out for taking control.

    --Bill


Log in to reply