Using pf to mitigate DDoS - discussion at openbsd-misc



  • There was a discussion about DDoS mitigation recently at openbsd-misc, which I thought some people here might find interesting.

    It seems that from a variety of attacks, a big SYN flood is the most harmful.

    From: Roger S.
    Subject: How to deal with DDoS ?
    Newsgroups: gmane.os.openbsd.misc
    Date: 2012-02-20 16:57:05 GMT (1 week, 6 days, 3 hours and 16 minutes ago)
    Greetings misc@

    I am facing regular and consequent DDoS, and I would like to know how
    the OpenBSD community deal with these. Hints and inputs welcome.

    The obvious first : my input pipes are not filled, there is plenty of
    bandwith available for my regular users. As OpenBSD is not enough (in
    my setup, I am sure there is a solution) to mitigate such attacks we
    use a proprietary product, but this solution has some undesirable
    side-effects and is not a viable long term solution for us.

    Methodology is more or less always the same :

    • massive UDP flood          :  2 Gbps / 150 Kpps -> dropped
      directly on the router, not a problem
    • moderate ICMP flood        :  10 Mbps /  12 Kpps
    • moderate IP fragments flood : 380 Mbps /  57 Kpps
    • moderate TCP RST flood      :  10 Mbps /  30 Kpps
    • massive TCP SYN flood      : 640 Mbps /  2 Mpps -> yup, that hurts

    So, UDP never ever reaches my OpenBSD box. The SYN are made with a
    very vicious method : each used IP send exactly one SYN, but there are
    millions of them (traffic probably spoofed, but can not use uRPF as we
    have asymmetric traffic and routes). I tried to set limit states with
    1M entries, and it was quickly filled (tried 5M but the box collapses
    way before that). So in the end, the state table collapses and no
    traffic can pass, even for regular users with already established
    connections.

    I ran some experiments in a lab trying to reproduce this, with a box
    roughly identical to what I have in production (but much weaker, of
    course). The box collapses at 600 Kpps SYN (100% interrupts), but
    handles everything very gently (less than 50% interrupts and no packet
    loss) if the first rule evaluated is block drop in quick from !
    <whitelisted_users>. So it seems that my bottleneck is PF here, not
    the hardware. A consequence of this saturation : both my main firewall
    and my backup claims MASTER ownership of the CARP (split brain
    syndrome). CARP works just fine when I add the block rule, though.

    Some configuration details :

    • OS  : OpenBSD 5.0/amd64 box, using GENERIC.MP
    • CPU : Intel X3460 CPU (4 cores, 2.80GHz)
    • RAM : 4GB
    • NIC : 2x Intel 82576 (2 ports each)

    Each network card has the following setup : one port to the LAN, one
    port to the WAN. Each pair (LAN1/LAN2 and WAN1/WAN2) is trunked using
    LACP. Already bumped net.inet.ip.ifq.maxlen, as all NICs are
    supported. My benchmarks did highlight two interesting things : amd64
    has better performance than i386 (roughly 5-10% less interrupts, with
    same rules and traffic), but the difference between GENERIC and
    GENERIC.MP is insignificant.

    My current idea is to hack a daemon to track established connections
    (extracting them la netstat), and inject my block rule in an anchor ( la relayd) when needed (watching some stats from pf, with its ioctl
    interface). Pros: regular users the firewall saw before the attack can
    still use the service. Cons: no new users are allowed until the
    removal of the rule, obviously. Better than nothing, but I welcome any
    other hints :)

    One other solution may be to add boxes. I tried a carpnodes cluster,
    but at 600 Kpps I got a "split brain" with both nodes claiming MASTER
    for each carpnode. Maybe if I configure ALTQ it could help this ? As I
    have more boxes, I could deal with the performance impact of ALTQ.

    I am willing to test any patch/suggestion you may have, of course.
    Even just hints about kernel code, as I am currently messing with PF
    code myself. I did compile a profiled kernel, I must now check the
    results but that will be another story.

    To finish, here is the typical load on the box (errors are from
    various DDoS, not related to normal use) :

    Status: Enabled for 77 days 02:17:58            Debug: err

    Interface Stats for trunk1            IPv4            IPv6
      Bytes In                  8885330383273                0
      Bytes Out                72449316050298            20224
      Packets In
        Passed                    48738702875                0
        Blocked                    10152865611                0
      Packets Out
        Passed                    67293792876              281
        Blocked                    4557637133                0

    State Table                          Total            Rate
      current entries                    37135
      searches                    130771929548        19632.2/s
      inserts                      4718030394          708.3/s
      removals                      4717993259          708.3/s
    Source Tracking Table
      current entries                    7455
      searches                      4951426366          743.3/s
      inserts                        623672861          93.6/s
      removals                      623665406          93.6/s
    Counters
      match                        5600111978          840.7/s
      bad-offset                            0            0.0/s
      fragment                        3591379            0.5/s
      short                            2500133            0.4/s
      normalize                          10968            0.0/s
      memory                            71750            0.0/s
      bad-timestamp                          0            0.0/s
      congestion                      3863476            0.6/s
      ip-option                              0            0.0/s
      proto-cksum                            0            0.0/s
      state-mismatch                  3722058            0.6/s
      state-insert                          0            0.0/s
      state-limit                            0            0.0/s
      src-limit                      234360390          35.2/s
      synproxy                    13817759263        2074.4/s
    Limit Counters
      max states per rule                    0            0.0/s
      max-src-states                    89727            0.0/s
      max-src-nodes                          0            0.0/s
      max-src-conn                          0            0.0/s
      max-src-conn-rate                      0            0.0/s
      overload table insertion              0            0.0/s
      overload flush states                  0            0.0/s</whitelisted_users>

    From: Roger S.
    Subject: Re: How to deal with DDoS ?
    Newsgroups: gmane.os.openbsd.misc
    Date: 2012-02-22 08:39:51 GMT (1 week, 4 days, 11 hours and 31 minutes ago)
    On Tue, Feb 21, 2012 at 9:51 PM, Joachim Schipper
    <joachim <at="">joachimschipper.nl> wrote:

    Just the most obvious idea, since you mention that this sort-of-works if
    you put "block drop in quick from !<whitelisted_users>": does it handle
    this load if you turn off pf, or only include one or two trivial rules?

    Did not try to turn off pf (I need it anyway), and my pf.conf is very
    simple and already optimized following the good book of pf and some
    undeadly posts.

    It certainly suggests that you may be well-served by optimizing your
    pf.conf… (also, you've probably found the "synproxy" directive? If
    not, try that too.)

    I already use synproxy, the problem is that I get so much SYN that
    pf/state table collapses.

    Also, state tracking is apparently faster than stateless pf for normal
    firewalls. I'd double-check if this is still true in your case, though;
    if nothing else, stateless pf makes a CARP'ed setup easier.

    I am not sure to understand here. I want to use synproxy to protect my
    backend servers, so I need state stracking.

    I'm pretty sure you can muck with the rules without dropping existing
    connections. (pf essentially does "does this packet match a known state?
    If not, look at pf.conf".) This is almost certainly easier than your
    proposed daemon.

    Sure thing, the daemon is only a workaround to provide degraded but
    working service when under attack.

    A final, rather hackish, idea that probably does need a bit of
    programming: greylisting for SYNs. Legitimate users will send you a
    second SYN, so you could do something like (this has not even been
    syntax-checked!)
      block drop log in quick from ! <syn_seen>no state flags S/SA

    I like the idea. This may need some programming indeed, but it seems
    even better than my idea. Thanks, I'll take a look at this.

    and then add every logged IP to syn_seen. Obviously, this will slow down
    access to the service for legitimate users, which may or may not be
    acceptable.

    We are speaking of a slower but working service, or no service at all.
    I prefer the first alternative :)</syn_seen></whitelisted_users></joachim>



  • A flood of any type of traffic you're passing is bad news for every firewall. The lowest performance limit on anything any firewall does is new connections per second, and you'll hit that pretty quickly under a decent sized flood, or if the box is fast enough to handle that, you'll hit the state table limit quickly regardless of how high it is. Traffic you're blocking has little impact though. That's generically true of every firewall, they're almost always the most susceptible thing on any network to DDoS attacks (if you have adequate bandwidth to where you aren't knocked offline entirely and at the mercy of your provider).


Log in to reply