How do I handle this? DDOS?



  • Hi there,
    I've been struggling with this for the last 48 hours non-stop. I have a PFsense firewall bridged and behind the firewall I have several servers running severals sorts of OS (mostly freebsd and a few windows). I have a 100mbit uplink which is pounded for the last 48 hours. Most of the servers are unreachable or extremely slow since.
    I have also bandwidthd installed and it gives an increase of tcp traffic going out. Normally it's around 4Mbit, but now it's 180Mbit. That's also what the stats at my provider say. But when I look at the individual servers, I have about 12 which have an extra of 200Kbit traffic. All together nowhere near the 180Mbit total.
    WHen I look on such a server I don't see active connections or strange programs.
    I don't know where I could look any further and any help would be apreciated.

    Thanks,

    Roger


  • Banned

    You cannot handle DDoS on your pfSense. Get in touch with your ISP. And kindly make sure you are NOT serving DNS, NTP or SNMP on your WAN.



  • thanks I know I can't handle ddos, but I should be able to see something I think. I do run DNS but no NTP or SNMP. Only a few servers are running DNS, but all are acting strange.

    Thanks,
    Roger


  • LAYER 8 Global Moderator

    So pfsense is connected to your uplink on its wan.. Then sniff and take a look see at the traffic..  Sniff on your lan connection and see what boxes are doing it - if you say you run dns open to the public my first guess would be your being used in an amplification attack..  That is really not a ddos against that you would need isp help with.

    You just need to fix the services that are being used to attack someone else, etc..  This is why sniffing the traffic will show you exactly what is going on.


  • Banned

    If you run manual outbound nat, then close the NAT one by one to see which server origins the traffic if they are not on seperate VLANS which I would recommend.

    Then reset the states and see what servers come up with traffic.

    Then shut the DNS service down and patch it if you can.



  • Pfsense is bridged, so no NAT. I will see if I can find something by sniffing. Would Darkstat be OK on a bridge?
    I have blocked DNS now, but still see the high traffic, which are, BTW, peaks every 30 sec.,

    Thanks,

    Roger



  • OK,
    I am not an expert ;-) but can this be the problem:
    IP Hostname MAC Address In Out Total Last seen
    121.40.54.90 00:00:5e:00:01:65 366,534,792 0 366,534,792 (never)
    121.40.50.249 (none) 00:00:5e:00:01:65 355,893,408 0 355,893,408 (never)
    121.41.53.152 (none) 00:00:5e:00:01:65 355,620,096 0 355,620,096 (never)
    121.41.54.220 (none) 00:00:5e:00:01:65 351,194,688 0 351,194,688 (never)

    I have setup pfblocker to block China, but these seem to get past it.


  • Banned

    As I noted, this is a completely futile effort. Blocking the packets on your firewall does not stop the traffic from killing your connectivity.


  • Banned

    No but the handling of the packages is where a true Enpterprise system differs from this SOHO shit :D


  • LAYER 8 Global Moderator

    what are you viewing this in?

    IP  Hostname  MAC Address  In  Out  Total  Last seen
    121.40.54.90      00:00:5e:00:01:65  366,534,792  0  366,534,792  (never)
    121.40.50.249  (none)  00:00:5e:00:01:65  355,893,408  0  355,893,408  (never)

    You say you blocked dns – how exactly did you do that?  if your connection is full of traffic.. Looking at a sniff of a few seconds should tell us what the problem is...  You say it peaks ever 30 seconds or so..  Well do a sniff when its peaking - and lets see what is all the fuss about.



  • Thanks guys,
    I started blocking outbound traffic to the IP ranges which showed in darkstat and the traffic went away. It still showed on the LAN side for a while and disappeared there also. I finally could get some sleep  8).
    I just have to find out what it was and how thay did it. Darkstat showed every connection was to a different chinese address. very strange.
    Thanks a lot guys!


  • Banned

    You really should do something about the public-facing DNS servers. Otherwise you'll end up cut off sooner or later by your ISP.



  • Well,
    They are primary and secundary servers for domains. However, only for local domains, so it's not an open DNS.

    Greetings,
    Roger


  • Banned

    Better double-check with these:

    http://openresolver.com/
    http://openresolverproject.org/

    Also, even for authoritative servers, some sort of rate limiting should be set up on the DNS servers.


  • LAYER 8 Global Moderator

    I never understand why people want to host their own dns..  I don't see it as productive - when you can let companies that do it fore their bread and butter host it on networks designed just to do that - and let them worry about all the exploits to dns, etc..

    Your never going to be able to host a dns network like they do – and the cost is pennies!!!  Something like dnsmadeeasy for example.. You can get enterprise hosting for pennies

    http://www.dnsmadeeasy.com/home/pricing-customization/

    Small companies, amounts of domains can be done for less than it would cost to run the hardware for elec,etc..  The only dns you should have to worry about is internal facing - if its public, let the people that do that for a living do it ;)



  • Well - I'm sure there are times when, for security reasons, running your own private DNS server is a good thing.

    But other than that, I agree.


  • Banned

    @johnpoz:

    Your never going to be able to host a dns network like they do – and the cost is pennies!!!  Something like dnsmadeeasy for example.. You can get enterprise hosting for pennies
    http://www.dnsmadeeasy.com/home/pricing-customization/

    The HE DNS is completely free for 50 domains/zones.


  • LAYER 8 Global Moderator

    "for security reasons, running your own private DNS server is a good thing."

    Can you give an example??  These are names that you want the WORLD to resolve..What security could you be worried about.. What you want is HA, Speed..  Do you have dns around the globe?  Do you have anycast setup?  Who do you think pays more attention to security concerns with dns than hosts that provide dns for a shitload of customers??

    Other than local dns, I can not see a point to host your own..  Its sure and the hell not cost effective!!  And your never going to be able to do it as good as the hosts can..

    I love dns, would love more than nothing to host it to the public - it just doesn't make sense to do so!!


  • Banned

    If everyone said that, then no host would be found… ;)



  • I think there are just times when its good for certain business, organizations etc to control how their DNS gets resolved.

    Lots of DNS servers out there, so apparently I'm not alone.


  • LAYER 8 Global Moderator

    Because they don't think it through would be my take on why so many.. Hosting with dns company X does not take away how anything is resolved.. It gives you better infrastructure to host your data is all.  You know what the cost would be to have servers around the globe with anycast, etc..

    Why should I set that up, pay for it, manage it.. When it comes down to it all I am worried about is that www.something.tld points to the IP I want it too.. The hardware, the location, the software that is done on is really nothing to do with what you need for your public dns.

    You want www.something.tld to resolve to an IP, you want it to do it quickly from anywhere on the planet - you want it to do it always do it 99.999%  You don't want some script kiddy hit your dns with a shit lot of queries and take down normal users being able to resolve it.  You want to be able to change www.something.tld to resolve to different IP when you want, etc..

    All of these things is what dns hosting companies do for a living..  That you think your company that does Y for a living should host dns to the public X is just crazy..  Do the math, it just make no sense to host your own public dns.. You can not come close to feature set, reliability, etc.  And your  going to spend more doing it..

    Just my take on it - Do you really think the big companies host their own?  Sorry they do not, its not cost effective!!  And the same can be said if you have 1 domain that gets 10 queries a day or 10 million..

    To the OP.. What do you think your getting by hosting public dns on your own stuff??  Other than problem like this! ;)



  • You could pretty much use that logic for any service.  No need to run any yourself because others are doing it better already.



  • @kejianshi:

    You could pretty much use that logic for any service.  No need to run any yourself because others are doing it better already.

    I think his main point is that DNS has a high risk for low reward, and there's a lot of competition, so the prices are hard to beat compared to finding the talent and training your staff. Not to mention the large infrastructure investment required to do it "correctly".

    DNS is also critical infrastructure. Your web app not working is fine, because you can return an HTTP error, but DNS not working is like your Internet link going down.



  • I manage our DNS.  We have two Linux VMs that I use with BIND for primary and secondary resolution.  I like the flexibility and control that having it local gives me.  I don't need global octo mega multicasting.  If our DNS is down then it's pretty likely that we're totally down.



  • @KOM:

    I manage our DNS.  We have two Linux VMs that I use with BIND for primary and secondary resolution.  I like the flexibility and control that having it local gives me.  I don't need global octo mega multicasting.  If our DNS is down then it's pretty likely that we're totally down.

    Of course they have you over there, so they have the talent  :-)  But how many companies really have people who know how things really work? I've met quite a few people who are in charge of very important things, but only know what they've been told with no understanding of how things work.

    I could see it being a good rule of thumb to recommend people purchase DNS services than self host. I have little experience in this, so I'm only appreciating the arguments, but I have no experience, but I read enough security articles about how whatever common service is misconfigured on 80% of servers.


  • LAYER 8 Global Moderator

    So in the same geographic location, prob even same network I take it? Yeah not something I would call robust HA sort of setup ;)

    Small dns setups can be hosted for FREE or like $29 a year.  While it sounds like your just leveraging spare cycles on your VM infrastructure and most likely free linux distros so cost is very low.

    From your very knowledgeable posts on all sorts of topics, I take it your very familiar with the subject and you have chosen this route because of your familiarity with the service and how to host it, etc…  Also from your past posts I am fairly sure you would know how to handle an attack against the service, and or how to harden the setup against such attacks in the first place..

    Not all companies IT personal have such skillsets.  You see it all the time where full recursion is allowed on public facing dns that is suppose to be authoritative for 1 or 2 domains.



  • Part of the issue (for me, at least) is that I manage 230+ domains for just the one company (don't ask or you will make me scream again.  Hint: executive buzzword bingo as applied to domains…).  I've used some domain registrar's DNS tools and they were painful and slow.  I couldn't imagine using them to actually deal with hundreds of domains.  Give me WinSCP and root credentials and I will happily manage DNS at the filesystem level, thank you very much.


  • Banned

    @KOM:

    Part of the issue (for me, at least) is that I manage 230+ domains for just the one company (don't ask or you will make me scream again.  Hint: executive buzzword bingo as applied to domains…).  I've used some domain registrar's DNS tools and they were painful and slow.  I couldn't imagine using them to actually deal with hundreds of domains.  Give me WinSCP and root credentials and I will happily manage DNS at the filesystem level, thank you very much.

    Uh. Normally you get some kind of API access for similar amount of domains from quite a couple of registrars here… (No clicking over web forms.)


Log in to reply