PfSense is causing a massive massive DHCP Flood on WAN
-
We're apparently flooding our ISP with DHCP requests… up to 32,000 per second. I do not have DHCP enabled anywhere on pfSense. Our configurations with our ISPs are static address and we have a separate DHCP server on our LAN.
Our setup is an I350T4 (quad port nic) running on an ASRock Intel Avoton C2750 C2750D4I 8 core.
I did a pcap on the interface... It's our MAC address is the source. Disabling the gateway and re-enabling it, or rebooting pfSense seems to fix the problem for awhile.
This only happens with this ISP. We have another ISP hooked up to the same card and have never experienced this issue.
What in the hell is going on? Our ISP told us don't enable your port until you figure this out. Please for the love of all that is good... help us!
![Screen Shot 2015-03-07 at 11.18.50 AM.png](/public/imported_attachments/1/Screen Shot 2015-03-07 at 11.18.50 AM.png)
![Screen Shot 2015-03-07 at 11.18.50 AM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-03-07 at 11.18.50 AM.png_thumb) -
Can you set your WAN port to use a static IP until someone here comes up with a real solution?
-
Did you create a loop? Broadcast packets and loops will mess your network up.
-
That does sound like a layer 2 loop or similar. Any bridging involved? Is DHCP or DHCP6 enabled on any interface? Can you share that pcap? It should be really telling. Can email to me (cmb at pfsense dot org) with a link to this thread if you don't want to put it out publicly.
-
Thanks everyone for the answers.
Can you set your WAN port to use a static IP until someone here comes up with a real solution?
We only use static. The firewall doesn't even handle DHCP for the LAN.
That does sound like a layer 2 loop or similar.
I'm not sure how the current topology could form a loop. Attached an image of our setup. Maybe if somehow the two firewalls looped through each other?
Any bridging involved?
No, we're a pretty simple setup. We have two ISPs, we load-balance outgoing connections, use CARP for the gateway address and CARP for incoming hosting.
Can you share that pcap?
Yes, emailed. If anyone else wants a look, just let me know and I'll send it to you.
![Screen Shot 2015-03-07 at 2.50.42 PM.png](/public/imported_attachments/1/Screen Shot 2015-03-07 at 2.50.42 PM.png)
![Screen Shot 2015-03-07 at 2.50.42 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2015-03-07 at 2.50.42 PM.png_thumb) -
Is DHCP or DHCP6 enabled on any interface?
No, nowhere on any interface. IPV6 is disabled on the router.
Anyone know where the dhcp client executable is at on the filesystem? I'm thinking moving it to another location so it can't be executed.
-
@j@svg:
Anyone know where the dhcp client executable is at on the filesystem? I'm thinking moving it to another location so it can't be executed.
Don't bother, the pcap shows it's definitively not the firewall itself initiating the requests.
The hostname in all those DHCP requests is "kali" which doesn't match your firewall's hostname (from the details in your email). There are two diff MACs making the requests, one is a HP MAC and one is Gemtek, where your firewall has Intel NICs. That's the MACs within the requests, meaning something other than the firewall is initiating them. The source MACs on the requests are those of the WAN NIC of the primary and backup firewalls both, so they are forwarding them, but they're not the source.
It appears there are only two requests that are getting looped endlessly somewhere (judging by the transaction ID). It didn't appear you had a bridge configured on either from the details in your email, which would have been my first guess on how that could happen. Maybe DHCP relay enabled? If it's enabled on both to forward requests to WAN, and somehow that forwarding ends up with the DHCP request being relayed back to LAN (probably if both LAN and WAN are on the same broadcast domain, which would be bad in general), it would create a never-ending loop of DHCP requests as fast as it could relay and re-relay them.
-
DHCP Relay definitely not enabled on the primary or slave (It's unchecked). What is the name of the executable for the DHCP relay daemon? I'll run a ps -aux to be doubly sure.
I'm absolutely baffled how these are getting forwarded though… maybe a bug in pfSense? I'll add a floating firewall rule to drop UDP traffic source or destination port 67 and 68 on all interface with the "take action immediately" flag set for both in/out.
Great call on the hostname, kali is the hostname and the requested IP is not one of ours either. I didn't peer into the actual content of the DHCP request... I let our ISP know.
AND THANK YOU! We've used pfSense professional support before, but now a new pfSense Gold subscription will be coming Monday!
-
Anyone know the name of the DHCP relay daemon?
-
@j@svg:
Anyone know the name of the DHCP relay daemon?
dhcrelay. Worth checking whether that's running, though if it's not configured under Services>DHCP Relay it won't be. Even if it is, it can't loop things endlessly in a properly setup network. Not a bad next step in trying to figure out how the requests are being forwarded at all.