Bug or hacking by udp flooding ?
-
Thanks Harvy, but no, the symptoms are quite clear. Every now and then, we get this same dhcp (sorry, I must have said dns earlier but its dhcp) request which repeat itself at high rate (at least 30000 pps).
A network loop wouldn't give that result. By hacking, I meant : "Is it possible to hack into pfsense and make it flood the network like that ?"As I said in my post, I don't find any incoming traffic into the fw, only out of it.
Thanks
-
Could you draw up a schematic of the network layout? (Including addresses)
Dhcp requests originate from clients? Or are these on your wan side?
-
A single mac sure from a phone that is connected over wifi is sure not going to give you 30000pps either ;)
You prob have some sort of loop with a broadcast storm or dhcp relay you have setup since that looks to be directed.
A drawing and actual sniff of the packets would be helpful.
-
Thank you for your answers. I'll provide a network schema in a next post.
Basically, our network is made of several VLANs in the range(s) 10.16.x.y/24 with a different x for each VLAN. We also have a third-party (vending machines) who's running a few machines in the 192.168.1.x range (not optimal, I know). The default gateway in each VLAN is 10.16.x.1, which points to the pfsense machines.The flooding DHCP request always comes from an adress in the 192.168.182.x range (with x varying randomly between 10 and 20). That range doesn't belong in our network. The destination is always 192.168.182.1 BUT is a broadcast (FF:FF:FF:FF…..).
I emphazize on the fact that it's ALWAYS strictly the same frame that floods the network. It's currently occuring randomly about 1 to 3 times per week. The only thing that changes is the source address 192.168.182.x.
Here's a complete captured packet :
Frame 7: 350 bytes on wire (2800 bits), 350 bytes captured (2800 bits) on interface 0
Interface id: 0 (enp0s25)
Encapsulation type: Ethernet (1)
Arrival Time: Mar 3, 2017 09:02:59.334596194 CET -
"The flooding DHCP request always comes from an adress in the 192.168.182.x range (with x varying randomly between 10 and 20). "
You have some sort of broadcast storm happening.
So client X is on network ABC, got IP address 192.168.182.X – then connects to different network.. That client is most likely going to be asking for that same IP again..
So you got some device moving from network X to your network Y and requesting the IP it had on the other network.. Is there some network around that has the same SSID.. Normally you see this happen when clients switch AP..
As to why your seeing so many packets.. Most likely something odd in your network causing a storm..
-
Johnpoz, thanks for your answer, it makes sense.
Although…
- Is a BOOTP request something normal for an android device connecting to a wifi network ? I don't think so.
- as I've told before, I've checked the port counters on the switches, and it appears that all the traffic seems to originate from the firewall. When I tried to follow the traffic's flow, I saw :
- on the FW switch : incoming frames from the fw and outgoing frames to the next switch
- on the GW switch : incoming frames from the FW switch and outgoing frames to the gateway ports.
- BUT, and that's the important point, no incoming frames from the switch to the internal FW interface !
So, where does this traffic comes from, if not from the fw itself ? And I insist again on the fact that either deleting the state in the FW or restarting it just kills the problem.
Sorry for being nerdy…
Thanks,
Thierry
-
Since you say you are using VLANs.
Do you by chance have a place where you have tagged and untagged traffic on the same wire?Are you using static routes from your pfSense pointing to the gateway going to the 192.168 net?
-
Hi GreenFrog,
Yes, this is possible.
As for the routes, the firewall is doing the routing between the different VLANs, and has several intefaces configured, about one per VLAN plus a few extras. There is an interface for the 192.168.1 network, and a VLAN as well.Thank you
Thierry
-
I'd enable broadcast storm protection on your access ports & ap links.
Also, block that mac-address on your WiFi controller.
There must be a configuration issue somewhere because 30kps won't be from any WiFi devices.
-
Broadcast control, I've done it already. I just didn't block the MAC address yet because I wanted to find a way to identify the owner…. but don't see how.
Thks,
Thierry
-
When they can't get on your network - pretty sure they will contact you ;)
-
I asked about the mixing of tagged and untagged traffic because i once had a misconfiguration where a device which wasn't vlan aware started leaking frames between the tagged vlans and the untagged subnet.
This lead to frames circulating between the tagged and untagged parts.
–> Essentially a logical loop even if no physical loop was present.If you can capture with a tap (or a hub..) the wire which could carry tagged/untagged mixed you should see it because you see the same frame over and over tagged and untagged.
How i solved my problem:
Got rid of the untagged traffic on this segment.
Some disagree, but i still am of the opinion that the rule-of-thumb of "don't mix tagged and untagged traffic on the same cable" applies. -
You might be hitting something like this: https://redmine.pfsense.org/issues/2073
Basically, you might have a policy routing rule (rule with a gateway set) that matches too much traffic. If the firewall tries to policy route broadcast traffic, bad things like this can happen. Now it's usually smart enough to not do that for addresses in a subnet it knows directly, but if you have a rule to policy route traffic with a source of "any" and it matches some other subnet's broadcast packet, it doesn't know the difference.
Fix the rule, clear the states or reboot, and you might be OK again. Assuming that's the problem, but the symptoms fit.
-
"but i still am of the opinion that the rule-of-thumb of "don't mix tagged and untagged traffic on the same cable" applies."
I would be one that disagrees ;) And with some devices its not even possible.. Case in point would be management of say an AP that does not allow you to tag its management vlan. But there is only 1 wire to connect the AP. So the wifi/ssids are tagged vlans while traffic to manage the AP is not tagged. Ie a native or PVID of the interface on the switch.
This should never be a problem if your switching environment is correctly configured to keep this untagged management vlan isolated to only the ports that it needs to be on.
-
Thanks for your help. I will investigate further on those grounds.
I'll keep you filed.Thierry
