Bug or hacking by udp flooding ?

[Thilroy]

Hello everybody,

I'm in charge of the network of a Brussels'school, in begium thus. We've got two redundant psense 2.3.2 running and lately, they've been subject to what seems to be an uncanny attack.
Symptom was that our internet connections became really slow and often went offline with packet drops over the 80%.

First problem was that I could not see any logging of this traffic in pfsense, only that the Gateways were flooded to the neck.
I ran a trace on the switches and here's what I found :

7 0.000006783 192.168.182.10 192.168.182.1 DHCP 350 DHCP Request  - Transaction ID 0x4dfc8324
this transaction rate was about 30000 packets/sec !! These adresses are NOT in our network.

A little drill-down in the frame revealed this :

Client MAC address: SamsungE_e2:04:d4 (08:21:ef:e2:04:d4)
and this as well :

Host Name: android-f0afc3a0b252fefc

This is the MAC address of one of our student(or teacher)s'phone. I found it on the DHCP server AND on the wifi controller (we run an unifi solution)

Clearly, someone was playing hacker here. But even stranger was that, when I tried to follow the traffic flow on our cisco switches, it appeared that this traffic came out of the fw then into the switch then back on the switch ports where our gateways are connected, BUT NO incoming traffic on the fw itself. Which leads me to the conclusion that the traffic was generated on the fw itself.

The demonstration of the fact came later when I restarted the fw : that flooding disappeared completely.

I'm not expert in hacking techniques, but is it possible that a malicious code is uploaded from that phone on our fw, causing that flooding to happen ? I guess that a fw bug is totally out of question ?

The incident has repeated itself quite a few times. Now I learned by experience that if I deleted the state associated with this 192.168.182.x address in the states table of the fw, the flooding stopped completely. I didn't take definitive measures to stop it (apart from activating broadcast storm control on the switch)  because I'm still hoping to catch the culprit, but how ?

So my two questions are :

1. Are we sure this is hacking ?
2. If yes, is there a way to find who's doing it, given that he connects through our school wifi ?

Many thanks in advance for your expert advice,

Thierry Noël

[Harvy66]

You can't actually hack by flooding. DOS attacks are a great way to make it difficult to tell that you're getting hacked because of the noise, but it an of itself is not a hack.

Is it possible you have a network loop?

[Thilroy]

Thanks Harvy, but no, the symptoms are quite clear. Every now and then, we get this same dhcp (sorry, I must have said dns earlier but its dhcp) request which repeat itself at high rate (at least 30000 pps).
A network loop wouldn't give that result. By hacking, I meant : "Is it possible to hack into pfsense and make it flood the network like that ?"

As I said in my post, I don't find any incoming traffic into the fw, only out of it.

Thanks

[heper]

Could you draw up a schematic of the network layout? (Including addresses)

Dhcp requests originate from clients? Or are these on your wan side?

[johnpoz]

A single mac sure from a phone that is connected over wifi is sure not going to give you 30000pps either ;)

You prob have some sort of loop with a broadcast storm or dhcp relay you have setup since that looks to be directed.

A drawing and actual sniff of the packets would be helpful.

[Thilroy]

Thank you for your answers. I'll provide a network schema in a next post.
Basically, our network is made of several VLANs in the range(s) 10.16.x.y/24 with a different x for each VLAN. We also have a third-party (vending machines) who's running a few machines in the 192.168.1.x range (not optimal, I know). The default gateway in each VLAN is 10.16.x.1, which points to the pfsense machines.

The flooding DHCP request always comes from an adress in the 192.168.182.x range (with x varying randomly between 10 and 20). That range doesn't belong in our network. The destination is always 192.168.182.1 BUT is a broadcast (FF:FF:FF:FF…..).

I emphazize on the fact that it's ALWAYS strictly the same frame that floods the network. It's currently occuring randomly about 1 to 3 times per week. The only thing that changes is the source address 192.168.182.x.

Here's a complete captured packet :

Frame 7: 350 bytes on wire (2800 bits), 350 bytes captured (2800 bits) on interface 0
    Interface id: 0 (enp0s25)
    Encapsulation type: Ethernet (1)
    Arrival Time: Mar  3, 2017 09:02:59.334596194 CET

[johnpoz]

"The flooding DHCP request always comes from an adress in the 192.168.182.x range (with x varying randomly between 10 and 20). "

You have some sort of broadcast storm happening.

So client X is on network ABC, got IP address 192.168.182.X – then connects to different network.. That client is most likely going to be asking for that same IP again..

So you got some device moving from network X to your network Y and requesting the IP it had on the other network..  Is there some network around that has the same SSID.. Normally you see this happen when clients switch AP..

As to why your seeing so many packets.. Most likely something odd in your network causing a storm..

[Thilroy]

Johnpoz, thanks for your answer, it makes sense.

Although…

• Is a BOOTP request something normal for an android device connecting to a wifi network ? I don't think so.
• as I've told before, I've checked the port counters on the switches, and it appears that all the traffic seems to originate from the firewall. When I tried to follow the traffic's flow, I saw :
      - on the FW switch : incoming frames from the fw and outgoing frames to the next switch
      - on the GW switch : incoming frames from the FW switch and outgoing frames to the gateway ports.

- BUT, and that's the important point, no incoming frames from the switch to the internal FW interface !

So, where does this traffic comes from, if not from the fw itself ? And I insist again on the fact that either deleting the state in the FW or restarting it just kills the problem.

Sorry for being nerdy…

Thanks,

Thierry

[GruensFroeschli]

Since you say you are using VLANs.
Do you by chance have a place where you have tagged and untagged traffic on the same wire?

Are you using static routes from your pfSense pointing to the gateway going to the 192.168 net?

[Thilroy]

Hi GreenFrog,

Yes, this is possible.
As for the routes, the firewall is doing the routing between the different VLANs, and has several intefaces configured, about one per VLAN plus a few extras. There is an interface for the 192.168.1 network, and a VLAN as well.

Thank you

Thierry

[heper]

I'd enable broadcast storm protection on your access ports & ap links.

Also, block that mac-address on your WiFi controller.

There must be a configuration issue somewhere because 30kps won't be from any WiFi devices.

[Thilroy]

Broadcast control, I've done it already. I just didn't block the MAC address yet because I wanted to find a way to identify the owner…. but don't see how.

Thks,

Thierry

[johnpoz]

When they can't get on your network - pretty sure they will contact you ;)

[GruensFroeschli]

I asked about the mixing of tagged and untagged traffic because i once had a misconfiguration where a device which wasn't vlan aware started leaking frames between the tagged vlans and the untagged subnet.
This lead to frames circulating between the tagged and untagged parts.
–> Essentially a logical loop even if no physical loop was present.

If you can capture with a tap (or a hub..) the wire which could carry tagged/untagged mixed you should see it because you see the same frame over and over tagged and untagged.

How i solved my problem:
Got rid of the untagged traffic on this segment.
Some disagree, but i still am of the opinion that the rule-of-thumb of "don't mix tagged and untagged traffic on the same cable" applies.

[jimp]

You might be hitting something like this: https://redmine.pfsense.org/issues/2073

Basically, you might have a policy routing rule (rule with a gateway set) that matches too much traffic. If the firewall tries to policy route broadcast traffic, bad things like this can happen. Now it's usually smart enough to not do that for addresses in a subnet it knows directly, but if you have a rule to policy route traffic with a source of "any" and it matches some other subnet's broadcast packet, it doesn't know the difference.

Fix the rule, clear the states or reboot, and you might be OK again. Assuming that's the problem, but the symptoms fit.

[johnpoz]

"but i still am of the opinion that the rule-of-thumb of "don't mix tagged and untagged traffic on the same cable" applies."

I would be one that disagrees ;)  And with some devices its not even possible.. Case in point would be management of say an AP that does not allow you to tag its management vlan.  But there is only 1 wire to connect the AP.  So the wifi/ssids are tagged vlans while traffic to manage the AP is not tagged.  Ie a native or PVID of the interface on the switch.

This should never be a problem if your switching environment is correctly configured to keep this untagged management vlan isolated to only the ports that it needs to be on.

[Thilroy]

Thanks for your help. I will investigate further on those grounds.
I'll keep you filed.

Thierry