• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Bug or hacking by udp flooding ?

Scheduled Pinned Locked Moved General pfSense Questions
17 Posts 6 Posters 2.0k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • T
    Thilroy
    last edited by Mar 27, 2017, 11:37 AM

    Hello everybody,

    I'm in charge of the network of a Brussels'school, in begium thus. We've got two redundant psense 2.3.2 running and lately, they've been subject to what seems to be an uncanny attack.
    Symptom was that our internet connections became really slow and often went offline with packet drops over the 80%.

    First problem was that I could not see any logging of this traffic in pfsense, only that the Gateways were flooded to the neck.
    I ran a trace on the switches and here's what I found :

    7 0.000006783 192.168.182.10 192.168.182.1 DHCP 350 DHCP Request  - Transaction ID 0x4dfc8324
    this transaction rate was about 30000 packets/sec !! These adresses are NOT in our network.

    A little drill-down in the frame revealed this :

    Client MAC address: SamsungE_e2:04:d4 (08:21:ef:e2:04:d4)
    and this as well :

    Host Name: android-f0afc3a0b252fefc

    This is the MAC address of one of our student(or teacher)s'phone. I found it on the DHCP server AND on the wifi controller (we run an unifi solution)

    Clearly, someone was playing hacker here. But even stranger was that, when I tried to follow the traffic flow on our cisco switches, it appeared that this traffic came out of the fw then into the switch then back on the switch ports where our gateways are connected, BUT NO incoming traffic on the fw itself. Which leads me to the conclusion that the traffic was generated on the fw itself.

    The demonstration of the fact came later when I restarted the fw : that flooding disappeared completely.

    I'm not expert in hacking techniques, but is it possible that a malicious code is uploaded from that phone on our fw, causing that flooding to happen ? I guess that a fw bug is totally out of question ?

    The incident has repeated itself quite a few times. Now I learned by experience that if I deleted the state associated with this 192.168.182.x address in the states table of the fw, the flooding stopped completely. I didn't take definitive measures to stop it (apart from activating broadcast storm control on the switch)  because I'm still hoping to catch the culprit, but how ?

    So my two questions are :

    1. Are we sure this is hacking ?
    2. If yes, is there a way to find who's doing it, given that he connects through our school wifi ?

    Many thanks in advance for your expert advice,

    Thierry Noël

    1 Reply Last reply Reply Quote 0
    • H
      Harvy66
      last edited by Mar 27, 2017, 3:05 PM

      You can't actually hack by flooding. DOS attacks are a great way to make it difficult to tell that you're getting hacked because of the noise, but it an of itself is not a hack.

      Is it possible you have a network loop?

      1 Reply Last reply Reply Quote 0
      • T
        Thilroy
        last edited by Mar 28, 2017, 6:19 AM

        Thanks Harvy, but no, the symptoms are quite clear. Every now and then, we get this same dhcp (sorry, I must have said dns earlier but its dhcp) request which repeat itself at high rate (at least 30000 pps).
        A network loop wouldn't give that result. By hacking, I meant : "Is it possible to hack into pfsense and make it flood the network like that ?"

        As I said in my post, I don't find any incoming traffic into the fw, only out of it.

        Thanks

        1 Reply Last reply Reply Quote 0
        • H
          heper
          last edited by Mar 28, 2017, 6:27 AM

          Could you draw up a schematic of the network layout? (Including addresses)

          Dhcp requests originate from clients? Or are these on your wan side?

          1 Reply Last reply Reply Quote 0
          • J
            johnpoz LAYER 8 Global Moderator
            last edited by Mar 28, 2017, 9:29 AM

            A single mac sure from a phone that is connected over wifi is sure not going to give you 30000pps either ;)

            You prob have some sort of loop with a broadcast storm or dhcp relay you have setup since that looks to be directed.

            A drawing and actual sniff of the packets would be helpful.

            An intelligent man is sometimes forced to be drunk to spend time with his fools
            If you get confused: Listen to the Music Play
            Please don't Chat/PM me for help, unless mod related
            SG-4860 24.11 | Lab VMs 2.8, 24.11

            1 Reply Last reply Reply Quote 0
            • T
              Thilroy
              last edited by Mar 28, 2017, 10:05 AM

              Thank you for your answers. I'll provide a network schema in a next post.
              Basically, our network is made of several VLANs in the range(s) 10.16.x.y/24 with a different x for each VLAN. We also have a third-party (vending machines) who's running a few machines in the 192.168.1.x range (not optimal, I know). The default gateway in each VLAN is 10.16.x.1, which points to the pfsense machines.

              The flooding DHCP request always comes from an adress in the 192.168.182.x range (with x varying randomly between 10 and 20). That range doesn't belong in our network. The destination is always 192.168.182.1 BUT is a broadcast (FF:FF:FF:FF…..).

              I emphazize on the fact that it's ALWAYS strictly the same frame that floods the network. It's currently occuring randomly about 1 to 3 times per week. The only thing that changes is the source address 192.168.182.x.

              Here's a complete captured packet :

              Frame 7: 350 bytes on wire (2800 bits), 350 bytes captured (2800 bits) on interface 0
                  Interface id: 0 (enp0s25)
                  Encapsulation type: Ethernet (1)
                  Arrival Time: Mar  3, 2017 09:02:59.334596194 CET

              1 Reply Last reply Reply Quote 0
              • J
                johnpoz LAYER 8 Global Moderator
                last edited by Mar 28, 2017, 10:12 AM

                "The flooding DHCP request always comes from an adress in the 192.168.182.x range (with x varying randomly between 10 and 20). "

                You have some sort of broadcast storm happening.

                So client X is on network ABC, got IP address 192.168.182.X – then connects to different network.. That client is most likely going to be asking for that same IP again..

                So you got some device moving from network X to your network Y and requesting the IP it had on the other network..  Is there some network around that has the same SSID.. Normally you see this happen when clients switch AP..

                As to why your seeing so many packets.. Most likely something odd in your network causing a storm..

                An intelligent man is sometimes forced to be drunk to spend time with his fools
                If you get confused: Listen to the Music Play
                Please don't Chat/PM me for help, unless mod related
                SG-4860 24.11 | Lab VMs 2.8, 24.11

                1 Reply Last reply Reply Quote 0
                • T
                  Thilroy
                  last edited by Mar 28, 2017, 10:43 AM

                  Johnpoz, thanks for your answer, it makes sense.

                  Although…

                  • Is a BOOTP request something normal for an android device connecting to a wifi network ? I don't think so.
                  • as I've told before, I've checked the port counters on the switches, and it appears that all the traffic seems to originate from the firewall. When I tried to follow the traffic's flow, I saw :
                        - on the FW switch : incoming frames from the fw and outgoing frames to the next switch
                        - on the GW switch : incoming frames from the FW switch and outgoing frames to the gateway ports.

                  - BUT, and that's the important point, no incoming frames from the switch to the internal FW interface !

                  So, where does this traffic comes from, if not from the fw itself ? And I insist again on the fact that either deleting the state in the FW or restarting it just kills the problem.

                  Sorry for being nerdy…

                  Thanks,

                  Thierry

                  1 Reply Last reply Reply Quote 0
                  • GruensFroeschliG
                    GruensFroeschli
                    last edited by Mar 28, 2017, 11:03 AM

                    Since you say you are using VLANs.
                    Do you by chance have a place where you have tagged and untagged traffic on the same wire?

                    Are you using static routes from your pfSense pointing to the gateway going to the 192.168 net?

                    We do what we must, because we can.

                    Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                    1 Reply Last reply Reply Quote 0
                    • T
                      Thilroy
                      last edited by Mar 28, 2017, 11:09 AM

                      Hi GreenFrog,

                      Yes, this is possible.
                      As for the routes, the firewall is doing the routing between the different VLANs, and has several intefaces configured, about one per VLAN plus a few extras. There is an interface for the 192.168.1 network, and a VLAN as well.

                      Thank you

                      Thierry

                      1 Reply Last reply Reply Quote 0
                      • H
                        heper
                        last edited by Mar 28, 2017, 11:26 AM

                        I'd enable broadcast storm protection on your access ports & ap links.

                        Also, block that mac-address on your WiFi controller.

                        There must be a configuration issue somewhere because 30kps won't be from any WiFi devices.

                        1 Reply Last reply Reply Quote 0
                        • T
                          Thilroy
                          last edited by Mar 28, 2017, 12:06 PM

                          Broadcast control, I've done it already. I just didn't block the MAC address yet because I wanted to find a way to identify the owner…. but don't see how.

                          Thks,

                          Thierry

                          1 Reply Last reply Reply Quote 0
                          • J
                            johnpoz LAYER 8 Global Moderator
                            last edited by Mar 28, 2017, 12:44 PM

                            When they can't get on your network - pretty sure they will contact you ;)

                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                            If you get confused: Listen to the Music Play
                            Please don't Chat/PM me for help, unless mod related
                            SG-4860 24.11 | Lab VMs 2.8, 24.11

                            1 Reply Last reply Reply Quote 0
                            • GruensFroeschliG
                              GruensFroeschli
                              last edited by Mar 28, 2017, 1:29 PM

                              I asked about the mixing of tagged and untagged traffic because i once had a misconfiguration where a device which wasn't vlan aware started leaking frames between the tagged vlans and the untagged subnet.
                              This lead to frames circulating between the tagged and untagged parts.
                              –> Essentially a logical loop even if no physical loop was present.

                              If you can capture with a tap (or a hub..) the wire which could carry tagged/untagged mixed you should see it because you see the same frame over and over tagged and untagged.

                              How i solved my problem:
                              Got rid of the untagged traffic on this segment.
                              Some disagree, but i still am of the opinion that the rule-of-thumb of "don't mix tagged and untagged traffic on the same cable" applies.

                              We do what we must, because we can.

                              Asking questions the smart way: http://www.catb.org/esr/faqs/smart-questions.html

                              1 Reply Last reply Reply Quote 0
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by Mar 28, 2017, 1:32 PM

                                You might be hitting something like this: https://redmine.pfsense.org/issues/2073

                                Basically, you might have a policy routing rule (rule with a gateway set) that matches too much traffic. If the firewall tries to policy route broadcast traffic, bad things like this can happen. Now it's usually smart enough to not do that for addresses in a subnet it knows directly, but if you have a rule to policy route traffic with a source of "any" and it matches some other subnet's broadcast packet, it doesn't know the difference.

                                Fix the rule, clear the states or reboot, and you might be OK again. Assuming that's the problem, but the symptoms fit.

                                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • J
                                  johnpoz LAYER 8 Global Moderator
                                  last edited by Mar 28, 2017, 2:43 PM

                                  "but i still am of the opinion that the rule-of-thumb of "don't mix tagged and untagged traffic on the same cable" applies."

                                  I would be one that disagrees ;)  And with some devices its not even possible.. Case in point would be management of say an AP that does not allow you to tag its management vlan.  But there is only 1 wire to connect the AP.  So the wifi/ssids are tagged vlans while traffic to manage the AP is not tagged.  Ie a native or PVID of the interface on the switch.

                                  This should never be a problem if your switching environment is correctly configured to keep this untagged management vlan isolated to only the ports that it needs to be on.

                                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                                  If you get confused: Listen to the Music Play
                                  Please don't Chat/PM me for help, unless mod related
                                  SG-4860 24.11 | Lab VMs 2.8, 24.11

                                  1 Reply Last reply Reply Quote 0
                                  • T
                                    Thilroy
                                    last edited by Mar 30, 2017, 7:30 AM

                                    Thanks for your help. I will investigate further on those grounds.
                                    I'll keep you filed.

                                    Thierry

                                    1 Reply Last reply Reply Quote 0
                                    1 out of 17
                                    • First post
                                      1/17
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                      This community forum collects and processes your personal information.
                                      consent.not_received