DHCP Broadcasts cluttering up log from LAN



  • I just pushed my pfSense box into production after doing some testing, it replaced an aging NetGear box.  I've been checking the logs just to make sure everything is okay, and I'm seeing tons of these messages in the Firewall log:

    block  Aug 29 09:49:10 LAN 0.0.0.0:68 255.255.255.255:67 UDP
    block  Aug 29 09:49:28 LAN 0.0.0.0:68 255.255.255.255:67 UDP
    block  Aug 29 09:49:29 LAN 0.0.0.0:68 255.255.255.255:67 UDP
    block  Aug 29 09:49:30 LAN 0.0.0.0:68 255.255.255.255:67 UDP

    DHCP appears to be working fine on the LAN, so it's not causing any issues, and the rules for the LAN interface right now allow any IPv4 traffic.  So, I'm a bit confused as to why this is cluttering up the logs.  I saw a post on quieting this noise on the WAN interface, but not sure if I should use that method as I don't want it to affect DHCP internally.  Although, as I said it seems to be working fine now.

    Anyone have any insight as to why I would be getting this with internally?



  • I noticed this problem too, after I added an explicit "block everything" rule that could log what was getting blocked instead of letting them go invisibly discarded. There are several special IP's, like DHCP broadcasts in this case, that don't need to go through the firewall (from LAN to WAN), so there is no point in logging each time the firewall blocks them. One easy way to remove them from your logs is to add explicit block rules for each of the IP's you want to "go away", and just don't check the box for logging the matches to the rule. Also, be sure to put the explicit IP blocks before the "block everything" rule ("block everything" should be last). If you don't, they will get logged.

    I just sorted all this out in the last few minutes, so I might have overlooked something, but I think this will work for you. I attached a screenshot of what my rules look like at the bottom of the list of rules. I assume you're familiar with how to use the logging checkbox for each rule (or you can figure it out easily), so I didn't bother to make a screenshot of that.

    ![explicit non-logged block rules to unlog mundane traffic.png](/public/imported_attachments/1/explicit non-logged block rules to unlog mundane traffic.png)
    ![explicit non-logged block rules to unlog mundane traffic.png_thumb](/public/imported_attachments/1/explicit non-logged block rules to unlog mundane traffic.png_thumb)


  • LAYER 8 Global Moderator

    So pfsense is not dhcp server for your lan segment?



  • Actually it is, but for some reason, DHCP still works. I suspect the firewall is blocking the DHCP packets AFTER they reach the DHCP server, because the firewall is only blocking LAN to WAN, not LAN to LAN. I'm guessing that the firewall assumes 255.255.255.255 must be outside the LAN, which is wrong. If you look at this page in your pfSense:

    https://192.168.1.1/services_dhcp.php

    You will see this:

    Available range 192.168.1.1 - 192.168.1.254

    So, the available range does not include 255.255.255.255 or 0.0.0.0, and that could be why the firewall assumes it must be beyond the LAN. pfSense probably wrongly assumes those IP's are not truly LAN to LAN, so it logs a block action in the firewall. That could be a bug. I will investigate if the issue has been reported before, and if I'm confident it's a valid unreported bug, I will report it.





  • @badon:

    I suspect the firewall is blocking the DHCP packets AFTER they reach the DHCP server…

    I agree.

    …because the firewall is only blocking LAN to WAN, not LAN to LAN.

    The firewall isn't processing LAN-LAN packets unless they are destined for the firewall (since broadcasts are destined for all nodes on the network, they are also processed by pfSense).

    I'm guessing that the firewall assumes 255.255.255.255 must be outside the LAN…

    No it doesn't.

    If you look at this page in your pfSense:

    https://192.168.1.1/services_dhcp.php

    You will see this:

    Available range 192.168.1.1 - 192.168.1.254

    So, the available range does not include 255.255.255.255 or 0.0.0.0, and that could be why the firewall assumes it must be beyond the LAN.

    No it's because neither network nor broadcast addresses should ever be handed out to clients by the DHCP server.

    The available range information on the DHCP server page isn't documenting what is or isn't within the LAN network, it is telling you what is the possible range of addresses to serve to clients on the network.

    pfSense probably wrongly assumes those IP's are not truly LAN to LAN, so it logs a block action in the firewall. That could be a bug.

    Let's start with the basics. A router, firewalling or not, should never forward broadcasts to any other of it's interfaces.

    The broadcasts are, as they should be, present and kept within your LAN network. They shouldn't be forwarded to other interfaces, so stopping them as your pfSense is doing is correct.



  • Thank you for the clarification.


  • LAYER 8 Global Moderator

    Here is the thing - if he is running dhcp server, then there needs to be a rule to allow that into pfsense - so it wouldn't be in the block log

    Look in your full rule set
    https://doc.pfsense.org/index.php/How_can_I_see_the_full_PF_ruleset
    pfctl -sa

    So here is sniff from mine showing that dhcp is allowed - so how would it be blocked and logged?

    pass in quick on vmx3f1 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "allow access to DHCP server"
    pass in quick on vmx3f1 inet proto udp from any port = bootpc to 192.168.1.253 port = bootps keep state label "allow access to DHCP server"
    pass out quick on vmx3f1 inet proto udp from 192.168.1.253 port = bootps to any port = bootpc keep state label "allow access to DHCP server"

    My guess would be not actually running dhcp, and something else is dhcp on the network would be how the clients are getting addresses, etc.

    Or he has rule setup blocking and its not the default rule blocking.. But if he had a rule blocking then dhcp wouldn't work!!  I don't see how he can say dhcp is working, yet the dhcp discover packets are being blocked.



  • Here's a screenshot of some DHCP leases. I assume this is conclusive evidence that DHCP is truly working despite there being explicit block rules in my config. Of course, I could just as easily make explicit pass rules, because the only objective here is to remove the noise from the firewall logs.

    ![pfSense DHCP leases.png](/public/imported_attachments/1/pfSense DHCP leases.png)
    ![pfSense DHCP leases.png_thumb](/public/imported_attachments/1/pfSense DHCP leases.png_thumb)


  • LAYER 8 Global Moderator

    How can it log something that it clearly passed if you got a lease?

    Can you post your output from the pfctl -sa for the rules

    And post your actual rules on this lan and if you have any floating rules setup.



  • Your guess is as good as mine as to how that's happening. I don't have any floating rules. The other rules contain secret information, and the output from pfctl -sa is loaded with more secret information. I could sanitize it before posting it, but I'd be afraid of missing something. Maybe you can set up a test system to see if you can duplicate it? I've duplicated it myself on 2 different machines, so it ought to be straightforward. I will be curious if you can figure out what's happening.


  • LAYER 8 Global Moderator

    secret info??  Like what that you block outboud from something called lan net to a public IP address??

    Secret info – yeah not so much.

    I run dhcp on my 32bit 2.1.5 pfsense on multiple segments, 2 wired only and 1 with wireless clients even and have NOT seen this issue not even one hit it firewall logs.  I have setup a test pfsense with 64bit 2.1.5 that also provides dhcp to clients and have never seen this issue.

    Because it doesn't seem possible - there is a rule when you turn on dhcp that allows the dhcp broadcast to your interface dhcp server is listening on.  So if the packet is passed in this rule, how it would it be listed as block in another rule??

    Seems highly highly unlikely -- if is a was  bug in code, then you would think lots of people would be reporting this.  I have only seen this 1 thread with any such info.  So would seem like something specific to your setup.

    If you do not want to put the "secret" firewall rules in a public forum - then feel free to PM them to me and can take a look see.  But without seeing your rules there is nothing else to do here from my point of view.

    if you have duplicated this on 2 setups up pfsense - then I would think the default rules you believe are the issue.  Or are you setting them up with your "secret" rules?  I would suggest you create a clean install with NO "secret" rules and do you get the error then?

    so you see in my output from pfctl -sa the rules for the dhcp.

    bootpc = port 68, and bootps = 67

    So that rule says let ANY source from port 68 using UDP to 255.255.255.255 (full broadast) to 67 (bootps) allow/pass  So if this rule allows the traffic that you are showing blocked.  How would some other rule block the same traffic?

    The next rule says hey if its direct traffic from same ports to the IP address of my interface that dhcp server is running on also allow it.  This would be part of the dhcp process.  But from 0.0.0.0 to 255.255.255.255 is a dhcpdiscover packet.  And has to be allowed in for new clients to get a lease.

    Have you modified these rules at a low level - are they not in there any you had to create some rule to allow dhcp to work?  What are these modifications or rules you have created?

    Have you somehow disabled this rule and only have your direct rule - so renews would work?  Please validate the rules I show are in your pfctl -sa output.  That clearly is not any sort of "secret" info ;)



  • Okay, this issue is more complicated than what I understood initially. :-[

    Since I was having the same block log entries I enabled [b]Status, System Logs, Settings, Filter descriptions and enabled it. That way I could find out it is the block bogon rule causing the log entry.

    Then I found that 0.0.0.0/8 is part of the bogon list, which explains the block.

    Since I also have working pfSense DHCP servers on all my local networks, I'm now also interested in finding out exactly how the broadcast packets can still reach port 67 in pfSense despite being logged as blocked. :o :o :o

    I can see two reasons for why this behaviour isn’t noticed by everyone and haven’t been discussed previously: not everyone enables bogon blocks on internal interfaces and even when doing so, DHCP still works…



  • @badon:

    I just sorted all this out in the last few minutes…

    Your workaround doesn’t work because any non-logging rules will be below the bogon rule that is the culprit.

    I have so far only come to think of two possible workarounds to get rid of the log entries:

    1. Go to Status, System Logs, Settings, Log Firewall Default Blocks and untick the “Log packets blocked by 'Block Bogon Networks' rules” option.
    2. Go to Interfaces, LAN and untick the "Block bogon networks" option.

    Unfortunately both of my workarounds have other consequences also.


  • LAYER 8 Global Moderator

    Why would anyone block bogon on the lan side??  The rules on the lan would be limited to source networks of your lan - so what would be the point of blocking bogon?  Your already blocking EVERYTHING that does not match your lan segment as source..



  • @johnpoz:

    Why would anyone block bogon on the lan side??

    Let me speculate here: as a newcomer it is hard to immediately understand each and every setting in detail and it may be tempting on initial installation to add something that seemingly increases the protection.

    The rules on the lan would be limited to source networks of your lan - so what would be the point of blocking bogon?  Your already blocking EVERYTHING that does not match your lan segment as source..

    Your posts here given me a much better understanding of how pfSense works and what you say makes sense but the configuration is still possible.

    In my opinion the only interesting question is why that configuration leads to a situation where the behaviour is in conflict with our present understanding of the software design. How can the DHCP server actually still work despite the critical initiating broadcast being reported as blocked?


  • LAYER 8 Global Moderator

    "to add something that seemingly increases the protection."

    Not sure I would agree with newcomer to "pfsense" - new comer to firewalls and network as a whole maybe? ;)  Bogon is not setup on the lan interfaces out of the box if I recall - such a rule is pointless with the default rule on the lan interface.

    With the way dhcp works - a client will send out discover that will be a broadcast to 255.255.255.255 from 0.0.0.0 - I agree with you if that traffic blocked the dhcp server should never see the discover, and then could never send an offer for the client to accept.

    Even on a renew the source would still be 0.0.0.0 – possible some clients send a renew it would be from source IP of the client??  I am not sure on every single client out there, but in general from the protocol level if I recall correctly he renew would be sourced from 0.0.0.0.

    In such a case if that traffic is blocked it should not be possible to get a dhcp lease - if pfsense is actually blocking it as it reports, then it should not be possible.  If there is a rule that allows the traffic as I posted from the pfctl -sa output it should never get to a block rule to be logged that it was blocked.

    What I can do when I get a chance is enable bogon on my test segment while doing a sniff and see exactly what is going.. If in fact the discover is listed as blocked but not then that is a huge bug if you ask me.



  • @johnpoz:

    With the way dhcp works - a client will send out discover that will be a broadcast to 255.255.255.255 from 0.0.0.0 - I agree with you if that traffic blocked the dhcp server should never see the discover, and then could never send an offer for the client to accept.

    Yet, that is what I'm and badon is seeing.

    I redid the test once again with the following output:

    16:40:19.967957 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 300
    16:40:19.969717 IP 192.168.11.1 > 192.168.11.14: ICMP echo request, id 32978, seq 0, length 28
    16:40:20.972225 IP 192.168.11.1.67 > 192.168.11.14.68: UDP, length 300
    16:40:21.854685 IP 0.0.0.0.68 > 255.255.255.255.67: UDP, length 308
    16:40:21.858652 IP 192.168.11.1.67 > 192.168.11.14.68: UDP, length 300
    
    

    And

    In such a case if that traffic is blocked it should not be possible to get a dhcp lease - if pfsense is actually blocking it as it reports, then it should not be possible.

    Exactly, and that is the only reason I still find it interesting to participate in this thread.

    What I can do when I get a chance is enable bogon on my test segment while doing a sniff and see exactly what is going..

    That is what I have done so, please do.



  • LAYER 8 Global Moderator

    Ok so I turned on bogon, and I enabled logging of bogon.. And now you have my attention - clearly something is ODD here..

    So if you look at /tmp/rules.debug – this should show you order of the processing of the rules

    So scroll down and you find


    block bogon networks (IPv4)

    http://www.cymru.com/Documents/bogon-bn-nonagg.txt

    block in log quick on $DMZ from <bogons>to any label "block bogon IPv4 networks from DMZ"

    block bogon networks (IPv6)

    http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt

    block in log quick on $DMZ from <bogonsv6>to any label "block bogon IPv6 networks from DMZ"
    antispoof for vmx3f3

    allow access to DHCP server on DMZ

    pass in quick on $DMZ proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
    pass in quick on $DMZ proto udp from any port = 68 to 192.168.3.253 port = 67 label "allow access to DHCP server"
    pass out quick on $DMZ proto udp from 192.168.3.253 port = 67 to any port = 68 label "allow access to DHCP server"

    So clearly blocking bogon's is before the dhcp rules..  and the "quick" is set -- so that means no other rules are processed and should be dropped and should not get to dhcp server..

    And I do see it in the log with the exact timestamp of my dhcp lease - see attached.

    If you do a pfctl -s rules

    block drop in log quick on vmx3f3 from <bogons>to any label "block bogon IPv4 networks from DMZ"</bogons>
    block drop in log quick on vmx3f3 from <bogonsv6>to any label "block bogon IPv6 networks from DMZ"
    block drop in on ! vmx3f3 inet from 192.168.3.0/24 to any
    block drop in inet from 192.168.3.253 to any
    block drop in on vmx3f3 inet6 from fe80::20c:29ff:fe1e:18c2 to any
    pass in quick on vmx3f3 inet proto udp from any port = bootpc to 255.255.255.255 port = bootps keep state label "allow access to DHCP server"
    pass in quick on vmx3f3 inet proto udp from any port = bootpc to 192.168.3.253 port = bootps keep state label "allow access to DHCP server"
    pass out quick on vmx3f3 inet proto udp from 192.168.3.253 port = bootps to any port = bootpc keep state label "allow access to DHCP server"

    Its listed above the dhcp rules as well

    Going to need one of the developers to let us know how it is that we get a log of something being blocked, yet still passed??  And from what I know of the way the rules are done - it looks like that should just be dropped and should never get to the dhcp server.  That is kind of big issue if you ask me.

    But this really should never come in to play really - because why would bogon be enabled on LAN interface?  Your rules should limit to source of your LAN network, or sub networks even.  Everything else would be blocked by the default block rule.


    </bogonsv6></bogonsv6></bogons>



  • @johnpoz:

    Why would anyone block bogon on the lan side??  The rules on the lan would be limited to source networks of your lan - so what would be the point of blocking bogon?  Your already blocking EVERYTHING that does not match your lan segment as source..

    P3R, thanks for chiming in with excellent info that helps us make progress on figuring all this out. I think I had reached my limit on how much I could help here.

    johnpoz, correct me if I'm wrong, but I think another thing to consider is the possibility that "LAN side" packets never even touch pfSense if there is a managed switch or router in the network somewhere that can route packets to the correct LAN port immediately without needing to spam the other devices and network segments with packets that are not intended for them. In other words, blocking bogon networks on the LAN side may not even be effective, even if there were some good reason to do that, which we aren't aware of at the moment.

    If so, then that may motivate us to question why such a feature exists in pfSense, and thus cause us to consider the possibility of removing the feature, or adding some documentation (preferably next to the checkbox itself) that explains the circumstances when the feature may not work as expected.

    Another thing to consider is whether 0.0.0.0 and 255.255.255.255 are truly bogon IP's, because of their universal deployment as a standard for DHCP services. I'm not aware of any other uses for those IP's, but I'm skeptical that there are any "official" standardization rules that define them, so they might be a de facto standard in practice solely because that's what everyone decided to use those IP's for. Determining whether those DHCP IP's are a standard of some kind might aid a decision of whether there is something here that needs to be fixed (like documentation, bugs, feature enhancements, etc).



  • @badon:

    I noticed this problem too, after I added an explicit "block everything" rule that could log what was getting blocked instead of letting them go invisibly discarded. There are several special IP's, like DHCP broadcasts in this case, that don't need to go through the firewall (from LAN to WAN), so there is no point in logging each time the firewall blocks them. One easy way to remove them from your logs is to add explicit block rules for each of the IP's you want to "go away", and just don't check the box for logging the matches to the rule. Also, be sure to put the explicit IP blocks before the "block everything" rule ("block everything" should be last). If you don't, they will get logged.

    I just sorted all this out in the last few minutes, so I might have overlooked something, but I think this will work for you. I attached a screenshot of what my rules look like at the bottom of the list of rules. I assume you're familiar with how to use the logging checkbox for each rule (or you can figure it out easily), so I didn't bother to make a screenshot of that.

    I'm a little late in the discussion, but I just found this. On this page:

    Status: System logs: Settings
    https://192.168.1.1/diag_logs_settings.php

    There is this option:

    Log Firewall Default Blocks > Log packets blocked by the default rule
    Hint: packets that are blocked by the implicit default block rule will not be logged if you uncheck this option. Per-rule logging options are still respected.

    So, it wasn't even necessary for me to add a block everything rule to get logs. It's interesting how unintended clumsy abuse of features can reveal unexpected behavior, bugs, and security problems. In this case, there might be a security problem if it is indeed a bug that DHCP is getting through pfSense when it should not. There could be a circumstance when DHCP should be blocked, like if all hosts are supposed to have self-assigned static IP's. An attacker could run his own DHCP server to gain control over the network traffic of hosts that do not have static IP's configured properly.

    I'm not sure what an attacker would be able to gain by that, but if something like that is plausible, it's still a major bug of higher-than-usual importance for pfSense, which specializes in preventing such things from happening, when configured to do so as we have done here.


  • LAYER 8 Global Moderator

    Here is the thing about your attack scenario..  When you enable dhcp server, pfsense puts in the rules to allow dhcp.  if your not running dhcp then that rule is not there - and even it was no harm no foul since pfsense is not listening.

    As to someone running a dhcp server on your network - that is not something pfsense can control anyway - pfsense has no interaction with traffic between devices on the same segment.. Only when you leave that segment would pfsense be able to firewall traffic.

    Ie this statement is somewhat true.
    "LAN side" packets never even touch pfSense if there is a managed switch or router in the network"

    But pfsense is going to see a broadcast 255.255.255.255 and if its a dhcp discover then the pfsense dhcp server would answer.

    " blocking bogon networks on the LAN side may not even be effective"
    "If so, then that may motivate us to question why such a feature exists in pfSense"

    In a normal setup yes, where your rule is that source has to be "lan net" then no it wouldn't be effective because all bogons would by that rule would be blocked anyway - since source is not valid lan net address anyway.  But you don't know what the owner of the firewall might want to do, maybe they have lots of sub network that connect to that interface what source would not be lan net.  So they leave it as any, but don't want any funny stuff going on so they block invalid traffic.

    It makes great sense on the wan side where you have no idea what the source IP might be on the internet - but your sure its not suppose to be an invalid IP that not in use, etc.

    0.0.0.0/8 - Addresses in this block refer to source hosts on "this"
      network.  Address 0.0.0.0/32 may be used as a source address for this
      host on this network; other addresses within 0.0.0.0/8 may be used to
      refer to specified hosts on this network [RFC1700, page 4].

    That by definition would not be routed on the public internet.  Again goes to blocking bogon on your "lan" interface normally would never be done - but sure if you were running dhcp server on that you could have problems.  You should be smart enough to know that if your wanting to block bogon on your lan for some reason

    You would never see legit 0.0.0.0 traffic inbound on your wan that I can think of - why would you be running a dhcp on your wan interface?  Now you would send out from 0.0.0.0 on your wan, and that answer would be allowed in by the state when you sent out the request.

    While agree clicking on shit you don't understand can cause you grief ;) If you block bogon on your LAN that your running a dhcp server on for example - but not sure why its listed as block when clearly its not.  But this scenario should never really come up.

    Maybe I am missing something in the rules.  But from what I see if you enable bogon on your lan – and 0.0.0.0/8 is in the list and the rule says drop quick on stuff in the list, and it does log it as the rule also says.  Not sure how it actually did not get blocked?

    edit:
    To me if I am reading the rules right if you turned on bogon blocking you should break your dhcp server ability to hand out leases because it shouldn't see the discover and wouldn't know any clients are even asking for one.



  • Once again, thank you very much for the clarification.



  • @badon:

    P3R, thanks for chiming in with excellent info that helps us make progress on figuring all this out. I think I had reached my limit on how much I could help here.

    johnpoz, correct me if I'm wrong, but I think another thing to consider is the possibility that "LAN side" packets never even touch pfSense if there is a managed switch or router in the network somewhere that can route packets to the correct LAN port immediately without needing to spam the other devices and network segments with packets that are not intended for them. In other words, blocking bogon networks on the LAN side may not even be effective, even if there were some good reason to do that, which we aren't aware of at the moment.

    A switch, managed, unmanaged or otherwise will not broadcast traffic to all ports, unless it's specifically broadcast traffic. Traffic only flows from the source to the destination (unless it's broadcast, as stated) on all unmanaged, lvl2 managed, lvl3 managed (technically speaking a router) and (if $pocket_depth permits) higher switches.

    An unmanaged switch is NOT a hub.


Log in to reply