ARP reports bogons
-
Huh, well seems like something more significant.
Do you see a crash report when it boots back?
-
@stephenw10 I do not, would I see this in the logs directly?
-
No, if it panicked and crashed you would normally see a crash report as an alert on the dash when you log back in after it reboots.
If if just resets instantly with no kernel panic that's more likely to be a hardware issue.
Do you see anything at the console when that happens?
Steve
-
@stephenw10 This is the same time pfSense VM went down, so I guess the server rebooted, not pfSense.
The computer has rebooted from a bugcheck. The bugcheck was: 0x00020001 (0x0000000000000011, 0x00000000002182ae, 0x0000000000001005, 0x0000010000003b80). A dump was saved in: C:\Windows\MEMORY.DMP. Report Id: 3fc6cc5c-a540-425f-9522-d0f8dfb9b40a.
-
Ah, it's a VM. Then yes, almost certainly. Is that what's causing every pfSense reboot? That's pretty bad if so.
-
@stephenw10 ok checked the AP, there is definitely no leases being handed out on that network. Everything seems to be working fine on the AP.
-
@stephenw10 im getting so many bogons on subnets that dont even exist on my network.
Also for auto configuration IPs
Im getting heaps!
Oct 10 18:19:52 arpwatch 74390 bogon 192.168.1.112 02:2d:1e:a0:e2:de Oct 10 18:19:53 arpwatch 74390 bogon 192.168.1.112 02:2d:1e:a0:e2:de Oct 10 21:14:47 arpwatch 73733 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 10 21:14:48 arpwatch 73733 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 10 21:14:49 arpwatch 73733 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 10 21:14:50 arpwatch 73733 bogon 169.254.192.108 c4:9d:ed:89:ed:05 Oct 10 21:14:50 arpwatch 73733 bogon 169.254.192.108 c4:9d:ed:89:ed:05 Oct 10 21:14:52 arpwatch 73733 bogon 169.254.192.108 c4:9d:ed:89:ed:05 Oct 10 21:14:59 arpwatch 73733 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 10 21:15:00 arpwatch 73733 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 10 21:15:01 arpwatch 73733 bogon 0.0.0.0 c4:9d:ed:89:ed:05 -
But those MAC addresses are valid hosts there?
Are you only seeing this from hosts connected via the AP?
Steve
-
@deanfourie if you do not want to see bogon log entries from arpwatch, then turn them off.. These are normal to see. Just turned it on - see bogon reported, because I didn't tell arpwatch not to log them.
Not sure what rabbit hole you have gone down - but yeah arpwatch will log bogons, its really just log spam.. Turn if off if you don't wan to see it..
This has zero to do with your problem -- clients reporting 169.254 is a sign that dhcp is not working... Clients that are set for dhcp, and don't get a lease will give themselves a 169.254.x.x address.. This is a link-local address, and called zero conf, or APIPA, etc.
Wireless clients moving about can also sometimes use their old IP from the previous network, etc. since all of rfc1918 space is listed in bogon, then sure arpwatch would be correct in reporting 192.168.1.112 as bogon..
To be honest I would turn off arpwatch for now - its confusing you.. There is nothing arpwatch is reporting that you have shown that says anything about your problem.. Now if you had arpwatch reporting that some other mac was using your pfsense IP on this network - that could be problematic for sure. Or if your device is having a problem, because again someother client on the network with different mac is using your IP.
But none of the info you have shown from arpwatch points to any sort of problem. You know what is problematic for a working network - your VM host just rebooting.. Kind of hard for pfsense to do its thing when its not on, etc. .
-
The only thing that might concern me is that it appears to be a symptom of clients failing to pull a dhcp lease. Or renew an existing one.
That 'feels' like an AP problem though.
-
@stephenw10 but that is the thing he should be looking to directly.. Which I stated before - actually troubleshoot the issue.
If your client is not working - does it have an IP, can it ping pfsense? Can it do dns, can it ping an external IP..
Looking at arpwatch log, and saying clients don't work doesn't get to the root of the problem.. Is the problem all the clients all of sudden loose their dhcp address? seems unlikely unless they were all turned on at the same time?
So for now forget what your seeing in arpwatch - and from a box that your saying is loosing internet - troubleshoot why it no longer has internet.. Not having an IP for sure not going to have internet, not being able to talk to its gateway (pfsense) again internet going to work - no dns, again no internet, etc.. Find out exactly what the problem is vs worry about some bogon log spam..
-
@johnpoz sure, I understand that arpwatch is just reporting bogons, but it still shows me that something is floating around on 192.168.1.112 which is not a subnet I use. So why is it even there?
It is DHCP, clients loose their IP, and I cannot renew.
But this can happen like 3 time within 10 minutes. Why would the client even be asking for 3 leases in 10 minutes?
Something just doesn't add up.
-
@deanfourie said in ARP reports bogons:
asking for 3 leases in 10 minutes?
because it didn't actually renew?
As a client gets closer and closer to when it expires - it will ask more and more frequently. When it fails then it would send a discover, etc.
I would suggest you troubleshoot a specific client that is failing - what exactly is failing? Do you see your lease over the 50% mark, if you do this is example of it not renewing for some reason.. A dhcp client gets a lease, and then would try and renew at the 50% mark, if you see a lease that is older than 50% ish of the lease, something is wrong with the renew process.
I would suggest you troubleshoot with a client that allows you to see such info, etc.. Like a windows box.
example.. Here just switched my pc to dhcp, see the lease is for 4 days..
If its past say 2:15PM on the 12th and still have this same lease - something isn't right because it should of renewed right around the 50% mark, so if its say 3pm on 12 and still showing this same lease - something is wrong for sure..
As to that device with 112 address. What is that device? I don't see 02:2d:1e as a lookup, could be a private mac - wireless tablets and phones and depending on the OS etc of the device can use random mac addresses, etc. But track down how that device is connected to your network - what switch port is it on? Is it connected to your AP, etc.
Again devices can send out that info - say I take my phone and connected to network X, and then moves to your network it could send out a probe or gratuitous with its old IP, a device can try and reuse the IP it had on a previous network, what should happen is your dhcp server should send a NAK, and then the client should send out a discover. And get an IP from your dhcp server, etc. Seeing such entries could also suggest as mentioned already another dhcp server on your network handing out IPs in that range..
That c4:9d:ed mac is listed as Microsoft.
-
@johnpoz also keep getting these in my logs.
Oct 12 00:05:59 nginx 2022/10/12 00:05:59 [error] 92058#100543: send() failed (54: Connection reset by peer)and these
Oct 12 00:07:00 php 18242 servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon) -
@deanfourie said in ARP reports bogons:
@johnpoz also keep getting these in my logs.
Oct 12 00:05:59 nginx 2022/10/12 00:05:59 [error] 92058#100543: send() failed (54: Connection reset by peer)and these
Oct 12 00:07:00 php 18242 servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)Never put Suricata or Snort under Service Watchdog! Service Watchdog does not understand how the two IDS/IPS packages operate and it will restart them needlessly-- sometimes resulting in multiple duplicate instances running on the same interface.
I am the developer/maintainer of both IDS/IPS packages, so I know what I am talking about ...
.While it may not be related to your immediate issue, you should never configure the IDS/IPS packages to be monitored by Service Watchdog.
-
@bmeeks Ok thanks,
I removed Suricata just to eliminate it as being an issue.
This did not resolve the issue. Last night I had the same problems, multiple interruptions and disconnects at around midnight.
I cleared all pfSense logs, and the ONLY thing I can see that occurs during this time is that in my system logs it reports the bogon at 0.0.0.0.
Oct 12 00:50:17 arpwatch 45709 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 12 00:50:18 arpwatch 45709 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 12 00:50:18 arpwatch 45709 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 12 00:50:33 arpwatch 45709 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 12 00:50:34 arpwatch 45709 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 12 00:50:35 arpwatch 45709 bogon 0.0.0.0 c4:9d:ed:89:ed:05 Oct 12 01:00:00 php 22959 [pfBlockerNG] Starting cron process. Oct 12 01:00:06 php 22959 [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload Oct 12 01:25:11 arpwatch 45709 bogon 0.0.0.0 c0:33:5e:31:9e:87 Oct 12 01:25:12 arpwatch 45709 bogon 0.0.0.0 c0:33:5e:31:9e:87 Oct 12 01:25:13 arpwatch 45709 bogon 0.0.0.0 c0:33:5e:31:9e:87One thing I don't understand is, say a user asks where the gateway is, who has 192.168.1.1, and the response is 192.168.1.1 is at 0.0.0.0 isnt this going to cause the exact issues im having?
I really suspect this is what is happening.
Your thoughts?
-
the response is 192.168.1.1 is at 0.0.0.0
that is not what is happening... Where are you seeing that 192.168.1.1 is at 0.0.0.0??
There is no way you can tell that from what arpwatch is saying there.. Lets see the sniff of these arps.
I would really suggest you just turn off arpwatch, or tell it not to log bogon - what is in your arpwatch db? It only seems to be confusing the issue for you.
My db is very small because I only had arpwatch on for short time to catch that it was showing bogon on my network as well
Lets see your pfsense arp table..
Here is snip of mine
Are you not seeing IPs in there you should see, do you see anything listed at 0.0.0.0 are you seeing any incompletes? for something?
so I pinged something from pfsense that doesn't exist, see how it shows incomplete in the arp table, because it arped for that IP and go not response
So while you were having this problem did you actually check what was going on one of your machines. Did it not have a lease, did it not have an IP, could it not ping its gateway, could it not do a dns query?
-
@deanfourie said in ARP reports bogons:
response is 192.168.1.1 is at 0.0.0.0
That makes no sense. You might possibly see 0.0.0.0 is at (MAC address) aa:bb:cc:dd:ee:ff. But you shouldn't since nothing should be using that IP.
Or you might see 'who has <gateway IP> tell 0.0.0.0'. But the response would be <gateway IP> is at <MAC address>.The only way clients get a gateway is via DHCP.
DHCP is failing here and from everything you've said it seems most likely the cause is something in the AP.
Steve
-
I did do some testing during these periods.
I was not able to ping anything Local or remote (internet). I checked my IP settings I still had a current lease, and all IP settings were correct.Internet and ALL LAN access just abruptly stops.
Now to me, if my local ARP table was being updated to point say the gateway, or all devices to 0.0.0.0, then this is the exact behavior I would expect to see.
Isn't this exactly what ARP spoofing does? Can send traffic anywhere with ARP
-
@deanfourie said in ARP reports bogons:
I was not able to ping anything Local or remote (internet).
so you could not ping pfsense IP? 192.168.1.1? Did you look in this devices arp table? What did it show for this IP? Nothing?
Is this device wired or wireless?
