Constant arpwatch bogon reports .... related to my own network !!??? (*strange andannoying!!*)

louis2

I noticed that arpwatch as it is now is a package, I did install some where in the past. I removed it since it is causing thousands of 'bogon' messages.

Also note that in my case the messages were related to a Mokerlink switch its management IP. No idea why that did trigger arpwatch.

Bogon is of course not correct, but what else could have triggered arpwatch ???

johnpoz

@louis2 If it sees an IP that is not suppose to be on this network. So bogon..

If pfsense is seeing say 192.168.10.3 on network that is suppose to be say 192.168.20.0/24 then that is bogon to that network.

I duplicated what your seeing by just setting a devices IP to wrong IP for the network it was on - and then yup lots and lots of bogon log entries.

If your switches management IP was seen on an interface for a different network then yup arpwatch will log bogon.

A bogon is an IP that is not suppose to be on a network - ie rfc1918 should not be on the internet. IP ranges that have not been assigned to be on the internet yet are bogon. This list has gotten smaller since IPv4 space that has not been assigned yet has gotten smaller.

But if you see IP X on network Y - where X should not be there, that yes that could be considered bogon - because it shouldn't be there.

A better term might be martian - which is a packet that shows up , that would not be returned to the sender.. if some X ip shows up on an interface in network Y as the source - the return traffic would not come back to the sender, because return traffic wanting to go to this X ip would be routed to the X network, not the Y network.

dennypage

@louis2 Arpwatch has been a package for years. ANDwatch, which replaces Arpwatch, is new.

louis2

@johnpoz

OK good to know that also local addresses not belonging to the subnet are bogon. I did not know that!
I would have looked differently to the alarms if I had known.

What does not take away that the messages I did saw where related to the switch its IP, so that looked completely OK!

I wonder if that address has been seem in multiple vlan's ... however the message is so limited, it does not give enough info! :(

What ever it should not happen. The weird behavoir is around a cheap Mokerlink switch. I am using Mokerlink 2.5/10G switches as 'room switches', since they are small and cheap and apart from this, seems to work OK.

johnpoz

@louis2 said in Constant arpwatch bogon reports .... related to my own network !!??? (*strange andannoying!!*):

cheap Mokerlink switch

do they actually support vlans? Since they have management IPs I would guess they should. But do they support the management vlan on a tagged vlan?

Yeah might be easier to know exactly what was going on if arpwatch listed what interface it saw this traffic on that it considers bogon because the network doesn't match the interface it was seen on.

louis2

@johnpoz

Yep, I only use vlan's !!

The switch is connected to the upper layer switch via a trunk, all tagged.

Since I would like to understand the problem, I am going to install ANDwatch (as it becomes availailable) at least temporarily, hoping that that packages give more precise messages.

After reading the definition you gave for bogon's, I am starting to fear that the management IP of the cheap switch is showing up in multiple vlan's. Which is really bad of course ..

On the other hand, I an not the only one having this trouble, so not sure if the problem is the package or the switch.

dennypage

@louis2 said

Since I would like to understand the problem, I am going to install ANDwatch (as it becomes availailable) at least temporarily, hoping that that packages give more precise messages.

After reading the definition you gave for bogon's, I am starting to fear that the management IP of the cheap switch is showing up in multiple vlan's. Which is really bad of course ..

Yes, ANDwatch will report what interface the IP address appears on. However, it does not offer any judgments about the IP address (bogon, etc.).

johnpoz

@louis2 said in Constant arpwatch bogon reports .... related to my own network !!??? (*strange andannoying!!*):

On the other hand, I an not the only one having this trouble, so not sure if the problem is the package or the switch.

I have not seen this exact problem before that I recall - mostly issues with dhcp being flagged as bogon, etc. Where the source IP is all zeros. Or if there was this issue reported, didn't recognize as this at the time.. where your seeing network A source traffic into network B.

For the package to report it - it has to be seen by the package.. And you shouldn't be seeing network A as source into network B. The only time you should see other network IPs into some network interface is if that network is a transit network.

Is your switch management IP a downstream network being routed over a transit/connector network?

If you had something like this, where some downstream router was routing over a transit into pfsense to be routed, then yeah pfsense seeing traffic sourced from say 192.168.2.x into its 172.16.0.1/30 interface would be normal and shouldn't be reported as bogon by arpwatch.

arpwatch should have a method of listing what source networks could be seen on an interface, so not to report such traffic as bogon without having to turn off reporting of bogon completely.

It is also possible that arpwatch is looking at traffic it shouldn't be to report that. If you have an physical interface say igb0 and then you have sub interfaces for vlans igb0.10 igb0.20 etc.. arpwatch shouldn't be reporting that traffic meant for igb0.20 for example with a tag of vlan ID being seen on igb0 which it would be but tagged shouldn't be reported as bogon.

dennypage

@johnpoz said

arpwatch should have a method of listing what source networks could be seen on an interface, so not to report such traffic as bogon without having to turn off reporting of bogon completely.

It does, but it’s not exposed in the pfSense package. Also, it affects all of Arpwatch, instead of a single interface, so it wouldn’t be very useful in this case.

I recommend ANDwatch (obviously), which works on individual interfaces and should provide a clearer picture.

johnpoz

@dennypage I would recommend ANDwatch as well - and I haven't even given it a spin yet. I don't really use arpwatch any more - I stopped using it when there was an issue way back when - after it ran for a while it would lock up my pfsense.

When andwatch becomes integrated into the normal pfsense packages I will for sure give it a spin ;)

You might know - does arpwatch just listen on the parent interface in promiscuous mode and ignore tags? If so then yeah any other vlans on that parent interface could cause it to list bogons from your vlans that are tagged.

dennypage

@johnpoz said:

@dennypage I would recommend ANDwatch as well - and I haven't even given it a spin yet.

When andwatch becomes integrated into the normal pfsense packages I will for sure give it a spin ;)

I hope you’ll give it a go. Btw, if you don’t want to trust my build of the core, you can just install the package from upstream FreeBSD. The pfSense package itself (php code) is easily inspectable.

You might know - does arpwatch just listen on the parent interface in promiscuous mode and ignore tags? If so then yeah any other vlans on that parent interface could cause it to list bogons from your vlans that are tagged.

I’d have to recheck code, which I can’t do right now, but I believe Arpwatch listens on whatever interface(s) you specify. If you specifiy a parent interface then that’s what it will use. Not sure if you can spec a parent interface with pfSense however, unless you make the mistake of mixing tagged and untagged on a single interface…

johnpoz

@dennypage oh I will for sure give it a spin - and completely trust your coding.. But I normally don't install stuff that is not from the pfsense repository - once it shows up there I will be playing with it for sure.

"unless you make the mistake of mixing tagged and untagged on a single interface…"

Not sure what call that a mistake - it is a valid configuration. I have an interface that has a native network on it, and then vlans on it as well. But yeah some people are if vlans on interface, then no untagged.. But it is a valid configuration - you just need to know what your doing ;) or yeah you could mess it up.

dennypage

@johnpoz said:

"unless you make the mistake of mixing tagged and untagged on a single interface…"

Not sure what call that a mistake - it is a valid configuration. I have an interface that has a native network on it, and then vlans on it as well. But yeah some people are if vlans on interface, then no untagged.. But it is a valid configuration - you just need to know what you’re doing ;) or yeah you could mess it up.

True. However…

While mixing tagged and untagged packets on an interface is a valid configuration for pfSense, and most/all switches, I would generally call it a mistake for most folk. Mixing tagged and untagged on a single port requires a good deal of attention to detail, and has a high cost of failure. There is good opportunity for confusion, particularly with add ons such as Arpwatch, Avahi, etc.

YMMV

johnpoz

@dennypage your not wrong in that you have to understand what your doing or sure you could mess it up.. Maybe that is what is causing his flood of bogons..

But running an untagged native vlan on an interface (single network) while also having tag is valid configuration. If something like arpwatch can't run like that - then that is a problem with arpwatch. Not with configuration of native and tagged vlans on the same physical interface.

Shoot many devices would actually require that sort of connection. Many cheap smart switches do not allow you to put the management IP on a tagged vlan. Unifi not that long ago did not support management on a tagged vlan.

So if you kept tagged off an interface for management - you would need to leverage another interface for your tagged vlan traffic.

JonathanLee

On the lan interface do you have bogons enabled ??? If so that only needs to be on wan side

dennypage

@johnpoz said:

@dennypage your not wrong in that you have to understand what your doing or sure you could mess it up.. Maybe that is what is causing his flood of bogons..

But running an untagged native vlan on an interface (single network) while also having tag is valid configuration. If something like arpwatch can't run like that - then that is a problem with arpwatch. Not with configuration of native and tagged vlans on the same physical interface.

[This deserves a better write-up than I can do from a phone in the middle of nowhere, but here goes...]

It's not a problem with Arpwatch, or with ANDwatch for that matter. It is an intrinsic host OS behavior resulting from mixing tagged and untagged packets on a physical interface.

As you read, keep in mind these two points:

VLANs are tags, not encapsulations.
VLAN interfaces are virtual interfaces build on top of physical interfaces.

Tcpdump, like Arpwatch and ANDwatch, uses the pcap library, which in turn uses the BPF kernel interface. It's how user space applications get sniff packets on an interface.

Arpwatch is essentially this:

tcpdump -I <ifname> arp

So we'll use tcpdump to illustrate the behavior.

We have these three interfaces:

igc0 - native physical interface with a mix of tagged and untagged packets. The untagged traffic is VLAN 1.
igc0.2 - virtual interface derived from igc0 for VLAN 2.
igc0.3 - virtual interface derived from igc0 for VLAN 3.

What happens when you run this command?

tcpdump -I igc0.3

Just what you would intuitively expect. You receive all packets that pass through virtual interface igc0.3. These will be all the packets that are tagged as VLAN 3 that pass through interface igc0. You will have a similar experience when you run tcpdump on igc0.2.

But what happens when you run this command?

tcpdump -I igc0

Again, just what you would intuitively expect. You receive all packets that pass through the interface. However, this time it is a physical interface and there are a mix of three different VLANs running through it. So you will receive VLAN 1, VLAN 2 and VLAN 3, all together as one stream of packets. It's intuitive when you run tcpdump, but it may come as a surprise when you run other programs.

This is exactly what will happen with Arpwatch if you tell it to watch a physical interface that has a mix of VLAN traffic. It gets the whole big stream of packets. And when it receives a VLAN 2/3 packet on the physical interface, it will view it as a bogon because it doesn't match the native (untagged) interface.

It seems there is always another gotcha when you mix tagged and untagged traffic on an interface. This is one of the more esoteric ones.

It's generally okay to mix tagged and untagged on a switch, and sometimes required as you say. However, it's not a good idea with a host OS such as FreeBSD (pfSense) or Linux if you can avoid it.

@louis2, going back and reading your original post, I expect this may be what is generating your thousands of bogon reports.

dennypage

@JonathanLee said:

On the lan interface do you have bogons enabled ??? If so that only needs to be on wan side

See above - bogon means something different in Arpwatch.

You would not want to run Arpwatch on the WAN interface.

johnpoz

@dennypage said in Constant arpwatch bogon reports .... related to my own network !!??? (*strange andannoying!!*):

It's generally okay to mix tagged and untagged on a switch, and sometimes required as you say. However, it's not a good idea with a host OS such as FreeBSD (pfSense) or Linux if you can avoid it.

But if switch is connected to pfsense (firewall/host) and to manage it has to be untagged then interface on pfsense has to have untagged traffic. If that switch can not use tagged for its management.

Great write up - and nothing I didn't already understand. I am the one that brought it up ;) but seems to me that is lack of configuration on arpwatch part. Should be able to tell it hey on igb0, that also has igb0.10 and igb0.20 since your too stupid to understand that .10 network is for the .10 network only and not .20 - don't mark networks abc you see on igb0 or any of subs as bogon.

But yeah he either has his L2s not isolated, or yup he is running into this. But turning off bogon did stop the reporting from my testing of it.

So does arpwatch allow for this config, but its not exposed in the gui? Where I can tell it - hey if you see networks A,B(vlanX) or C(vlanY) on the parent interface - not to report it as bogon?

I take it andwatch allows for this - and will be able to configure it, I would think it could be auto figured out to be honest if you tell it to listen on igb0 and igb0.x and igb0.y etc..

dennypage

@johnpoz said:

but seems to me that is lack of configuration on arpwatch part. Should be able to tell it hey on igb0, that also has igb0.10 and igb0.20 since your too stupid to understand that .10 network is for the .10 network only and not .20 - don't mark networks abc you see on igb0 or any of subs as bogon.

So does arpwatch allow for this config, but its not exposed in the gui? Where I can tell it - hey if you see networks A,B(vlanX) or C(vlanY) on the parent interface - not to report it as bogon?

No, Arpwatch does not offer any vlan configuration. Arpwatch itself allows you to say net/cidr (nothing to do with vlans) should be considered local, but it is not exposed in the pfSense package.

I take it andwatch allows for this - and will be able to configure it, I would think it could be auto figured out to be honest if you tell it to listen on igb0 and igb0.x and igb0.y etc..

No, ANDwatch does not offer any vlan configuration either. ANDwatch does allow you to specify extensions of the pcap filter, so in theory you could exclude vlan tagged packets if your implementation allowed it, but this would be OS/bpf/pcap build dependent.

From my pov, best practice is to avoid the whole situation by not mixing tagged and untagged traffic on physical interfaces.

YMMV.

dennypage

Finally back from traveling and had a chance to look at mixed tagged/untagged interfaces under pfSense with ANDwatch. pcap on pfSense 25.03, and presumably 2.8.0, is in fact compiled with vlan filter support.

So, with a mixed interface configuration like this:

igc0: untagged, presumably vlan 1
igc0.2: tagged, vlan 2
igc0.3: tagged, vlan 3

You have a choice of either running a single ANDwatch instance on igc0, which will monitor all of the subnets, or running individual ANDwatch instances on each of the interfaces.

If you want to run individual instances, and have ANDwatch exclude the vlan tagged packets for interface igc0, you would set the "PCAP Filter" on igc0 to "Custom filter" and include "not vlan" in the filter.