Constant arpwatch bogon reports .... related to my own network !!??? (*strange andannoying!!*)

louis2

Yep, I only use vlan's !!

The switch is connected to the upper layer switch via a trunk, all tagged.

Since I would like to understand the problem, I am going to install ANDwatch (as it becomes availailable) at least temporarily, hoping that that packages give more precise messages.

After reading the definition you gave for bogon's, I am starting to fear that the management IP of the cheap switch is showing up in multiple vlan's. Which is really bad of course ..

On the other hand, I an not the only one having this trouble, so not sure if the problem is the package or the switch.

dennypage

@louis2 said

Since I would like to understand the problem, I am going to install ANDwatch (as it becomes availailable) at least temporarily, hoping that that packages give more precise messages.

After reading the definition you gave for bogon's, I am starting to fear that the management IP of the cheap switch is showing up in multiple vlan's. Which is really bad of course ..

Yes, ANDwatch will report what interface the IP address appears on. However, it does not offer any judgments about the IP address (bogon, etc.).

johnpoz

@louis2 said in Constant arpwatch bogon reports .... related to my own network !!??? (*strange andannoying!!*):

On the other hand, I an not the only one having this trouble, so not sure if the problem is the package or the switch.

I have not seen this exact problem before that I recall - mostly issues with dhcp being flagged as bogon, etc. Where the source IP is all zeros. Or if there was this issue reported, didn't recognize as this at the time.. where your seeing network A source traffic into network B.

For the package to report it - it has to be seen by the package.. And you shouldn't be seeing network A as source into network B. The only time you should see other network IPs into some network interface is if that network is a transit network.

Is your switch management IP a downstream network being routed over a transit/connector network?

If you had something like this, where some downstream router was routing over a transit into pfsense to be routed, then yeah pfsense seeing traffic sourced from say 192.168.2.x into its 172.16.0.1/30 interface would be normal and shouldn't be reported as bogon by arpwatch.

arpwatch should have a method of listing what source networks could be seen on an interface, so not to report such traffic as bogon without having to turn off reporting of bogon completely.

It is also possible that arpwatch is looking at traffic it shouldn't be to report that. If you have an physical interface say igb0 and then you have sub interfaces for vlans igb0.10 igb0.20 etc.. arpwatch shouldn't be reporting that traffic meant for igb0.20 for example with a tag of vlan ID being seen on igb0 which it would be but tagged shouldn't be reported as bogon.

dennypage

@johnpoz said

arpwatch should have a method of listing what source networks could be seen on an interface, so not to report such traffic as bogon without having to turn off reporting of bogon completely.

It does, but it’s not exposed in the pfSense package. Also, it affects all of Arpwatch, instead of a single interface, so it wouldn’t be very useful in this case.

I recommend ANDwatch (obviously), which works on individual interfaces and should provide a clearer picture.

johnpoz

@dennypage I would recommend ANDwatch as well - and I haven't even given it a spin yet. I don't really use arpwatch any more - I stopped using it when there was an issue way back when - after it ran for a while it would lock up my pfsense.

When andwatch becomes integrated into the normal pfsense packages I will for sure give it a spin ;)

You might know - does arpwatch just listen on the parent interface in promiscuous mode and ignore tags? If so then yeah any other vlans on that parent interface could cause it to list bogons from your vlans that are tagged.

dennypage

@johnpoz said:

@dennypage I would recommend ANDwatch as well - and I haven't even given it a spin yet.

When andwatch becomes integrated into the normal pfsense packages I will for sure give it a spin ;)

I hope you’ll give it a go. Btw, if you don’t want to trust my build of the core, you can just install the package from upstream FreeBSD. The pfSense package itself (php code) is easily inspectable.

You might know - does arpwatch just listen on the parent interface in promiscuous mode and ignore tags? If so then yeah any other vlans on that parent interface could cause it to list bogons from your vlans that are tagged.

I’d have to recheck code, which I can’t do right now, but I believe Arpwatch listens on whatever interface(s) you specify. If you specifiy a parent interface then that’s what it will use. Not sure if you can spec a parent interface with pfSense however, unless you make the mistake of mixing tagged and untagged on a single interface…

johnpoz

@dennypage oh I will for sure give it a spin - and completely trust your coding.. But I normally don't install stuff that is not from the pfsense repository - once it shows up there I will be playing with it for sure.

"unless you make the mistake of mixing tagged and untagged on a single interface…"

Not sure what call that a mistake - it is a valid configuration. I have an interface that has a native network on it, and then vlans on it as well. But yeah some people are if vlans on interface, then no untagged.. But it is a valid configuration - you just need to know what your doing ;) or yeah you could mess it up.

dennypage

@johnpoz said:

"unless you make the mistake of mixing tagged and untagged on a single interface…"

Not sure what call that a mistake - it is a valid configuration. I have an interface that has a native network on it, and then vlans on it as well. But yeah some people are if vlans on interface, then no untagged.. But it is a valid configuration - you just need to know what you’re doing ;) or yeah you could mess it up.

True. However…

While mixing tagged and untagged packets on an interface is a valid configuration for pfSense, and most/all switches, I would generally call it a mistake for most folk. Mixing tagged and untagged on a single port requires a good deal of attention to detail, and has a high cost of failure. There is good opportunity for confusion, particularly with add ons such as Arpwatch, Avahi, etc.

YMMV

johnpoz

@dennypage your not wrong in that you have to understand what your doing or sure you could mess it up.. Maybe that is what is causing his flood of bogons..

But running an untagged native vlan on an interface (single network) while also having tag is valid configuration. If something like arpwatch can't run like that - then that is a problem with arpwatch. Not with configuration of native and tagged vlans on the same physical interface.

Shoot many devices would actually require that sort of connection. Many cheap smart switches do not allow you to put the management IP on a tagged vlan. Unifi not that long ago did not support management on a tagged vlan.

So if you kept tagged off an interface for management - you would need to leverage another interface for your tagged vlan traffic.

JonathanLee

On the lan interface do you have bogons enabled ??? If so that only needs to be on wan side

dennypage

@johnpoz said:

@dennypage your not wrong in that you have to understand what your doing or sure you could mess it up.. Maybe that is what is causing his flood of bogons..

But running an untagged native vlan on an interface (single network) while also having tag is valid configuration. If something like arpwatch can't run like that - then that is a problem with arpwatch. Not with configuration of native and tagged vlans on the same physical interface.

[This deserves a better write-up than I can do from a phone in the middle of nowhere, but here goes...]

It's not a problem with Arpwatch, or with ANDwatch for that matter. It is an intrinsic host OS behavior resulting from mixing tagged and untagged packets on a physical interface.

As you read, keep in mind these two points:

VLANs are tags, not encapsulations.
VLAN interfaces are virtual interfaces build on top of physical interfaces.

Tcpdump, like Arpwatch and ANDwatch, uses the pcap library, which in turn uses the BPF kernel interface. It's how user space applications get sniff packets on an interface.

Arpwatch is essentially this:

tcpdump -I <ifname> arp

So we'll use tcpdump to illustrate the behavior.

We have these three interfaces:

igc0 - native physical interface with a mix of tagged and untagged packets. The untagged traffic is VLAN 1.
igc0.2 - virtual interface derived from igc0 for VLAN 2.
igc0.3 - virtual interface derived from igc0 for VLAN 3.

What happens when you run this command?

tcpdump -I igc0.3

Just what you would intuitively expect. You receive all packets that pass through virtual interface igc0.3. These will be all the packets that are tagged as VLAN 3 that pass through interface igc0. You will have a similar experience when you run tcpdump on igc0.2.

But what happens when you run this command?

tcpdump -I igc0

Again, just what you would intuitively expect. You receive all packets that pass through the interface. However, this time it is a physical interface and there are a mix of three different VLANs running through it. So you will receive VLAN 1, VLAN 2 and VLAN 3, all together as one stream of packets. It's intuitive when you run tcpdump, but it may come as a surprise when you run other programs.

This is exactly what will happen with Arpwatch if you tell it to watch a physical interface that has a mix of VLAN traffic. It gets the whole big stream of packets. And when it receives a VLAN 2/3 packet on the physical interface, it will view it as a bogon because it doesn't match the native (untagged) interface.

It seems there is always another gotcha when you mix tagged and untagged traffic on an interface. This is one of the more esoteric ones.

It's generally okay to mix tagged and untagged on a switch, and sometimes required as you say. However, it's not a good idea with a host OS such as FreeBSD (pfSense) or Linux if you can avoid it.

@louis2, going back and reading your original post, I expect this may be what is generating your thousands of bogon reports.

dennypage

@JonathanLee said:

On the lan interface do you have bogons enabled ??? If so that only needs to be on wan side

See above - bogon means something different in Arpwatch.

You would not want to run Arpwatch on the WAN interface.

johnpoz

@dennypage said in Constant arpwatch bogon reports .... related to my own network !!??? (*strange andannoying!!*):

It's generally okay to mix tagged and untagged on a switch, and sometimes required as you say. However, it's not a good idea with a host OS such as FreeBSD (pfSense) or Linux if you can avoid it.

But if switch is connected to pfsense (firewall/host) and to manage it has to be untagged then interface on pfsense has to have untagged traffic. If that switch can not use tagged for its management.

Great write up - and nothing I didn't already understand. I am the one that brought it up ;) but seems to me that is lack of configuration on arpwatch part. Should be able to tell it hey on igb0, that also has igb0.10 and igb0.20 since your too stupid to understand that .10 network is for the .10 network only and not .20 - don't mark networks abc you see on igb0 or any of subs as bogon.

But yeah he either has his L2s not isolated, or yup he is running into this. But turning off bogon did stop the reporting from my testing of it.

So does arpwatch allow for this config, but its not exposed in the gui? Where I can tell it - hey if you see networks A,B(vlanX) or C(vlanY) on the parent interface - not to report it as bogon?

I take it andwatch allows for this - and will be able to configure it, I would think it could be auto figured out to be honest if you tell it to listen on igb0 and igb0.x and igb0.y etc..

dennypage

@johnpoz said:

but seems to me that is lack of configuration on arpwatch part. Should be able to tell it hey on igb0, that also has igb0.10 and igb0.20 since your too stupid to understand that .10 network is for the .10 network only and not .20 - don't mark networks abc you see on igb0 or any of subs as bogon.

So does arpwatch allow for this config, but its not exposed in the gui? Where I can tell it - hey if you see networks A,B(vlanX) or C(vlanY) on the parent interface - not to report it as bogon?

No, Arpwatch does not offer any vlan configuration. Arpwatch itself allows you to say net/cidr (nothing to do with vlans) should be considered local, but it is not exposed in the pfSense package.

I take it andwatch allows for this - and will be able to configure it, I would think it could be auto figured out to be honest if you tell it to listen on igb0 and igb0.x and igb0.y etc..

No, ANDwatch does not offer any vlan configuration either. ANDwatch does allow you to specify extensions of the pcap filter, so in theory you could exclude vlan tagged packets if your implementation allowed it, but this would be OS/bpf/pcap build dependent.

From my pov, best practice is to avoid the whole situation by not mixing tagged and untagged traffic on physical interfaces.

YMMV.

dennypage

Finally back from traveling and had a chance to look at mixed tagged/untagged interfaces under pfSense with ANDwatch. pcap on pfSense 25.03, and presumably 2.8.0, is in fact compiled with vlan filter support.

So, with a mixed interface configuration like this:

igc0: untagged, presumably vlan 1
igc0.2: tagged, vlan 2
igc0.3: tagged, vlan 3

You have a choice of either running a single ANDwatch instance on igc0, which will monitor all of the subnets, or running individual ANDwatch instances on each of the interfaces.

If you want to run individual instances, and have ANDwatch exclude the vlan tagged packets for interface igc0, you would set the "PCAP Filter" on igc0 to "Custom filter" and include "not vlan" in the filter.

johnpoz

@dennypage nice! I think the running 3 instances and filtering vlans on 0 (untagged traffic) would eliminate any sort of bogon because the source IP range is different than the actual network being seen on.