Use of bridge and span interface for traffic analysis

darcey · Jun 4, 2022, 10:18 AM

In pfSense, I am trying to use a bridge with a single member interface and single span interface to replicate traffic to a monitoring host.
A single monitoring host is connected to the span interface. The idea then is to be able to add/remove member interfaces to/from the bridge in order to monitor traffic of interest. The monitoring host is currently an ntopng instance in an LXC container. Everything is virtualised in Proxmox.

This appears to work. If I make one of the internal LAN interfaces sole member of the bridge, I begin to see expected traffic in ntopng. However, this does not work when I set the WAN or an OpenVPN server interface as sole bridge member. Then, I see no traffic going to the Span interface. I verified this in ntopng and also using wireshark over ssh on the span interface in pfsense.

The pfSense WAN interface is a PPPOE device and the VPN interface is one explicitly assigned to a VPN server instance.

I wondered why I would not see traffic mirrored when I set either of those interfaces as the bridge member.

stephenw10 · Jun 5, 2022, 5:05 AM

Hmm, I'm not sure I've ever tried that with a PPPoE interface. I'd expect it to work with OpenVPN though since that's commonly used with bridges for TAP mode.
Maybe an MTU mismatch prevents it? Though the span port should always be higher so you wouldn't expect that to be an issue.

Steve

darcey · Jun 5, 2022, 5:05 AM

@stephenw10 Thanks for the sanity check. I will have another look.

darcey · Jun 5, 2022, 10:16 AM

It seems the OpenVPN interface may need to be tap based in order to function as a bridge member. At least that's what the docs say when it comes to bridging LAN and VPN clients. Then I guess PPPoE WAN interface presents other issues. I'm not too concerned as I am primarily interested in watching a couple of other regular interfaces. One thing I have noticed is ntopng (in container not pfsense) consumes rather more cpu than I expected, given the moderate volume of traffic that is passing the SPAN interface.

stephenw10 · Jun 5, 2022, 1:33 PM

Mmm, interesting. I have seen issues with bridging disimilar NIC types before. Generally you can only bridge NICs of the same type but I've never looked into exactly what the restrictions are there. The bridge man page does specify the MTU must be the same for all bridge members.
I know that you can bridge wifi NICs to Ethernet for example which are significantly different.
Also OpenVPN TUN interfaces use MTU 1500.

Can you bridge to the PPPoE parent NIC?

Steve

darcey · Jun 5, 2022, 2:41 PM

@stephenw10 said in Use of bridge and span interface for traffic analysis:

Mmm, interesting. I have seen issues with bridging disimilar NIC types before. Generally you can only bridge NICs of the same type but I've never looked into exactly what the restrictions are there. The bridge man page does specify the MTU must be the same for all bridge members.
I know that you can bridge wifi NICs to Ethernet for example which are significantly different.
Also OpenVPN TUN interfaces use MTU 1500.

Can you bridge to the PPPoE parent NIC?s

I had not tried that. I'd have to assign an interface to the parent as it's currently unassigned (vtnet1).
MTU is 1492 for pppoe0.
I am guessing it will not be a compatability issue between member interfaces as I'm only ever assigning one member interface at a time. The SPAN port remains constant. All the vtnet0...vtnetn hardware presented to pfsense is underlain by bridges on the proxmox host. Some of those bridges have vlans associated with them.

auto vmbr1
iface vmbr1 inet manual
        bridge-ports bond0.40
        bridge-stp off
        bridge-fd 0
#WAN

auto vmbr2
iface vmbr2 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
#DMZ

auto vmbr6
iface vmbr6 inet manual
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        bridge-ageing 0
#Span port

stephenw10 · Jun 5, 2022, 2:44 PM

Mmm, bridges in Proxmox should not make any difference. pfSense cannot see those, only the vtnet NIC it's presented with.
You might be better off spanning the ports in Proxmox though. I've never tried that. You wouldn't see the traffic inside PPPoE of course.

Steve

darcey · Jun 5, 2022, 2:51 PM

@stephenw10 said in Use of bridge and span interface for traffic analysis:

Mmm, bridges in Proxmox should not make any difference. pfSense cannot see those, only the vtnet NIC it's presented with.
You might be better off spanning the ports in Proxmox though. I've never tried that. You wouldn't see the traffic inside PPPoE of course.

That was my original plan. I set bridge aging to zero and connected an lxc monitoring host to the same proxmox bridge. However that seems inefficient once you have several or more guests on same bridge. A span port seems more efficient and more like a real switch.
I'm reading up on ovs vs tradional bridge/tc methods. Both are somewhat harder to understand but hope to get there eventually!

darcey · Jun 5, 2022, 3:00 PM

@stephenw10 said in Use of bridge and span interface for traffic analysis:

You might be better off spanning the ports in Proxmox though. I've never tried that. You wouldn't see the traffic inside PPPoE of course.

I can see it on the PVE host with tcpdump -i vmbr1 -U -s0 -w - pppoes. But whether it's straightforward to see it with an attached network analysis guest, I have not tried yet.