So what I would like to achieve is the following to allow monitoring…

rfinch23

So what I would like to achieve is the following to allow monitoring of all network traffic showing the original IP address that initiated the connection

I would like to have 3 x unifi layer 2 switches connected to two x unifi layer 3 switches with traffic being monitored by one, or two pfsense devices if required with the firewall and NAT disabled this then passing traffic to a unifi Dream Machine Pro that is connected to the internet by a adsl modem

There are four networks setup on the UDMP using vlans I.e 192.168.6.210, 192.168.8.210 etc with the default gate way pointing to the pfsense. The pfsense gateway points to the WAN interface on the pfsense that is connected to the UDMP.

Everything seems to work with just one layer 3 switch however I am finding it difficult to introduce the second layer 3 switch for redundancy.

The UDMP is handing out DHCP with the pfsense interface as the default gateway for the different networks.

So in theory a device connected to a layer 2 switch and who’s port is allocated to the correct vlan gets its IP address from the UDMP with a gateway ip address of the pfsense interface that is connected to the layer 3 switch and uses the WAN interface on the pfsense to send the traffic to the UDMP and then out to the internet via the adsl modem.

Is there a way to then add the second layer 3 switch, would I need to somehow balance to pfsense devices?

Is this even possible, is there a better way to achieve the same goal?

Thanks

Roy

JKnott

@rfinch23

PfSense includes Packet Capture, which can monitor anything passing through it. Since you have managed switches, you can use port mirroring, to monitor traffic through the switch. However, I know of no way to monitor all the traffic on a network, as the bandwidth required alone would overwhelm an interface.

Maybe you could refine your requirements a bit.

rfinch23

@jknott That’s a shame as I was hoping to use a combination of ntopng and snort on one or two pfsense devices with i5 cpu, 8GB ram, 128 GB drive and 6 intel nic’s

Roy

JKnott

@rfinch23

You have to understand how networks work. For example, pfsense can only see traffic that passes through it. This means something that passes through the switches, but not pfsense, will not be seen. Managed switches may have something, but that has nothing to do with pfsense. Even with port mirroring, you will only see the ports you mirror.

johnpoz

Why do you call it a layer 3 switch, and why do you want to.. Your routing with it?? I have your post like 3 times and can not make out what your trying to accomplish?

How about you draw up what your wanting to do..

Routing all your traffic through what exactly? Your udmp?

rfinch23

@jknott Ok I may have this completely wrong. The switches are using vlans to separate the traffic, all networks/ vlans go to the pfsense on their own interface and all networks/ vlans have their gateway set to the corresponding interface on the pfsense so all traffic over the vlans should go through pfsense other than it’s own network/ vlan subnet. This aside what I am really after is monitoring all traffic destined for the internet from all the vlans going through the pfsense outbound to the UDMP.

rfinch23

@johnpoz Hi I will put something together tomorrow and post it.

JKnott

@rfinch23

VLANs going to pfsense is not enough. The traffic must pass through a pfsense port. Also, with switches, the traffic is forwarded to it's destination port based on MAC address. So, if a packet is not already heading to pfsense, then pfsense won't see it, because the packet won't even have passed through the switch port pfsense is connected to.

JKnott

@rfinch23 said in So what I would like to achieve is the following to allow monitoring…:

This aside what I am really after is monitoring all traffic destined for the internet from all the vlans going through the pfsense outbound to the UDMP.

Then all you have to monitor is the WAN port. It will pass all traffic for the Internet, no matter what LAN or VLAN it's on.

johnpoz

Yeah I have a layer 3 capable switch, but currently its not doing any routing an only using as layer 2.. So while its a layer 3 switch, I wouldn't call that out unless it was actually routing.

Looking forward to your drawing so can see what your wanting to actually do - and then we can figure out if what you draw will do what you want, etc. It doesn't have to be fancy - a crayon and napkin and snap a pic with your phone will get the conversation started.

rfinch23

@johnpoz Hi, what I am really after is monitoring all traffic destined for the internet from all the vlans/Networks going through the pfsense outbound to the UDM and elimiate bunny hopping the 3 layer 2 switches, any vlan routing done by the layer 3 switches would be a bonous

Is there a way to then add the second layer 3 switch, would I need to somehow balance two pfsense device?
Is this even possible, is there a better way to achieve the same goal.
unifi diag new proposal.jpg

JKnott

@rfinch23

Why do you have that UDM and pfsense? One gets in the way of the other.

johnpoz

You have 3 freaking routers in this layout?? with only 4 vlans? Why??? so many??

And 2 layer 3 - doing routing?? This makes zero sense at all..still have no idea what your trying to do other than use up equipment to do nothing.. You say layer 3 and mention them doing routing, but then you have the interfaces on pfsense labeled with the 4 vlans? I still don't know what your wanting to do..

And you want an HA pair of pfsense - or just another on its own another hop?

If you want to setup HA pair of pfsense
https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html

what is the point of setting up all the HA stuff, when you then have 2 single points of failure - the edge router and then your udm are both single points that could fail..

rfinch23

@jknott you are right, but this is what we have ended up with, as mentioned the goal is to be able to better monitor the internet traffic.
So the bullet points would be:

make sure we are not bunny hopping the three bottom switches by using a pair of switches
make sure the pfsense does not become a single point of failure and can monitor all internet traffic.
we could have a hot standby UDMP id required. The UDMP handles all the unifi switches, AP's and some cameras and the WAN links
there may be a second internet connection now the UDMP can handle two WAN connection.
was just throwing this out there for some suggestion on if this is possable, if so how.

i hope this clarifies what we are trying to do.

rfinch23

@jknott the goal is to be able to better monitor the internet traffic. but still use the UDMP to manage the unifi estate that consists of Unifi switches, AP's ans some cameras.

johnpoz

@rfinch23 what is the point of pfsense? If you want to leverage UDMP? Seems like your throwing in another complexity and just something else that could fail?

And those switches you show are they part of the unifi infrastructure?

Where exactly are the AP going to connect? You are most likely going to have hairpinning or have to deal with L3 adoption if being controlled by the UDMP..

edit: Is your goal to have pfsense monitor traffic, so your not wanting to leverage DPI on the UDMP? If that is the case there is no reason to route through it.. If your going to leverage dpi on udmp - what is the point of pfsense?

rfinch23

@johnpoz yes all the switches shown are under the UDMP.

Currently there are 75 cable connection and 61 WiFi connections

Mixture of PC’s, servers, tablets ,some IoT’s, phones, intercoms and cameras.

JKnott

@rfinch23

Your network is so messed up, I missed that 3rd router @johnpoz mentioned. You need one, count 'em, one router for a network. The way you have it, you now have at least 3 single points of failure.

but still use the UDMP to manage the unifi estate that consists of Unifi switches, AP's ans some cameras.

I run the Unifi software on my main desktop system. Regardless there are other ways, such as the Unifi controller or a Raspberry Pi running Ubuntu Linux and the Unifi software. You could also run it on any Windows systems you have handy.

johnpoz

He can still leverage the UDMP to manage his unifi stuff, but there was another thread where that user was trying to route through the udm, when really there was no reason to..

Just because something can route, doesn't mean it has too ;) Just like a layer 3 switch that can do routing.. You can just use it as layer 2, or you could use it as combo where it routes some stuff but only layer 2 for other vlans.

Same goes for this udmp - you don't have to leverage its routing if you don't want too.. They don't go into this over on the unifi forums and docs - because why would anyone ever want to do that ;) heheh

Are you trying to leverage the DPI monitoring you can do with the UDMP - or are you wanting to use the pfsense to monitor. I get using the udmp for your unifi infrastructure. But if your going to leverage it at the edge - just not seeing what is the point of pfsense?

I don't believe the udmp supports lacp, so is that what your wanting leverage pfsense for? Just seems like this overly complex without any clear direction of exactly wanting to accomplish..

Since it seems pretty impossible to leverage the udmp for any sort of HA, something that has been requested for some time over on the unifi forums.. So if HA is your goal - I would just use the udmp for your cameras and other unifi infrastructure and not try and leverage its routing or dpi.. But I don't get all the desire of HA when you clearly have multiple single points of failure upstream, especially if you try and use the udmp at the edge.. If your going to be ok with spare swap box for the udmp.. Why not just have a swap switch as well - and then just take pfsense out of the picture?

rfinch23

@johnpoz This sounds good removing the UDMP from the routing and using it just as a controller. So would I then be better also offloading the DHCP to the pfsense as well? It will however be a shame loosing the DPI as a layer of protection.

If I concentrate on the pfsense would it be possible to use a vigor 130 modem and use the pfsense to authenticate to the ISP? Then turn on the pfsense firewall and NAT to control traffic and any NAT that is currently used?

How would I configure the pfsense interfaces to connect to both the top switches and support all the internal networks?

This sounds like it would all help to simplify the network. I could then look at using HA with the pfsense.

Many thanks for both your suggestions to date
As you can see from all my question I am new to the pfsense.

So really appreciate all your help.