Packet Capture not seeing traffic within internal network communication

Just.Enuff

Hello,

This post is a bit scatter brain as I have been trying to accomplish new pfsense install and unraid segmentation for sometime and wrapping my head around how to accomplish this and learning it all has been trying for sure. So please bare with me here and i apologize if wrong forum or for what may be lack of understanding here.

Below this question I have posted my setup and such if you would like more insight into what i am doing and how i arrived here.

My current problem.

I am trying to packet capture on my networks in understanding current communication happening so that I can create rules and such going forward. However I believe i am not seeing all data in the capture and cannot understand why.

It started with Unraid SMB not working when server was on segmented / separate network (OPT1). Now realizing cannot see the traffic even on same network (LAN).

Lan network capture problem.

On this network have two computers.

Laptop = Comp #1 IP address 192.168.7.10
Unraid = Comp #2 IP address 192.168.7.20

Utilizing Comp #1 I start packet capture on LAN in pfsense, then I open another tab on Comp #1 and login to Comp #2 web gui unraid admin panel. From Unraid admin panel i start array, run dockers, etc... Then i go into the share directories from Comp #1 utilizing local computer file manager. Everything works as it should and then I stop the capture.

When I check the capture i see nothing listed for Comp #2 IP when in comes to communicated to Comp #1. I do not see any information on how comp #1 talked to comp #2 at all. Why? I do capture random traffic such as NTP lookup or similar on Comp #2, but not cross talk with #1 and #2

I have tested this quite a bit and the only time I can see traffic from Comp #1 to Comp #2 is when I run Plex docker in Unraid. If I run Plex it will show Comp #2 is communicating to Comp #1 via udp with IP 192.168.7.255 . This does not seem to be constant communication as it only logs this traffic intermittently.

I have also checked system logs firewall and it gives me same information as packet capture, no more or less. Its a bit baffling to me and makes me feel more over my head each minute goes by.

More information below if you please.

I am fairly new to Pfsense and networking in general, so I ask to please go easy on me. I am in the process of setting up my network and before doing so I am playing around with pfsense to understand its capabilities more before i run everything through it. My setup will probably basic to most, but fairly complex to me at the moment.

My Setup.

Computer running latest Pfsense with quad nic card and currently only utilizing three ports. WAN, LAN, OPT1

WAN communicating to ISP Modem.

LAN communicating to wireless router to which most things are on. Router is in access point mode and appears to pass all information to PFsense in the way of listing the actual IP's, etc... So not double NAT Per say ( Still new, may not be exact term to use here)

Opt1 has Unraid server in trail mode at the moment. I have taken on testing it as well, which may be more than i should be doing right now, but i jump head first at times.

What am i trying to do?

Ultimately i would like to have LAN for desktops / laptops and OPT1 for unraid and Internet of things (IOT).

Current Progress.

Pfsense is running, all computers on LAN connect and work as currently requested of them.

Unraid on OPT1 does not work as requested of it and currently troubleshooting why.

More Details on Current Issues

Unraid being biggest issue at this point. If I connect Unraid directly into LAN network via the router, all computers on LAN network can see and connected to allowed shares via SMB. But if i put Unraid on OPT1 network, computers on LAN cannot see the shares at all, however I am still able to login to Unraid WEB GUI admin from a computer on the LAN Network.

I understand that LAN and OPT1 are network segmented and will need to have firewall rules, routing or bridging to connect them, but yet cannot wrap my head around this and why default i can login to unraid web gui admin panel without rules, routes or bridges.

This had lead me to understanding the communication going between the networks and trying to read firewall logs and capture packets.

Things that have worked, but lead to more questions

When I have unraid on OPT1 and Laptop on LAN. I can access the shares if I set up a default setting bridge in pfsense admin under interface / assignments / bridges. This allows access, but leaves some concern as without fully understanding bridges the scenario seems to be possibly defeating the purposes of segmented separate networks LAN & OPT1. Would bridging be similar to just having the Unraid server on the same network (LAN) any how? what would the benefit of segmented network only to connect back via bridge, if I could just keep them on same LAN network via switch? Maybe i am misunderstanding bridge and this could be the correct way, just uncertainty for me and creating more questions.

JKnott

@just-enuff

If you're trying to capture traffic between two devices on your LAN, Packet Capture will never see it. It can only capture traffic that passes through it and that doesn't happen within the same LAN. You'll have to create a Data Tap from a managed switch and place it between one of the devices and your switch

Gertjan

..... or, if you can find one, get a so called hub .

A hub is far more stupid as a switch : it repeats on all its ports what comes in on a port. Just perfect to eavesdrop on your network.

Gertjan

@just-enuff said in Packet Capture not seeing traffic within internal network communication:

But if i put Unraid on OPT1 network, computers on LAN cannot see the shares at all, however I am still able to login to Unraid WEB GUI admin from a computer on the LAN Network.

Totally known and normal.

What you describe as "see" is which devices are known as in its "network neighbourhood" (a Microsoft term ?).
With a Microsoft PC, or actually : any OS - shows you what's on it's LAN segment, like 192.168.1.1 to 192.168.1.254. These OS's will not try to go across routers as they can't and don't know what behind a router. Your explorer shows you the devices that it can detect 'locally'.
If you were able to see devices behind a router, your explorer would freeze a while (a couple of years, probably) and then finally show you :
All the devices it find and recognize on that other network port called WAN : the entire Internet (please, please, take a screen shot !!!!).
As it will take some time to enumerate from 0.0.0.0 to 255.255.255.255.

Well, consider this : your "LAN" network is your "network neighbourhood".
If you have another LAN (or WAN, at least) 'behind' your router, then that is another "network neighbourhood".

As you have found out, you can still connect to these devices using their host name like my-ap.localdomain-athome.lan or 192.168.2.4 using the IP.
I guess you understand that your router, pfSese is also a firewall, so, if you want to connect FROM a device on your LAN TO a device on your other LAN (OPT1), then a pfSense firewall rule on your LAN must make this possible. The default LAN firewall does this.

JKnott

@gertjan said in Packet Capture not seeing traffic within internal network communication:

Just perfect to eavesdrop on your network.

And kill your performance. Most hubs are 10 Mb. 100 Mb hubs are rare. Then it forces half duplex, killing performance even more. A 5 port managed switch is cheap these days.

Just.Enuff

@jknott said in Packet Capture not seeing traffic within internal network communication:

@just-enuff

If you're trying to capture traffic between two devices on your LAN, Packet Capture will never see it. It can only capture traffic that passes through it and that doesn't happen within the same LAN. You'll have to create a Data Tap from a managed switch and place it between one of the devices and your switch

Well that explains that, so much time wasted on the impossible, lol. Appreciate your response as that was driving me crazy. Though it does make me wonder why I cannot capture any information if i segment LAN from OPT1. If I packet capture on OPT1 and communicate to LAN, Doesn't information now pass through OPT1? OR?? does it only capture packets destined for what is has some how determined to be outside local networks? If this the case, what triggers that knowledge something in the request or tables?

I do have a switch that should be capable of the "data tap" but i was hoping to not incorporate more hardware and learning at this point as i am already somewhat over my head with this. I have read your first post from the link and its a little confusing to me, through i grasp why it will work.

I now have to convince myself this is the direction i need to take to fully understand the communication happening. Right now I cannot picture the cross talk happening and its blocking me mentally from understanding networking.

Thanks for the info and the push in the right direction, I suspect i will be back with more questions for you on this.

Just.Enuff

@gertjan said in Packet Capture not seeing traffic within internal network communication:

@just-enuff said in Packet Capture not seeing traffic within internal network communication:

But if i put Unraid on OPT1 network, computers on LAN cannot see the shares at all, however I am still able to login to Unraid WEB GUI admin from a computer on the LAN Network.

Totally known and normal.

What you describe as "see" is which devices are known as in its "network neighbourhood" (a Microsoft term ?).
With a Microsoft PC, or actually : any OS - shows you what's on it's LAN segment, like 192.168.1.1 to 192.168.1.254. These OS's will not try to go across routers as they can't and don't know what behind a router. Your explorer shows you the devices that it can detect 'locally'.
If you were able to see devices behind a router, your explorer would freeze a while (a couple of years, probably) and then finally show you :
All the devices it find and recognize on that other network port called WAN : the entire Internet (please, please, take a screen shot !!!!).
As it will take some time to enumerate from 0.0.0.0 to 255.255.255.255.

Well, consider this : your "LAN" network is your "network neighbourhood".
If you have another LAN (or WAN, at least) 'behind' your router, then that is another "network neighbourhood".

As you have found out, you can still connect to these devices using their host name like my-ap.localdomain-athome.lan or 192.168.2.4 using the IP.
I guess you understand that your router, pfSese is also a firewall, so, if you want to connect FROM a device on your LAN TO a device on your other LAN (OPT1), then a pfSense firewall rule on your LAN must make this possible. The default LAN firewall does this.

Yes, makes sense that my computer would not try to index / connect other "network neighborhoods" automatically. But cannot understand how to tell my computer that their is one that exist on another near by neighboring network. With how i am understanding your logic it makes me feel as if I can login to admin GUI on OPT1 lan because I am specifically stating the ip and port to do so on LAN computer, so it lets it through. If that is the case, why can i just not connect to share server via file manager on ubuntu with no more firewall rules in pfsense? Is it because default pfsense rules lets only HTTP ports through and not smb?

johnpoz

@just-enuff said in Packet Capture not seeing traffic within internal network communication:

Though it does make me wonder why I cannot capture any information if i segment LAN from OPT1.

That is not right - if device A is on lan, and device B is on opt1 - then for sure you would HAVE to see traffic between them on pfsense.. Not possible to not see it. So either your not sniffing the correct interface? Or your devices are not on different segments like you think - or something else is routing between them..

As to your rules question? Lan defaults to any any rules.. What rules did you put on Opt1 interface - and what direction are you trying to create the connection?

If your server with the shares are on opt1, and your on lan - and you didn't change the default rules you would for sure be able to access anything on opt1 network. Unless its running a firewall on the device which blocks it, or the device is using something other than opt1 address of pfsense for its gateway? Or doesn't have a gateway set, etc.

If you could draw up your network - even if on a napkin, and give some details of your networks, lan is 192.168.1/24 opt is 192.168.2/24 ?? example

And post up screenshots of your rules you have on lan and opt1 - we would be more able to help you figure out your problems.

But no - you are not going to be able to browse for some device on opt1 from lan with any sort of discovery protocols, like windows network neighborhood, etc.

Back to this

cannot capture any information if i segment LAN from OPT1.

If your opt1 rules do not allow the traffic, and your sniffing on the lan interface of pfsense - then no you woudn't see anything.. Since you didn't allow it on opt1.. If you sniffed on opt1 you would see the traffic trying to go to lan, be it you allow it or not.

Gertjan

@jknott said in Packet Capture not seeing traffic within internal network communication:

And kill your performance. Most hubs are 10 Mb. 100 Mb hubs are rare. Then it forces half duplex, killing performance even more.

I was mentioning the advantages.
There are also reasons, real disadvantages, that explain why hubs are "not preferred" anymore.

@jknott said in Packet Capture not seeing traffic within internal network communication:

A 5 port managed switch is cheap these days.

and the spying part on LAN using pfSense ? That's not gona happen with a switch.

@johnpoz said in Packet Capture not seeing traffic within internal network communication:

Not possible to not see it.

Crosstalk ?
(Ok, I'm out)

johnpoz

@gertjan said in Packet Capture not seeing traffic within internal network communication:

Crosstalk ?

Then he is not setup like he thinks he is.. But sure for all we know he has 1 flat network running multiple layer 3 on the same L2, and yeah they are just talking to each other directly..

My guess is he put no rules on opt, and sniffed on lan - so yeah no traffic in a sniff.

Just.Enuff

thanks to @JKnott I now know that i cannot capture packets within the same network since traffic is not passing through the interface i am monitoring. That never dawned on me prior as i pictured everything always passing through the gateway on the LAN.

So in my head I saw it sort of like this with both computers on same LAN / network.

gateway? / pfsense = 192.168.1.1
Laptop comp #1 ip = 192.168.1.10
unraid comp #2 ip = 192.168.1.20

All of that being on LAN. So if i capture packets on the LAN interface in Pfsense wouldn't it do it from the gateway perspective with ip of 192.168.1.1?

If it did it through that perspective then comp #1 passes through gateway? (192.168.1.1) to communicate with comp#2

Then something should be logged/ Captured, Yet it wasn't ??

So, that was wrong and for reasons i don't completely understand yet. Cause the information isn't being passed like that on same network to be monitored. I am guessing that since the computers are on the same network they just talk directly to one another without passing through the gateway per-say on a data level.

I would think that at some point though, maybe periodically or at beginning of communication something can be logged between the two to indicate they exist to one another, its just that was not the information i was looking for at the time.

Learning this was key to me moving forward along with @johnpoz reassurance that i should have seen communication when i did have my networks segmented. This cemented that i was thinking correctly as my last question to @JKnott was directed at why that packet capture scenario didn't work for me earlier as that surely should involve passing of information on the network in the capture.

So today to reinforce this i hooked my systems back up to be segmented like i had done before to find out why it wasn't working if it should have

So to help picture the setup. will paint the basic setup.

(PREFERRED DESIGN)
Laptop comp #1 on LAN = ip 192.168.1.10
Unraid comp #2 on OPT1 = ip 192.168.2.20

In this scenario before i could not connect to comp#2 from comp#1 in the normal file sharing way. I also was not logging any information to indicate why it wouldn't connect in the system firewall logs or via Packet Capture. But i could always connect to comp#2 from comp#1 with web browser and admin panels. I attributed this to the possibility of maybe port 80/443 being open for web traffic by default in pfsense?

What this scenario really did was drive me more to troubleshooting on same network to watch what communication was allowed to happen to try and figure out why it was being blocked on segmented networks. Truth be told, i put more effort here as i knew the systems were communicated correctly and what better way to learn then to see some actually in action. Problem with that is you cannot see that with pfsense as i have learned in this thread. Back to solving this .

So today I focused on PREFERRED DESIGN as all things in the thread indicate it should work and also communication should be able to be captured. This now makes is better for troubleshooting as i can see the data without creating @JKnott "DATA TAP". But why wouldn't it work if what @johnpoz suggested was true and that default settings should allow my devices to communicate Comp#1 to Comp#2 and vice versa .

The ANSWER

My problem turned out to be nothing to do with PFSense. It was the i had been connected to a VPN. The VPN was not allowing local communication to happen. Why could i access the admin panels via HTTP browser? Split tunneling is why.

So turned off VPN open up file browser and cannot see the network of course. It will not automatically try and index the world of networks out there. So i tell it to connect via SMB and wooolaaaaa... It works. I can also capture the packets on OPT1. What a feeling. I have been working on this on and off for weeks, not kidding. Makes my brain feel lighter actually. Many thanks to @Gertjan, @JKnott & @johnpoz that helped me here in this thread!

New Problems Arise

My celebrations are short lived. In hindsight I should have soaked that award in longer before bogging down again. Being so excited to move forward for first time in awhile i now need to understand how LAN and OPT1 communicate and i want to hone in on locking it up and not just some wide door open. So first I want to break it, create rules so hard it doesn't work and then open it up again and see it flow.

Well turns out i can't do that just yet. No matter what firewall rules i put on OPT1, I cannot stop Comp#1 on the LAN from SMBing the files from Comp#2 on OPT1.

This is the exact opposite problem from before. I can now connect and cannot block it. Hrmmm.. initial investigation is leaning me towards default "let out anything from firewall host itself" but not sure. Its quite a learning process for me, but i slowly move forward thanks to all your help. I will post back if i cannot solve this, but likely in another thread if no response to this here.

Once again, many thanks. You have no idea how good it is for you to help me move from the above problem. Have a good day!

johnpoz

@just-enuff said in Packet Capture not seeing traffic within internal network communication:

Well turns out i can't do that just yet. No matter what firewall rules i put on OPT1, I cannot stop Comp#1 on the LAN from SMBing the files from Comp#2 on OPT1.

If you want to stop lan from talking to opt, the rules would be on LAN, not opt.

Firewall rules are evaluated as traffic enters an interface from the network attached too..

lan ---> lan (pfsense) opt ---> opt

If you want to stop lan from going to opt, rules would be on lan... If you want to stop opt from going to lan, rules would be on opt.

Top down, first rule to trigger wins, no other rules are evaluated.

Rules are evaluated inbound to an interface, not outbound. While you can do outbound rules, those would have to be done in floating. And are for another day, once you have figured out the basics..

Also keep in mind that if you create traffic from lan to opt, a state is created. If you then create a rule that blocks lan from opt.. You would have to make sure the state is no longer there. Before that block rule would be effective. States are evaluated before rules.

So if there is an existing state for the traffic, creating a rule to block will no work until that state has timed out and goes away, or you remove the state by hand via the state table. Or the states are reset, be by hand or say a reboot, etc.

Gertjan

@johnpoz
Add two lines.
And carve it into stone.

JKnott

@just-enuff said in Packet Capture not seeing traffic within internal network communication:

All of that being on LAN. So if i capture packets on the LAN interface in Pfsense wouldn't it do it from the gateway perspective with ip of 192.168.1.1?

No. Read up on how switches work. Switches rely on the MAC address to forward the frame to the appropriate port. So, if A sends to B, pfSense on C will not even see it, let alone capture it.