local LAN traffic blocked
I've been using pfSense for several years with no major issues. Made some changes in my environment yesterday and am now seeing some traffic being blocked between local devices on my LAN.
2.4.5-RELEASE (amd64)/FreeBSD 11.3-STABLE. Running pfBlockerNG and Suricata. Only 2 interfaces - WAN and LAN. pfSense is handling DHCP for the LAN.
Prior to yesterday my physical configuration was -
Internet -> pfSense -> wifi router -> LAN switch - pfSense was only seeing traffic that had been NAT'd through the wifi router - only 1 IP. No issues at all.
Several things happened yesterday. Upgraded Internet bandwidth and moved from DHCP supplied WAN addressing to static WAN IP. Also dropped old wifi router and installed new Ubiquiti AP. Physical layout is now-
Internet -> pfSense -> LAN switch with AP connected - going to be changing but currently 1 flat network, 1 subnet for family on the LAN side.
Changed the WAN and LAN addressing on the pfSense box. Everything was working when I initially switched over. As far as I can tell now, everything is working with 1 exception - I have several wifi cameras and a machine running DVR software (all currently on a single LAN subnet) that are being blocked from communicating by the pfSense firewall. I noticed there was a period of where the camera feed was displaying and where it seemed to randomly drop. Now, none of the cameras are getting video feeds to the software. You can ping them from the DVR machine, access their web UI for settings/etc, but the video feed is being blocked. pfSense firewall shows camera-IP to DVRPC-ip:port as blocked with "The rule that triggered this action is: @9(10000000103) block drop in log inet all label "Default deny rule IPv4""
I have all sorts of things on the LAN behind the pfsense box - Cisco ASA with PTP VPN to another location (with other devices/different subnet behind it), family phones/ PCs/printers. EVERYTHING (work, games, kids remote school work and connections) seems to be working with the exception of the camera feeds.
I have tried created an alias with all camera IPs, created 2 allow rules stating all IPv4 traffic between cams and DVR-PC is allowed. I have created "easy rules" from the instances in the logs. Moved all these rules to the top of the LAN rule list. Stopped Suricata service. No changes.
Any ideas on what I'm missing/what's causing this behavior?
Thanks in advance
am now seeing some traffic being blocked between local devices on my LAN
PfSense cannot affect traffic that's entirely on the local LAN. It only affects traffic that passes through it. The only thing pfSense provides for the local LAN is DHCP and DNS servers.
I agree theoretically BUT, I have 1 cable connecting the pfsense box to the local network, one to the WAN. No LAN traffic should really be passing through the pfSense box. Based on that, I originally suspected the new AP - checked and all is well. BUT, camera feeds aren't working and I can see the traffic being blocked in the firewall logs - X date LAN Internal IP Internal IP:port .
Where is that traffic trying to go? You could completely disconnect pfSense from the network and it would have no effect on local traffic.
I agree with what both of you are saying, that is why I'm having issues resolving this. I understand how it is supposed to work but that is not what I am seeing. Again, I've been using pfSense before netgate ever took over. Here's the layout again-
ISP router - cable - WAN_pfsense_LAN - cable - LAN switch - all other devices (excluding devices behind inner fw)
cameras and DVR_PC exist on 192.168.111.x/24. I can ping from cameras to DVR_PC. I can ping from DVR_PC to cameras. DVR_PC is receiving no feeds from cameras. The pfSense firewall logs are filled with -
x Apr 13 14:27:52 LAN 192.168.111.104:80 192.168.111.73:60015 TCP:SA
x Apr 13 14:27:50 LAN 192.168.111.103:80 192.168.111.73:60017 TCP:SA
x Apr 13 14:27:45 LAN 192.168.111.103:80 192.168.111.73:60017 TCP:SA
x Apr 13 14:27:44 LAN 192.168.111.103:80 192.168.111.73:60013 TCP:SA
x Apr 13 14:27:44 LAN 192.168.111.104:80 192.168.111.73:60015 TCP:SA
x Apr 13 14:27:42 LAN 192.168.111.103:80 192.168.111.73:60017 TCP:SA
x Apr 13 14:27:40 LAN 192.168.111.103:80 192.168.111.73:60017 TCP:SA
x Apr 13 14:27:39 LAN 192.168.111.104:80 192.168.111.73:60015 TCP:SA
.100-.107 are cameras. .73 is the DVR PC. Prior to yesterday, everythign worked fine. After the cutover yesterday, cameras started falling out of view. Now, none connect and the firewall logs are full of blocked connections between the cameras and the DVR_PC. I have no idea why these are even being seen/acted upon by the firewall.
I have added these rules to the top of the LAN fw rules and they appear to have no effect.
IPv4 * cams * 192.168.111.73 * * none
IPv4 * 192.168.111.73 * cams * * none
That is asymmetrical traffic.. You have a mask wrong would be my guess.. .73 sent syn to .103, .104, They said hey thats not on my network... Send to my gateway, pfsense said F off, I have no state for that traffic.. So your Syn,Ack is blocked!
If these devices were actually on the same network, pfsense would never see it.
(excluding devices behind inner fw)
So you have some downstream firewall/router?
As to mask, if you had say a /27 or higher on the .103 and .104 boxes your .73 would be outside its network.
Entirely possible I have fat-fingered something :) but I've attempted to double-check everything. I will be more than happy if it is that simple. Checking.
See my edit, say a /27 or /28 etc.. on the .103 or .104 boxes would put .73 outside their networks. So traffic would be sent to its gateway.. (pfsense)
Just checked one - can only intermittently reach over network to the web UI. Manually plugged into the eth port (as opposed to wifi) to connect and take a look. Mask was correct - /24, but I had also changed each camera to point to a non-existent GW setting so that video stream could not route outside. Changed that (GW) back and it connected immediately. Should not have made a difference as cameras and DVR_PC are on same subnet. Will check the rest to confirm but assume that is it as I'd changed GW on all.
Can you not just leave the gateway blank? If gateway was wrong, how would it ever have sent data to pfsense at all?
Clearly missing piece of the puzzle..
I agree 100%. Have never seen this type of behavior. Blank is what I normally do but these cameras seem to validate the entries for each octet to be between 1-254, and for the entire ip to match subnet, so I couldn't make them blank. I'll create a fw rule instead.
That was it. Still doesn't make sense to me but I changed the GW back to the valid GW for each camera and they started communicating correctly again. Sorry for the confusing post but I was stumped. Couldn't understand (and still don't) why a block was showing in the pfSense fw logs for traffic that shoudl never pass through the fw. It is working now. Thanks for having me take another look.
I expect that gateway was elsewhere, along with misconfigured address, etc. caused packets to try to leave the network.
Makes no sense at all.. There would be no way the device would of sent traffic to pfsense at all, if its wasn't the gateway..
And if the devices on the same network per their masks, there would be no reason to send traffic to the gateway, no matter what it was, etc.
Clearly some piece of the puzzle is missing.
Only thing that would make sense is the devices for whatever reason thought that pfsense IP was the .73 box.. Can you look in their arp table?
I guess no one believes the masks were correct :) but I can confirm the mask was correct on each - not even a single fat-finger. These cameras have been running for over a year and the subnet they sit on hasn't changed at all. The only changes I made were to the gateway settings hoping to add another layer of difficulty to exporting camera data out of the subnet. Didn't even change the wifi settings, just migrated the SSID/encryption/etc to the new AP. The only thing I changed to restore functionality was the gateway from a non-existent .12 to the correct host.
After having to manually attach to and change each, I will be creating DHCP reservations so I never have to do that again in the future :)
These are pretty good but relatively basic cameras (Amcrest) and I don't see any CLI access in the webUI to check arp tables. Don't see any ssh settings either and could't reach with ssh/putty.
I believe you - but this is why some info is missing that would explain what you were seeing.
If two devices on the same network, they would never send any traffic to the gateway. Pfsense was blocking out of state traffic SA. So for whatever reason these devices on .103 and .104 were sending their syn,ack to pfsense.
So either the device thought that .73 was not on its network, and would need to send to its gateway, but you stated that pfsense was not even the gateway, it was some other IP..
The only thing that really makes any sense was that these devices thought that the .73 they were sending their SA too was pfsense mac address. The question is how or why?
Again, I agree with what you're saying :) pfSense is the valid gateway, but they were not pointing to it. These cameras have always pointed to .73. All of the involved devices have always lived on the 192.168.111.x/24 network. The pfSense ip has always been the gw for that subnet although pfSense just moved there - pfSense was originally the gw on an outer subnet. Before and after the cutover, the cameras and the DVR_PC could continue to ping each other by IP, so they had each other's MAC resolved and the path to each was known - no hops in between. I had even reset the switch these connect to earlier today because I thought something may have been wrong/bad/corrupted in it's MAC tables. There was never a point where the devices failed to ping each other.
I did change the .73 host from a static IP of .73 to a DHCP reservation for it's MAC so that pfSense now hands it .73 but that shouldn't have affected anything either.
Changing from a wrong gw on the cameras to the correct one is the only change I made to resolve what I was seeing. I'm sure there is a piece that I don't see that would give us, as Paul Harvey used to say, the rest of the story. I will be making other changes to this network and it is driving me nuts not seeing the missing link so I can say I know what was happening.
Thanks for all of the responses. Keeps me on my toes.