pfSense keeps blocking access from one subnet to another most of the time (but not always)
-
Thank you @johnpoz for clarifying with the rule order, I heard many times that the rules work from top to bottom I guess I'm too dumb to understand it practically hahahah )) Placed deny rules for IP cams on top.
About my pfSense problem:
@johnpoz said in pfSense keeps blocking access from one subnet to another most of the time (but not always):
If you can not ping your nas on opt1, then you have firewall running on the nas, is not using pfsense as its gateway.. Or its mask is wrong, or its multihomed?
No, it's not about NAS, right now I have three devices turned on on OPT1 -- TrueNAS on 10.0.74.12, NAS on 10.0.74.11 and iMac on 10.0.74.14 -- neither of them is accessible while I'm pinging them on my macbook on wifi via its terminal. Both subnets are /24, 255.255.255.0.
We go over this like every other day it seems.. Your saying from the lan, you can not even ping pfsense IP on your opt1 interface?
When I'm on LAN, I can't ping anything on OPT1 EXCEPT pfSense -- it is on 10.0.73.1 (LAN) and 10.0.74.1 -- I can ping 10.0.74.1 when I'm on 10.0.73.xx and I can access 10.0.74.1 on web interface, but other 10.0.74.xx devices are unreachable. And -- when I log on to pfSense via web interface and go to Diagnostics > Ping, and try to ping 10.0.74.12 (TrueNAS), 10.0.74.11 (Syno) and 10.0.74.14 (iMac) -- 100% packet loss. Again, this applies only when my Macbook is on LAN (wifi/wired -- doesn't matter).
Is your client on lan your pinging from maybe on wifi and wire at the same time? Is it running some security software that would stop you from pinging stuff?
No, either Wi-Fi either eternet (when trying to reach devices on OPT1 while on LAN)
Did you set its IP up, maybe you have a mask wrong?
My laptop's IP varies -- static on WiFi (DHCP Static based on MAC address), random on wired LAN (DHCP), static on OPT1, but, as I said, this whole issue goes away when I work wired to OPT1.
Do you maybe have some wifi router with a leg in lan that might have its dhcp server still running?
No, my Tenda Mesh Wi-Fi is in bridge mode, all routing/IP stuff is done on pfSense.
Thank you very much! ))
-
Forgot: there are no deny rules on OPT1, only allowing ones )
-
Dang! I connected my macbook to OPT1 by wire, checked that I can access everything on OPT1 (as expected). Disconnected the cable, went back on LAN Wi-Fi and now I can reach OPT1! It could be something with my laptop but it's not! When I had Mac mini wired to LAN it too couldn't get to OPT1 most_of_the_time and how on earth this situation could affect Diagnostics > Ping from web interface? If I'm logged on to pfSense Web GUI that means I'm inside the router and it should ping anything, but no, it can't ping OPT1 devices that "most of the time" but now, when I plugged my Macbook to OPT1 for a few seconds and disconnected -- everything works! I'm no IT specialist but I think this situation can be considered stupidly weird by an IT specialist either ))
-
@throttlenerd so question are you using say the new ethernet rules? Or captive portal?
have you setup any static arp entries?
I'm inside the router and it should ping anything, but no, it can't ping OPT1 devices that "most of the time"
This is true - from the firewall you should be able to ping anything connected to networks pfsense is attached too. Reasons why you might not be able to, firewall rules on the device your trying to ping. Not able to arp for the devices IP, or wrong mac in the arp table for that IP..
I was thinking what could cause interment sort of issues, invalid mac comes to mind, duplicate IP where you have the correct mac for what your wanting to talk to, and then other times have wrong one, etc..
There is has been some ongoing issues where if you set a static mac it goes away and looks like just dynamic mac, and then can set back to static on restart of setting the static arp, etc.
-
@johnpoz All devices on OPT1 have static IPs and they are set up in those devices' settings, no MAC-based static leases. I have MAC-based static leases on LAN interface -- for my laptop, mobile phones, Wi-Fi AP (3-unit mesh). Today I figured out the problem goes away if I (by "I" I mean my laptop) connect to OPT1 by wire and then unplug it and connect to LAN wirelessly -- everything is accessible. But since "most of the time" this issue is "live" -- I will ssh into my other NAS which is on LAN (Surveillance, Plex) and will try to reach my work NAS on OPT1 -- it shouldn't be able to ping it even though right now it can, because I was on OPT1 a few minutes ago. Yes yes this is weeeeird. Will keep you posted thank you very much for helping me! Happy New Year! ))
-
So I waited a bit, now the issue is alive again -- no LAN to OPT1 access. Right now I'm (laptop) on Wi-Fi. I turned on SSH on my other Syno NAS which is on LAN. I open Terminal on my Mac, ssh into 10.0.73.12 (LAN NAS), pinging 10.0.74.1 (pfSense on OPT1) -- all good. Pinging two devices on OPT1 -- no luck. But I was able to ping them both from my mac and/or LAN NAS just after disconnecting my laptop from OPT1. So, ehmmmm. Still more questions than answers ahaha )
-
@throttlenerd you sure your actually isolated at layer 2?
You shouldn't be seeing the mac of the devices if your on another L2 network.. I would plug your laptop back into opt network.. ping your devices, now you say you can ping them while your on lan? Look in your mac address table, a simple arp -a should show only the stuff that is on your current network.. You should no longer see mac from devices on opt network.
-
But I was able to ping them both from my mac and/or LAN NAS
Oh by saying "mac" I meant my Macintosh laptop )) The only time I dealt with MAC addresses was when I assigned static DHCP leases to some devices on LAN interface.
Yes, arp -a while I'm on LAN shows only 10.0.73.xx (LAN) devices, no OPT1
-
And! Just checked another aspect which proves this black magic issue goes beyond anything related to my laptop and/or MAC addresses. My "work" Synology NAS which is on OPT1 has a Synology QuickConnect feature, so I can safely log into it while I'm out of my home network, i.e. anywhere in the world (no port forwarding/open ports, it works via Synology relay server). So, "most of the time", as this problem is "active" I can't QuickConnect to my OPT1 NAS via my phone DS Files app. I made a quick experiment just now:
The laptop was disconnected from OPT1 for a while, opened DS Files on my iPhone -- app says my Quick Connect ID NAS is unreachable. Connected laptop to OPT1 by wire -- QuickConnect still doesn't work. Unplugged ethernet from my laptop, went on Wi-Fi -- boom/voila -- I can log into my NAS via QuickConnect with DS Files app on my phone no matter how it connects to the internet, via home Wi-Fi or LTE (Wi-Fi off on the phone). Ehhhhh!!! What could it be ahah
-
@throttlenerd I would look to what you nas for example has in his arp table when not working, and then when working.
-
@johnpoz said in pfSense keeps blocking access from one subnet to another most of the time (but not always):
@throttlenerd I would look to what you nas for example has in his arp table when not working, and then when working.
Hi @johnpoz, seems like I can't get an arp table of my OPT1 NAS "when not working", because I can get into it either by OPT1 wire or Wi-Fi (LAN) as soon as I plug OPT1 wire into my laptop, so it becomes "working". One thing I notice -- when this LAN to OPT1 block releases I, as far as I can guess/remember, always get this Safari (browser) notice about shady https certificate, but since it's my home network I don't care and click "trust". There's no certificate on OPT1 iMac, I use it with Apple Screen Sharing feature to delegate some tasks I don't want to do on my laptop. Could it be smth inside pfSense related to "certificate refreshing" or whatever..
-
And sshing into OPT1 NAS, then arp -a lists devices connected to OPT1 which are currently powered on -- pfSense, iMac, my laptop, nothing out of ordinary
-
Found this in pfSense logs regarding certificates:
32379 Certificate: webConfigurator default (6370da2b7d127) (6370da2b7d127): Expired 16 days ago
-
@throttlenerd that would have zero to do with anything - that would just give you a warning from browser when accessing the gui.
-
@throttlenerd said in pfSense keeps blocking access from one subnet to another most of the time (but not always):
seems like I can't get an arp table of my OPT1 NAS "when not working",
can you not use some device on this opt 1 you can access locally? you just need one device to look at the mac table - or just look on pfsense - what does it see? if you can not find the mac it can not ping anything
-
can you not use some device on this opt 1 you can access locally? you just need one device to look at the mac table - or just look on pfsense - what does it see? if you can not find the mac it can not ping anything
Ehmmm I can't understand ) Yes I can unplug anything for the sake of the experiment -- do I need to plug OPT1 NAS directly to pfSense's OPT1 interface bypassing switches and stuff? It's totally doable but maybe I understood it wrong. Right now I'm on Wi-Fi (LAN) and can reach OPT1, like if when I plug into OPT1 by wire pfSense gives me some hours of LAN to OPT1 access. Then I will receive an email from may second NAS (on LAN) that it has lost connection to UPS (actually meaning it has lost connection to OPT1 NAS which also acts as a UPS server)
Regarding ARP table -- I can see every active device when arping from pfSense and only active OPT1 devices when arping from OPT1 NAS. Arping from LAN NAS results in a list of LAN devices. I think there's nothing strange here but I'm no expert )
-
@throttlenerd said in pfSense keeps blocking access from one subnet to another most of the time (but not always):
Regarding ARP table -
Well if pfsense can see the mac, and you ping it and it doesn't answer - that is not a pfsense problem..
@throttlenerd said in pfSense keeps blocking access from one subnet to another most of the time (but not always):
If I go to Diagnostics > Ping -- even pfSense can't ping both OPT1
So while your pinging some IP on your opt network from pfsense - do a sniff do you see pfsense send the ping, do you not get a response - then its not pfsense problem the thing your pining with the correct mac address in pfsense arp table doesn't answer..
-
Dear @johnpoz, I waited until the problem is active again (OPT1 unpingable), went into pfSense > Diagnostics > ARP Table. I'm (laptop) on W-Fi (LAN). There was only OPT1 NAS turned on, arp saw its IP but says "(incomplete)" instead of mac address. And it sees its own IP (10.0.74.1) with mac address. Then I turned on TrueNAS box and iMac, both on OPT1. Refreshed the page -- it showed their IPs but no mac addresses -- (incomplete). Then I refreshed the page and new instances (TrueNAS and iMac) disappeared. Refreshed -- Syno NAS disappeared. Refreshed -- Syno NAS appeared with no mac address. Then Syno NAS disappeared again. Then I disconnected from WiFi, connected to OPT1 by wire -- of course arp table now shows every OPT1 device with their mac addresses. Then I unplugged the OPT1 cable, went on WiFi -- "access from LAN to OPT1 is granted for now" -- I can ping any device, pfSense's arp shows every device with macs ))
-
@throttlenerd and what exactly is opt1 connected too.. Your not going to be able to talk to anything with mac incomplete.. that normally means you arped for it, but got no mac back..
They never going to work if you don't see the mac, pfsense won't even send the traffic on or even send a ping from its own IP if there is no mac to send too.
is the switch pfsense opt1 interface going down, into sleep or something and not passing on traffic until you connect to it with another device.. What is the make and model of this switch? Do you have maybe green ethernet enabled on it? Or some kind of power saving feature?
-
@johnpoz Hmmm, sleep mode! Sounds interesting! The switch is Cisco SG100D-05 V2 but it's unmanaged and there are no settings to be changed other than on/off. What if the NIC sleeps OPT1 interface down? But that would be stupid.. The NIC is HP NC360T.. I'll plug the NAS directly into OPT1 now and leave the switch on but unplugged from network to see if it changes its status LED in a day or so. Thank you!
