PfSense blocks traffic coming from SubnetA to SubnetB
-
Hello.
I'm having a firewalling problem, and after scratching my head for days, I'm turning to you to have your experience with PfSense.
I have the following interfaces:
WAN - Internet Access DNCH
SITEA - Subnet 192.168.2.0/24
SITEB - Subnet 192.168.88.0/24The SITEA and SITEB interface are connected using a WiFi bridge running on 10.0.0.0/29 subnet.
After configuring the routing, I'm not able to ping devices from SITEA to SITEB, but I can ping all SITEB devices from SITEA. I tought I had a routing issue, but after disabling the PfSense firewall (using command pfctl -d), the traffic from SITEA to SITEB starts working.
Here is a ping test from 192.168.2.35 (computer) to 192.168.88.10 (computer) when firewall is active:
ping 192.168.88.10 Pinging 192.168.88.10 with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out. Ping statistics for 192.168.88.10: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),On the PFSense router, I started a tcpdump before the test as follow:
[2.7.1-RELEASE][admin@gw.yqsi.ca]/root: tcpdump -nni igb1.2 net 192.168.88.0/24 and icmp tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on igb1.2, link-type EN10MB (Ethernet), snapshot length 262144 bytes 12:55:56.355299 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12964, length 40 12:56:01.088205 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12965, length 40 12:56:06.069825 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12966, length 40 12:56:11.080297 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12967, length 40On the SITEB's interface (igb2), there is nothing coming in.
Now, let's disable the firewall using command
pfctl -d:# pfctl -d pf disabledThen let's start the same ping test from the same computer:
ping 192.168.88.10 Pinging 192.168.88.10 with 32 bytes of data: Reply from 192.168.88.10: bytes=32 time=2ms TTL=126 Reply from 192.168.88.10: bytes=32 time=1ms TTL=126 Reply from 192.168.88.10: bytes=32 time=1ms TTL=126 Reply from 192.168.88.10: bytes=32 time=2ms TTL=126 Ping statistics for 192.168.88.10: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 1ms, Maximum = 2ms, Average = 1mstcpdumpfrom SITEA interface (igb1.2):tcpdump -nni igb1.2 net 192.168.88.0/24 and icmp tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on igb1.2, link-type EN10MB (Ethernet), snapshot length 262144 bytes 13:00:51.773295 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12976, length 40 13:00:51.775061 IP 192.168.88.10 > 192.168.2.35: ICMP echo reply, id 2, seq 12976, length 40 13:00:52.782661 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12977, length 40 13:00:52.784380 IP 192.168.88.10 > 192.168.2.35: ICMP echo reply, id 2, seq 12977, length 40 13:00:53.798504 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12978, length 40 13:00:53.800278 IP 192.168.88.10 > 192.168.2.35: ICMP echo reply, id 2, seq 12978, length 40 13:00:54.809514 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12979, length 40 13:00:54.811362 IP 192.168.88.10 > 192.168.2.35: ICMP echo reply, id 2, seq 12979, length 40And the
tcpdumpfrom SITEB interface (igb2):tcpdump -nni igb2 icmp and not net 10.0.0.0/29 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on igb2, link-type EN10MB (Ethernet), snapshot length 262144 bytes 13:00:51.773320 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12976, length 40 13:00:51.775036 IP 192.168.88.10 > 192.168.2.35: ICMP echo reply, id 2, seq 12976, length 40 13:00:52.782685 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12977, length 40 13:00:52.784370 IP 192.168.88.10 > 192.168.2.35: ICMP echo reply, id 2, seq 12977, length 40 13:00:53.798529 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12978, length 40 13:00:53.800252 IP 192.168.88.10 > 192.168.2.35: ICMP echo reply, id 2, seq 12978, length 40 13:00:54.809538 IP 192.168.2.35 > 192.168.88.10: ICMP echo request, id 2, seq 12979, length 40 13:00:54.811350 IP 192.168.88.10 > 192.168.2.35: ICMP echo reply, id 2, seq 12979, length 40```So I'm totally lost here.
Looking at pfTop from GUI when firewall is enabled:
Here are the routes configuration:
# netstat -rWn Routing tables Internet: Destination Gateway Flags Nhop# Mtu Netif Expire 0.0.0.0/1 10.6.112.1 UGS 17 1500 ovpnc1 default x.x.x.x UGS 13 1500 igb0 10.0.0.0/29 link#3 U 18 1500 igb2 10.0.0.3 link#8 UHS 19 16384 lo0 10.6.112.0/24 link#16 U 14 1500 ovpnc1 10.6.112.165 link#8 UHS 15 16384 lo0 24.53.40.0/24 link#1 U 10 1500 igb0 24.53.40.1 link#1 UHS 12 1500 igb0 24.53.40.150 link#8 UHS 11 16384 lo0 127.0.0.1 link#8 UH 2 16384 lo0 128.0.0.0/1 10.6.112.1 UGS 17 1500 ovpnc1 179.61.197.193 24.53.40.1 UGHS 16 1500 igb0 192.168.2.0/24 link#11 U 8 1500 igb1.2 192.168.2.250 link#8 UHS 3 16384 lo0 192.168.70.0/24 link#15 U 6 1500 ovpns2 192.168.70.1 link#8 UHS 9 16384 lo0 192.168.88.0/24 10.0.0.4 UGS 20 1500 igb2 192.168.114.0/24 link#13 U 1 1500 igb1.114 192.168.114.1 link#8 UHS 5 16384 lo0 192.168.147.0/24 link#14 U 4 1500 igb1.147 192.168.147.1 link#8 UHS 7 16384 lo0Firewall rules on SITEA interface (called VLAN2):
Firewall rules on SITEB interface (called SITEB):
Looking at the firewall logs in PfSense, for interface SITEB, I have this:
Remember that subnet 10.0.0.0/29 is the bridge subnet I'm using between the two sites. That is a point-to-point bridge.
The ping test from SITEA to SITEB is still running, and and I look at the firewall log, search for destination ip of 192.168.88.10 (the ip I'm pinging), there is nothing to display...
Oh, and by the way, even if I created an outbound NAT for subnet 192.168.88.0/24 on the WAN Address, the SITEB can't access the Internet.
So I know that is a very long post. I've tried to make it the mose defailled as possible. I really hope someone could help me out figuring why the firewall is blocking communication between those two subnets. Since I found that the routing is working when I disable the firewall, I'm totally screwd!
Since a picture worth thousands words, here is a simple schema of my setup:
I wanna thank you all for the time taking reading my post. If you need anymore information, please feel free to ask,
Thank you and best regards,
Yanick -
@yquirion
At B you have to disable blocking of private networks in the WAN interface settings. -
@viragomann Hi! Thanks for your answer. On site B, the router is not pfsense, but a Mikrotik router with firewall rule of allow any -> any traffic.
Thanks!
-
@yquirion It won't fix your problem but on SiteA the last rule for 192.168.88.x is irrelevant because 1) it's below the "allow any" rule and 2) 192.168.88.x is not a subnet on that interface so packets will never arrive from 192.168.88.x. Also the rule above it is for the subnet of "VLAN2 subnets" correct? so also will never trigger.
Your firewall log for SiteB is mostly DNS (port 53).
Do you have static routes set up on both ends?
https://docs.netgate.com/pfsense/en/latest/routing/static.html#example-static-route -
From what I am seeing your SITE-B rule allows the traffic you are testing, however you do not have logging enabled on the rule to verify. Your screenshot of the firewall logs do not show the traffic between the test computers being blocked.
Would it be possible to show your static routes as well as enable logging for the SITE-B allow rule and then show use the firewall logs while you are testing the connectivity?
-
@SteveITS said in PfSense blocks traffic coming from SubnetA to SubnetB:
@yquirion It won't fix your problem but on SiteA the last rule for 192.168.88.x is irrelevant because 1) it's below the "allow any" rule and 2) 192.168.88.x is not a subnet on that interface so packets will never arrive from 192.168.88.x. Also the rule above it is for the subnet of "VLAN2 subnets" correct? so also will never trigger.
Your firewall log for SiteB is mostly DNS (port 53).
Do you have static routes set up on both ends?
https://docs.netgate.com/pfsense/en/latest/routing/static.html#example-static-routeThank you both for your suggestion.
SiteA the last rule for 192.168.88.x is irrelevant because 1) it's below the "allow any"...
I'm totally agree, but I tried everything I could to find the issue. When the firewall is disable, the test ping pass. But I will remove it.Here are the static routing I've made:
As per @skenigma suggestion, I've also enabled the logging on some rules:
Interface: VLAN2 (SiteA) - 192.168.2.0/24
Interface: SiteB - 192.168.88.0/24
I also look at the link provided by @SteveITS and the option
Bypass firewall rules for traffic on the same interfacewas already enabled.Also from that article, I read the following section:
Alternatively, firewall rules may be added manually to allow similar traffic. Two rules are needed, one on the interface tab where the traffic enters (e.g. LAN) and another on the Floating tab:
From that steps, I read this:
Click Add to add a new rule to the top of the list
So I made those two rules including there Advanced Features, and then it started to work:
[PS] C:\Users\yanqui$ ping 192.168.88.10 -t Pinging 192.168.88.10 with 32 bytes of data: Reply from 192.168.88.10: bytes=32 time=2ms TTL=126 Reply from 192.168.88.10: bytes=32 time=2ms TTL=126So I tried to fins which one of the rules, the one from the SiteA (VLAN2) interface, or the one at the floating level, still using interface VLAN2. It appears that the only thing I had to do was to bring up the existing rule I've already created:
... on the top of the list. No need the floating rules or advanced option to be enabled.To be honest, I don't have any idea what those advanced options and floating rules does, but it seems I don't need them.
Here is a
traceroutefrom SiteA to SiteB:tracert -d 192.168.88.10 Tracing route to 192.168.88.10 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.2.250 2 1 ms 1 ms 1 ms 10.0.0.4 3 1 ms 1 ms 1 ms 192.168.88.10Now, the only issue I have, is there is no Internet access from SiteB.
If I ping an address from SiteB to SiteA, even the gedault gateway of SiteA, it woks. Here is a
treceroutecommand:tracert -d 192.168.2.3 Tracing route to 192.168.2.3 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.88.1 2 1 ms 1 ms 2 ms 10.0.0.3 3 2 ms 2 ms 2 ms 192.168.2.3When doing a
tracerouteto 8.8.8.8, I'm getting this:tracert -d 8.8.8.8 Tracing route to 8.8.8.8 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.88.1 2 * * * Request timed out. 3 *Of course, I added a NAT rule on the PfSense firewall:
After trying different interface there, nothing seems to work.
So with your help I'm getting very closer to. I don't know why the rule I've created have to be in the top of the list to work, but at least it is now working.
There is only the internet access from SiteB to solve. If you have any clue, please me know. In the meantime, if I find something, I will post it.
Regards!
Yanick -
@yquirion
if you are wanting Site B to access the internet through Site A, you will need a rule allowing 192.168.88.0/24 to ANYthen you would need to make sure the routes ate site B are set to the default route to Site A
-
@skenigma said in PfSense blocks traffic coming from SubnetA to SubnetB:
@yquirion
if you are wanting Site B to access the internet through Site A, you will need a rule allowing 192.168.88.0/24 to ANYthen you would need to make sure the routes ate site B are set to the default route to Site A
Hi @skenigma!
Thanks again I appreciate your help.
Before I saw your reply, I did create this rule on SITEB interface in the PfSense firewall:
Now, from the firewall logs, I see this:
From the SITEB Mikrotik router, I have those route defined:
If the traffic goes through SiteA to access the Internet, do I need an outbound NAT for 192.168.88.0/24? I already created it, but even if the firewall logs tells the ICMP pass, it is now working.
Thanks!
Yanick -
@yquirion
So probably you pointed the default route on B to A.If the traffic goes through SiteA to access the Internet, do I need an outbound NAT for 192.168.88.0/24?
Exactly.
I already created
Did you also enable the Hybrid mode?
If still no joy restart the firewall at A.
-
Hi @viragomann,
Thanks for your reply!
Did you also enable the Hybrid mode?
Well I will have to search what is Hybrid mode. I don't think this is enable.Here are an output of pfTop. The first one is when I ping 8.8.8.8 from a host at SiteA:
Then this second one, is when I ping 8.8.8.8 from a host in siteB:
We can see on the sfirst on the connection goes to my WAN IP address, while the In and Out are the same on SiteB...
Thanks!
Yanick -
Hi @viragomann!
Here is the answer to your previous question.
Did you also enable the Hybrid mode?
I set it to Manual. Should I change it to the second one (Hybrid Outbound NAT rule generation.
(Automatic Outbound NAT + rules below) or the first one (Automatic outbound NAT rule generation.
(IPsec passthrough included))?Thanks,
Yanick -
@yquirion
So the packets from B go out on A WAN, but are not natted.So presumably there is something wrong with the outbound NAT. Can you show it, please?
-
Hi @viragomann,
Here is almost all of it:
I'm using some VLANs as well as a VPN connection, but do not bother with those.
Thanks
Yanick -
@yquirion
Yeah, you're missing the rule for the B LAN.
Maybe you've chosen the wrong interface by mistake.
-
Hi @viragomann,
You're right. I change that to see what it will do, but it was WAN before. I will put it back, but unfortunately it won't solve the issue...
Here it is:
ping 8.8.8.8 Pinging 8.8.8.8 with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out.Thanks
Yanick -
@yquirion
Should work, since the packets are directed out on WAN.As suggested before, reboot the box. Or at least flush the states.
-
Hi @viragomann,
I just rebooted and it doesn't solve the issue.
I really don't understand this one!
Thank you so much for your help!
Regards,
Yanick -
@yquirion
Search the state table on A for the source IP.Use packet capture to see, what's going on there.
-
Hi @viragomann,
Here are some tests I've done.
From SiteB, when I ping anything on the Internet, let's take one more time Google DNS 8.8.8.8, I will have this result for command
tcpdump -nni igb2 icmp and host 8.8.8.8I execute on my PFSense. Interfaceigb2is the physical interface used by SITEB in pfsense:22:27:29.720748 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48849, length 40 22:27:34.632365 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48850, length 40 22:27:39.639399 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48851, length 40 22:27:44.644884 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48852, length 40In the meantime, running
tcpdumpon WAN interfaceigb0, will not produce any output:tcpdump -nni igb0 icmp and host 8.8.8.8 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytesSo it means the traffic from SiteB never redirected to the SiteA WAN's interface.
When doing that ping test from the SiteB's Mikrotik Router, the tcpdump running on interface
igb2on the pfsense router will show this:22:33:36.921387 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 0, length 36 22:33:37.924867 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 1, length 36 22:33:38.928295 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 2, length 36 22:33:39.931764 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 3, length 36 22:33:40.934412 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 4, length 3610.0.0.0/29 IPs from the router are normal because the P2P link between the two site is using this subnet.
When I ping 8.8.8.8 from SiteA computer (192.168.2.35), I will see that on the states page:
But when doing this from SiteB, I don't see any "icmp" traffic going through.
Correction: After looking again, I can see the state of ICMP from SiteB:
But still "Request timed out".
Will continue investigating!
Thanks again for your help!
Yanick
-
@yquirion said in PfSense blocks traffic coming from SubnetA to SubnetB:
In the meantime, running tcpdump on WAN interface igb0, will not produce any output:
Did you change the firewall rules?
What rules to you have at A on SITEB interface?@yquirion said in PfSense blocks traffic coming from SubnetA to SubnetB:
When I ping 8.8.8.8 from SiteA computer (192.168.2.35), I will see that on the states page:
Don't filter the interface, only the destination IP.
If the ping works I'd expect to see two states for this. One on the incoming interface (e.g. VLAN2) and one on the outgoing (WAN).I assume, if you try to access the internet from B the WAN state is missing.