PfSense blocks traffic coming from SubnetA to SubnetB
-
@SteveITS said in PfSense blocks traffic coming from SubnetA to SubnetB:
@yquirion It won't fix your problem but on SiteA the last rule for 192.168.88.x is irrelevant because 1) it's below the "allow any" rule and 2) 192.168.88.x is not a subnet on that interface so packets will never arrive from 192.168.88.x. Also the rule above it is for the subnet of "VLAN2 subnets" correct? so also will never trigger.
Your firewall log for SiteB is mostly DNS (port 53).
Do you have static routes set up on both ends?
https://docs.netgate.com/pfsense/en/latest/routing/static.html#example-static-routeThank you both for your suggestion.
SiteA the last rule for 192.168.88.x is irrelevant because 1) it's below the "allow any"...
I'm totally agree, but I tried everything I could to find the issue. When the firewall is disable, the test ping pass. But I will remove it.Here are the static routing I've made:
As per @skenigma suggestion, I've also enabled the logging on some rules:
Interface: VLAN2 (SiteA) - 192.168.2.0/24
Interface: SiteB - 192.168.88.0/24
I also look at the link provided by @SteveITS and the option
Bypass firewall rules for traffic on the same interfacewas already enabled.Also from that article, I read the following section:
Alternatively, firewall rules may be added manually to allow similar traffic. Two rules are needed, one on the interface tab where the traffic enters (e.g. LAN) and another on the Floating tab:
From that steps, I read this:
Click Add to add a new rule to the top of the list
So I made those two rules including there Advanced Features, and then it started to work:
[PS] C:\Users\yanqui$ ping 192.168.88.10 -t Pinging 192.168.88.10 with 32 bytes of data: Reply from 192.168.88.10: bytes=32 time=2ms TTL=126 Reply from 192.168.88.10: bytes=32 time=2ms TTL=126So I tried to fins which one of the rules, the one from the SiteA (VLAN2) interface, or the one at the floating level, still using interface VLAN2. It appears that the only thing I had to do was to bring up the existing rule I've already created:
... on the top of the list. No need the floating rules or advanced option to be enabled.To be honest, I don't have any idea what those advanced options and floating rules does, but it seems I don't need them.
Here is a
traceroutefrom SiteA to SiteB:tracert -d 192.168.88.10 Tracing route to 192.168.88.10 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.2.250 2 1 ms 1 ms 1 ms 10.0.0.4 3 1 ms 1 ms 1 ms 192.168.88.10Now, the only issue I have, is there is no Internet access from SiteB.
If I ping an address from SiteB to SiteA, even the gedault gateway of SiteA, it woks. Here is a
treceroutecommand:tracert -d 192.168.2.3 Tracing route to 192.168.2.3 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.88.1 2 1 ms 1 ms 2 ms 10.0.0.3 3 2 ms 2 ms 2 ms 192.168.2.3When doing a
tracerouteto 8.8.8.8, I'm getting this:tracert -d 8.8.8.8 Tracing route to 8.8.8.8 over a maximum of 30 hops 1 <1 ms <1 ms <1 ms 192.168.88.1 2 * * * Request timed out. 3 *Of course, I added a NAT rule on the PfSense firewall:
After trying different interface there, nothing seems to work.
So with your help I'm getting very closer to. I don't know why the rule I've created have to be in the top of the list to work, but at least it is now working.
There is only the internet access from SiteB to solve. If you have any clue, please me know. In the meantime, if I find something, I will post it.
Regards!
Yanick -
@yquirion
if you are wanting Site B to access the internet through Site A, you will need a rule allowing 192.168.88.0/24 to ANYthen you would need to make sure the routes ate site B are set to the default route to Site A
-
@skenigma said in PfSense blocks traffic coming from SubnetA to SubnetB:
@yquirion
if you are wanting Site B to access the internet through Site A, you will need a rule allowing 192.168.88.0/24 to ANYthen you would need to make sure the routes ate site B are set to the default route to Site A
Hi @skenigma!
Thanks again I appreciate your help.
Before I saw your reply, I did create this rule on SITEB interface in the PfSense firewall:
Now, from the firewall logs, I see this:
From the SITEB Mikrotik router, I have those route defined:
If the traffic goes through SiteA to access the Internet, do I need an outbound NAT for 192.168.88.0/24? I already created it, but even if the firewall logs tells the ICMP pass, it is now working.
Thanks!
Yanick -
@yquirion
So probably you pointed the default route on B to A.If the traffic goes through SiteA to access the Internet, do I need an outbound NAT for 192.168.88.0/24?
Exactly.
I already created
Did you also enable the Hybrid mode?
If still no joy restart the firewall at A.
-
Hi @viragomann,
Thanks for your reply!
Did you also enable the Hybrid mode?
Well I will have to search what is Hybrid mode. I don't think this is enable.Here are an output of pfTop. The first one is when I ping 8.8.8.8 from a host at SiteA:
Then this second one, is when I ping 8.8.8.8 from a host in siteB:
We can see on the sfirst on the connection goes to my WAN IP address, while the In and Out are the same on SiteB...
Thanks!
Yanick -
Hi @viragomann!
Here is the answer to your previous question.
Did you also enable the Hybrid mode?
I set it to Manual. Should I change it to the second one (Hybrid Outbound NAT rule generation.
(Automatic Outbound NAT + rules below) or the first one (Automatic outbound NAT rule generation.
(IPsec passthrough included))?Thanks,
Yanick -
@yquirion
So the packets from B go out on A WAN, but are not natted.So presumably there is something wrong with the outbound NAT. Can you show it, please?
-
Hi @viragomann,
Here is almost all of it:
I'm using some VLANs as well as a VPN connection, but do not bother with those.
Thanks
Yanick -
@yquirion
Yeah, you're missing the rule for the B LAN.
Maybe you've chosen the wrong interface by mistake.
-
Hi @viragomann,
You're right. I change that to see what it will do, but it was WAN before. I will put it back, but unfortunately it won't solve the issue...
Here it is:
ping 8.8.8.8 Pinging 8.8.8.8 with 32 bytes of data: Request timed out. Request timed out. Request timed out. Request timed out.Thanks
Yanick -
@yquirion
Should work, since the packets are directed out on WAN.As suggested before, reboot the box. Or at least flush the states.
-
Hi @viragomann,
I just rebooted and it doesn't solve the issue.
I really don't understand this one!
Thank you so much for your help!
Regards,
Yanick -
@yquirion
Search the state table on A for the source IP.Use packet capture to see, what's going on there.
-
Hi @viragomann,
Here are some tests I've done.
From SiteB, when I ping anything on the Internet, let's take one more time Google DNS 8.8.8.8, I will have this result for command
tcpdump -nni igb2 icmp and host 8.8.8.8I execute on my PFSense. Interfaceigb2is the physical interface used by SITEB in pfsense:22:27:29.720748 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48849, length 40 22:27:34.632365 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48850, length 40 22:27:39.639399 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48851, length 40 22:27:44.644884 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48852, length 40In the meantime, running
tcpdumpon WAN interfaceigb0, will not produce any output:tcpdump -nni igb0 icmp and host 8.8.8.8 tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytesSo it means the traffic from SiteB never redirected to the SiteA WAN's interface.
When doing that ping test from the SiteB's Mikrotik Router, the tcpdump running on interface
igb2on the pfsense router will show this:22:33:36.921387 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 0, length 36 22:33:37.924867 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 1, length 36 22:33:38.928295 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 2, length 36 22:33:39.931764 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 3, length 36 22:33:40.934412 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 4, length 3610.0.0.0/29 IPs from the router are normal because the P2P link between the two site is using this subnet.
When I ping 8.8.8.8 from SiteA computer (192.168.2.35), I will see that on the states page:
But when doing this from SiteB, I don't see any "icmp" traffic going through.
Correction: After looking again, I can see the state of ICMP from SiteB:
But still "Request timed out".
Will continue investigating!
Thanks again for your help!
Yanick
-
@yquirion said in PfSense blocks traffic coming from SubnetA to SubnetB:
In the meantime, running tcpdump on WAN interface igb0, will not produce any output:
Did you change the firewall rules?
What rules to you have at A on SITEB interface?@yquirion said in PfSense blocks traffic coming from SubnetA to SubnetB:
When I ping 8.8.8.8 from SiteA computer (192.168.2.35), I will see that on the states page:
Don't filter the interface, only the destination IP.
If the ping works I'd expect to see two states for this. One on the incoming interface (e.g. VLAN2) and one on the outgoing (WAN).I assume, if you try to access the internet from B the WAN state is missing.
-
Hi @viragomann,
I've made an error. Here are the new result from a fresh test
From SiteA - Interface VLAN2
From SiteB - Interface SITEB
So we can see that nothing from Interface SITEB in going to the WAN Interface.
Here are the rules on the WAN interface:
And the rules on the SITEB Interface:
Here are the interface configuration:
All those 3 interfaces are physical interfaces on my PfSense router.
Outbound rules:
I really don't understand why nothing is going to the WAN interface from SITEB interface.
Thanks again!
Yanick
-
Hi @viragomann,
I found something.
Just after sending the reply, I did change this:
Those two options were selected on the WAN interface.
After that I see I could ping 8.8.8.8, but nothing else... Then if I stop the ping to 8.8.8.8, it won't restarts until I play again with those parameter (re-enable them).
I reset all states; no luck, I rebooted the pfsense, same thing...
At this moment I have a running ping from SITEB to 8.8.8.8. It works. I see the corresponding traffic on the igb0 (WAN) interface using
tcpdump.Now, if I stop that ping, then wait for the states being cleared, then restart it, it is no longer working.
Maybe I am running into a bug with pfsense, but all of this has no sense at all.
Thanks
Yanick -
@yquirion
WTF! Just noticed, that you have "static port" checked in the outbound NAT rule.
It wasn't checked before!
Why did you change things, you're not advised to change? You needed just to edit the rule and change the interface to WAN.
This makes it hard to help you with your issues.Those two options were selected on the WAN interface.
At A WAN this setting should be checked. At B not, mentioned in my very first post.
At A VLAN2 it has to be unchecked. But this might have been the case before, otherwise you were not able to access A from B at all. -
Hi @viragomann,
That is very odd. I never notice it. Sorry about that. However, I just look at it and since that post, it has been corrected.
Regarding the 2 options, I don't have those on siteB because siteB is using a Mikrotik router. From the pfsense router on sitea, those options are disabled on all interface (was only enable on the WAN before).
I'm not home for the weekend so I cannot make any other test on this.. But that os very strange that, while pfsense is applying the change of the
Block private networks and loopback addresses, for a second the ping will work until I cancel it and the connection is being cleared of the state table...Thanks again, and I'm really sorry having made that mistake. I'm just testing stuff between replies!
Yanick
-
Hi @viragomann,
I found it!
It was because of a miss configuration into my client OpenVPN connection.
Here is a post that explain what was my problem:
https://www.reddit.com/r/PFSENSE/comments/i125ig/default_route_now_set_to_vpn_client_instead_of/
I was looking into this because of this:
# route get 1.1.1.1 route to: one.one.one.one destination: default mask: 128.0.0.0 gateway: 10.4.112.1 fib: 0 interface: ovpnc1 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0That gateway of 10.4.112.1 was the gateway of my OpenVPN client connection. When I stop the service for that VPN connection, that was the result of the same command:
/root: route get 8.8.8.8 route to: dns.google destination: default mask: default gateway: modemcable001.40-53-24.mc.videotron.ca fib: 0 interface: igb0 flags: <UP,GATEWAY,DONE,STATIC> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0So I made search on why the OpenVPN client was adding that route to the routing table:
0.0.0.0/1 <<VPN Interface Gateway IP>>And I found the post tjhat somve my issue.
I would like to thank you very much as well as @skenigma for your time helping me solving the issue.
Best Regards,
Yanick