PfSense blocks traffic coming from SubnetA to SubnetB

yquirion

@viragomann Hi! Thanks for your answer. On site B, the router is not pfsense, but a Mikrotik router with firewall rule of allow any -> any traffic.

Thanks!

SteveITS

@yquirion It won't fix your problem but on SiteA the last rule for 192.168.88.x is irrelevant because 1) it's below the "allow any" rule and 2) 192.168.88.x is not a subnet on that interface so packets will never arrive from 192.168.88.x. Also the rule above it is for the subnet of "VLAN2 subnets" correct? so also will never trigger.

Your firewall log for SiteB is mostly DNS (port 53).

Do you have static routes set up on both ends?
https://docs.netgate.com/pfsense/en/latest/routing/static.html#example-static-route

skenigma

@yquirion

From what I am seeing your SITE-B rule allows the traffic you are testing, however you do not have logging enabled on the rule to verify. Your screenshot of the firewall logs do not show the traffic between the test computers being blocked.

Would it be possible to show your static routes as well as enable logging for the SITE-B allow rule and then show use the firewall logs while you are testing the connectivity?

yquirion

@SteveITS said in PfSense blocks traffic coming from SubnetA to SubnetB:

@yquirion It won't fix your problem but on SiteA the last rule for 192.168.88.x is irrelevant because 1) it's below the "allow any" rule and 2) 192.168.88.x is not a subnet on that interface so packets will never arrive from 192.168.88.x. Also the rule above it is for the subnet of "VLAN2 subnets" correct? so also will never trigger.

Your firewall log for SiteB is mostly DNS (port 53).

Do you have static routes set up on both ends?
https://docs.netgate.com/pfsense/en/latest/routing/static.html#example-static-route

Hi @SteveITS and @skenigma,

Thank you both for your suggestion.

SiteA the last rule for 192.168.88.x is irrelevant because 1) it's below the "allow any"...
I'm totally agree, but I tried everything I could to find the issue. When the firewall is disable, the test ping pass. But I will remove it.

Here are the static routing I've made:

As per @skenigma suggestion, I've also enabled the logging on some rules:
Interface: VLAN2 (SiteA) - 192.168.2.0/24

Interface: SiteB - 192.168.88.0/24

I also look at the link provided by @SteveITS and the option Bypass firewall rules for traffic on the same interface was already enabled.

Also from that article, I read the following section:

Alternatively, firewall rules may be added manually to allow similar traffic. Two rules are needed, one on the interface tab where the traffic enters (e.g. LAN) and another on the Floating tab:

From that steps, I read this:

Click Add to add a new rule to the top of the list

So I made those two rules including there Advanced Features, and then it started to work:

[PS] C:\Users\yanqui$ ping 192.168.88.10 -t

Pinging 192.168.88.10 with 32 bytes of data:
Reply from 192.168.88.10: bytes=32 time=2ms TTL=126
Reply from 192.168.88.10: bytes=32 time=2ms TTL=126

So I tried to fins which one of the rules, the one from the SiteA (VLAN2) interface, or the one at the floating level, still using interface VLAN2. It appears that the only thing I had to do was to bring up the existing rule I've already created:

... on the top of the list. No need the floating rules or advanced option to be enabled.

To be honest, I don't have any idea what those advanced options and floating rules does, but it seems I don't need them.

Here is a traceroute from SiteA to SiteB:

tracert -d 192.168.88.10

Tracing route to 192.168.88.10 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.2.250
  2     1 ms     1 ms     1 ms  10.0.0.4
  3     1 ms     1 ms     1 ms  192.168.88.10

Now, the only issue I have, is there is no Internet access from SiteB.

If I ping an address from SiteB to SiteA, even the gedault gateway of SiteA, it woks. Here is a treceroute command:

tracert -d 192.168.2.3

Tracing route to 192.168.2.3 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.88.1
  2     1 ms     1 ms     2 ms  10.0.0.3
  3     2 ms     2 ms     2 ms  192.168.2.3

When doing a traceroute to 8.8.8.8, I'm getting this:

tracert -d 8.8.8.8

Tracing route to 8.8.8.8 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  192.168.88.1
  2     *        *        *     Request timed out.
  3     *

Of course, I added a NAT rule on the PfSense firewall:

After trying different interface there, nothing seems to work.

So with your help I'm getting very closer to. I don't know why the rule I've created have to be in the top of the list to work, but at least it is now working.

There is only the internet access from SiteB to solve. If you have any clue, please me know. In the meantime, if I find something, I will post it.

Regards!
Yanick

skenigma

@yquirion
if you are wanting Site B to access the internet through Site A, you will need a rule allowing 192.168.88.0/24 to ANY

then you would need to make sure the routes ate site B are set to the default route to Site A

yquirion

@skenigma said in PfSense blocks traffic coming from SubnetA to SubnetB:

@yquirion
if you are wanting Site B to access the internet through Site A, you will need a rule allowing 192.168.88.0/24 to ANY

then you would need to make sure the routes ate site B are set to the default route to Site A

Hi @skenigma!

Thanks again I appreciate your help.

Before I saw your reply, I did create this rule on SITEB interface in the PfSense firewall:

Now, from the firewall logs, I see this:

From the SITEB Mikrotik router, I have those route defined:

If the traffic goes through SiteA to access the Internet, do I need an outbound NAT for 192.168.88.0/24? I already created it, but even if the firewall logs tells the ICMP pass, it is now working.

Thanks!
Yanick

viragomann

@yquirion
So probably you pointed the default route on B to A.

If the traffic goes through SiteA to access the Internet, do I need an outbound NAT for 192.168.88.0/24?

Exactly.

I already created

Did you also enable the Hybrid mode?

If still no joy restart the firewall at A.

yquirion

Hi @viragomann,

Thanks for your reply!

Did you also enable the Hybrid mode?
Well I will have to search what is Hybrid mode. I don't think this is enable.

Here are an output of pfTop. The first one is when I ping 8.8.8.8 from a host at SiteA:

Then this second one, is when I ping 8.8.8.8 from a host in siteB:

We can see on the sfirst on the connection goes to my WAN IP address, while the In and Out are the same on SiteB...

Thanks!
Yanick

yquirion

Hi @viragomann!

Here is the answer to your previous question.

Did you also enable the Hybrid mode?

I set it to Manual. Should I change it to the second one (Hybrid Outbound NAT rule generation.
(Automatic Outbound NAT + rules below) or the first one (Automatic outbound NAT rule generation.
(IPsec passthrough included))?

Thanks,
Yanick

viragomann

@yquirion
So the packets from B go out on A WAN, but are not natted.

So presumably there is something wrong with the outbound NAT. Can you show it, please?

yquirion

Hi @viragomann,

Here is almost all of it:

I'm using some VLANs as well as a VPN connection, but do not bother with those.

Thanks
Yanick

viragomann

@yquirion
Yeah, you're missing the rule for the B LAN.
Maybe you've chosen the wrong interface by mistake.

yquirion

Hi @viragomann,

You're right. I change that to see what it will do, but it was WAN before. I will put it back, but unfortunately it won't solve the issue...

Here it is:

ping 8.8.8.8

Pinging 8.8.8.8 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Thanks
Yanick

viragomann

@yquirion
Should work, since the packets are directed out on WAN.

As suggested before, reboot the box. Or at least flush the states.

yquirion

Hi @viragomann,

I just rebooted and it doesn't solve the issue.

I really don't understand this one!

Thank you so much for your help!

Regards,
Yanick

viragomann

@yquirion
Search the state table on A for the source IP.

Use packet capture to see, what's going on there.

yquirion

Hi @viragomann,

Here are some tests I've done.

From SiteB, when I ping anything on the Internet, let's take one more time Google DNS 8.8.8.8, I will have this result for command tcpdump -nni igb2 icmp and host 8.8.8.8 I execute on my PFSense. Interface igb2 is the physical interface used by SITEB in pfsense:

22:27:29.720748 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48849, length 40
22:27:34.632365 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48850, length 40
22:27:39.639399 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48851, length 40
22:27:44.644884 IP 192.168.88.10 > 8.8.8.8: ICMP echo request, id 7, seq 48852, length 40

In the meantime, running tcpdump on WAN interface igb0, will not produce any output:

tcpdump -nni igb0 icmp and host 8.8.8.8
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), snapshot length 262144 bytes

So it means the traffic from SiteB never redirected to the SiteA WAN's interface.

When doing that ping test from the SiteB's Mikrotik Router, the tcpdump running on interface igb2 on the pfsense router will show this:

22:33:36.921387 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 0, length 36
22:33:37.924867 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 1, length 36
22:33:38.928295 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 2, length 36
22:33:39.931764 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 3, length 36
22:33:40.934412 IP 10.0.0.4 > 8.8.8.8: ICMP echo request, id 1711, seq 4, length 36

10.0.0.0/29 IPs from the router are normal because the P2P link between the two site is using this subnet.

When I ping 8.8.8.8 from SiteA computer (192.168.2.35), I will see that on the states page:

But when doing this from SiteB, I don't see any "icmp" traffic going through.

Correction: After looking again, I can see the state of ICMP from SiteB:

But still "Request timed out".

Will continue investigating!

Thanks again for your help!

Yanick

viragomann

@yquirion said in PfSense blocks traffic coming from SubnetA to SubnetB:

In the meantime, running tcpdump on WAN interface igb0, will not produce any output:

Did you change the firewall rules?
What rules to you have at A on SITEB interface?

@yquirion said in PfSense blocks traffic coming from SubnetA to SubnetB:

When I ping 8.8.8.8 from SiteA computer (192.168.2.35), I will see that on the states page:

Don't filter the interface, only the destination IP.
If the ping works I'd expect to see two states for this. One on the incoming interface (e.g. VLAN2) and one on the outgoing (WAN).

I assume, if you try to access the internet from B the WAN state is missing.

yquirion

Hi @viragomann,

I've made an error. Here are the new result from a fresh test

From SiteA - Interface VLAN2

From SiteB - Interface SITEB

So we can see that nothing from Interface SITEB in going to the WAN Interface.

Here are the rules on the WAN interface:

And the rules on the SITEB Interface:

Here are the interface configuration:

All those 3 interfaces are physical interfaces on my PfSense router.

Outbound rules:

I really don't understand why nothing is going to the WAN interface from SITEB interface.

Thanks again!

Yanick

yquirion

Hi @viragomann,

I found something.

Just after sending the reply, I did change this:

Those two options were selected on the WAN interface.

After that I see I could ping 8.8.8.8, but nothing else... Then if I stop the ping to 8.8.8.8, it won't restarts until I play again with those parameter (re-enable them).

I reset all states; no luck, I rebooted the pfsense, same thing...

At this moment I have a running ping from SITEB to 8.8.8.8. It works. I see the corresponding traffic on the igb0 (WAN) interface using tcpdump.

Now, if I stop that ping, then wait for the states being cleared, then restart it, it is no longer working.

Maybe I am running into a bug with pfsense, but all of this has no sense at all.

Thanks
Yanick