Communication between LANs does not work

obones

Hello,

I have a following setup here with three interfaces: 1 WAN with a public IP and two LANs on the following segments:

LAN18 = 10.10.18.0/24
LAN11 = 10.10.11.0/24

NAT is activated on automatic mode and the default "pass IPv4 to any" rules are present on both LANs.
With this, any host from any of the two LANs can see the Internet just fine.

However, if a machine inside LAN11 tries to ping or connect to any UDP/TCP socket on a LAN18 machine, the connection fails.
The same goes for the other way round, from a LAN18 machine to a LAN11 machine.

I have activated logging on the pass rules and I can see that an ICMP packet has reached pfSense when pinging LAN18 from LAN11 and that his packet was "passed".

I looked around this forum and other places and found a person who said that he added pfSense as the gateway on its LAN interface which solved its situation.
So I tried adding the pfSense LAN18 address as the gateway for LAN18 and this allowed LAN11 machines to ping LAN18 machines. However, TCP or UPD connections, do not work.
And this did not allow LAN18 to ping LAN11.
So I removed that gateway and set it back to None on LAN18.

If I go to the states dump, I see this:

LAN11 	udp 	10.10.11.20:50210 -> 10.10.18.253:53 	NO_TRAFFIC:SINGLE 	3 / 0 	186 B / 0 B 	
LAN18 	udp 	10.10.11.20:50210 -> 10.10.18.253:53 	SINGLE:NO_TRAFFIC 	3 / 0 	186 B / 0 B

It's as if pfSense does not want to NAT the packets.

I must be missing something obvious, but I could not find it either here nor in the documentation.

Any help would be greatly appreciated.

KOM

An allow all to any rule on both LANs is enough. Post a screencap of both rulesets and we can see what you're doing.

https://doc.pfsense.org/index.php/Connectivity_Troubleshooting

obones

Well, that's what I thought too, here are the screencaps:

LAN11

LAN18

I don't see what's wrong with the rules here.

One thing though, is that both LAN11 and LAN18 are VLAN tagged interfaces (vmx0.11 and vmx0.18) on the same physical interface (vmx0) which is plugged on the trunk port of the switch.
But I'm not sure this would have any interference as both LANs are capable of going to the outside world.

KOM

Yes, those rules look good. I'm weak on VLANs so I can't really help you there but I wouldn't be surprised if it's some weird VLAN thing. Otherwise, there isn't much left other than local client firewalls.

obones

Local client firewalls are off the picture, all machines on the same LAN can ping and connect to each other just fine.

johnpoz

@obones said in Communication between LANs does not work:

Local client firewalls are off the picture, all machines on the same LAN can ping and connect to each other just fine.

You understand for example that windows will allow ping on same sub, but other sub block - this is almost always a local clients host firewall issue.

Why would pfsense nat traffic between 2 local rfc1918 networks? Sniff on pfsense lan11 interface for example, then ping something on the lan11 from 18, do you see pfsense end the ping request out? But no answer?

Can your 18 client for example ping the IP of pfsense lan11 interface?

Other common mistakes are policy routing out some vpn or something on pfsense without allowing for intervlan traffic - this is not the case from your screenshots. Other common issue is device on vlans not even using pfsense as their gateway, etc.

Other common issue is wrong mask setup either on client or pfsense.

obones

@johnpoz said in Communication between LANs does not work:

You understand for example that windows will allow ping on same sub, but other sub block - this is almost always a local clients host firewall issue.

The firewall are turned off for the time being so that I'm sure they do not interfere.

Why would pfsense nat traffic between 2 local rfc1918 networks?

Well, that would allow me to segregate traffic from one LAN to the other. Granted, this is not what I'm doing at the moment, but I'm starting with a simple "all in" rule set to see how it goes. Then, later on, I will restrict what is allowed based on actual needs.

Sniff on pfsense lan11 interface for example, then ping something on the lan11 from 18, do you see pfsense end the ping request out? But no answer?

Sniffing on LAN11, I see only those packets:

17:31:42.009100 IP 10.10.11.20 > 10.10.18.253: ICMP echo request, id 1, seq 7, length 40
17:31:46.800152 IP 10.10.11.20 > 10.10.18.253: ICMP echo request, id 1, seq 8, length 40
17:31:51.800144 IP 10.10.11.20 > 10.10.18.253: ICMP echo request, id 1, seq 9, length 40
17:31:56.800140 IP 10.10.11.20 > 10.10.18.253: ICMP echo request, id 1, seq 10, length 40

If I sniff on LAN18, I get this:

17:34:51.730084 IP 10.10.11.20 > 10.10.18.253: ICMP echo request, id 1, seq 14, length 40
17:34:56.300474 IP 10.10.11.20 > 10.10.18.253: ICMP echo request, id 1, seq 15, length 40
17:35:01.300554 IP 10.10.11.20 > 10.10.18.253: ICMP echo request, id 1, seq 16, length 40
17:35:06.300584 IP 10.10.11.20 > 10.10.18.253: ICMP echo request, id 1, seq 17, length 40

So it seems 18.253 does not send anything back...

Can your 18 client for example ping the IP of pfsense lan11 interface?

No, 10.10.18.253 cannot ping 10.10.11.254 (pfSense)
However, 10.10.11.20 can ping 10.10.18.254 (pfSense)

Other common issue is device on vlans not even using pfsense as their gateway, etc.

They all are, if they would not they wouldn't be able to reach the Internet via pfSense.

Other common issue is wrong mask setup either on client or pfsense.

Just checked, it's /24 on all LAN interfaces inside pfSense.
However, you hit the nail on the head, the mask was right on DHCP based clients, but for some idiotic reason, I set it to /16 on the static IPs that all happen to be in the LAN18 network.
So obviously, those thought that 10.10.11.0/24 was within their reach.

Sorry for wasting your time you with something so obvious.

johnpoz

@obones said in Communication between LANs does not work:

Sorry for wasting your time you with something so obvious.

Not a waste of time - actually turned out to be a good run through of all the different common things.. Thanks for doing the sniff and showing pfsense was sending out the traffic.

And then reporting back that problem with mask on client.

This for sure could help the next guy!!

As to nat and this response?

Well, that would allow me to segregate traffic from one LAN to the other

Huh??? not understand your thought process here.. They are segmented already, you have a firewall between them, etc. What would be the purpose of natting??

KOM

I think he's confusing NAT with general access.

KOM

@scottsmith101 Read the thread. His problem is already solved.

obones

@johnpoz said in Communication between LANs does not work:

@obones said in Communication between LANs does not work:

Sorry for wasting your time you with something so obvious.

Not a waste of time - actually turned out to be a good run through of all the different common things.. Thanks for doing the sniff and showing pfsense was sending out the traffic.

And then reporting back that problem with mask on client.

I wrote the post as I tested the various things, so it felt as a waste of time to discard all I already had typed.

Well, that would allow me to segregate traffic from one LAN to the other

Huh??? not understand your thought process here.. They are segmented already, you have a firewall between them, etc. What would be the purpose of natting??

Maybe I did not make myself clear enough, but I have two LAN interfaces (LAN11 and LAN18) and I will be adding more interfaces in the future, all with their own 10.10.x.0/24 subnet.
So, to me, this means there is some sort of address translation when a packet goes from LAN11 to LAN18 and back. Pretty much the same when a packet goes from LAN11 to the Internet and back.
As it turns out, this works out of the box, so there is nothing to configure on pfSense for this to happen.

In the end, what I want is that all LANs can go to LAN18 or the outside world, but cannot send packets to each other. But that, I will manage with some firewall rules.

Derelict

So, to me, this means there is some sort of address translation when a packet goes from LAN11 to LAN18 and back. Pretty much the same when a packet goes from LAN11 to the Internet and back.

There is no translation there. Only routing.

When the LAN11 host sends traffic to a LAN18 host it sees the destination host is not on its local network so it consults its routing table and probably sends the traffic to its default gateway.

The router receives the traffic and consults its routing table and sees the LAN18 network matches, ARPs for the destination host if necessary and sends the traffic out that interface.

When the LAN18 host receives the traffic and has to reply, it sees the destination host is not on its local network so it consults its routing table and probably sends the traffic to its default gateway.

The router receives the traffic and consults its routing table and sees the LAN11 network matches, ARPs for the destination host if necessary and sends the traffic out that interface.

obones

Of course !

I'm so used to the fact that "private" IPs need to be translated before going to the outside world that I forgot about basic routing.

Thanks for the reminder.

morghan0321

This post is deleted!