Setting up new device on LAN

peterlecki · May 11, 2022, 4:54 PM

@johnpoz @stephenw10
This was an ID10T error. I had wireless active on the client device I was testing from so of course it was accessing the internet via the normal gateway. I now noticed this after you pointed it out and turned it off. Now it no longer can access the internet. Apologies for wasting your time and energy and thank you for pointing this out.

Anyway, now that I no longer complicate the situation even further with "fake news", let's get back to the issue at hand. John, blocked out in that screenshot was my real ISP gateway IP. The WANGW_TEMP was configured as the gateway for the WAN interface. So the public WANGW gateway was available but not used. Just to eliminate any potential problems, I now completely removed it:

And here are the icmp states:

stephenw10 · May 11, 2022, 5:31 PM

Ok, so the states look correct but your temporary WAN gateway is not responding so the pings go nowhere.
I assume it is not in the ARP table still?

So what is 192.168.4.1? How is the WAN interface connected to it?

I would not expect the public WAN gateway to be available at the same time if there is another router in between.

Steve

johnpoz · May 11, 2022, 8:38 PM

@peterlecki here is the thing if 192.168.4.1 is actually suppose to be pfsense gateway. It needs to answer arp.. If no then no there is no way its going to do anything, answer ping pass traffic nothing.

So either you do not have pfsense wan IP plugged into the correct port, or 192.168.4.1 is not the correct IP..

Your saying if you plug pfsense into this router it gets that wan IP via dhcp? But can not arp for 192.168.4.1?? You sure that is what is suppose to be used for the gateway?

If I had a router and its IP was say 192.168.4.254, I could hand out dhcp - in the 192.168.4.x network - but if I told dhcp clients 192.168.4.1 was the gateway when there is no 192.168.4.1 then no you wouldn't work couldn't arp for it..

So your saying when wifi clients are on wifi they get 192.168.4.1 as their gateway and they can ping it.. Well then seems whatever your plugging pfsense wire into isn't that..

peterlecki · May 11, 2022, 8:58 PM

@johnpoz @stephenw10

4.1 does answer ARP and I can see it from PC 4.100: 192.168.4.1 70-4f-57-01-24-b0
but pfSense still shows MAC of 4.1 as "incomplete" in the ARP table.

from 4.100 I can ping 4.244 and 4.1
from 4.1 I can ping 4.100 but cannot ping 4.244
so it does seem as if the wires were not correctly plugged in somewhere but 4.1 has just ONE wire going into it but only 4.100 can ping it but 4.244 cannot

stephenw10 · May 11, 2022, 10:08 PM

Try swapping the wires or the ports on the switch.

There is something low level failing.

I assume you see link LEDs on the WAN port and switch?

It could be a bad NIC you could reassign the WAN to a different port.

Steve

johnpoz · May 11, 2022, 10:19 PM

@peterlecki said in Setting up new device on LAN:

from 4.100 I can ping 4.244 and 4.1

And pfsense sees the mac of .100 in its arp table but not .1?

Is that just a dumb switch? Or a vlan cable switch? Or smart switch - could be doing private vlan setup that is not correct for how you want to use it, etc.

peterlecki · May 11, 2022, 11:48 PM

@stephenw10
LEDs are the same on WAN and LAN ports. I just switched them around and LAN works on the previous WAN port and WAN still doesn't work on the previous LAN port. So it's not the hardware.

peterlecki · May 11, 2022, 11:51 PM

@johnpoz
Correct, pfSense can see 4.100 but not 4.1

It's a dumb switch in between them.

stephenw10 · May 11, 2022, 11:52 PM

You tried swapping the swotch ports the pfSense WAN and laptop are connected to?

Because some sort of private VLAN setup on the switch could present like this as @johnpoz said.
Edit: Missed your update

Steve

stephenw10 · May 11, 2022, 11:53 PM

Try running a packet capture on WAN in promiscuous mode. You should see at least broadcast traffic from the other hosts in the subnet.

peterlecki · May 12, 2022, 12:06 AM

@stephenw10
Interesting!

ARP, Request who-has 192.168.4.1 tell 192.168.4.244

So the 4.1 gateway is not responding. Yet it responds to the 4.100 host. Plus the 4.1 device shows 4.244's MAC in its own ARP table. But never responds to the request? I am fucking tripping, man.

stephenw10 · May 12, 2022, 1:10 AM

Ah, well hallucinogenic substances is one explanation.

But is it fact responding and the pfSense WAN simply never receives it...

Try pinging the 4.100 host whilst running a pcap. It should ARP for that too and should see a response.

peterlecki · May 12, 2022, 1:10 AM

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

keyser · May 12, 2022, 8:40 AM

@peterlecki said in Setting up new device on LAN:

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

Any chance you have entered a subnetmask on the new pfSense interface by error as /25 or higher?

keyser · May 12, 2022, 10:17 AM

@peterlecki said in Setting up new device on LAN:

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

I just tried placing my SG-2100 behind my primary pfSense, and I am seeing the exact same issue. My downstream pfsense gets a DHCP IP from the primary, but after that any packets sent from the downstream device arrives at the primary, but NO packets are sent as a reply out the LAN interface. Even though states are created, allowed, and nothing is blocked on the primary pfSense.... It's as if it completely ignores that particular device.
A force ping towards the downlevel pfSense from the primary is never transmitted from the LAN interface. Any other Ping towards other devices on the same interface works just fine.

I'm baffled right now.....

keyser · May 12, 2022, 11:15 AM

@keyser WTF.....

When I force ping the downlevel Firewall from the primary, the Ping request goes out the WAN interface - regardless if I auto source it or select the LAN interface as source.

For the one particular IP address of the downlevel pfSense (its WAN), my primary pfSense ignores even local connected routing entries and transmits packets toward it on WAN (internet).

WTF?

EDIT: Looking at the primary's routing table there is a entry for the downlevel pfSenses IP address that uses the WAN gateway. So that entry was somehow created, and I just found out how:

This issue arises because there is configured an IPsec tunnel (s2s) between the devices based on DNS names (from ealier on) that obviously can't come up. But the gateway routing line comes from the IPsec S2S definition as that uses the DNS name of the downlevel pfSense (which I updated to a LAN address so I could reach it....)

So IPsec S2S was the culprit here.... My mistake.....

keyser · May 12, 2022, 2:59 PM

@keyser Final observation:

There seems to be a bug in pfSense as any static routes created out of WAN from a Site2Site gateway definition never expires or gets deleted.
To get rid of them requires a reboot.

As I change addresses on the downlevel device more and more static routes are added to the primary, and they have no expiration.
Neither do they get deleted if I stop the IPsec Service or disable the Site2Site VPN Phase1. Only a full reboot removes the entries.

stephenw10 · May 12, 2022, 4:02 PM

@keyser said in Setting up new device on LAN:

Any chance you have entered a subnetmask on the new pfSense interface by error as /25 or higher?

If that was the case it would not ARP for 4.100.

However a /25 mask on the upstream router might present like this.

Try changing the pfSense WAN IP to something inside that like 4.99.

Steve

luckman212 · May 12, 2022, 2:59 PM

@keyser That could be related to the bug I just encountered: https://redmine.pfsense.org/issues/13153

peterlecki · May 12, 2022, 4:02 PM

@stephenw10
I double checked the mask and it was 24. I also changed the IP to 4.99 but it made no difference. From 4.100 I'm able to ping 4.99 and vice versa, ping from 4.99 to 4.100 BUT no comm between 4.1 and 4.99 in either direction. My upstream is a basic SOHO consumer device so I can't see routing tables like @keyser saw in his. I'll try bypassing my upstream device and make pfSense the primary gateway.