Setting up new device on LAN

stephenw10 · May 11, 2022, 10:08 PM

Try swapping the wires or the ports on the switch.

There is something low level failing.

I assume you see link LEDs on the WAN port and switch?

It could be a bad NIC you could reassign the WAN to a different port.

Steve

johnpoz · May 11, 2022, 10:19 PM

@peterlecki said in Setting up new device on LAN:

from 4.100 I can ping 4.244 and 4.1

And pfsense sees the mac of .100 in its arp table but not .1?

Is that just a dumb switch? Or a vlan cable switch? Or smart switch - could be doing private vlan setup that is not correct for how you want to use it, etc.

peterlecki · May 11, 2022, 11:48 PM

@stephenw10
LEDs are the same on WAN and LAN ports. I just switched them around and LAN works on the previous WAN port and WAN still doesn't work on the previous LAN port. So it's not the hardware.

peterlecki · May 11, 2022, 11:51 PM

@johnpoz
Correct, pfSense can see 4.100 but not 4.1

It's a dumb switch in between them.

stephenw10 · May 11, 2022, 11:52 PM

You tried swapping the swotch ports the pfSense WAN and laptop are connected to?

Because some sort of private VLAN setup on the switch could present like this as @johnpoz said.
Edit: Missed your update

Steve

stephenw10 · May 11, 2022, 11:53 PM

Try running a packet capture on WAN in promiscuous mode. You should see at least broadcast traffic from the other hosts in the subnet.

peterlecki · May 12, 2022, 12:06 AM

@stephenw10
Interesting!

ARP, Request who-has 192.168.4.1 tell 192.168.4.244

So the 4.1 gateway is not responding. Yet it responds to the 4.100 host. Plus the 4.1 device shows 4.244's MAC in its own ARP table. But never responds to the request? I am fucking tripping, man.

stephenw10 · May 12, 2022, 1:10 AM

Ah, well hallucinogenic substances is one explanation.

But is it fact responding and the pfSense WAN simply never receives it...

Try pinging the 4.100 host whilst running a pcap. It should ARP for that too and should see a response.

peterlecki · May 12, 2022, 1:10 AM

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

keyser · May 12, 2022, 8:40 AM

@peterlecki said in Setting up new device on LAN:

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

Any chance you have entered a subnetmask on the new pfSense interface by error as /25 or higher?

keyser · May 12, 2022, 10:17 AM

@peterlecki said in Setting up new device on LAN:

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

I just tried placing my SG-2100 behind my primary pfSense, and I am seeing the exact same issue. My downstream pfsense gets a DHCP IP from the primary, but after that any packets sent from the downstream device arrives at the primary, but NO packets are sent as a reply out the LAN interface. Even though states are created, allowed, and nothing is blocked on the primary pfSense.... It's as if it completely ignores that particular device.
A force ping towards the downlevel pfSense from the primary is never transmitted from the LAN interface. Any other Ping towards other devices on the same interface works just fine.

I'm baffled right now.....

keyser · May 12, 2022, 11:15 AM

@keyser WTF.....

When I force ping the downlevel Firewall from the primary, the Ping request goes out the WAN interface - regardless if I auto source it or select the LAN interface as source.

For the one particular IP address of the downlevel pfSense (its WAN), my primary pfSense ignores even local connected routing entries and transmits packets toward it on WAN (internet).

WTF?

EDIT: Looking at the primary's routing table there is a entry for the downlevel pfSenses IP address that uses the WAN gateway. So that entry was somehow created, and I just found out how:

This issue arises because there is configured an IPsec tunnel (s2s) between the devices based on DNS names (from ealier on) that obviously can't come up. But the gateway routing line comes from the IPsec S2S definition as that uses the DNS name of the downlevel pfSense (which I updated to a LAN address so I could reach it....)

So IPsec S2S was the culprit here.... My mistake.....

keyser · May 12, 2022, 2:59 PM

@keyser Final observation:

There seems to be a bug in pfSense as any static routes created out of WAN from a Site2Site gateway definition never expires or gets deleted.
To get rid of them requires a reboot.

As I change addresses on the downlevel device more and more static routes are added to the primary, and they have no expiration.
Neither do they get deleted if I stop the IPsec Service or disable the Site2Site VPN Phase1. Only a full reboot removes the entries.

stephenw10 · May 12, 2022, 4:02 PM

@keyser said in Setting up new device on LAN:

Any chance you have entered a subnetmask on the new pfSense interface by error as /25 or higher?

If that was the case it would not ARP for 4.100.

However a /25 mask on the upstream router might present like this.

Try changing the pfSense WAN IP to something inside that like 4.99.

Steve

luckman212 · May 12, 2022, 2:59 PM

@keyser That could be related to the bug I just encountered: https://redmine.pfsense.org/issues/13153

peterlecki · May 12, 2022, 4:02 PM

@stephenw10
I double checked the mask and it was 24. I also changed the IP to 4.99 but it made no difference. From 4.100 I'm able to ping 4.99 and vice versa, ping from 4.99 to 4.100 BUT no comm between 4.1 and 4.99 in either direction. My upstream is a basic SOHO consumer device so I can't see routing tables like @keyser saw in his. I'll try bypassing my upstream device and make pfSense the primary gateway.

stephenw10 · May 12, 2022, 4:39 PM

Hmm, bizarre. Some stale ARP cache somewhere? MAC address conflict?

johnpoz · May 12, 2022, 4:42 PM

@stephenw10 what is the arp table look like on the 4.1 device?

If it has a entry for whatever mac pfsense interface IS? or the IP, etc.

stephenw10 · May 12, 2022, 4:53 PM

No way to see it on the ISP router.

peterlecki · May 12, 2022, 4:59 PM

@johnpoz @stephenw10
4.1 is not ISP, it's my private device and it has 4.244's correct MAC in its ARP table. All devices had multiple reboots to clear any caches.