Setting up new device on LAN

stephenw10 · May 11, 2022, 11:53 PM

Try running a packet capture on WAN in promiscuous mode. You should see at least broadcast traffic from the other hosts in the subnet.

peterlecki · May 12, 2022, 12:06 AM

@stephenw10
Interesting!

ARP, Request who-has 192.168.4.1 tell 192.168.4.244

So the 4.1 gateway is not responding. Yet it responds to the 4.100 host. Plus the 4.1 device shows 4.244's MAC in its own ARP table. But never responds to the request? I am fucking tripping, man.

stephenw10 · May 12, 2022, 1:10 AM

Ah, well hallucinogenic substances is one explanation.

But is it fact responding and the pfSense WAN simply never receives it...

Try pinging the 4.100 host whilst running a pcap. It should ARP for that too and should see a response.

peterlecki · May 12, 2022, 1:10 AM

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

keyser · May 12, 2022, 8:40 AM

@peterlecki said in Setting up new device on LAN:

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

Any chance you have entered a subnetmask on the new pfSense interface by error as /25 or higher?

keyser · May 12, 2022, 10:17 AM

@peterlecki said in Setting up new device on LAN:

@stephenw10
I do see the ARP request for 4.100 and the reply on the pfSense capture.
I also ran a promiscuous capture on the 4.100 host and can see ARP requests from 4.244 for 4.1 but 4.1 never responds. I can see it respond to 4.100 but it never responds to 4.244, as if it is completely ignoring any and all packets from that host.

I just tried placing my SG-2100 behind my primary pfSense, and I am seeing the exact same issue. My downstream pfsense gets a DHCP IP from the primary, but after that any packets sent from the downstream device arrives at the primary, but NO packets are sent as a reply out the LAN interface. Even though states are created, allowed, and nothing is blocked on the primary pfSense.... It's as if it completely ignores that particular device.
A force ping towards the downlevel pfSense from the primary is never transmitted from the LAN interface. Any other Ping towards other devices on the same interface works just fine.

I'm baffled right now.....

keyser · May 12, 2022, 11:15 AM

@keyser WTF.....

When I force ping the downlevel Firewall from the primary, the Ping request goes out the WAN interface - regardless if I auto source it or select the LAN interface as source.

For the one particular IP address of the downlevel pfSense (its WAN), my primary pfSense ignores even local connected routing entries and transmits packets toward it on WAN (internet).

WTF?

EDIT: Looking at the primary's routing table there is a entry for the downlevel pfSenses IP address that uses the WAN gateway. So that entry was somehow created, and I just found out how:

This issue arises because there is configured an IPsec tunnel (s2s) between the devices based on DNS names (from ealier on) that obviously can't come up. But the gateway routing line comes from the IPsec S2S definition as that uses the DNS name of the downlevel pfSense (which I updated to a LAN address so I could reach it....)

So IPsec S2S was the culprit here.... My mistake.....

keyser · May 12, 2022, 2:59 PM

@keyser Final observation:

There seems to be a bug in pfSense as any static routes created out of WAN from a Site2Site gateway definition never expires or gets deleted.
To get rid of them requires a reboot.

As I change addresses on the downlevel device more and more static routes are added to the primary, and they have no expiration.
Neither do they get deleted if I stop the IPsec Service or disable the Site2Site VPN Phase1. Only a full reboot removes the entries.

stephenw10 · May 12, 2022, 4:02 PM

@keyser said in Setting up new device on LAN:

Any chance you have entered a subnetmask on the new pfSense interface by error as /25 or higher?

If that was the case it would not ARP for 4.100.

However a /25 mask on the upstream router might present like this.

Try changing the pfSense WAN IP to something inside that like 4.99.

Steve

luckman212 · May 12, 2022, 2:59 PM

@keyser That could be related to the bug I just encountered: https://redmine.pfsense.org/issues/13153

peterlecki · May 12, 2022, 4:02 PM

@stephenw10
I double checked the mask and it was 24. I also changed the IP to 4.99 but it made no difference. From 4.100 I'm able to ping 4.99 and vice versa, ping from 4.99 to 4.100 BUT no comm between 4.1 and 4.99 in either direction. My upstream is a basic SOHO consumer device so I can't see routing tables like @keyser saw in his. I'll try bypassing my upstream device and make pfSense the primary gateway.

stephenw10 · May 12, 2022, 4:39 PM

Hmm, bizarre. Some stale ARP cache somewhere? MAC address conflict?

johnpoz · May 12, 2022, 4:42 PM

@stephenw10 what is the arp table look like on the 4.1 device?

If it has a entry for whatever mac pfsense interface IS? or the IP, etc.

stephenw10 · May 12, 2022, 4:53 PM

No way to see it on the ISP router.

peterlecki · May 12, 2022, 4:59 PM

@johnpoz @stephenw10
4.1 is not ISP, it's my private device and it has 4.244's correct MAC in its ARP table. All devices had multiple reboots to clear any caches.

johnpoz · May 12, 2022, 5:56 PM

@stephenw10 said in Setting up new device on LAN:

No way to see it on the ISP router.

You sure he stated

" Plus the 4.1 device shows 4.244's MAC in its own ARP table."

You validated this pfsense actual mac for its wan interface?

Change the interface on pfsense so you get a different mac, or clone the mac to something.. But again there is no way your going to talk to 4.1 if pfsense has no mac.

try setting a static mac for 4.1 on pfsense if your saying 4.1 has the mac address of pfsense.

But you got something crazy going on if your saying pfsense arps for 100 and all work fine, but 4.1 has the mac of pfsense but pfsense doesn't have the mac for 4.1.. You see pfsense send out arp for 4.1 but you do not get an answer?

edit:
Can you post the mac address of both your 4.1 device and pfsense interface your connecting.. There is zero reason to hide the mac.. but if you want you can leave off the last 3... Just curious if a multicast mac.. I had a device one time that was like a bridge for a current cost device that measured power usage for whole house. And it had some oddness - they had used a multicast mac on it... Which can cause some weirdness..

stephenw10 · May 12, 2022, 6:14 PM

You can see it here: https://forum.netgate.com/post/1041334
Seems fine.

My mistake on the ISP router. What exactly in the router at 4.1 then?

johnpoz · May 12, 2022, 8:22 PM

@stephenw10 yeah that mac is not multicast - but what is the mac of this 4.1 device?

This should be simple enough to figure out..

Sniff on both devices - arp from 4.1 for 4.244 or 4.99 whatever pfsense IP is.. Now arp from pfsense for 4.1

Do the other devices see the arp from the other device, do they respond or not.. If they respond is the other device actually seeing it in the sniff?

If device is not reply to the arp, or not sending it out the right interface - then there is something wrong with that device..

But not see a mac for 4.1 in your arp table - you are not going to be able to talk to 4.1 that is for damn sure.. You could see if setting static arp in pfsense for 4.1 allows communication - but that wouldn't be an actual fix, that would be just a work around. Understanding why the normal arp function is not working would be the fix, something is odd for damn sure..

if just a dumb switch.. there really is nothing it could be doing.. It would have an arp cache that is how it knows what mac is on what switch port... But that shouldn't really last that long, have you rebooted the switch? And you have moved pfsense to different port on the switch?

peterlecki · May 12, 2022, 9:21 PM

@johnpoz @stephenw10
I want to give you guys props for being so active in trying to help me with this. This was a very weird problem that was making me doubt myself. I finally gave up and reflashed. Now it works fine, so it must have been some odd setting I changed somewhere but it was escaping me. The reflash also resolved the login and save delays of almost two minutes every time.

johnpoz · May 12, 2022, 9:28 PM

@peterlecki said in Setting up new device on LAN:

I finally gave up and reflashed.

Reflashed what? Your 4.1 device? You did a clean install of pfsense? Not sure I would call that a flash? Flash is something you do to bios or firmware.. Installing pfsense clean is not something I would consider a "flash"

Redoing say dd-wrt firmware on some soho router is a flash ;)

Glad you got it sorted.. But now we will never actually know what was going on..