Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A

dwhacks

Been away for a couple days, so I haven't tried any suggestions. BUT everything worked for about 15 minutes when I tried it the last couple hours. It no longer works, and I can't even ping site B from site A....

When I try to ping from the shell on site A pfsense:

[2.7.2-RELEASE][admin@pfsense.localhacks.lan]/root: ping dwhacks.com
PING dwhacks.com (24.71.68.91): 56 data bytes
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied
ping: sendto: Permission denied
^C
--- dwhacks.com ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss

I can ping other hosts, like google.ca

I will try some of the suggestions tomorrow.

dwhacks

@Gblenn Traceroute seems to go nowhere:

1 traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1
 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1
 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1
 *
 2 traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1
 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1
 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1
 *
 etc (does this 18 times)

Gblenn

@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:

BUT everything worked for about 15 minutes when I tried it the last couple hours. It no longer works, and I can't even ping site B from site A....

Sounds like you have something that kicks in and blocks it? Are you running Suricata/Snort?

stephenw10

Yup 'permission denied' like that is a local block and I'd bet that Snort or Suricata in blocking mode. Unless it's the ISP router doing some active blocking.

dwhacks

@stephenw10 @Gblenn

Looks like you are both correct, and its SNORT. But I can't figure out why.

Here are some screenshorts of the blocked list after clearing it, and accessing the webpage and or SSHing into the server.

login-to-view
login-to-view
login-to-view

They don't seem to have the IP of site B, but its at this point when things stopped working.

With snort disabled, things seem to work, with a few little slowdowns/lockups over ssh. Not sure if its just SSH though, but my session will freeze after a couple commands and doesn't even say "broken pipe".... could be unrelated.

stephenw10

Hmm, the ssh failure could be some asymmetry in the route somehow.

I agree it doesn't look like anything in that list should be blocking it. Is that on site B? Could be blocking outbound on site A?

Gblenn

@stephenw10 said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:

Is that on site B? Could be blocking outbound on site A?

Yes check site B as well for clues... and look into the Alert page as well. And search for any references to the server IP (internal IP).
When testing, clear the Blocked IP list completely, and then run a ping towards the server and see when it shows up in the block list. Also try accessing it the normal way HTTP/HTTPS and try SSHing into it.

dwhacks

@stephenw10 said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:

........Is that on site B? Could be blocking outbound on site A?

SNORT is on site A, and this is from that network. Site B has a barebones PFsense, I don't think I have any packages installed yet.

I ran a capture on both machines at the same time while accessing the site with SNORT disabled on site A and they both look basically the same. During this, I was all SSHd into the server on site B. I ran Neofetch, and then nano test and save, and then nano test and it seemed to freeze. After a minute or two I notice the test file was now open in the terminal, but still frozen. If I leave the SSH terminal open it will eventually "broken pipe".

This is all with snort disabled

Here is Site A Capture
login-to-view

and Site B
login-to-view

Both while snort is disabled.

dwhacks

@Gblenn said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:

When testing, clear the Blocked IP list completely, and then run a ping towards the server and see when it shows up in the block list. Also try accessing it the normal way HTTP/HTTPS and try SSHing into it.

No internal IP's are referenced in snort on site A

When Re-enabling snort, it doesn't seem to be blocking access to site B completely. I will poke around on the site and it works fine, go away for a minutes and then it seems frozen, but If I refresh or F5 then it comes back fine.... this was not the behavior before.

Here's the downloads from snort while poking around on the site. SSH froze up/slowed down again.
I do not see any references to site B in the site A snort blocked table. I also didn't think snort did anything with outbound traffic.

snort_logs_2025-04-16-15-43-36_igb1.tar.gz snort_blocked_2025-04-16-15-43-56.tar.gz

stephenw10

Need to see the full pcap really. Or try filtering just the port 22 SSH traffic until it fails. That should show missing traffic on one side.

Or do you perhaps see ARP calls from either side when fails?

dwhacks

Seems like Snort finally blocked it again (still assuming its snort) but I don't see anything in the blocked list or alerts:
snort_blocked_2025-04-16-15-56-10.tar.gz
snort_logs_2025-04-16-15-56-01_igb1.tar.gz

Clearing the blocked table grants me access again....

until these 4 end up in the blocked table:
login-to-view
and this in the alerts:
login-to-view

I'm thinking its that IP ending in .188

I will disable snort, and to a pcap with ssh. and see what shows up

dwhacks

@stephenw10 said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:

Need to see the full pcap really. Or try filtering just the port 22 SSH traffic until it fails. That should show missing traffic on one side.

Here is a pcap from site A while SSH until it hangs. You can see where traffic significantly slows down. (I don't know if that shows anything)

packetcapture-igb1-20250416160523.pcap
login-to-view

I will try the same from Site B in a bit.

dwhacks

and here is the pcap file from Site B. I left pcap running for about 2 minutes after ssh halted.

packetcapture-vtnet0-20250416231312.pcap

stephenw10

Ah, OK. So you can see in the pcaps that at some point site B stops seeing the packets from site A.

Site A is still seeing the traffic from site B and is replying but that never reaches site B.

But check the MAC addresses on the traffic. There is traffic asymmetry.

Site A sends traffic:
Ethernet II, Src: IntelCor_ad:23:1f (90:e2:ba:ad:23:1f), Dst: Cadant_aa:08:45 (00:01:5c:aa:08:45)

But receives traffic:
Ethernet II, Src: bc:24:11:41:f2:d6 (bc:24:11:41:f2:d6), Dst: IntelCor_ad:23:1f (90:e2:ba:ad:23:1f)

Site B sends traffic:
Ethernet II, Src: bc:24:11:41:f2:d6 (bc:24:11:41:f2:d6), Dst: IntelCor_ad:23:1f (90:e2:ba:ad:23:1f)

But receives traffic:
Ethernet II, Src: Cadant_aa:08:45 (00:01:5c:aa:08:45), Dst: bc:24:11:41:f2:d6 (bc:24:11:41:f2:d6)

So it looks like site A is not sending to the site B MAC directly but via 00:01:5c:aa:08:45. Which I assume is the ISP router?

After some time some ICMP redircet probably expires and traffic is no longer forwarded.

So it looks like you have an ARP issue. Check the ARP table on each firewall. Make sure they are actually in the same subnet.

dwhacks

@stephenw10

Here is a screenshot of the arp table from site A:
login-to-view

And here is site B:
login-to-view

It seems to me like they should be ok?

EDIT: looks like site B has two different MAC's???? how? Possible from when it was connected to Site A?

Gblenn

@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:

EDIT: looks like site B has two different MAC's???? how? Possible from when it was connected to Site A?

Yes the MAC for WAN (24.71.68.91) on site B is Proxmox, whereas on site A it is Cadant and same as the .1 address (which I assume is related to your ISP?).

I would try/check these things:

Clear the ARP table on Site A to see if it shows up correctly.
Switch cables for Site A and B on the ISP modem, to see that it's not related to the modem. Possibly the problem would be the reverse if it is not configured correctly?
Check the bridge on Proxmox for the pfsense WAN, or rather reassign it as pass through.
This does require some work on your part: Remove the devices and add them back in the same order you added them before but select PCI device this time, RAW. Then in pfsense you need to open the pfsense VM shell during boot, and assign WAN and LAN again to the correct interface. They will change from vtnet to something else igb0, 1 or ix0, 1 depending on HW.

stephenw10

Site B looks correct. What lloks wrong is that site A has the same MAC for 24.71.68.1 and 24.71.68.91. The .91 address is the site B WAN and should have it's own MAC.

I assume both those are connected to the ISP router dircetly? But I had assumed those ports on the router were just a switch and that doesn't seem to be the case. The ISP router is responding to ARP requests for site B's WAN for some reason. A real switch wouldn't do that.

I would still argue you should be using just one pfSense instance here with both LAN behind it on separate subnets. That would remove this issue entirely.

But it looks like the problem here is that the ISPs router is misbehaving.

Edit: Good point, it could be Proxmox doing something odd. But I run that here and have never seen it do weird layer 2 stuff like that.

Gblenn

@stephenw10 said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:

What lloks wrong is that site A has the same MAC for 24.71.68.1 and 24.71.68.91

Exactly, and site A was the one that had .91 before. So if there is some stale info in the ARP table then flushing it may fix it I suppose?

@stephenw10 said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:

Edit: Good point, it could be Proxmox doing something odd. But I run that here and have never seen it do weird layer 2 stuff like that.

I agree, I use Proxmox for two firewalls in a similar setup, and I have had no trouble with vtnet or direct NIC's. Although I do have a switch on the WAN, not a router. But considering that Proxmox is the one thing that is different vs site A, I'm thinking it may be part of the problem.