Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
What should I be looking for in Diagnostics -> States ?
Looks for NAT'd traffic from the internal host you're testing from.
But if ping is failing then check the firewall rules on site B. You might just not be passing ping there but it would be much easier to test of you do.
Are those two public IPs actually in the same subnet?
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
as per the screenshots I have attached I do not see any conflicts.
But if site A has a host override for daynewaterlow.com pointing to 192.168.2.7, and that IP is no longer valid, since daynewaterlow.com is now located at Site B, of course it will not work...
-
@Gblenn daynewaterlow.com is on site A, netbird.dwhacks.com is on site B. netbird.dwhacks.com is not accessible from the network of site A.
I haven't moved all my webservers over to site B yet because I am unable to access them from site A so that makes admin stuff tricky. I would like to move them eventually.
@stephenw10 I cannot see anything in states that points to the IP of site B at any point. I think they are probable on the same subnet of my ISP but I'm not sure how to check. I believe PING is allowed.
-
I added a rule to firewall on site B and I can now ping from site A:
I still cannot access the netbird.dwhacks.com from site A though.
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
I cannot see anything in states that points to the IP of site B at any point.
In either firewall?
Is the client behind firewall A resolving the url to the firewall B WAN?
If it is you should at least see some states in firewall A. If you don't then it could be a firewall rule on the A LAN blocking it (or not passing it).
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
I did think of something while I was making another coffee: before setting up the router for site B I have dual WAN at site A. My ISP almost never changes the IP addresses, is it possible there is some residual config or states or something left over in my site A router making it think that the IP of site B is supposed to go to itself? Maybe is DHCP as thats what the routers use on WAN?
So Site A is (or was) "aware" of the IP that now resides on Site B then... well yes I suppose there could be some residual states or something which makes it think it should go back to itself. Similar to what I was after with the host override rules, which however you have clearly changed. I assume you have removed WAN2 completely, as well as any gateway groups etc that you may have had. And how about services Dynamic DNS, on site A is that also updated to reflect that dwhacks is no longer on that IP?
Try tracert to dwhacks.com to see what you get from pfsense on site A... That should give you a clue as to what it's doing.
Also you can run Pcap on WAN whilst pinging dwhacks to see if it shows up. Add a filter with the public IP for Site B so you only capture the important information.
-
Been away for a couple days, so I haven't tried any suggestions. BUT everything worked for about 15 minutes when I tried it the last couple hours. It no longer works, and I can't even ping site B from site A....
When I try to ping from the shell on site A pfsense:
[2.7.2-RELEASE][admin@pfsense.localhacks.lan]/root: ping dwhacks.com PING dwhacks.com (24.71.68.91): 56 data bytes ping: sendto: Permission denied ping: sendto: Permission denied ping: sendto: Permission denied ping: sendto: Permission denied ^C --- dwhacks.com ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet lossI can ping other hosts, like google.ca
I will try some of the suggestions tomorrow.
-
@Gblenn Traceroute seems to go nowhere:
1 traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 * 2 traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 * etc (does this 18 times) -
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
BUT everything worked for about 15 minutes when I tried it the last couple hours. It no longer works, and I can't even ping site B from site A....
Sounds like you have something that kicks in and blocks it? Are you running Suricata/Snort?
-
Yup 'permission denied' like that is a local block and I'd bet that Snort or Suricata in blocking mode. Unless it's the ISP router doing some active blocking.
-
Looks like you are both correct, and its SNORT. But I can't figure out why.
Here are some screenshorts of the blocked list after clearing it, and accessing the webpage and or SSHing into the server.
They don't seem to have the IP of site B, but its at this point when things stopped working.
With snort disabled, things seem to work, with a few little slowdowns/lockups over ssh. Not sure if its just SSH though, but my session will freeze after a couple commands and doesn't even say "broken pipe".... could be unrelated.
-
Hmm, the ssh failure could be some asymmetry in the route somehow.
I agree it doesn't look like anything in that list should be blocking it. Is that on site B? Could be blocking outbound on site A?
-
@stephenw10 said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
Is that on site B? Could be blocking outbound on site A?
Yes check site B as well for clues... and look into the Alert page as well. And search for any references to the server IP (internal IP).
When testing, clear the Blocked IP list completely, and then run a ping towards the server and see when it shows up in the block list. Also try accessing it the normal way HTTP/HTTPS and try SSHing into it. -
@stephenw10 said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
........Is that on site B? Could be blocking outbound on site A?
SNORT is on site A, and this is from that network. Site B has a barebones PFsense, I don't think I have any packages installed yet.
I ran a capture on both machines at the same time while accessing the site with SNORT disabled on site A and they both look basically the same. During this, I was all SSHd into the server on site B. I ran Neofetch, and then nano test and save, and then nano test and it seemed to freeze. After a minute or two I notice the test file was now open in the terminal, but still frozen. If I leave the SSH terminal open it will eventually "broken pipe".
This is all with snort disabled
Here is Site A Capture
and Site B
Both while snort is disabled.
-
@Gblenn said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
When testing, clear the Blocked IP list completely, and then run a ping towards the server and see when it shows up in the block list. Also try accessing it the normal way HTTP/HTTPS and try SSHing into it.
No internal IP's are referenced in snort on site A
When Re-enabling snort, it doesn't seem to be blocking access to site B completely. I will poke around on the site and it works fine, go away for a minutes and then it seems frozen, but If I refresh or F5 then it comes back fine.... this was not the behavior before.
Here's the downloads from snort while poking around on the site. SSH froze up/slowed down again.
I do not see any references to site B in the site A snort blocked table. I also didn't think snort did anything with outbound traffic.snort_logs_2025-04-16-15-43-36_igb1.tar.gz snort_blocked_2025-04-16-15-43-56.tar.gz
-
Need to see the full pcap really. Or try filtering just the port 22 SSH traffic until it fails. That should show missing traffic on one side.
Or do you perhaps see ARP calls from either side when fails?
-
Seems like Snort finally blocked it again (still assuming its snort) but I don't see anything in the blocked list or alerts:
snort_blocked_2025-04-16-15-56-10.tar.gz
snort_logs_2025-04-16-15-56-01_igb1.tar.gzClearing the blocked table grants me access again....
until these 4 end up in the blocked table:
and this in the alerts:
I'm thinking its that IP ending in .188
I will disable snort, and to a pcap with ssh. and see what shows up
-
@stephenw10 said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
Need to see the full pcap really. Or try filtering just the port 22 SSH traffic until it fails. That should show missing traffic on one side.
Here is a pcap from site A while SSH until it hangs. You can see where traffic significantly slows down. (I don't know if that shows anything)
packetcapture-igb1-20250416160523.pcap
I will try the same from Site B in a bit.
-
and here is the pcap file from Site B. I left pcap running for about 2 minutes after ssh halted.
-
Ah, OK. So you can see in the pcaps that at some point site B stops seeing the packets from site A.
Site A is still seeing the traffic from site B and is replying but that never reaches site B.
But check the MAC addresses on the traffic. There is traffic asymmetry.
Site A sends traffic:
Ethernet II, Src: IntelCor_ad:23:1f (90:e2:ba:ad:23:1f), Dst: Cadant_aa:08:45 (00:01:5c:aa:08:45)But receives traffic:
Ethernet II, Src: bc:24:11:41:f2:d6 (bc:24:11:41:f2:d6), Dst: IntelCor_ad:23:1f (90:e2:ba:ad:23:1f)Site B sends traffic:
Ethernet II, Src: bc:24:11:41:f2:d6 (bc:24:11:41:f2:d6), Dst: IntelCor_ad:23:1f (90:e2:ba:ad:23:1f)But receives traffic:
Ethernet II, Src: Cadant_aa:08:45 (00:01:5c:aa:08:45), Dst: bc:24:11:41:f2:d6 (bc:24:11:41:f2:d6)So it looks like site A is not sending to the site B MAC directly but via 00:01:5c:aa:08:45. Which I assume is the ISP router?
After some time some ICMP redircet probably expires and traffic is no longer forwarded.
So it looks like you have an ARP issue. Check the ARP table on each firewall. Make sure they are actually in the same subnet.
