Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A
-
Not sure where this fits so I'm putting it here.
I've been running PFsense for a long time, and my hardware has been upgraded a lot over the years, to the point where my router config is a bit of a mess, but it works. I have a webserver and host several sites, all very low traffic and mostly just for fun.
I recently found out that my ISP will grant a second IP without charging so I figured I would setup a second network and router for all of my out of home network accessible stuff. The second router is on a Proxmox box, and generally works fine with one big issue:
I cannot reach anything on Network B from network A through the "World Wide Web" unless I use a VPN or I'm not on my home (Network A). Maybe this has something to do with my ISP, but I have no idea what I would even tell them. Maybe its SNORT or something else on router A?
While writing this, I realized I never tried going from Network B to Network A. That seems to work fine. I was able to Ping and SSH into my webserver behind Network A's firewal/router from a system on network B over the world wide web.
Or maybe there is something obvious I am missing? Any ideas, or things I can check and get back would be excellent.
-
How are the two routers connected? What links them upstream?
How are you trying to connect? Using the internal private IPs dircetly?
Really I'd recommend not using two routers. There's little reason to do it and using one makes routing between subnets far easier.
-
@stephenw10 The routers are each connected to the bridged modem separately with their own WAN IP from the ISP.
Router A has been with me forever and has a webserver behind it: link text
Router B is new, and has a Netbird selfhosted install behind it: link text
My home networks is behind router A. When on my home network I cannot browse to netbird on the web, or connect to it. If I connect to a VPN first, and then type netbird.dwhacks.com then it works.
Im betting you can access both of those links, and they both have different public IP's. I cannot which I find weird.
-
@dwhacks if you ISP router/modem is truly in bridge mode and you are passing 2 public IPs to separate routers, then you essentially have 2 sites. Careful, the upstream ISP router, if not configured correctly, could be doing some NAT.
I agree with the former post, only one router should be needed. You can use subnetting to achieve service segregation.
If you stay with trying to manage two sites, then to go from one site to the other, you would need to go out to the Internet and come back in. This would involve careful forwarding of the needed ports and good firewalling practices at both the edge and your internal services.
I used to have an ISP that provided fiber optic service. They installed a router/modem device. I didn’t trust that their device would stay in bridge mode. I ran my own ethernet cable directly off their ONT in the garage, and connected it directly into my pfSense. Worked perfectly for 10 years. The only reason it stopped working was, I moved. Investigate whether you could potentially ditch their router. You don’t need it, you just need an ethernet connection. If you’re dealing with cable, then it’s even easier, you can buy your own modem that does not have a router built in.
-
One second thought, I just noticed you mentioned that one of your pfSense routers is in a virtual machine. That would seem to imply that you have a pfSense router behind a pfSense router. Your VM needs a host and that host has to be on an already existing private network. Do you have a bridge interface set up on your hypervisor? There are all sorts of complications potentially here. Double NAT, possibly triple NAT. My apologies for missing the VM part earlier.
-
@aljames I've been using the modem/router in bridge mode without issues for about 9 years. Only recently did I ask to have a second IP added. The ISP provided device has two ports which get two different public IP's. One port goes to the WAN nic on Pfsense A and one port goes to WAN nic on Pfsense B.
Router B is in proxmox, with a dedicated NIC connected to the pfsense VM. I don't think that the VM is the problem as I can connect from outside my home network no problem. For example, if I turn WIFI off on my phone, I can access daynewaterlow.com (webserver behind router A) and netbird.dwhacks.com (webserver behind router B). If I take my laptop to a coffee shop this works, or if I turn a VPN on on my laptop this also works.
I understand that if I want to access stuff behind router B from behind router A then I need to go out to the web and back, this is the configuration I am looking for. It seems like im not actually going "out" and that's the problem.
I want these to be two different "locations" as far as the internet is concerned. The idea is to have two different routers so I can try different setups on them (DNS Forwarder VS resolver, different packages, ETC), with different services behind them. Mostly to hopefully keep anything sketchy off on home network. It would be as simple as swapping the feed to my switch, or changing bridged ports on the VM to change which router it was going through.
Is there any way to test weather my router is the problem? Or is it something upstream in the ISP's side of things. I feel like this should just work.
This
And this
seem to find both IP'sWebserver A
Webserver B
will probably work for you. -
@dwhacks I agree, Router B in Proxmox is not an issue. Doesn't matter if you have passthru for the NIC or VirtIO, it works perfectly fine. I have a similar setup as you do with the main difference being that I have my fiber going to a switch and my two FW's connected to that (so no ISP router in between). But you clearly have public IP's, and different one's, on both A and B.
So if you can't access a service on site B from A, I'm thinking there must be an internal issue, on site A.
Perhaps you have some old rules still on site A, like Host overrides?
Check under Services > DNS Resolver - Host overrides... and look for your domain names (daynewaterlow and netbird.dwhacks.com).. -
So both firewalls have a public IP on their WAN? And I assume each firewall can ping the other no problem?
So you have port forwards setup on each to access the hosts behind them?
Since that traffic appears to come from/to the WAN subnet directly you might not matching it with an outbound NAT rule in one direction.
Check the states on both firewalls when you try.
-
@Gblenn I believe the "bridge mode" of the ISP's router/modem basically turns it into a switch. At least that's what its supposed to do.
I agree, the problem must be in my router from site A. I have just connected a spare wifi AP to site B's LAN and when on that network I can connect to services at both sites, so site B is working correctly.
Site A is using the DNS forwarder (not resolver). I set this up many years ago, and it has host overrides for the services to be access when on that network.
Site B is using DNS resolver and have the host override for dwhacks.com only
I don't either of these should be causing an issue.
-
@stephenw10 Both firewalls have public IP's but only ping from B to A:
Ping from PFsense on site B
Ping from PFsense on site A:
Ping clearly finds the correct IP.
What should I be looking for in Diagnostics -> States ?
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
Site A is using the DNS forwarder (not resolver). I set this up many years ago, and it has host overrides for the services to be access when on that network.
Well if you have host overrides on Site A that references the services that are now on Site B, it will not work. So remove or disable those host overrides and see if that fixes it...
-
@Gblenn as per the screenshots I have attached I do not see any conflicts.
I did think of something while I was making another coffee: before setting up the router for site B I have dual WAN at site A. My ISP almost never changes the IP addresses, is it possible there is some residual config or states or something left over in my site A router making it think that the IP of site B is supposed to go to itself? Maybe is DHCP as thats what the routers use on WAN?
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
What should I be looking for in Diagnostics -> States ?
Looks for NAT'd traffic from the internal host you're testing from.
But if ping is failing then check the firewall rules on site B. You might just not be passing ping there but it would be much easier to test of you do.
Are those two public IPs actually in the same subnet?
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
as per the screenshots I have attached I do not see any conflicts.
But if site A has a host override for daynewaterlow.com pointing to 192.168.2.7, and that IP is no longer valid, since daynewaterlow.com is now located at Site B, of course it will not work...
-
@Gblenn daynewaterlow.com is on site A, netbird.dwhacks.com is on site B. netbird.dwhacks.com is not accessible from the network of site A.
I haven't moved all my webservers over to site B yet because I am unable to access them from site A so that makes admin stuff tricky. I would like to move them eventually.
@stephenw10 I cannot see anything in states that points to the IP of site B at any point. I think they are probable on the same subnet of my ISP but I'm not sure how to check. I believe PING is allowed.
-
I added a rule to firewall on site B and I can now ping from site A:
I still cannot access the netbird.dwhacks.com from site A though.
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
I cannot see anything in states that points to the IP of site B at any point.
In either firewall?
Is the client behind firewall A resolving the url to the firewall B WAN?
If it is you should at least see some states in firewall A. If you don't then it could be a firewall rule on the A LAN blocking it (or not passing it).
-
@dwhacks said in Two IP's from ISP, Two PFsense routers (one a vm) cannot access B from A:
I did think of something while I was making another coffee: before setting up the router for site B I have dual WAN at site A. My ISP almost never changes the IP addresses, is it possible there is some residual config or states or something left over in my site A router making it think that the IP of site B is supposed to go to itself? Maybe is DHCP as thats what the routers use on WAN?
So Site A is (or was) "aware" of the IP that now resides on Site B then... well yes I suppose there could be some residual states or something which makes it think it should go back to itself. Similar to what I was after with the host override rules, which however you have clearly changed. I assume you have removed WAN2 completely, as well as any gateway groups etc that you may have had. And how about services Dynamic DNS, on site A is that also updated to reflect that dwhacks is no longer on that IP?
Try tracert to dwhacks.com to see what you get from pfsense on site A... That should give you a clue as to what it's doing.
Also you can run Pcap on WAN whilst pinging dwhacks to see if it shows up. Add a filter with the public IP for Site B so you only capture the important information.
-
Been away for a couple days, so I haven't tried any suggestions. BUT everything worked for about 15 minutes when I tried it the last couple hours. It no longer works, and I can't even ping site B from site A....
When I try to ping from the shell on site A pfsense:
[2.7.2-RELEASE][admin@pfsense.localhacks.lan]/root: ping dwhacks.com PING dwhacks.com (24.71.68.91): 56 data bytes ping: sendto: Permission denied ping: sendto: Permission denied ping: sendto: Permission denied ping: sendto: Permission denied ^C --- dwhacks.com ping statistics --- 4 packets transmitted, 0 packets received, 100.0% packet lossI can ping other hosts, like google.ca
I will try some of the suggestions tomorrow.
-
@Gblenn Traceroute seems to go nowhere:
1 traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 * 2 traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 *traceroute: wrote netbird.dwhacks.com 40 chars, ret=-1 * etc (does this 18 times)
