Using 2 gateways with different subnets on a single WAN interface
-
Hey Steve, thanks for replying.
I'll begin by mentioning that I asked my ISP today how exactly they would like me to handle this/How do they route this (Instead of all that guesswork), and their answer was somewhere along the lines of "Find someone who knows what they are doing, We can't teach you, Don't use Pfsense, etc...", So... Yea. Fun. Sadly, for... ehh... reasons... I can't leave them at this point. Back to guessing, I... guess.
-
Using a switch with two NICs showed the 2nd gateway as offline. Let me just verify we are talking about the same kind of setup: ISP Modem -> A single cable to a "dumb" switch -> Additional two cables from that switch to two different NIC's on PfSense, each NIC with its own GW - Correct? If so, then it did not work. I can obviously try it again, who knows... If that's not what you meant, please, enlighten me :)
-
Tried using IP Alias (VIP?) on WAN, with manual outbound rules, not even setting anything related to gateway. If that's what you meant, that also didn't work. It was actually the first thing that I tried.
-
Interesting, but sadly, I've hit a brick wall with them and god knows what do they expect. Actually, God probably has his doubts on this matter too. I tried rebooting the modem a number of times, but not between each and every config attempt. I'll definitely try rebooting it once more and between each test.
-
"How were you testing this?" - More than happy to answer, but testing what, exactly? Sending/Receiving traffic?
-
I tried sending traffic from an external network to the 2nd IP and nothing seem to arrive. It's not getting blocked based on a firewall, it's just not arriving. Pinging the GW address (171.203.44.158) does get a response, but I'm not sure I can draw any conclusion based on that, I guess that's the modem/ISP handling that. EDIT: When I ping the GW, while showing UP, from PfSense itself, I don't get a response, Tried pinging from both addresses. If I remove the 2nd GW (System -> Routing) and leave the VIP intact, I can ping it from the first IP, but not the 2nd (VIP). Not sure this info contributes anything to understanding, but I guess it can't hurt..
-
Eventually, this 2nd IP will only be used for a PBX and it will definitely have a specific Outbound route rule. Do you mean that in this 2 subnets/one layer scenario, I will have to set an outbound NAT rule to every device/network?
Just a quick edit: The only situation where the GW is showing up, is when the virtual IP is set (on WAN), but then again, other than it showing UP, I can't get anything to talk
I really appreciate your time and your willingness to help. Thank you and have a good night/evening!
-
-
If it shows as UP it's responding to gateway monitoring pings. If you didn't set the monitoring IP to something custom it uses the gateway IP itself. So that probably implies the gateway is responding to pings.
Since the gateway address is inside the VIP subnet when you ping out it should use that as the source.
Check the state table in Diag > States.Since you first tried with the VIP that uses the same MAC as the WAN NIC the ISP may be locked to that. Did you reboot the modem?
Did the ISP give you any documentation at all when enabling this?
-
@stephenw10 Hey,
Nop, I didn't set a different monitoring IP.
The only thing I can see in the States table regarding the 2nd IP/GW is an ICMP entry, from the 1st IP to the 2nd GW address (Not the 2nd IP), with a state of 0:0. Other than that - Nada. I tried resetting the states (Current config is: VIP of 2nd IP exist + 2nd GW exists and is up) and I do see that the one server (PBX) that has an outbound rule which translates its address to the 2nd IP does have some states (It is attempting to connect to the SIP Trunk, as it should), but they reach nowhere, as far as I understand: NO_TRAFFIC & SYN_SENT:CLOSED.
Sadly, I haven't been able to reboot the modem as I'm currently away from the office, I'll be there Sunday and that's the first thing I'll try. As I don't have any management access to it, my only way of rebooting it is by physically unplugging it from the wall.
ISP gave me nothing but a heartache. In all seriousness though, they are difficult. They basically gave me nothing but the 2nd IP address, GW & Subnet. When I asked them to elaborate on how they want/expect me to set it up, they told me something along the lines of "You should find a PBX guy who knows what he's doing", even though my question was 100% not about PBX, but about the 2nd IP assignment. Pretty frustrating.
One thing that I should have probably mentioned earlier, is that the "modem" is not exactly a simple consumer modem, but a RAD ETX-203AX Carrier Ethernet Demarcation device. It was supplied by the ISP and I have zero access to it. I actually tried ssh-ing my way in, to at least see the config, but it is disabled. I'm guessing that the console port is enabled, but chances are that they have changed the default password. I'm reading the manual of that device (Found it online). So far, I've seen something about setting one port with 2 IP ranges ("Classification key – Provides the opportunity to use multiple IP address ranges
to route packets via the port"), but I'm not done reading yet, so maybe this will be further explained down the manual, but then again, there is a limit to how much I can understand how a device is configured, just by reading the manual, without actually logging in to the device, something that is probably blocked on some level.I keep trying "playing" with the VIP, the GW, FW rules, custom routes, but I'm currently not getting anywhere...
As always, I'm very thankful for your help and I'll be more than happy to provide any additional info.
Good night :)
-
@eaglex said in Using 2 gateways with different subnets on a single WAN interface:
The only thing I can see in the States table regarding the 2nd IP/GW is an ICMP entry, from the 1st IP to the 2nd GW address (Not the 2nd IP), with a state of 0:0.
Ok that's interesting because I expect it to use the VIP as source but only if the subnet is correct. Make the the IPAlias VIP is setup with the correct subnet mask.
Make sure the correct subnet shows in the routing table.
-
I have to agree that it looked odd to me too. My ISP gave me 255.255.255.252 as the subnet, so /30, and that's what I set (And if I somehow getting it wrong, let me know). Is it possible they gave me the wrong subnet? To be honest, they gave me a lot of wrong information so far, so that's not completely unimaginable. In this case, is there a way for me to find out the correct subnet, other than trying /30 /29 /28 and so on, manually, one by one?
When I put the actual address they provided me into ipinfo.io, it actually says that it's in a /26 subnet, but is it accurate? For good measures, I try editing my VIP with a /26 alias, to no success.
Also, there's this option under the GW setting: "Use non-local gateway through interface specific route." - Is it possible that I need to turn it on? Tried but didn't see a different result.
I'm attaching screenshots of my config, after deleting VIP & GW, clearing states and then re-adding VIP & GW and killing states again. That ping behavior still exists. I didn't set any static route, as far as I know... I have to admit I'm a bit uneducated (But willing to learn) in this area.
Gateways:
WAN:
VIP Alias:
States:
Diagnostics -> Routes:
System logs -> Routing:
Thanks.
-
As long as the gateway and VIP are actually inside the same /30 it should be fine. And I imagine they must be because otherwise you could not add the gateway.
Try pinging whilst setting the VIP as the source IP.
-
@stephenw10 Tried that, no response...
-
Sorry I meant pinging the gateway from the VIP as source.
-
@stephenw10 It's a no go in this case :(
-
Hmm, that seems like it should definitely work.
Try running a pcap for that traffic on WAN. Make sure it's actually leaving and has the expected MAC address.
If you send traffic to the VIP address from something external you should at least see ARP requests for it arriving in a pcap.
-
So I don't know if it makes sense or not (Honestly, at this point, not a lot makes sense to me anymore
), but when I tried pinging the 2nd GW from the 2nd IP, just like the screenshot in my previous reply shows, I don't see it happening in a pcap. The only thing that uses that VIP that I can see in the capture, is my PBX, which has a NAT outbound rule to use the VIP, trying to connect to the trunk, using the correct VIP (And using WAN interface MAC address, which makes sense, as it's a virtual IP on that interface, correct?). Also, and I don't know if it means anything, the destination for that SIP is the MAC address of the first GW (As appears on the ARP table on PfSense), not the 2nd one. (Which doesn't even appear on the ARP table). Weird?Other than that, the only ICMP action I'm seeing in the capture is:
- From the 1st IP ("Main" WAN IP) to the 1st gateway + Correct reply.
- From the 1st IP to the 2nd gateway... + Correct reply.
Nothing is coming out from the 2nd IP, the VIP, the one I choose as the source address when pining (Other than that SIP).
Thank you :)
-
Also...
-
Forgot to check incoming traffic - Again, nothing. Tried pinging from an external network, nothing shows on the packet capture.
-
I forgot to mention that I'm running PfSense in a VM on Proxmox, but I'm using PCI pass-through of the NIC (Intel I350) to the VM, so each port has its own native individual MAC address - I don't think it should matter in this case, but probably worth mentioning.
-
-
Do you see the 2nd gateway IP address in the pfSense ARP table? Is it using the same MAC address as the 1st gateway?
-
Nop...
Other than devices that are on my LAN/VLAN's, VLAN's interfaces and so on, I have:
- 1st IP (With the WAN MAC address and a hostname of my ISP) + 1st GW (With a Cisco MAC address).
- 2nd IP (With the WAN MAC address and nothing more).
The 2nd GW is just not in the ARP table.
I also tried restarting the modem today and tried doing it between every config change, for example, I tried the dumb switch method again, restarted it before, restarted it after, etc, didn't do anything. I guess that's not it.
-
Ah, sorry I see you already reported the WAN2 GW is not in the ARP table.
OK in the pcap you did where it is pinging the WAN2 GW what MAC address is it using?
Since it's pinging from the WAN1 IP it's probably routing via the WAN1 GW using that MAC.This starts to look like they require a different MAC for the local IP...
-
-
So when IP1 is pinging GW2, the source MAC address is the WAN MAC address and the destination MAC address is GW1 MAC address, not GW2 MAC address (Which I don't know its value...). Does it make sense? Does it actually tell us anything? Can I try force it to route otherwise?
-
Do you mean that the ISP might ask me what's the PBX NIC's MAC address, for example, whitelist it on their side and tell me to connect it directly there? Something like that? Well, other than just giving me the dry details and yelling at me, I got no additional info from them :|.
Thanks again, really appreciate your ongoing willingness to help :)
-
-
Mmm, two interesting things. You are able ping the GW2 IP but only via GW1. That is routed by the ISP. I don't know if the IP you posted above is the real GW2 IP but it doesn't respond to pings for me externally.
It doesn't respond when you try to ping it from VIP which is inside the same subnet. When you do that it will ARP for the address directly and it isn't responding since you don't have a MAC for it.
It's possible the ISP will only allow the MAC address to be used for one connection. If that was the case though I would have expected the two NIC setup via the switch to work.
There are really only two ways the ISP can provide this two you. Either they route the subnet to you via the existing WAN IP or they provide it directly on the WAN L2 segment. It appears to be the latter because in the routed subnet the GW2 IP would not respond to pings at all unless you had added it.
So I would go back to the two NIC setup and test there. Or try just using a different client on the switch configured to use the WAN2 IP and GW2.
-
Hey there, sorry for the late reply, had some personal issues and I wasn't available. I'm gonna try again and update as soon as I can. ISP is sadly still pretty unresponsive...
Thanks again.
