Using 2 gateways with different subnets on a single WAN interface

eaglex · Mar 2, 2024, 2:47 PM

I have to agree that it looked odd to me too. My ISP gave me 255.255.255.252 as the subnet, so /30, and that's what I set (And if I somehow getting it wrong, let me know). Is it possible they gave me the wrong subnet? To be honest, they gave me a lot of wrong information so far, so that's not completely unimaginable. In this case, is there a way for me to find out the correct subnet, other than trying /30 /29 /28 and so on, manually, one by one?

When I put the actual address they provided me into ipinfo.io, it actually says that it's in a /26 subnet, but is it accurate? For good measures, I try editing my VIP with a /26 alias, to no success.

Also, there's this option under the GW setting: "Use non-local gateway through interface specific route." - Is it possible that I need to turn it on? Tried but didn't see a different result.

I'm attaching screenshots of my config, after deleting VIP & GW, clearing states and then re-adding VIP & GW and killing states again. That ping behavior still exists. I didn't set any static route, as far as I know... I have to admit I'm a bit uneducated (But willing to learn) in this area.

Gateways:

WAN:

VIP Alias:

States:

Diagnostics -> Routes:

System logs -> Routing:

Thanks.

stephenw10 · Mar 2, 2024, 10:08 PM

As long as the gateway and VIP are actually inside the same /30 it should be fine. And I imagine they must be because otherwise you could not add the gateway.

Try pinging whilst setting the VIP as the source IP.

eaglex · Mar 2, 2024, 10:39 PM

@stephenw10 Tried that, no response...

stephenw10 · Mar 2, 2024, 11:21 PM

Sorry I meant pinging the gateway from the VIP as source.

eaglex · Mar 3, 2024, 12:04 PM

@stephenw10 It's a no go in this case :(

stephenw10 · Mar 3, 2024, 4:12 PM

Hmm, that seems like it should definitely work.

Try running a pcap for that traffic on WAN. Make sure it's actually leaving and has the expected MAC address.

If you send traffic to the VIP address from something external you should at least see ARP requests for it arriving in a pcap.

eaglex · Mar 3, 2024, 7:14 PM

@stephenw10

So I don't know if it makes sense or not (Honestly, at this point, not a lot makes sense to me anymore ), but when I tried pinging the 2nd GW from the 2nd IP, just like the screenshot in my previous reply shows, I don't see it happening in a pcap. The only thing that uses that VIP that I can see in the capture, is my PBX, which has a NAT outbound rule to use the VIP, trying to connect to the trunk, using the correct VIP (And using WAN interface MAC address, which makes sense, as it's a virtual IP on that interface, correct?). Also, and I don't know if it means anything, the destination for that SIP is the MAC address of the first GW (As appears on the ARP table on PfSense), not the 2nd one. (Which doesn't even appear on the ARP table). Weird?

Other than that, the only ICMP action I'm seeing in the capture is:

From the 1st IP ("Main" WAN IP) to the 1st gateway + Correct reply.
From the 1st IP to the 2nd gateway... + Correct reply.

Nothing is coming out from the 2nd IP, the VIP, the one I choose as the source address when pining (Other than that SIP).

Thank you :)

eaglex · Mar 3, 2024, 8:23 PM

Also...

Forgot to check incoming traffic - Again, nothing. Tried pinging from an external network, nothing shows on the packet capture.
I forgot to mention that I'm running PfSense in a VM on Proxmox, but I'm using PCI pass-through of the NIC (Intel I350) to the VM, so each port has its own native individual MAC address - I don't think it should matter in this case, but probably worth mentioning.

stephenw10 · Mar 3, 2024, 9:02 PM

Do you see the 2nd gateway IP address in the pfSense ARP table? Is it using the same MAC address as the 1st gateway?

eaglex · Mar 3, 2024, 9:34 PM

Nop...

Other than devices that are on my LAN/VLAN's, VLAN's interfaces and so on, I have:

1st IP (With the WAN MAC address and a hostname of my ISP) + 1st GW (With a Cisco MAC address).
2nd IP (With the WAN MAC address and nothing more).

The 2nd GW is just not in the ARP table.

I also tried restarting the modem today and tried doing it between every config change, for example, I tried the dumb switch method again, restarted it before, restarted it after, etc, didn't do anything. I guess that's not it.

stephenw10 · Mar 3, 2024, 10:14 PM

Ah, sorry I see you already reported the WAN2 GW is not in the ARP table.

OK in the pcap you did where it is pinging the WAN2 GW what MAC address is it using?
Since it's pinging from the WAN1 IP it's probably routing via the WAN1 GW using that MAC.

This starts to look like they require a different MAC for the local IP...

eaglex · Mar 5, 2024, 10:56 AM

@stephenw10

So when IP1 is pinging GW2, the source MAC address is the WAN MAC address and the destination MAC address is GW1 MAC address, not GW2 MAC address (Which I don't know its value...). Does it make sense? Does it actually tell us anything? Can I try force it to route otherwise?
Do you mean that the ISP might ask me what's the PBX NIC's MAC address, for example, whitelist it on their side and tell me to connect it directly there? Something like that? Well, other than just giving me the dry details and yelling at me, I got no additional info from them :|.

Thanks again, really appreciate your ongoing willingness to help :)

stephenw10 · Mar 5, 2024, 2:17 PM

Mmm, two interesting things. You are able ping the GW2 IP but only via GW1. That is routed by the ISP. I don't know if the IP you posted above is the real GW2 IP but it doesn't respond to pings for me externally.

It doesn't respond when you try to ping it from VIP which is inside the same subnet. When you do that it will ARP for the address directly and it isn't responding since you don't have a MAC for it.

It's possible the ISP will only allow the MAC address to be used for one connection. If that was the case though I would have expected the two NIC setup via the switch to work.

There are really only two ways the ISP can provide this two you. Either they route the subnet to you via the existing WAN IP or they provide it directly on the WAN L2 segment. It appears to be the latter because in the routed subnet the GW2 IP would not respond to pings at all unless you had added it.

So I would go back to the two NIC setup and test there. Or try just using a different client on the switch configured to use the WAN2 IP and GW2.

eaglex · Mar 12, 2024, 3:57 PM

@stephenw10

Hey there, sorry for the late reply, had some personal issues and I wasn't available. I'm gonna try again and update as soon as I can. ISP is sadly still pretty unresponsive...

Thanks again.