Netgate 4200 : Static route to a subnet behind a LAN host — connections keep dying after ~20-40s. Configuration mistake, or has anyone hit this on a 4200?

jsibre

Router: Netgate 4200, pfSense Plus 26.03.1

Setup, simplified:

[LAN client, 192.168.1.41] ---- [pfSense LAN, 192.168.1.1] ---- [Proxmox, 192.168.1.2]
                                                                       |
                                                          (Proxmox bridges to a second
                                                           network of its own)
                                                                       |
                                                              [192.168.2.0/24]

pfSense LAN is 192.168.1.0/24.
192.168.1.2 is a Proxmox host on that same LAN (not a VLAN, not a separate pfSense interface — just another host on 192.168.1.0/24) that bridges to a second internal network of its own, 192.168.2.0/24 (a Proxmox Linux bridge, vmbr1).
On pfSense, I defined a gateway using 192.168.1.2 and a static route: destination 192.168.2.0/24, using that gateway.
This means pfSense receives LAN client traffic on its LAN interface, and — per the static route — has to send it back out that same LAN interface toward 192.168.1.2 to reach 192.168.2.0/24.

What I expect: this should just work — it's a standard "router with a static route to a subnet reachable via another host on the same LAN" setup, nothing exotic.

What actually happens: long-lived TCP connections (I've been testing with SSH) from a LAN client to a host on 192.168.2.0/24 reliably die after roughly 20-40 seconds (sometimes longer). Short requests and light traffic are fine; anything that stays open for more than about half a minute drops. No firewall log entries, no rule counter anomalies — the connection just goes silent and eventually times out.

Questions for the community:

Is there a "correct" way to configure this that I'm missing — some setting specific to routing traffic back out the interface it came in on? (I've already tried: State Killing Behavior: "Do not kill states on gateway failure", "Disable Gateway Monitoring", "Disable Gateway Monitoring Action", State Type "Sloppy" and State Type "None" on the relevant firewall rules, disabling ICMP redirects, and recreating the gateway/static route from scratch — none of it changed the behavior.)
Has anyone gotten a setup like this — a static route to a subnet behind a same-LAN host — working reliably on a Netgate 4200 specifically? Curious whether this is a config issue on my end or something particular to this hardware/software combo.

Other bits to know:

An SSH connection to Proxmox (192.168.1.2 - the same physical interface used as the gateway to reach 192.168.2.2) works fine - no issues there.
Everything downstream of that interface is virtual (vmbr1, 192.168.2.2) on the Proxmox.
This same Proxmox host and topology worked fine with my old UniFi UDM Pro. (But a lot of hardware has been changed - not just the UDMPros/Netgate swap.)

I do have a much more detailed writeup (packet captures from three points simultaneously, NIC driver checks, tunable tests, etc.) if anyone wants to dig deeper — happy to share, didn't want to lead with it here since I'm mainly hoping someone's just done this exact simple thing and can tell me if I'm missing a step.

I plan to do some more testing to try to create a simplest test case, so I may have follow-ups.

Thanks for any clues, or confirmation that you've seen this issue before, too.

jsibre

I've completed a few additional tests today:

I made a backup of my config (mostly working - just this one lingering issue with the connections over the static route dying), performed a factory reset, and duplicated the error in a minimal config. More on that below. After completing that, I restored my original configuration.
I set up a second subnet on PORT3 to see if 'normal' routing (no hairpin, no static route required) worked ok. It worked ok - no problems there.
I then extended the second test case by essentially duplicating the first scenario on PORT3 using two Raspberry Pis and got the same failure.

More detail, Test #1
As I said, I factory reset the router. I then disconnected it from the WAN, and just hooked up my Mac and my Proxmox. I set PORT2 to be 192.168.1.1/24, left DHCP as it defaulted (it was sufficient to give my Mac an IP address, and the Proxmox was static at 192.168.1.2, below the lease range). I designated 192.168.1.2 as a gateway and created a static route for 192.168.2.0/24 via that gateway. Again, SSH connections from the Mac to 192.168.2.2 failed fairly quickly (I don't have the figure handy, but is was in the same general range of 20-40 seconds).

I restored my backup and put everything back as it was.

Test #2
I set up PORT3 to be 192.168.3.1/24 and tried tests to a Raspberry Pi at 192.168.3.10 from my Mac on 192.168.1.101. Worked fine - SSH connection lasted for hours before I killed it.

Test #3
I basically duplicated the original test, except by leaving my original config for WAN and PORT2LAN intact, and just duplicating the original topology on PORT3: two physical hosts (hanging off a dumb switch - not VLAN aware), one (still) sitting at 192.168.3.10, and another (call it pi2) at 192.168.3.11. But then I gave pi2 an alias of 192.168.4.2 on the same eth0. So, pi1 is playing the part of my Mac, and pi2 is playing the part of my Proxmox. The first time I ran the test, the SSH connection died in 32 seconds. The second time I ran the test, it failed in 37 seconds.

Conclusions
Because test 3 happened on PORT3, I'm ruling out the idea that there's something defective about PORT2LAN. Or at least, nothing that's not also defective in PORT3. Maybe these ports can't handle that rapid turnaround, but if so, it's consistent, and the drivers will need to compensate. So, I don't think the hardware is defective in a way that warrants special attention, but I do think there's something wrong here.

This type of routing should be trivial, but I can consistently break it, and worse, I haven't found a way to work around it that doesn't involve adding more hardware on the network (i.e., running a second cable to a second NIC on my Proxmox from, e.g., PORT3 on the router). I have tried doing that same thing using VLANs, and that didn't make the issue go away. It seems like if hairpin routing (in and out on the same port) happens on a physical port, it will eventually fall apart. It doesn't matter if it's a different logical interface (as can happen with VLANs) - if the underlying physical port is the same, it will eventually fail.

I have a backup of the configuration used in the test case from test #1 above - very simple setup - if anyone wants to look at it or even try to replicate it.

I'm increasingly certain I've found a bug here, and I'm hoping someone can prove me wrong, or that we can get it fixed.

No one else is seeing this issue? I can't be the only person here running Proxmox, and I'll bet a lot of folks running it have the same need. ...But maybe not that many that have switched from one vendor (where it was trivial) to another (where it's seemingly impossible to get it working well) and thus deal with it by using Proxmox as a jumphost... Flip side: Netgate is handling source IPs from WAN traffic better than the UDM Pro was!

Thanks

PeteRage

The 4200 is not a network switch and it sounds like you have multiple interfaces carrying 192.168.1.0/24?

If that's the case you are on an uncommon used topology and could be encountering a bug or a configuration issue. Recommendation would be to put a switch in and only have a single connection for 192.168.1.0/24 - that'll also make your network faster since you won't be software routing between ports on pfsense.

If that's not, try putting a route on the MAC for 192.168.2.0/24 and that'll bypass pfsense completely.

SteveITS

@jsibre there’s a 30 second timer for https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html. Is the gateway on PVE the pfSense? (Same path both directions)

jsibre

@PeteRage
Thanks for the reply - I did try to set up two interfaces on the Netgate in some of my tests, but in each case, no - they were not on the same network. And anyway, those were just tests.

My real configuration is using port 1 for WAN, and port 2 for LAN. Ports 3 and 4 are idle and unconfigured. The LAN is 192.168.1.0/24, and the 192.168.2.0/24 network is accessible inside the PVE which sits at 192.168.1.2.

By the way, I've found a work-around that's working fairly well. I'm still curious about the nature of this configuration and why I couldn't get it to work, but the work-around has solved my problem, frankly, in a better way. That work-around is using DHCP option 121 to push routing information to the clients on the network, so that they all know to send traffic intended for 192.168.2.0/24 via 192.168.1.2. I say it's better because it avoids sending that traffic via the router in the first place.

Thanks again for the reply!

Oh, and yes, configuring the Mac (manually, or via option 121) does work fine - I just didn't want to rely on configuring DHCP clients manually like that - once I learned about option 121, I was set.

jsibre

@SteveITS
Thanks Steve - I hadn't found that page before, but I may try it again. (I say try it again, I mean, try that config again, rather than the DHCP option 121 that's working for me now).

That page does seem to address my issue directly, however... I think I've already tried everything detailed there, and various combinations, to no avail (which is why I think there's a bug). I do like that the process described at the top seems to describe my problem almost identically. Almost. (The ICMP Redirects were being sent, but my Mac seemed to be disregarding them - packet captures on the netgate showed that the Mac was still sending traffic to the router that should have gone direct to the 192.168.1.2 if the redirects were having any effect)

I tried disabling ICMP Redirects (on the off chance that it was connected to the router killing states)
I tried firewall rules with State Type "global default", "Sloppy", and even "None"
Bypass firewall rules for traffic on the same interface - I think I tried it. Honestly, I'm not sure - I may have ruled it out as sounding too permissive (this is where, had I found the link you shared, I might have had more success)
State Killing Behavior: "Do not kill states on gateway failure"
Disable Gateway Monitoring
Disable Gateway Monitoring Action

And, these were tried in various combinations.
So, what I'm not sure about is, did I find and try the correct combination? Did I apply things like Sloppy to the correct rules. Not sure.

Also, the PFSense will tell you when it needs to reboot for a change to take effect. Therefore, for the changes that didn't say they needed a reboot, I didn't reboot it. What I didn't try was rebooting it after all these config changes - so, maybe something really did need a reboot, and I didn't reboot after the change. Not sure how likely a problem that is.

I may try this again now that I've got that page to refer to, but, as I mentioned in another reply a few minutes ago, I think my work-around that's in place now is actually a better solution - it prevents the router from even having to be involved (and, technically, there's no reason it needs to be involved). The DHCP Option 121 config allows me to get the hosts on the network configured so that they all know to use 192.168.1.2 as the next hop for 192.168.2.0/24.

Also, you asked, "Is the gateway on PVE the pfSense?" - Yes, the PVE is using 192.168.1.1 (PORT2LAN on the Netgate) as it's default route/gateway. The PVE is also statically configured - I think it's the only thing on my network (besides the Netgate) that is not using DHCP.

Thanks for the reply and tips!

SteveITS

@jsibre FWIW I've not actually tried routing by PVE directly, though I've seen it's possible. The one use case we had was for a virtualized network so we set up a pfSense VM and that routes/NATs out, with those VMs attached to an SDN VLAN behind it. Which is maybe not relevant here, but an alt solution.

It sure seems like asymmetic routing, it's a matter of finding out why.

jsibre

@SteveITS
Oh, I do think it's asymmetric routing - for sure. I have packet captures (from before setting up the routes on the clients via 121) showing traffic FROM the client going via pfsense, but return traffic... just not appearing on the pfsense - but still getting back to the client.

So, my understanding is that the traffic went from Mac to switch to pfsense to switch to PVE to LXC (192.168.2.2) and return traffic went from LXC to PVE (I guess - actually, I'm a bit fuzzy on if it made a 'stop' at the PVE) to switch to Mac. Bypassing the Pfsense/Netgate completely.

Now, with the routes configured via 121, it's just what you'd expect Mac - switch - PVE/LXC, and LXC/PVE - switch - Mac. Works well. Less load on router. Better performance. All around a better solution - I just didn't know that options existed before. (My old UniFi handled the static route with no hiccups, so I never had to dig deeper/learn more).

Thanks