IPsec VPN Up But Not Passing Traffic (identical config has been for years)
-
This issue came up a few months ago, updating to 25.07.1 seemed to have fixed it, but it came back up today.
Between a CARP pair of 1541s and another remote 1541, we have a VPN setup, at a random time (usually during the night, of course) this connection will stop passing traffic even though it'll show as up on both firewalls management GUIs.
A restart of the VPN fixes it every time, until it happens again.
All 3 firewalls in this setting are fully patched, and not much has really changed with them, yet it's been fine for about 3 years with basically zero downtime.
I rebuilt the VPN on both sides (quadruple checking I didn't miss something stupid the first time), and it was an identical config, and the issue went away. Or so I thought, it roared it's ugly head again today when I had to failover to our backup firewall (I've got another post about that).
VPNs to 6 other sites with pfSense firewalls, with nearly identical configurations (other than subnets), will work flawlessly the entire time, it's only this single one that goes down.
With the other issue I had today, I am starting to wonder if there is some hardware issue going on, but would love to get some advice on troubleshooting this further.
If I have to I'll start back up TAC and reach out, but since I can fix it quickly when it happens, I'd honestly like to learn what the cause of this is and figure out a proper fix and file a bug report on Redmine if there is something weird going on.
-
@planedrop Identical configuration? You are aware that you should configure slightly different rekey timers in both ends? Having identical settings can accidentally cause a traffic disruption due to ke-keying failure.
-
@keyser This is good to know, I actually forgot about that lol.
I can give that a shot, though the rekey timers are identical between sites for all other sites and they've never had an issue.
Again the IPsec Status page will show them as up on both firewalls, but no traffic will pass.
I should have noted these are policy VPNs too, not routed, and the rules will be hit but no traffic shows up on the other side.
I will give this a shot, once I'm done troubleshooting the other issue I had that required my failover in the first place (which was honestly the oddest issue I've seen in my carrier, but that's an entirely different post/issue).
-
@planedrop Yeah, It’s a long shot I know, but I pressume there is a reason they note this - perhaps those systems just have a tendency to use the same random generator every once in a while.
On another note? If it happens at night… could it be that you should configure keep-alive on the tunnels, because nighttime inactivity can cause a state or NAT session timeout on an inpath firewall?
-
@keyser Yeah it's worth a shot at least, I'll give this a go.
I have seen others online were duplicate SAs will show up, but from everything I've seen that normally doesn't prevent traffic from flowing, maybe I'm remembering wrong though.
We do use these VPNs 24/7, we have a night crew, so I don't think it's related to the keep alive. Nonetheless I'll make sure it's enabled (it's not right now, but I am 99% sure I had it enabled in the previous setup when this issue started).
Thanks for the tips though, greatly appreciate it.
