OpenVPN + BGP finally stabilized
-
Many many trials and travails later, I have finally stabilized my OpenVPN + MPLS MultiWAN + BGP setup. The problem was simply that fairly often pf would block BGP session packets traveling over OpenVPN tunnels, causing the session to go idle, and switching the session over to its MPLS link (slower in this case, but used as a backup).. then after some time it would switch back. This was continuous, and causing annoying hiccups in our application.
Other OpenBGP clients in the full mesh on OpenVPN behaved similarly, but not identically. Essentially, in most cases pf would suddenly block all BGP packets and the session would drop for "awhile".
I have not had the time to do a TCP analysis as to what could be causing this, however my suspicion is simply that OpenBGPD has a faulty TCP mechanism, and occasionally misbehaves.
There was simply no way to avoid the dreaded "default deny" rules from time to time. Floating rules did not help, and etc.
On some of my pfSense boxes, simply checking the option that prevents gateway state flushing worked (apparently a common problem with OpenVPN) On some, particularly this one, it did not.
So, I heaved a sigh and edited /etc/inc/filter.inc and put in, as the very first rule before the default deny(s):
pass quick on { ovpnc2 } from any to any no state label "USER_RULE"
save and reloaded the conf, and things worked.
I have no trouble doing this, as the IPs inside that tunnel are not routable to anywhere else, and no traffic of any kind except the BGP traffic is taking place there.
I'd suggest that folks look into this. I am using UDP OpenVPN, fairly standard setup with no-replay and slightly larger buffers. (no-replay was also necessary, as something was fudging my time stamps). My BGP sessions are directly on the neighbors using the OpenVPN IPs, which are individual /30 blocks. I have also specified the local OpenVPN address as "local-address" in my BGP neighbor definitions.
I'd appreciate not having to do this.. I don't like manually kludging things! If someone can replicate this behavior I would love to hear what you find!
-
Just out of curiosity, did you have a rule in the OpenVPN rule tab under the firewall?
-
Yes I do. All/All Pass.
Its definitely odd behavior… I have rules on OpenVPN, and All/All pass on each OpenVPN interface, assigned and set. And the block would show as coming from that interface. See, TCP SYN packets get through.. its something to do with state keeping. I am not a pf savvy guy (I know the basics, but analyzing the blocks is a bit beyond me at the moment)