Consistently intermittent traffic for pfSense with multiple OpenVPN tunnels

v1k0d3n

Finally figured out what causes this strange behavior, but I would like to see if others have experienced this - to rule out anything obvious.

I've had an OpenVPN deployment configured for a while. The internal address range is 192.168.80.0/25. This tunnel is configured as a TLS + Auth tunnel for users against the local database. It's been working really well.

This week I needed to configure a new Site-to-Site tunnel with a remote lab using shared keys (to keep it easy for now, as this is a temporary arrangement). I created the lab end as a server, and my pfSense as an OpenVPN client. Both firewalls are running pfSense 2.5.2. The tunneled network is 192.168.116.100`.

I followed Tom's video circa 2017 (Lawrence Systems) - which is close, with only minor changes to a few fields. It's still basically the same. This video should give you an idea of how this environment was configured though, for reference.

Immediately I realized that every-other-packet was being sent down the tunnel correctly. The host I was coming from is 192.168.0.1/23 and the hosts that I'm testing across the tunnel are 192.168.116.100 and 192.168.116.100 (as shown below).

❯ sudo hping -S -p 22 192.168.116.100
HPING 192.168.116.100 (en0 192.168.116.100): S set, 40 headers + 0 data bytes
len=44 ip=192.168.116.100 ttl=62 DF id=0 sport=22 flags=SA seq=0 win=29200 rtt=27.7 ms
len=44 ip=192.168.116.100 ttl=62 DF id=0 sport=22 flags=SA seq=2 win=29200 rtt=28.4 ms
len=44 ip=192.168.116.100 ttl=62 DF id=0 sport=22 flags=SA seq=4 win=29200 rtt=27.2 ms
len=44 ip=192.168.116.100 ttl=62 DF id=0 sport=22 flags=SA seq=6 win=29200 rtt=26.9 ms
len=44 ip=192.168.116.100 ttl=62 DF id=0 sport=22 flags=SA seq=8 win=29200 rtt=27.2 ms
len=44 ip=192.168.116.100 ttl=62 DF id=0 sport=22 flags=SA seq=10 win=29200 rtt=26.7 ms
len=44 ip=192.168.116.100 ttl=62 DF id=0 sport=22 flags=SA seq=12 win=29200 rtt=28.2 ms
len=44 ip=192.168.116.100 ttl=62 DF id=0 sport=22 flags=SA seq=14 win=29200 rtt=44.9 ms

Ok, well that's really strange.

Then I realized that, for some odd reason, the traffic was being sourced differently through - what looks like the same ovpnc1 interface?

Working:

Action   Time               Interface   Source           Destination        Protocol
Allow    Jul 23 11:49:16    ovpnc1      192.168.185.2    192.168.116.184    ICMP
Allow    Jul 23 11:49:16    LAN         192.168.0.1      192.168.116.184    ICMP

Not Working:

Action   Time               Interface   Source           Destination        Protocol
Allow    Jul 23 11:49:16    ovpnc1      192.168.80.1     192.168.116.184    ICMP
Allow    Jul 23 11:49:16    LAN         192.168.0.1      192.168.116.184    ICMP

This every-other behavior was present until I turned down the first tunnel entirely. My routing tables on each firewall look absolutely fine. Making things even more strange is that I have a VLAN which was working perfectly fine, consistent each time with no issues. This network was 192.168.4.0/22. Every ICMP request was returned with a reply, and traffic when down the expected path each time.

Can someone give me an idea of what might be going on or what I should look at config-wise? This smells like a bug to me, but I'm not too proud to say that other people are smarter than I am. I don't have an ego - don't care to be wrong either. I just figured if I'm running into this, than someone else could be as well. I'm happy to report it as a bug, if someone from Netgate thinks it's a bug. I would expect there to be two ovpnc interfaces and routing to occur as expected, but that doesn't seem to be happening.