Running a GRE Tunnel over OpenVPN

vladivanovic

Hi all,

I'm in a bit of a complex situation, last night I swapped out the main OpenVPN Server to replace it with PFSense. At first everything seems to work well, all traffic is flowing between multiple sites - however a single openvswitch gre tunnel which runs over said site-to-site vpn has stopped working.

I saw the following weird states for GRE in the state table on both and for some reason the firewall is adding the WAN IP's to the source of the packets state.

Quick Layout:

For example, a normal TCP Session that worked fine:
ovpnc1 tcp 10.223.13.101:5563 <- 10.130.70.201:57494 CLOSED:SYN_SENT

Only GRE Sessions seemed to have weird sessions where it would rewrite the packet on to the WAN despite the PFSense instance having a VPN Session with the correct routes being pushed down (i.e. the TCP transactions above worked fine as did ICMP and others)

The GRE session from the Branch site:
WAN gre 192.168.1.28 (10.223.13.100) -> 10.12.0.132 SINGLE:NO_TRAFFIC
LABB gre 10.12.0.132 <- 10.223.13.100 MULTIPLE:MULTIPLE

The GRE session from the Main Site:
LAN gre 10.223.13.100 <- 10.12.0.132 NO_TRAFFIC:SINGLE
WAN gre 202.167.236.80 (10.12.0.132) -> 10.223.13.100 SINGLE:NO_TRAFFIC
LAN gre 10.223.13.100 <- PUBLIC-IP(sanitised) NO_TRAFFIC:SINGLE
ovpns1 gre PUBLIC-IP(sanitised) -> 10.223.13.100 SINGLE:NO_TRAFFIC

This session should essentially be establishing between 10.223.13.100 and 10.12.0.132 - however rather than the state showing up on the opnvn interface and lan interface only, its actually somehow showing up on the WAN interface and rewriting parts of it.

I tried to kill the sessions but nothing changed.

I disabled NAT'ing and then killed the sessions again, bam it came up and started working just fine with the following in the state table which is what I would have originally expected to see.

GRE from the Branch site:
ovpnc1 gre 10.223.13.100 <- 10.12.0.132 MULTIPLE:MULTIPLE
LABB gre 10.12.0.132 <- 10.223.13.100 MULTIPLE:MULTIPLE

GRE from the Main site:
LAN gre 10.223.13.100 <- 10.12.0.132 MULTIPLE:MULTIPLE
ovpns1 gre 10.12.0.132 -> 10.223.13.100 MULTIPLE:MULTIPLE

Bear in mind I also tried resetting the openvpn session from the branch site multiple times which changed nothing, both the branch and main site are barely under load and state tables are at a minimum.

I have no tried to reproduce this error as of yet, I can if required for further debugging but can anyone explain to me why PFSense NAT'ing module seemed to think it should automatically try to NAT GRE traffic rather than punt it up to the tunnel? :) I don't necessarily need to do a post implementation review but I know that my colleagues and boss will be interested.</pfsense></internet></pfsense></branch>