GRE Tunnel using Proxy ARP

Xuap · Apr 6, 2024, 5:00 PM

Hey.

My current setup is a two pfsense firewalls that connect each other by creating a GRE connection.

The intended goal is to pass the network from 185.113.141.1 to the local. I'm trying to do this so I can use remote IPV4s into my network.

The remote pfsense has the following data:

Gateway: 185.113.141.1
Avaliable IPs:

185.113.141.90
185.113.141.139
185.113.141.145
185.113.141.146
185.113.141.147

On the remote pfsense

Every IP Address on the remote pfsense is selected as Proxy ARP just like the following image:

What I did so far was disabling the outbound NAT on the remote pfsense, and enabling NAT as hybrid on the local pfsense making it so that every local IP address skips NAT.

To make the IP work in the local pfsense I created a static route on the remote pfsense just like the following image:

The firewall rules on the remote pfsense are empty on the WAN and just 80 and 443 port opened in the GRE interface.

On the local pfsense

To make it work like it was a local gateway, I set up the LAN interface like this:

(I've also tried creating a non-local gateway but it didn't work aswell)

This is the current NAT setup:

The firewall rules on the WAN interface are:

On the LAN interface (The local 185.113.141.1):

On the GRE interface:

Both tunnels connect really well, I can ping 10.0.2.X on each machine and it works perfectly.

I'm currently passing the network through a Linux Bridge on my proxmox, it worked perfectly before, but recently we had a network upgrade to 500mbps and extra ip addresses and it suddently stopped working.

For instance, if I do this:

root@teste:~# ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=60 time=11.0 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=60 time=10.6 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=60 time=11.1 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=60 time=11.1 ms
^C
--- 1.1.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 10.636/10.940/11.086/0.180 ms
root@teste:~# curl -v icanhazip.com
*   Trying 104.16.185.241:80...
* Connected to icanhazip.com (104.16.185.241) port 80 (#0)
> GET / HTTP/1.1
> Host: icanhazip.com
> User-Agent: curl/7.74.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sat, 06 Apr 2024 16:55:16 GMT
< Content-Type: text/plain
< Content-Length: 16
< Connection: keep-alive
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET
< Set-Cookie: __cf_bm=kiTZnWs5HGdQlIbxFQVbSdYv9fFtdT.hSJ4bFYpo.bM-1712422516-1.0.1.1-D8aXzP1LkZR9DAXsabeWBDb3qHhYsNsf3eQUtJEAJ.kLoB_RGLk52WOLP7JksU7Spf1ORCLneUsAzOTGDkFAQg; path=/; expires=Sat, 06-Apr-24 17:25:16 GMT; domain=.icanhazip.com; HttpOnly; SameSite=None
< Server: cloudflare
< CF-RAY: 87034df58c56489b-LIS
< alt-svc: h3=":443"; ma=86400
< 
185.113.141.145
* Connection #0 to host icanhazip.com left intact
root@teste:~#

It works just fine, but if I try to connect to https, it takes a really long time and then replies this:

root@teste:~# curl -v https://google.com
*   Trying 142.250.178.174:443...
* Connected to google.com (142.250.178.174) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to google.com:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to google.com:443 
root@teste:~#

I've tried every rule, every change and it still doesn't work. If I traceroute an IP like cloudflare's warp, it goes through the tunnel, but if it's from the inner network of the remote pfsense, like 185.113.141.4 or the dns server 185.113.141.253, it doesn't go through.

Example of traceroute to the one.one.one.one:

root@teste:~# traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
 1  10.0.2.2 (10.0.2.2)  9.811 ms  9.770 ms  9.757 ms
 2  gw-141.i4w.pt (185.113.141.1)  9.849 ms  9.835 ms  9.976 ms
 3  194.38.148.179 (194.38.148.179)  10.615 ms  10.601 ms  10.626 ms
 4  195.245.143.5 (195.245.143.5)  10.568 ms  10.955 ms  10.943 ms
 5  RC02-12 (195.245.142.194)  10.926 ms RC01-12 (195.245.142.166)  10.912 ms RC02-12 (195.245.142.194)  10.897 ms
 6  Cloudflare.AS13335.gigapix.pt (193.136.250.200)  11.477 ms  11.252 ms  11.229 ms
 7  one.one.one.one (1.1.1.1)  11.199 ms  10.716 ms  10.602 ms
root@teste:~#

And if I connect to 185.113.141.145 by http, it goes like lightning, but if I try to https it, it doesn't go through, or if it does, take a really really long time and sometimes doesn't even load at all.

Any idea how can I solve it? Thanks a lot, I've been exhausted trying to fix it :'(

stephenw10 · Apr 6, 2024, 10:02 PM

@Xuap said in GRE Tunnel using Proxy ARP:

If I traceroute an IP like cloudflare's warp, it goes through the tunnel, but if it's from the inner network of the remote pfsense, like 185.113.141.4 or the dns server 185.113.141.253, it doesn't go through.

Where are those IPs? They are not in the list you gave. If those are targets it will fail because they are inside the local LAN subnet. Which is all kinds of wrong.

If you don't have the proxy ARP VIPs does it stop working entirely?

Can I assume this is the actual same issue as this thread:
https://forum.netgate.com/topic/187069/problem-with-tcp-and-gre-tunnel

Steve

Xuap · Apr 7, 2024, 10:27 AM

@stephenw10 Hey stephen, so, answering your questions:

Where are those IPs? They are not in the list you gave. If those are targets it will fail because they are inside the local LAN subnet. Which is all kinds of wrong.

The 185.113.141.4 is a random server on the inner network of the 185.113.141.1 remote gateway, that the network admin said we could try to traceroute.

The 185.113.141.253 is the DNS server of the remote network (Redirects to 8.8.8.8 afaik).

If you don't have the proxy ARP VIPs does it stop working entirely?

I removed the 185.113.141.145/32 IP from the VIPs in the remote pfsense and the network still goes the same. I can ping 1.1.1.1, curl some http host but still the same, not working for https.

Can I assume this is the actual same issue as this thread:
https://forum.netgate.com/topic/187069/problem-with-tcp-and-gre-tunnel

StomperG is a colleague of mine and is on the same boat with me on this one. I'm a little more experienced with pfsense since I've been using it for a year or so, therefore he asked me to make a topic about the matter and since I know what is working or not I can safely explain it to you.

(His topic can be closed if you want)

Any idea what can be done to force the local IP to go through the 185.113.141.145?

Best regards.
Xuap

stephenw10 · Apr 7, 2024, 9:50 PM

The basic problem here is that you're trying to use something that is expected to be in the same layer 2 but route it across a layer 3 tunnel.

So the two solutions would be to either make it routed properly as I suggested in the other thread. Or to switch to a completely layer 2 setup using something like OpenVPN TAP mode and bridges.

You said you tried to add the gateway as outside the subnet and it didn't work?

Xuap · Apr 9, 2024, 4:59 PM

@stephenw10 How would it work by making it go over a complete layer 2 using OpenVPN in TAP mode?

Yes, it basically considers the gateway as non-local but doesn't work.

We've tried using Wireguard before but it just went deprecated, so any idea on the OpenVPN? I would need to close the GRE tunnels and make everything go through the vpn?

Best regards

stephenw10 · Apr 9, 2024, 5:58 PM

Wireguard is not deprecated but it is layer 3. It won't help here.

To use OpenVPN create the tunnel with both ends configured in TAP mode. Then bridge the remote end to the WAN and the local end to the LAN. The local LAN will then be in the same layer 2 segment as the VPC side WAN so devices will see each other and everything will use the VPC side gateway directly. The local LAN could use one of the available IPs you have or have no IP address if the local pfSense does not require access to the subnet directly.

https://docs.netgate.com/pfsense/en/latest/recipes/openvpn-bridged.html

Xuap · Apr 10, 2024, 1:51 PM

@stephenw10 Ok, so if I'm getting this right, my local pfsense should connect to the OpenVPN as a TAP client and get the openvpn interface assigned to the lan 185.113.141.0/24 network?

stephenw10 · Apr 10, 2024, 2:23 PM

It doesn't matter which end is client but I would probably choose the local end, yes.

The OpenVPN interface is not assigned as LAN. You have to bridge it to the LAN by creating a bridge and adding both LAN and OpenVPN to it. At the other end it's bridged to the WAN.

How much latency is there between the pfSense instances? High latency across a layer 2 link can cause problems.

Xuap · Apr 12, 2024, 10:32 PM

@stephenw10 Whenever I start the OpenVPN interface, after a few seconds it just stops my connection to pfsense, any idea why?

The latency is 9ms. It always worked fine, without any issues...

Xuap · Apr 12, 2024, 10:36 PM

@Xuap Just forget the part where it stops my connection. It was lacking the port 80 pass rule loool

Xuap · Apr 16, 2024, 2:28 PM

@Xuap Ok.. so I tried creating the tunnel as follows:

Remote

Local

Local pfsense rules:

Remote pfsense rules:

Any error here?

Because the status is just that it don't connect...

Xuap · Apr 16, 2024, 2:28 PM

@stephenw10 Any idea?

TwentyTwo · Apr 17, 2024, 7:45 PM

@Xuap I tried the same setup as you and got the same problem.

stephenw10 · May 7, 2024, 12:55 AM

Sorry missed the replies here.

It looks like you're using the webgui cert as the server cert? It has to be a cert created against the server CA.

It also looks like the TLS key is different. Both ends must have the sane TLS key.

You also still have a bunch of routed tunnel settings like pushing routes and adding gateways. But I'd fix up the cert/key first before looking at that.

Steve