Wireguard Gateway Status

Hossimo

Moving this to the correct forum:

I finally switched my home router to a SG2100 so I can easily setup a WireGuard tunnel between to the Colo. Fortunately Christian Just did a wonderful video on setting up a site to site tunnel that after I followed I had a perfect bidirectional tunnel that's way faster than my previous openVPN tunnel and learned a few things too!

The only thing I'm not sure about is how to get the Gateway Status to lineup with reality.

Here its showing up as 100% packet loss, the thing is the tunnel is working perfectly.

So I know the simple answer is to just turn off System/Routing/Gateways - Gateway Monitoring, or perhaps pointing Monitor IP to something that responds to a ping, but I'm wondering what the "Right" thing to do is.

Looking at the System Logs/System/Gateways i see that I'm getting an sendto: error: 93 about twice a second.

I'm not exactly sure what the error actually means but if it's a system call error perhaps a permission error?

Anyway I tried fully open rules on all the relevant interfaces, but that did not seems to work.

Any ideas?

Thanks

cmcdonald

@hossimo what subnet are you using for the tunnel network?

All that you need for gateway monitoring is for the gateway address to respond to ICMP. Can you ping 10.222.222.0 from pfSense? You can use the ping tool under Diagnostics > Ping

Hossimo

@cmcdonald I'm running the following:

Site A:
LAN 192.168.10.1/24
Tunnel 10.222.222.0/31

Site B:
Lan 10.201.0.254/24
Tunnel 10.222.222.1/31

I forgot to mention the my ping and traceroute tests.
From Site A LAN to 10.222.222.1 I get host unreachable.
From Site A Tunnel to 10.222.222.1 I also get unreachable

From Site B LAN to 10.222.222.0 I get unreachable
From Site B Tunnel to 10.222.222.0 I get unreachable

This afternoon I was able to ping from the tunnel but not from the LAN, not sure what I changed there.

The both ends of the tunnel currently have an IPv4* * * * * * rules

The LAN on both ends have a IPv4* LAN net * * * * rule

For traceroute, From the Site A LAN to a server in site B I get:
1 * * *
2 10.201.0.1 18.767 ms * 19.731 ms

From the Site B LAN to my machine in site A I get (allowed pings from different subnet)
1 * * *
2 192.168.10.171 12.921 ms * 19.183 ms

So forsure I can't ping the tunnel for some reason, I probably missed something simple but I looked for a while and didn't see anything a miss. Also not sure why I'm always missing the middle packet, MTU and MSS are set to 1420 on both ends, IPv4 traffic only, but I tried 1420 and 1440.

Just as a test I just tried a specific rule for ping (allow ICMP echo request and echo reply) on both tunnel ends and still no luck.

Hossimo

Anyone have any idea what I might have done wrong.

The Tunnel works great, I can ping "through" the tunnel but not the other side.

It's not affecting the usage of the gateway just wondering what I might have configured wrong to not allow the gateway monitor to work correctly.

cmcdonald

@hossimo Assuming your firewall rules are correct, you should be able to ping across the tunnel from both sites.

i.e. pinging from site b to 10.222.222.0 and pinging from site a to 10.222.222.1. Both of those should work.

I'd run pcaps from the perspective of the tunnel interface on both sides. Anything showing up in firewall logs?

Hossimo

Yes I find it very strange as well.

I'm obviously doing something wrong but I can't figure out what type of rule is more permissive then allow anything from anywhere :)

Here is a ping test I just did from my computer to each of the following:

The left side is my home network, while the left is the colo network, captures were both from the perspective of the Tunnel Interface.

Test 1
Missing from both Local and Remote packet captures, but I did get a response back, I assume from the LAN interface not the tunnel.

>ping 10.222.222.0
Pinging 10.222.222.0 with 32 bytes of data:
Reply from 10.222.222.0: bytes=32 time<1ms TTL=64
Reply from 10.222.222.0: bytes=32 time<1ms TTL=64
Reply from 10.222.222.0: bytes=32 time<1ms TTL=64
Reply from 10.222.222.0: bytes=32 time<1ms TTL=64

Test 2
Lines 1 thru 4 on the Local capture From my laptop to the other side of the tunnel. so it's getting to the interface just not across it?

>ping 10.222.222.1
Pinging 10.222.222.1 with 32 bytes of data:
Reply from 192.168.10.1: Destination host unreachable.
Reply from 192.168.10.1: Destination host unreachable.
Reply from 192.168.10.1: Destination host unreachable.
Reply from 192.168.10.1: Destination host unreachable.

Test 3

Lines 6 thru 16, ping from my laptop to a server in the colo. zips right through.

>ping 10.201.0.1
Pinging 10.201.0.1 with 32 bytes of data:
Reply from 10.201.0.1: bytes=32 time=11ms TTL=126
Reply from 10.201.0.1: bytes=32 time=13ms TTL=126
Reply from 10.201.0.1: bytes=32 time=11ms TTL=126
Reply from 10.201.0.1: bytes=32 time=13ms TTL=126

I put a log onto an allow all icmp rule on the remote and filtered the source to my IP all I see is the traffic that gets trough, so Its not default blocked there.

With the same rule on the local router I don't get any log entries, actually a default block to an aws server for some reason, in the future!? (that's for another day)

This is definitely odd, and probably shouldn't take any of mine or anyone else's time, but it seems it should be so simple to solve.

Hossimo

I finally figured it out

Spot the issue:

It was a dumb misconfiguration on my end and I honestly didn't think of looking at until I realized it was only pinging within the tunnel and to the other side of the tunnel that was affected.

The issue was that within both peer configurations (Peers -> Address Config -> Allowed IPs) I had the incorrect subnet.

When I originally was going to set-up the tunnel I was going to use 10.254.254.0/31 but ended up changing to 10.222.222.0/31 instead.

Well I forgot to change the peers allowed IPs so nether the near side of the tunnel (10.222.222.0) could send a ping, and I could not ping the far side (10.222.222.1).

As soon as I fixed both mistakes it started working correctly, I knew it was something simple, just not what I expected (is it ever?)

So with that, is there any real need to not automatically allow both gateways? outside of allowing the clients to ping within the tunnel (which can be blocked via a rule) it seems both (or all) ends of the gateways should automatically be added to the allowed list based on the tunnel IP and it's static route gateways.

Anway way simple mistake and I'm glad I found it, I can stop obsessing now.

cmcdonald

@hossimo ha glad to hear it's working! Sometimes the simplest mistakes are the hardest to hunt down. I hit that more than I'd like to admit.