2.7.2-RELEASE GRE Tunnel problem
-
I've been running pfSense 2.7.0 for years, without issue.
A few days ago, I upgraded my hardware in preparation for 10G fiber coming in. The new hardware came up fine, and 2.7.2-RELEASE was installed on it.
I use GRE tunnels as part of my home network to get to my Cisco-based gear in colo. These tunnels have been running on 2.7.0 without issue. They don't seem to work on 2.7.2-RELEASE, so I broke something, but I've been staring at this for 6 hours now, and can't find the problem.
The GRE tunnel comes up. Traffic from pfSense to Cisco gets encapsulated correctly, exits via the correct WAN interface, gets to Cisco GRE without issue. Traffic from Cisco to pfSense gets encapsulated correctly, exits via its correct WAN interface, reaches the WAN interface on pfSense, and vanishes. I removed all rules on the firewall (again, this has been working for years on 2.7.0).
Here's an example ping from pfSense to Cisco, on the Wan interface of pfSense:
01:06:19.459474 IP 1.1.1.1 > 2.2.2.2: GREv0, length 33: IP 172.17.1.226 > 172.17.1.225: ICMP echo request, id 22302, seq 22, length 9
01:06:19.483790 IP 2.2.2.2 > 1.1.1.1: GREv0, length 33: IP 172.17.1.225 > 172.17.1.226: ICMP echo reply, id 22302, seq 22, length 9But on gre0:
01:18:39.806776 IP 172.17.1.226 > 172.17.1.225: ICMP echo request, id 22302, seq 1424, length 9
01:18:40.339025 IP 172.17.1.226 > 172.17.1.225: ICMP echo request, id 22302, seq 1425, length 9
just the outgoing traffic. What am I missing?Under states, I see unusual results:
WANATT gre 1.1.1.1 -> 2.2.2.2 SINGLE:NO_TRAFFIC 8.571K / 0 528 KiB / 0 B
(on 2.7.0, I see the gre state as MULTIPLE:MULTIPLE)
-
Testing a local tunnel, between 2.7.0 and 2.7.2 on the LAN works fine. So, there is something I'm not understanding about the WAN interface (which, in my case, is a VLAN on a 10g port, ix1.35.)
The test GRE tunnel was configured on ix1.1, and works as expected.
-
I assume you don't see anything blocked in the logs?
If you're running that pcap on the VLAN interface then it must be arriving there as expected. I would try a pcap on ix1 directly though to check the VLAN tags are correct in there.
-
Excellent suggestion - but they are for sure arriving on the correct vlan, and into the ix1.35 interface.
Nothing is blocked in the logs. The "firewall" entry is simply "Pass ipv4+ipv6 any protocol, any address" for both the GRE interface and the WAN interface.
I've demonstrated (to myself) that I can build a function GRE tunnel on the LAN interface, without issue, to another pfSense box, and bring it up.
I've confirmed that the GRE interface looks correct:
gre0: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1476
description: ATTtoEdge3
options=80000<LINKSTATE>
tunnel inet 1.1.1.1 --> 2.2.2.2
inet 172.17.1.226 --> 172.17.1.225 netmask 0xfffffffc
inet6 fe80::2e0:67ff:fe27:6d54%gre0 prefixlen 64 scopeid 0x12
inet6 2001:[blah]::1:2 --> 2001:[blah]::1:1 prefixlen 128
groups: gre
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>It's exactly as if the packets arrive on the WANATT interface, and can't figure out where to go to get to the GRE interface, so get discarded. I've even tried turning off hardware checksum, just in case that was breaking something.
Is there a way to trace the flow of the packets internally on FreeBSD? Adding additional rules? I'm comfortable with any outlandish suggestions anyone can offer.
-
Following on to your suggestion, I removed the ix1.35 interface, and connected it directly using a 1g Ethernet port on the 2.7.2-RELEASE router, eliminating the potential VLAN processing error.
First, I nuked the existing WAN and GRE configuration, and deleted ix1.35.
So now I have external WAN -> igc3 (1g port).I configured the igc3 port as WANATT, using DHCP and DHCP6, and ticked (only) the block private/block bogon to create default rules.
I then built the GRE tunnel. Then I added the GRE tunnel as a new interface, did not tick the "block private networks and bogon" boxes.
I then added appropriate PASS rules.
Exactly the same issues, and the same "state":
WANATT gre 1.1.1.1 -> 2.2.2.2 SINGLE:NO_TRAFFIC 1.521K / 0 94 KiB / 0 BSame traffic reaching the CIsco side, same responses coming in via igc3:
listening on igc3, link-type EN10MB (Ethernet), snapshot length 262144 bytes
04:04:43.607766 IP 1.1.1.1 > 2.2.2.2: GREv0, length 33: IP 172.17.1.226 > 172.17.1.225: ICMP echo request, id 32511, seq 7, length 9
04:04:43.607773 IP 1.1.1.1 > 2.2.2.2: GREv0, length 53: IP6 2001:[blah]::1:2 > 2001:[blah]::1:1: ICMP6, echo request, id 32912, seq 7, length 9
04:04:43.632239 IP 2.2.2.2 > 1.1.1.1: GREv0, length 33: IP 172.17.1.225 > 172.17.1.226: ICMP echo reply, id 32511, seq 7, length 9
04:04:43.633238 IP 2.2.2.2 > 1.1.1.1: GREv0, length 53: IP6 2001:[blah]::1:1 > 2001:[blah]::1:2: ICMP6, echo reply, id 32912, seq 7, length 9
04:04:44.118472 IP 1.1.1.1 > 2.2.2.2: GREv0, length 53: IP6 2001:[blah]::1:2 > 2001:[blah]::1:1: ICMP6, echo request, id 32912, seq 8, length 9
04:04:44.128981 IP 1.1.1.1 > 2.2.2.2: GREv0, length 33: IP 172.17.1.226 > 172.17.1.225: ICMP echo request, id 32511, seq 8, length 9
04:04:44.143506 IP 2.2.2.2 > 1.1.1.1: GREv0, length 53: IP6 2001:[blah]::1:1 > 2001:[blah]::1:2: ICMP6, echo reply, id 32912, seq 8, length 9
04:04:44.153758 IP 2.2.2.2 > 1.1.1.1: GREv0, length 33: IP 172.17.1.225 > 172.17.1.226: ICMP echo reply, id 32511, seq 8, length 9 -
Hmm, is that new test tunnel still to the Cisco device? Can you test to a remote pfSense device on the WAN?
For some reason the state is not matching the incoming packets. That could be because the gre interface itself is not recognising those as valid.
The gre interface has a few options that could potentially be an issue though I'm not aware of anything that changed from 2.7.0 to 2.7.2.
I would open those packets in wireshark and compare them in detail with the working LAN side tunnel.
Nothing significant I see here: https://github.com/pfsense/FreeBSD-src/commits/devel-main/sys/net/if_gre.c
-
@stephenw10
That's a great idea!I created a tunnel from 2.7.2 to the old 2.7.0 system, using the WAN interfaces.
Guess what?
Works fine.
WANATT gre 3.3.3.3 -> 1.1.1.1 MULTIPLE:MULTIPLE 512 / 344 28 KiB / 18 KiB
I can ping both sides.
I did just IPv4, initially. Let me add ipv6...And that breaks it, but IPv4 still works. So it does look line an issue... I'll continue poking.
-
@DaveRand
The IPv6 breakage was caused by the firewall entry on the GRE not permitting IPv6.So, GRE via WAN to older pfSense boxes work.
GRE from older boxes to Cisco works.
GRE from 2.7.2 to Cisco boxes work, but 2.7.2 fails to recognize the GRE packets from Cisco
GRE from 23.09.1 to older pfSense boxes works.
GRE from 23.09.1 to Cisco boxes work, but 23.09.1 fails to recognize the GRE packets from Cisco.I think there may be a bug.
-
Yup, possibly.
I would compare the gre reply packets from pfSense to those from Cisco coming into the 2.7.2 (or 23.09.1) install and see what the difference is. There must be some reason the interface isn't seeing those incoming packets.
-
@stephenw10
The problem was due to an AT&T modem keeping an ARP cache alive far longer than is reasonable. It was sending packets to the wrong MAC address. That's why the interface was seeing the traffic (promiscuous mode), but not accepting it.Sigh.
I never would have suspected it. Netgate TAC rocks.
-
Aha, nice!
