GIF monitoring traffic goes via wrong parent interface on pfSense 2.4.4

Samsonov

With pfSense 2.4.3, my 6in4 tunnels were set up like this:

gif0 to Broker1 via pppoe1 — online;
gif1 to Broker2 via pppoe1 — online;
gif2 to Broker1 via pppoe2 — online.

After upgrading to pfSense 2.4.4, it now works like this:

gif0 to Broker1 via pppoe2 — offline;
gif1 to Broker2 via pppoe1 — online;
gif2 to Broker1 via pppoe1 — offline.

By using tcpdump and filtering by Broker1 address, it can be seen that gateway monitoring pings of gateways corresponding to gif0 and gif2 use parent interfaces of each other — no wonder this traffic is discarded by Broker1 and both interfaces are considered offline. Even more so, from time to time (I have rebooted several times, and also tried changing GIF parent interfaces back and forth) gif2 pings cannot be seen on either pppoe2 or pppoe1, while gif0 pings always go through pppoe2. Whether or not gif1 selects proper interface concisely or by pure luck, I have no idea.

How do I debug this? I have checked gateway and interface settings, routing and firewall rules, tried changing interface binding and then re-setting it back to normal, but could not spot any configuration error nor quirk solution. In fact, I did not change the settings since April, and everything worked until I upgraded to pfSense 2.4.4.

chrismacmahon

On system - routing Do you have a default gateway defined, or is it on automatic?

Samsonov

@chrismacmahon I have tried various options for IPv4 and IPv6 default gateways:

Automatic;
default gateway group defined by pfSense 2.4.4;
my own gateway group;
randomly selected single gateway;
single gateway corresponding to WAN interface being currently sniffed;
single gateway other than WAN interface being currently sniffed;
None.

In general, this had no effect on GIF pings. And why should it? Are not gateway monitoring pings supposed to travel via parent interface only?

Well, I once observed the traffic to start flowing — with randomly selected default gateways, during the short period of time while the new gateway settings were applied — but it stopped as soon the settings were applied and I could not repeat this later.

I also had successfully used ping6 and traceroute6 from pfSense host shell even before that, i. e. without changing default gateways. Either by default selection of by manual selection through -s argument, gif0 or gif2 source address was displayed by those utilities; I did not check that it actually used proper interfaces though — when I started tcpdump in parallel later, the traffic stopped flowing and did not resume anymore (I tried quitting tcpdump in order to check whether promiscuous mode may be the one to blame, but that did not help).

Samsonov

Additional noteworthy observations.

There was one strange thing about GIF configuration on pfSense 2.4.3 (and before?). I had to disable Outer Source Filtering on gif0 for the traffic to flow — otherwise even gateway monitoring pings were discarded upon reception: that is, if I remember correctly, ping replies were received on parent interface but rejected at GIF level. Those ping replies had proper source and destination addresses for both IPv4 and IPv6 and came in via proper interface. Of course, the IPv6 network for GIF tunnel itself was not the same as for overlaid network — but that is the case for all tunnels of all brokers. In particular, gif2 to the same broker was functioning well with Outer Source Filtering enabled by default, as well as gif1 to another broker.

Right before upgrading from 2.4.3 to 2.4.4, I noticed that gif2 also needs disabling Outer Source Filtering. I had no idea on why this happened and how long ago — just switched the offending setting, and the tunnel became operational for about a couple of hours until the update took place. Same as earlier, however, gif1 to another broker was functioning with Outer Source Filtering enabled by default, and used proper parent interface even after upgrading to pfSense 2.4.4.

Now that pfSense 2.4.4 is installed, I tried switching Outer Source Filtering back on and then off again — just in case — but observed no effect. That was expected indeed, as the primary issue is not with ingress filtering on local side: outgoing traffic is filtered by remote end because of improper source addresses caused by improper parent interface being used.

I also tried Disable Gateway Monitoring for both gateways corresponding to gif0 and gif2. That allowed the traffic to flow out unconditionally, but only showed that any kind of traffic — not just ICMP pings — chose wrong parent interface. I once again tried changing default gateway settings, and the outcome was equally negligible. That is, sometimes I saw small bursts of legitimate traffic pass out and then in (such as my NTP server making a request and receiving a reply), but it is hard to correlate to settings change as those bursts stop soon. The other times I see legitimate inbound traffic entering proper parent interface, but somehow filtered on local side — such as incoming NTP and DNS requests with no reply from my home server [because pfSense filtered those requests out]. :puzzled: