Traffic of Google domains is blocked over IPSec tunnel

tiiash

What I am experiencing is similar to the situation described in the topic "[solved] Allow rule not working?!". However, as my setup differs I didn't want to pirate that thread and created a fresh one instead.

First, let me briefly summarize my (non-average) setup.
I am currently setting up two individual pfSense installations for use at our small company. We are located in two different rooms inside a shared-office space. The only networking connection between these two offices is through an untrusted cable running through the cellar of the building. Therefore, we have decided to encrypt the channel via IPSec and treat both offices as separate "location" if you will.
One office, let's call it "B", has no individual Internet uplink of any kind and is only connected via this IPSec tunnel.
The other office ("A") has two Internet uplinks which are connected to pfSense.
For now, all traffic should be routed over WAN1. I am just mentioning this for the sake of completeness.
Both pfSenses are configured with multiple VLANs to separate traffic.

WAN1 -v
      pfSense "A"  <------ Routed IPSec ------->   pfSense "B"
WAN2 -^

I decided to use "routed IPSec" to connect both pfSenses. While "A" uses IP addresses in the range of "10.1.0.0/16" for all VLANs, "B" uses 10.2.0.0/16. That way, I can route traffic for both offices easily by specifying a single static routing entry per firewall.

Fast forward to my current situation:
Clients connected directly to site "A" apparently access the Internet without issues.
Clients connected to site "B" can access most websites without issues as well. However, as soon as any Google domain is involved (plain old google.com, but also googleusercontent.com, fonts.gstatic.com, even ads from doubleclick.net), the problems start.

Wireshark and tcpdump show that the first few packages of each TLS handshake are transmitted without issue, but before the connection is established, pfSense "A" blocks the connection, referencing the "Default deny rule IPv4".

First of all, I don't understand how this default deny rule could be triggered at all. I created an explicit allow anything via IPv4 rule for testing purposes on all interfaces, including the WAN and IPSec ones.

Second, even if such a rule were present, I don't understand why the traffic is caught by it. Somehow, it seems that traffic coming from Google IP addresses is not seen as "related/established", although the connection has been initiated by the client a few seconds earlier (nothing could have timed out, no states should have been cleared in the meantime).

Again, this only seems to affect Google domains. Anything else that I could think of as a test can be accessed perfectly fine.

I am out of ideas. Can anyone help?:)

tiiash

The problem still persists. I had some time to debug it further and can provide the following additional info:
pfSense A blocks traffic based on the rule block out log inet all tracker 1000000104 label "Default deny rule IPv4", specified on the ipsec interface.
As far as my understanding of IPSec VTI and firewalling in pfSense is, I cannot (easily) manually override this rule because it blocks outgoing traffic, but the "IPSec" tab in the Firewall WebGUI is meant to configure rules for incoming traffic on this interface. Is this correct?

I therefore decided to temporarily "disable all packet filtering" on pfsense A in order to debug more easily. The next thing that happened was that the same traffic got caught in the firewall of pfSense B, rule block drop in log inet all label "Default deny rule IPv4". As this is incoming traffic, I should be able to easily allow it using the WebGUI(?). However, as I mentioned in my first post, I already have a manual "allow all" rule (all IPv4 protocols, all sources, all destinations, all ports), which apparently does not match for some reason.

If I also disable packet filtering on pfSense B, Google domains start to work.
A manual test against the Alexa Top ~20 shows that this problem does not only affect Google domains (google.com, youtube.com), but also:

weibo.com
yahoo.com
live.com
part of the resources loaded by Netflix.com (more specifically, *.nflxext.com)

So far I did not test if the issue could be mitigated by switching from VTI to tunneled IPSec. Could it be possible that there are some subtle issues due to the VTI option not being as mature as the other IPSec variants?

tiiash

We now went live with the setup and I observed another detail.

Until now, I had only seen the weird behavior when connecting to external websites. As these were mostly hosted by Google, I suspected it might have been related to Google using particularly modern protocols or implementations of their own server software.

When trying to access the web interface of an old VoIP phone we have at one of the sites, I however experienced the same issue. The device definitely uses the opposite of modern protocols/software, it even only supports TLS1.0/1.1. I managed to take some screenshots depicting the problem:

Log output in pfSense saying that the connection is blocked. It looks as if the connection would originate from the phone (10.16.103.56) when in fact, it was initiated by my client (10.25.102.100) to the HTTPS port of the device.
pfSense rule log

I also managed to grab a wireshark dump of the connection attempt. Please ignore the differing IP address "10.137.0.37". This is due to me using Qubes OS, an operating system which heavily makes use of virtual machines which automatically receive internal IP addresses. For this issue, you can read 10.137.0.37 == 10.25.102.100.
Wireshark output showing issue

The output shows that a TCP handshake is established correctly and the TLS handshake starts as well, but is interrupted immediately. I think that this does not always happen at the same time. When accessing the Google domains I remember seeing a "Server Hello" message.

If I start a dummy webserver (Linux laptop running Apache) with the same IP address as the VoIP phone I can access the Apache default webpage without any issues, both over plaintext and HTTPS connections. This leads me to the conclusion that this is indeed a bug in pfSense which occurs only together with some particular servers. I don't know how else this could be explained.

I will file a bug report for this issue.