No incoming S2S IPSec VPN connections. Manual telnet requests show up.

halb9

Hi folks,
I can't seem to establish a working VPN connection between my two sites. It would be much appreciated if someone could have a look at it with fresh eyes. It probably is some minor misconfiguration I am currently missing. Thanks. Here we go.
Oh and I put this in the general section of the forum as of now I don't really know where exactly the issue is. I suggest to move it to the appropriate category once we find out where I screwed up ;)

What I have:

Mainsite has two pfSense units, configured in a HA setup.
Offsite has one unit.

Both sites have a "normal" internet connection and and LTE uplink.
The LTE uplink is purely for web/surf redundancy. No VPN failover required.

On the mainsite primary internet comes from a VDSL PPPoE dialup via the FritzBox router. FritzBox is set up to act in an 'Exposed Host' configuration, meaning every request to the FritzBox from the outside is directly forwarded to the firewall (my pfSense) behind it. In this case since it is a HA setup the CARP IP is set for the exposed host.
Theoretically I could use a simple modem and do the PPPoE login directly via my pfSense - and have done so in the past - however due to ISP wisdom this does not work very well. And is not recommended for a HA setup anyways.
Since the FritzBox is doing the PPPoE dialup the public IP lays on the WAN interface of the FritzBox.

On the offsite primary internet comes from a cable company via their modem. Normally this modem acts at the same time as a router and does pretty much the same as the FritzBox would. However I activated the so called 'BridgeMode' meaning it only does docsis to ethernet modulation and gives the connected device a public IP address. That's it. Therefore the public IP lays on the WAN Interface of the pfSense unit. No exposed host necessary.

Both primary internet uplinks are consumer grade lines. No static IP. No special SLA. Just plain homeuser internet.
Both secondary LTE uplinks are for emergency purpose. They are behind CGNAT so you can't really do anything with it anyways, except web stuff.

For both sites a dynamic dns service is configured. So a public domain resolvs to the current public IP of the respective primary uplink.

For both sites, access to the webgui via the IP from the opposite site is granted (feed by the dyndns update of course)

What I want:
Site-to-Site VPN tunnel between both sides. Basically both servers should be able to communicate over the VPN with eachother.
To keep it simple both servers are located in the LAN subnet. This might change later but for debugging purposes we will keep it that way for now.
VPN should use the VDSL / cable line. Not the LTE uplink.

What I did?

Setup both my units as a HA unit according to netgate docs.
https://docs.netgate.com/pfsense/en/latest/solutions/reference/highavailability/index.html
Configured both units for IPSEC tunnel.
Phase1:

Phase2:

What works / does not work?
VPN tunnel will not establish.

My troubleshooting steps so far.
When I'm connected via ssh to the offsite-a and try to open a telnet session on some random port I see those incoming connections on the mainsite.

However when trying to establish the VPN tunnel via the webgui from the offsite I see no incoming connections. This is regardless of me trying to connect to the domain or the IP.

My question:
It seems like the offsite does not send out the vpn request. Any idea why? How can I dig deeper? What am I missing?
Thanks for taking the time!
All the best!

halb9

Good news. I found some things that we're wrong.

Some of them I was able to fix. Others not yet.

I had a look in the IPSec log from the webgui. The initiator on the offsite showed received NO_PROPOSAL_CHOSEN error notify as an error.
I doublechecked my security proposals, set my Encryption Algorithms in phase 2 to just AES with 256 bits. Nothing else.
Still no connection.
But I had a better look at
https://docs.netgate.com/pfsense/en/latest/troubleshooting/ipsec.html
and especially
https://docs.netgate.com/pfsense/en/latest/troubleshooting/ipsec.html#mismatched-identifier-with-nat

I edited that part so on the offsite for the remote identifier it says 192.168.0.10.

Still no connection. But just out of curiosity I switched the initiator for the vpn connection. From offsite -> mainsite to mainsite -> offsite. Et voila, working vpn connection!

Which is not exactly the way I wanted it. But still better than nothing.

I switched back again to have a look at the log again.

I see outgoing vpn connections from the offsite but no incoming connections on the mainsite. So there has to be some issue on the FritzBox with the port forward / exposed host setting.

In the ARP table of the FritzBox I see the following.

192.168.0.6 has nothing to do with this and should be ignored.

192.168.0.12 with :68:F8 is the secondary cluster unit with real ip and real mac-address.
192.168.0.11 with :01:02 is the primary cluster unit with real ip and virtual mac-address.
192.168.0.10 with :69:be is the virtual cluster unit with virtual ip and real mac-adress (of primary cluster unit).

According to
https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability.html#use-a-different-vhid-on-each-carp-vip

my understanding is for both real ip adresses the real mac should be in the arp table. And for the virtual ip the virtual mac should be there.

For my LAN subnet this seems to be the case.

Any idea why it is not doing that on the WAN interface?

Thanks

halb9

I believe I got it.
Turns out, the FritzBox (at least in regards to virtual ip/mac) is crap. What you see is not what you get.
I crossreferenced what I saw on the FritzBox with my local computer.

On my local computer it seems to work as expected.

Furthermore I disabled the exposed host functionality and went for a simple port forward.

With this, initiating a vpn connection from the offsite works without any problems.

I'll mark this as solved.
Thanks!

//edit: Ok seems I'm unable to edit my first post. Anyway for me this works now. Have great day!