network issues - vti - gateway_alarm restarts all tunnels

[shalles]

Hi there,

our company is running three pfSense Firewalls (pfSense 2.4.4 p2) connected together via ipsec and vti setup.
One pfSense is running at AWS and so has to use nat-t.
While the connection to AWS is established successfully without any problem, we experience short network hiccups from time to time.
We suggest that the issue is related with child_sa rekeying.
We already tried several options, but the situation did not improve.

The similar setup between our onpremise location and our office is running without any problems by the way.
The only difference we see is that no nat-t is involved there.

Can anyone please have a look on this, because I am getting more and more desperate.
I added our config below.
I actually didn´t provide any logs because i don´t know what to look for.

Thank you,
Sebastian

onpremise_ipsec_statusall.txt
onpremise_ipsec.conf.txt
onpremise_ipconfig_ipsec2000.txt
aws_ipsec_statusall.txt
aws_ipsec.conf.txt
aws_ipconfig_ipsec1000.txt

[Konstanti]

This post is deleted!

[shalles]

Hi,

i still try to debug our problems. Yesterday we again had connection problems and i found some interesting log entries, i would like to share.

At 16:02:40 dpinger raises a gateway alarm and then all ipsec tunnels get restartet.

<12>1 2019-02-12T16:02:40+01:00 technik-prod-transitgw-eu-central-1-001 dpinger - - - SAEAST1_VTIV4 169.254.3.17: Alarm latency 230233us stddev 522us loss 21%
<30>1 2019-02-12T16:02:40+01:00 technik-prod-transitgw-eu-central-1-001 rc.gateway_alarm 82072 - - >>> Gateway alarm: SAEAST1_VTIV4 (Addr:169.254.3.17 Alarm:1 RTT:230.233ms RTTsd:.522ms Loss:21%)
<13>1 2019-02-12T16:02:40+01:00 technik-prod-transitgw-eu-central-1-001 check_reload_status - - - updating dyndns SAEAST1_VTIV4
<13>1 2019-02-12T16:02:40+01:00 technik-prod-transitgw-eu-central-1-001 check_reload_status - - - Restarting ipsec tunnels
<13>1 2019-02-12T16:02:40+01:00 technik-prod-transitgw-eu-central-1-001 check_reload_status - - - Restarting OpenVPN tunnels/interfaces
<13>1 2019-02-12T16:02:40+01:00 technik-prod-transitgw-eu-central-1-001 check_reload_status - - - Reloading filter

During this restart we have packet loss on all tunnels and it takes several minutes to build up all tunnels and routes.
I have added a logfile with more details below.

Is this a intended behaviour to restart all tunnels?

Thank you,
Sebastian

0_1550048573297_ipsec_aws_16_00_log2.txt

[Konstanti]

@shalles
Hey
The logic of PFSense in this case looks like this

1. you configure remote gateway monitoring /System/Routing/ SAEAST1_VTIV4/Advanced
   There are also parameters that , if exceeded, dpinger which calls the script
   rc.gateway_alarm
2. rc.gateway_alarm calls an external command pfSctl
3. pfSctl calls several scripts ( one of which rc.newipsecdns is designed to initialize IPSEC ) with the help of the daemon check_reload_status

In the settings, you can disable the gateway monitoring o ( then PF will assume that the gateway is UP all the time )

[jimp]

You could also leave monitoring enabled but disable gateway actions for the VTI gateway.

[shalles]

Thank you, for your feedback!
I will give it a try.

Sebastian