pfSense and IPSEC lan to lan: a big doubt about the correct implementation

[mauro.tridici]

@stephenw10 yes, the pings are running and I'm making some tests on both the directions.

I checked again the firewall logs. no rule is blocking the traffic.
I'm trying to understand where is the error

[stephenw10]

If there are still no icmp states then either there are firewall rules blocking that traffic or the pings never arrive at pfSense. A pcap on the pfSense B em2 interface should show that.

[mauro.tridici]

VPCS on the left can ping every hosts on the schema (except the 192.168.120.1)
VPCs on the left is not able to ping 192.168.120.0/24

VPCS on the right can ping every hosts on the schema (except the 10.99.0.0/24)
VPCs on the left is not able to ping 192.168.118.0/24

If I'm not wrong, it seems that the ping traffic is not passing through the tunnel.
Sorry for this stupid information...

[mauro.tridici]

@stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

If there are still no icmp states then either there are firewall rules blocking that traffic or the pings never arrive at pfSense. A pcap on the pfSense B em2 interface should show that.

I'm going to check this. I will let you know

[mauro.tridici]

@stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

If there are still no icmp states then either there are firewall rules blocking that traffic or the pings never arrive at pfSense. A pcap on the pfSense B em2 interface should show that.

pcap on pfsense B em2 interface is completely blank/empty

[stephenw10]

Then the client PC there is not attached to em2 as it shows in the disagram or it is sending pings some other way, it has the wrong default route.

[mauro.tridici]

@stephenw10

VPCS on the left:

VPCS> show

NAME IP/MASK GATEWAY MAC LPORT RHOST:PORT
VPCS1 192.168.118.10/24 192.168.118.1 00:50:79:66:68:01 20096 127.0.0.1:20097
fe80::250:79ff:fe66:6801/64

VPCs on the right:

VPCS> show

NAME IP/MASK GATEWAY MAC LPORT RHOST:PORT
VPCS1 192.168.120.10/24 192.168.120.1 00:50:79:66:68:00 20098 127.0.0.1:20099
fe80::250:79ff:fe66:6800/64

Both the client PC an reach internet and other machines in the schema

For example, the client PC on the left:

VPCS> ping www.google.it
www.google.it resolved to 142.250.184.35

84 bytes from 142.250.184.35 icmp_seq=1 ttl=126 time=22.655 ms
^C
VPCS> ping 10.99.0.2

84 bytes from 10.99.0.2 icmp_seq=1 ttl=63 time=3.166 ms
^C
VPCS> ping 10.99.0.1

84 bytes from 10.99.0.1 icmp_seq=1 ttl=64 time=1.025 ms
^C
VPCS> ping 192.168.122.180

84 bytes from 192.168.122.180 icmp_seq=1 ttl=63 time=2.851 ms

But if I try to make a ping from one client PC to the other one the pcap is empty.

[stephenw10]

Do you see those successful pings in pcaps?

Those VPCS client machines only have one NIC?

Either they are not sending the pings via pfSense or the pcap is filtering them out.

[mauro.tridici]

@stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

Do you see those successful pings in pcaps?

Yes, I can see them in caps

Those VPCS client machines only have one NIC?

Yes, each VPCS has only one NIC

Either they are not sending the pings via pfSense or the pcap is filtering them out.

I don't think that pcap is filtering the pings.
Maybe something related to IPSEC tunnel because I just tried to ping pfsense B LAN IP (192.168.120.1) directly from pfsense A LAN IP (192.168.118.1) with no success.

During the next minutes, I would like to verify that static route on routerA is working as expected.

[stephenw10]

If you try to ping 192.168.118.1 from the client behind pfSense B you must see those ping packets entering the interface at pfSense B.

What have you named that interface? (em2) Because it isn't the LAN. What firewall rules have you added on it?

[mauro.tridici]

@stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

If you try to ping 192.168.118.1 from the client behind pfSense B you must see those ping packets entering the interface at pfSense B.

Yes, I can confirm that. But the ping packets have not a response

What have you named that interface? (em2) Because it isn't the LAN. What firewall rules have you added on it?

• em2 interface on pfsenseB is named "ACCESS"
• I added a very open rule "from any to any"

I don't know if it can help, but please note that, in the IPSEC configuration page, I didn't set the name of the subnet "ACCESS" but the networks:

This is for the pfsenseB. I did the same for the pfsenseA (changing the order).

[stephenw10]

Ok, that looks good.

Earlier though you said those pings did not appear in a pcap in pfSense on the Access interface. Is that still the case?
It looks like that pcap was done on the client directly.

With that rule if they are arriving it should definitely create at least a state on the Access interface. And since the P2 matches it it should go out over the tunnel creating a state on IPSec too.

Steve

[mauro.tridici]

@stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

Ok, that looks good.
Earlier though you said those pings did not appear in a pcap in pfSense on the Access interface. Is that still the case?

it is no longer happening.

With that rule if they are arriving it should definitely create at least a state on the Access interface. And since the P2 matches it it should go out over the tunnel creating a state on IPSec too.

I just repeated the test pinging 192.168.118.1 from PC with IP 192.168.120.10.
Ping is not working and the states for ACCESS interface are 0.

Attaching the virtual Wireshark to the em0 port of RouterA I see these lines :

168 2022-12-01 11:35:34.467546 192.168.120.10 192.168.118.1 ICMP 146 Echo (ping) request id=0x004b, seq=956/48131, ttl=63 (no response found!)
169 2022-12-01 11:35:34.468024 192.168.122.230 192.168.122.180 ICMP 174 Destination unreachable (Protocol unreachable)

[stephenw10]

Ok, the states exist though and on both ACESS and IPSec. You can see both show 193 packets outbound and no replies.

So check the packet counters in the IPSec status again.

I would not expect to be able to see that traffic on the WAN dircetly, it should be encrypted.

However that 193 packet AH state on WAN looks suspicious. Do you have the Phase 2 protocol set to AH instead of ESP at one end?
It should always be ESP there, AH is unencrypted.

[mauro.tridici]

@stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

Ok, the states exist though and on both ACESS and IPSec. You can see both show 193 packets outbound and no replies.
So check the packet counters in the IPSec status again.
I would not expect to be able to see that traffic on the WAN dircetly, it should be encrypted.

Yes, you are right.

However that 193 packet AH state on WAN looks suspicious. Do you have the Phase 2 protocol set to AH instead of ESP at one end?
It should always be ESP there, AH is unencrypted.

P2 protocol was the same on both endpoints (it was AH, you are right).

Anyway, everything works as expected now. It was a Nat problem on the RouterA.
Many thanks for your help. You are a genius :)

As soon as possible I will share what was happening. right now I'm travelling

[mauro.tridici]

Hello Steve, @stephenw10

a last question for you about this case.
I was able to deploy the IPSEC Lan to Lan successfully (thanks to you).
Now it works using the transport network as you suggested. It is almost done.

Anyway, the throughput is a little bit slow (280Mbits/s but 1Gb/s link is available). I noticed that, during the previous tests between two PFsense 2.6CE instances, the throughput was about 900 Mbits/s. The test instances had each one a public IP.

Now, as in the previous test, AES-NI is enabled on both the endpoints, but the pfsense CE versions are different (2.6 and 2.5.2).
Do you think that this could be the reason of the poor performance?

In one of my previous post, a user said:

" If AES_NI is in the game pfSense since 2.6 CE or Plus version will benefit from that"

Thanks

[stephenw10]

It's probably just a factor of the latency between the sites. I imagine it's significantly higher than in your test setup?

How are you testing the throughput?

Steve

[mauro.tridici]

Good Morning Steve,

here some additional info about the test and real scenario.

TEST environment details:

• n. 2 virtual machines with pfSense 2.6 CE with public IPs on WAN interfaces (let's say, 1.1.1.1/25 and 2.2.2.2/25);
• both the virtual machines have been deployed on two VMware ESXi 6.7 U3 hypervisors (one per site);
• the test has been made in order to test IPSEC feature and connectivity
• both the hypervisors have AES-NI enabled on bios
• both the virtual machines have the NetGate suggested configuration (AES-GCM, and so on).
• the throughput has been checked using iperf2 (iperf2 client running in the LAN at site A, iperf2 server running in the LAN at site B)
• the throughput is very near the 1Gb/s link capacity.

REAL/PRODUCTION environment:

• n. 2 virtual machines, one with pfSense 2.6 CE and the other with 2.5.2 CE;
• only one pfsense (at site B) has a public IPs on WAN interface (let's say, 2.2.2.3/25). The other one is behind the router (please, refer to the schema above).
• both the virtual machines have been deployed on two VMware ESXi 6.7 U3 hypervisors (one per site) - the same hypervisors mentioned above;
• both the hypervisors have AES-NI enabled on bios
• both the virtual machines have the NetGate suggested configuration (AES-GCM, and so on).
• the throughput has been checked using iperf2 (iperf2 client running in the LAN at site A, iperf2 server running in the LAN at site B)
• the throughput is about 280Mbits/s.
• please, note that the iperf test between the router WAN interface (1.1.1.2/25) at site A and the pfsense Wan interface (2.2.2.3/25) at site B returns 890 Mbits/s.

Do you think that the router at site A is doing something bad? Or it is really due to the different versions of pfSense?

Thank you,
Mauro

[stephenw10]

Are those two environments actually between the same sites?

What is the latency across the tunnel in each case? Or just between the WANs outside the tunnel?

What were the iperf test conditions? Single TCP stream?

If you can pass close to 900Mbps outside the tunnel I would expect to see better inside it. That test does not include pfSense at all though at site A? The router at site A could be restricting something.

Is there any reason one of those VMs is still running 2.5.2?

Steve

[mauro.tridici]

@stephenw10 said in pfSense and IPSEC lan to lan: a big doubt about the correct implementation:

Are those two environments actually between the same sites?

Yes, the two environments are between the same sites (site A and site B) and I'm using the same hypervisors on each site.

What is the latency across the tunnel in each case? Or just between the WANs outside the tunnel?

I made a ping between one host in the LAN at site A (192.168.118.13/24) and one host in the LAN at site B (192.168.120.114/24) and viceversa.

ping from 192.168.118.13 and 192.168.120.114

PING 192.168.120.114 (192.168.120.114) 56(84) bytes of data.
64 bytes from 192.168.120.114: icmp_seq=1 ttl=61 time=1.93 ms
64 bytes from 192.168.120.114: icmp_seq=2 ttl=61 time=1.57 ms
64 bytes from 192.168.120.114: icmp_seq=3 ttl=61 time=1.67 ms

ping from 192.168.120.114 and 192.168.118.13

PING 192.168.118.13 (192.168.118.13) 56(84) bytes of data.
64 bytes from 192.168.118.13: icmp_seq=1 ttl=61 time=1.46 ms
64 bytes from 192.168.118.13: icmp_seq=2 ttl=61 time=1.86 ms
64 bytes from 192.168.118.13: icmp_seq=3 ttl=61 time=1.59 ms

Or just between the WANs outside the tunnel?

This is the ping output between the WAN interface of the router at site A and the WAN interface of pFSense at site B:

ping x.x.177.2
PING x.x.177.2 (x.x.177.2) 56(84) bytes of data.
64 bytes from x.x.177.2: icmp_seq=1 ttl=57 time=1.06 ms
64 bytes from x.x.177.2: icmp_seq=2 ttl=57 time=1.13 ms
64 bytes from x.x.177.2: icmp_seq=3 ttl=57 time=1.18 ms
64 bytes from x.x.177.2: icmp_seq=4 ttl=57 time=1.25 ms
64 bytes from x.x.177.2: icmp_seq=5 ttl=57 time=0.947 ms

This is the ping output between the WAN interface of the pfsense behind the router at site A and the WAN interface of pFSense at site B:

[2.6.0-RELEASE][admin@L2L01.home.arpa]/root: ping x.x.177.2
PING x.x.177.2 (x.x.177.2): 56 data bytes
64 bytes from x.x.177.2: icmp_seq=0 ttl=56 time=1.381 ms
64 bytes from x.x.177.2: icmp_seq=1 ttl=56 time=1.488 ms
64 bytes from x.x.177.2: icmp_seq=2 ttl=56 time=1.465 ms
64 bytes from x.x.177.2: icmp_seq=3 ttl=56 time=1.222 ms
64 bytes from x.x.177.2: icmp_seq=4 ttl=56 time=1.143 ms

What were the iperf test conditions? Single TCP stream?

I made an iperf test (between the same hosts mentioned above) using single TCP stream and multiple TCP stream.

This is the result with a single TCP stream:

iperf2 -c 192.168.120.114

Client connecting to 192.168.120.114, TCP port 5001
TCP window size: 85.3 KByte (default)

[ 3] local 192.168.118.13 port 42192 connected with 192.168.120.114 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 286 MBytes 240 Mbits/sec

And these are the results with 4, 8 and 16 parallel processes:

iperf2 -c 192.168.120.114 -P 4

Client connecting to 192.168.120.114, TCP port 5001
TCP window size: 390 KByte (default)

[ 5] local 192.168.118.13 port 43148 connected with 192.168.120.114 port 5001
[ 3] local 192.168.118.13 port 43146 connected with 192.168.120.114 port 5001
[ 4] local 192.168.118.13 port 43150 connected with 192.168.120.114 port 5001
[ 6] local 192.168.118.13 port 43152 connected with 192.168.120.114 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 153 MBytes 128 Mbits/sec
[ 5] 0.0-10.0 sec 152 MBytes 127 Mbits/sec
[ 3] 0.0-10.0 sec 154 MBytes 129 Mbits/sec
[ 6] 0.0-10.0 sec 152 MBytes 128 Mbits/sec
[SUM] 0.0-10.0 sec 611 MBytes 512 Mbits/sec

iperf2 -c 192.168.120.114 -P 8

Client connecting to 192.168.120.114, TCP port 5001
TCP window size: 390 KByte (default)

[ 8] local 192.168.118.13 port 43168 connected with 192.168.120.114 port 5001
[ 6] local 192.168.118.13 port 43166 connected with 192.168.120.114 port 5001
[ 7] local 192.168.118.13 port 43164 connected with 192.168.120.114 port 5001
[ 3] local 192.168.118.13 port 43160 connected with 192.168.120.114 port 5001
[ 5] local 192.168.118.13 port 43162 connected with 192.168.120.114 port 5001
[ 9] local 192.168.118.13 port 43172 connected with 192.168.120.114 port 5001
[ 10] local 192.168.118.13 port 43170 connected with 192.168.120.114 port 5001
[ 4] local 192.168.118.13 port 43158 connected with 192.168.120.114 port 5001
[ ID] Interval Transfer Bandwidth
[ 9] 0.0-10.0 sec 61.1 MBytes 51.3 Mbits/sec
[ 4] 0.0-10.0 sec 61.1 MBytes 51.3 Mbits/sec
[ 6] 0.0-10.0 sec 59.0 MBytes 49.4 Mbits/sec
[ 7] 0.0-10.0 sec 61.0 MBytes 51.1 Mbits/sec
[ 3] 0.0-10.0 sec 61.0 MBytes 51.1 Mbits/sec
[ 5] 0.0-10.0 sec 61.0 MBytes 51.1 Mbits/sec
[ 8] 0.0-10.0 sec 61.0 MBytes 51.1 Mbits/sec
[ 10] 0.0-10.0 sec 60.9 MBytes 51.0 Mbits/sec
[SUM] 0.0-10.0 sec 486 MBytes 407 Mbits/sec

iperf2 -c 192.168.120.114 -P 16

Client connecting to 192.168.120.114, TCP port 5001
TCP window size: 390 KByte (default)

[ 18] local 192.168.118.13 port 43204 connected with 192.168.120.114 port 5001
[ 6] local 192.168.118.13 port 43178 connected with 192.168.120.114 port 5001
[ 4] local 192.168.118.13 port 43174 connected with 192.168.120.114 port 5001
[ 7] local 192.168.118.13 port 43182 connected with 192.168.120.114 port 5001
[ 9] local 192.168.118.13 port 43184 connected with 192.168.120.114 port 5001
[ 3] local 192.168.118.13 port 43180 connected with 192.168.120.114 port 5001
[ 5] local 192.168.118.13 port 43176 connected with 192.168.120.114 port 5001
[ 8] local 192.168.118.13 port 43186 connected with 192.168.120.114 port 5001
[ 10] local 192.168.118.13 port 43188 connected with 192.168.120.114 port 5001
[ 14] local 192.168.118.13 port 43196 connected with 192.168.120.114 port 5001
[ 12] local 192.168.118.13 port 43192 connected with 192.168.120.114 port 5001
[ 15] local 192.168.118.13 port 43198 connected with 192.168.120.114 port 5001
[ 13] local 192.168.118.13 port 43194 connected with 192.168.120.114 port 5001
[ 16] local 192.168.118.13 port 43200 connected with 192.168.120.114 port 5001
[ 11] local 192.168.118.13 port 43190 connected with 192.168.120.114 port 5001
[ 17] local 192.168.118.13 port 43202 connected with 192.168.120.114 port 5001
[ ID] Interval Transfer Bandwidth
[ 7] 0.0-10.0 sec 41.0 MBytes 34.4 Mbits/sec
[ 5] 0.0-10.0 sec 41.0 MBytes 34.4 Mbits/sec
[ 14] 0.0-10.0 sec 41.1 MBytes 34.5 Mbits/sec
[ 12] 0.0-10.0 sec 41.1 MBytes 34.5 Mbits/sec
[ 15] 0.0-10.0 sec 41.1 MBytes 34.5 Mbits/sec
[ 4] 0.0-10.0 sec 41.1 MBytes 34.5 Mbits/sec
[ 3] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
[ 8] 0.0-10.0 sec 41.1 MBytes 34.4 Mbits/sec
[ 13] 0.0-10.0 sec 41.1 MBytes 34.4 Mbits/sec
[ 17] 0.0-10.0 sec 41.1 MBytes 34.4 Mbits/sec
[ 6] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
[ 9] 0.0-10.0 sec 41.1 MBytes 34.4 Mbits/sec
[ 10] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
[ 11] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
[ 18] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
[ 16] 0.0-10.0 sec 41.2 MBytes 34.5 Mbits/sec
[SUM] 0.0-10.0 sec 658 MBytes 550 Mbits/sec

Please, note that the results are not the always the same. Sometimes, It seems to be unstable. The basic value is always about 240/280 Mbits/s.
Sometimes, "write failed: Connection reset by peer" message appears in the iperf2 output when I use parallel streams.
The error message doesn't aappear if I use a single stream, but the bitrate is very low.

iperf2 -c 192.168.120.114 -P 16

Client connecting to 192.168.120.114, TCP port 5001
TCP window size: 390 KByte (default)

[ 16] local 192.168.118.13 port 43260 connected with 192.168.120.114 port 5001
[ 3] local 192.168.118.13 port 43238 connected with 192.168.120.114 port 5001
[ 8] local 192.168.118.13 port 43242 connected with 192.168.120.114 port 5001
[ 4] local 192.168.118.13 port 43234 connected with 192.168.120.114 port 5001
[ 5] local 192.168.118.13 port 43236 connected with 192.168.120.114 port 5001
[ 9] local 192.168.118.13 port 43246 connected with 192.168.120.114 port 5001
[ 7] local 192.168.118.13 port 43240 connected with 192.168.120.114 port 5001
[ 6] local 192.168.118.13 port 43244 connected with 192.168.120.114 port 5001
[ 11] local 192.168.118.13 port 43250 connected with 192.168.120.114 port 5001
[ 13] local 192.168.118.13 port 43254 connected with 192.168.120.114 port 5001
[ 14] local 192.168.118.13 port 43258 connected with 192.168.120.114 port 5001
[ 15] local 192.168.118.13 port 43256 connected with 192.168.120.114 port 5001
[ 12] local 192.168.118.13 port 43252 connected with 192.168.120.114 port 5001
[ 10] local 192.168.118.13 port 43248 connected with 192.168.120.114 port 5001
[ 18] local 192.168.118.13 port 43264 connected with 192.168.120.114 port 5001
[ 17] local 192.168.118.13 port 43262 connected with 192.168.120.114 port 5001
write failed: Connection reset by peer
write failed: Connection reset by peer
[ ID] Interval Transfer Bandwidth
[ 18] 0.0- 0.0 sec 358 KBytes 229 Mbits/sec
[ 17] 0.0- 0.0 sec 358 KBytes 220 Mbits/sec
[ 16] 0.0-10.7 sec 3.62 MBytes 2.84 Mbits/sec
[ 3] 0.0-10.7 sec 3.64 MBytes 2.86 Mbits/sec
[ 4] 0.0-10.7 sec 3.62 MBytes 2.84 Mbits/sec
[ 5] 0.0-10.7 sec 3.44 MBytes 2.70 Mbits/sec
[ 9] 0.0-10.7 sec 3.62 MBytes 2.84 Mbits/sec
[ 7] 0.0-10.7 sec 3.41 MBytes 2.67 Mbits/sec
[ 6] 0.0-10.7 sec 3.59 MBytes 2.82 Mbits/sec
[ 11] 0.0-10.7 sec 3.62 MBytes 2.83 Mbits/sec
[ 13] 0.0-10.7 sec 3.64 MBytes 2.86 Mbits/sec
[ 14] 0.0-10.7 sec 2.52 MBytes 1.97 Mbits/sec
[ 15] 0.0-10.7 sec 3.62 MBytes 2.84 Mbits/sec
[ 10] 0.0-10.7 sec 3.41 MBytes 2.68 Mbits/sec
[ 8] 0.0-10.7 sec 3.62 MBytes 2.83 Mbits/sec
[ 12] 0.0-10.8 sec 3.46 MBytes 2.69 Mbits/sec
[SUM] 0.0-10.8 sec 49.5 MBytes 38.5 Mbits/sec

I also tried to change the net.core.somaxconn in sysctl.conf on both the endpoints (as suggested in the link below), but the behaviour didn't change.

https://spyff.github.io/net/2019/07/14/many-tcp-flow-with-iperf/#the-problem-of-small-backlog

If you can pass close to 900Mbps outside the tunnel I would expect to see better inside it.

Yes, it is exactly what I would expect.

That test does not include pfSense at all though at site A?

Yes, some test I made include pfsense at site A (iperf test through the tunnel)

The router at site A could be restricting something.

I hope that the router is not the cause...I'm your hands :)

Is there any reason one of those VMs is still running 2.5.2?

No, it has been created some months ago and it is in production. If you think that it should updated, we will schedule a downtime to do it.

Steve