IPSec traffic fails.



  • I am relatively new to pfsense and IPSec and unsure where to start on diagnosing my problem, hope you guys can help me figure it out.

    I am running 4 SG-5100s at 4 locations, 1 HQ location, and 3 remote locations. Each remote has an IPSec tunnel to HQ. All locations have an IPSec tunnel to an offsite Vendor.

    The HQ keeps dropping traffic to 1 remote location and the Vendor even though the Tunnels show to be good. The other 2 IPSec tunnel locations have no problems to HQ or to the Vendor. Restarting the tunnels by service or individually has no effect. Memory, CPU, and Temperature all look good.

    There are no 3rd party apps installed and all locations are running 2.4.4_2.

    Can y'all help me figure out what is going on please?!


  • LAYER 8 Netgate

    Going to need far more details.

    What do you mean by keeps dropping traffic?

    If the tunnels are up and traffic isn't getting to where you think it should be going, packet captures are probably in order. You can pcap on the hq ipsec interface for the traffic that is not behaving how you think it should. Is it being sent into the tunnel? Then pcap the other side. Then pcap the outgoing interface, etc.

    Status > System Logs, IPsec is your friend if there is a problem with the tunnel.



  • Thanks! That's exactly what I needed, a place to get started. I will run pcap and the logs and post my findings..



  • @derelict said in IPSec traffic fails.:

    Going to need far more details.

    What do you mean by keeps dropping traffic?

    If the tunnels are up and traffic isn't getting to where you think it should be going, packet captures are probably in order. You can pcap on the hq ipsec interface for the traffic that is not behaving how you think it should. Is it being sent into the tunnel? Then pcap the other side. Then pcap the outgoing interface, etc.

    Status > System Logs, IPsec is your friend if there is a problem with the tunnel.

    There is nothing in the IPSec logs that note a bad connection...

    Results from Packet Capture:
    First set is on the failed IPSec connection, 2nd set is on a working IPSec connection...

    19:53:11.850647 f4:8e:38:b4:b0:19 > 00:90:0b:7a:83:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 9098, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.84.100 > 10.240.1.5: ICMP echo request, id 1, seq 245, length 40
    19:53:16.644504 f4:8e:38:b4:b0:19 > 00:90:0b:7a:83:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 9136, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.84.100 > 10.240.1.5: ICMP echo request, id 1, seq 246, length 40
    19:53:21.643464 f4:8e:38:b4:b0:19 > 00:90:0b:7a:83:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 9172, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.84.100 > 10.240.1.5: ICMP echo request, id 1, seq 247, length 40
    19:53:26.643540 f4:8e:38:b4:b0:19 > 00:90:0b:7a:83:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 9224, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.84.100 > 10.240.1.5: ICMP echo request, id 1, seq 248, length 40

    19:53:36.401741 f4:8e:38:b4:b0:19 > 00:90:0b:7a:83:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 9275, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.84.100 > 10.251.85.5: ICMP echo request, id 1, seq 249, length 40
    19:53:36.419189 00:90:0b:7a:83:b8 > f4:8e:38:b4:b0:19, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 30774, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.85.5 > 10.251.84.100: ICMP echo reply, id 1, seq 249, length 40
    19:53:37.403521 f4:8e:38:b4:b0:19 > 00:90:0b:7a:83:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 9287, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.84.100 > 10.251.85.5: ICMP echo request, id 1, seq 250, length 40
    19:53:37.420886 00:90:0b:7a:83:b8 > f4:8e:38:b4:b0:19, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 30859, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.85.5 > 10.251.84.100: ICMP echo reply, id 1, seq 250, length 40
    19:53:38.404605 f4:8e:38:b4:b0:19 > 00:90:0b:7a:83:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 9297, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.84.100 > 10.251.85.5: ICMP echo request, id 1, seq 251, length 40
    19:53:38.422040 00:90:0b:7a:83:b8 > f4:8e:38:b4:b0:19, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 30888, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.85.5 > 10.251.84.100: ICMP echo reply, id 1, seq 251, length 40
    19:53:39.406527 f4:8e:38:b4:b0:19 > 00:90:0b:7a:83:b8, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 9317, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.84.100 > 10.251.85.5: ICMP echo request, id 1, seq 252, length 40
    19:53:39.423884 00:90:0b:7a:83:b8 > f4:8e:38:b4:b0:19, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 62, id 30934, offset 0, flags [none], proto ICMP (1), length 60)
    10.251.85.5 > 10.251.84.100: ICMP echo reply, id 1, seq 252, length 40

    Looking at the stats of the tunnel:
    Bytes-In: 0 (0 B)
    Packets-In: 0
    Bytes-Out: 14,736 (14 KiB)
    Packets-Out: 114


  • LAYER 8 Netgate

    Move the capture point hop-by-hop closer to the destination 10.240.1.5. Where does the traffic break down?



  • It breaks down at the tunnel's edge I am guessing. Most all of the Vendors systems refuse ICMP except for the test connection that I monitor.


  • LAYER 8 Netgate

    I wouldn't be guessing. I would be taking captures and knowing.

    If they block ICMP then you have to test using something that isn't blocked.



  • If I knew what else to test, I wouldn't be guessing. I am at a loss here. I am new to IPSec and unsure what other resources on pfsense can be used to help diagnose the problem.

    This is the PCAP from ICMP and the Vendor software on the IPSec Interface...

    21:39:59.453049 (authentic,confidential): SPI 0x4060e670: (tos 0x0, ttl 127, id 24409, offset 0, flags [none], proto ICMP (1), length 60, bad cksum 7014 (->7114)!)
    10.251.84.100 > 10.240.1.5: ICMP echo request, id 1, seq 296, length 40
    21:40:04.155649 (authentic,confidential): SPI 0x4060e670: (tos 0x0, ttl 127, id 24447, offset 0, flags [none], proto ICMP (1), length 60, bad cksum 6fee (->70ee)!)
    10.251.84.100 > 10.240.1.5: ICMP echo request, id 1, seq 297, length 40
    21:40:09.155699 (authentic,confidential): SPI 0x4060e670: (tos 0x0, ttl 127, id 24459, offset 0, flags [none], proto ICMP (1), length 60, bad cksum 6fe2 (->70e2)!)
    10.251.84.100 > 10.240.1.5: ICMP echo request, id 1, seq 298, length 40
    21:40:14.155717 (authentic,confidential): SPI 0x4060e670: (tos 0x0, ttl 127, id 24473, offset 0, flags [none], proto ICMP (1), length 60, bad cksum 6fd4 (->70d4)!)
    10.251.84.100 > 10.240.1.5: ICMP echo request, id 1, seq 299, length 40

    21:42:29.179633 (authentic,confidential): SPI 0x4060e670: (tos 0x0, ttl 127, id 28989, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 1d59 (->1e59)!)
    10.251.84.100.10877 > 10.240.1.223.4444: Flags [S], cksum 0x5073 (correct), seq 611660321, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    21:42:32.180712 (authentic,confidential): SPI 0x4060e670: (tos 0x0, ttl 127, id 29101, offset 0, flags [DF], proto TCP (6), length 52, bad cksum 1ce9 (->1de9)!)
    10.251.84.100.10877 > 10.240.1.223.4444: Flags [S], cksum 0x5073 (correct), seq 611660321, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
    21:42:38.181699 (authentic,confidential): SPI 0x4060e670: (tos 0x0, ttl 127, id 29143, offset 0, flags [DF], proto TCP (6), length 48, bad cksum 1cc3 (->1dc3)!)
    10.251.84.100.10877 > 10.240.1.223.4444: Flags [S], cksum 0x6482 (correct), seq 611660321, win 8192, options [mss 1460,nop,nop,sackOK], length 0


  • LAYER 8 Netgate

    OK so you are sending the echo requests and TCP SYN packets over the tunnel and there is no response. Need to capture on the other side and see if they are arriving and what is happening to the traffic over there.

    That's pretty much all you can do from that side.

    Hop by hop closer to the destination.



  • I can't track the other side, that is the Vendor. I don't have control or access to that.

    I can track the connection the company location that fails, but it is up at the moment. No problems right now.