IPSec tunnel - No traffic



  • We have about 15 sites with PFsense endpoints. All on 2.4.4. Some are upgrades and one fresh install.

    Traffic doesn't seem to pass thru the tunnels.
    Couple things I noticed.

    • Tunnels preexisting prior to upgrade to 2..4.4 are fine.

    • New tunnels don't pass traffic

    • On reinstall endpoints I can set up tunnels and establish connection but only pass traffic between new installs not upgrades.


  • Netgate Administrator

    So you see the tunnels come up at phase 1 and phase 2 in all cases?

    Do you see the traffic counters in Status > IPSec increasing at either end if you try to ping across it?

    These are all tunnels with pfSense at both ends?

    You should at least check that Async-Crypto is disabled in IPSec > Advanced. That is a new option ins 2.4.4 that in the vast majority of cases speeds up ipsec, sometimes significantly. But we are now seeing some situations where it prevents traffic.

    Steve



  • All tunnel endpoints are pfsense and Async-Crypto is disabled on all of them by default. The tunnels have zero traffic on them.


  • Netgate Administrator

    Hmm,
    Do you see any errors in the IPSec log or System log?
    If you run packet captures on the IPSec interface and try to ping across the tunnel do you see that traffic?
    If not do you see it leaving on the WAN incorrectly?

    Steve



  • I ran capture on both ipsec and wan. Wan at least shows some of the ipsec tunnel traffic between two endpoints but not traffic. Ipsec is just flat empty. I've never seen packet capture being completely empty.
    System logs show no errors for ipsec just a bunch of "activating new task" "nothing to initiate".
    Oh yeah so one of the time running packet capture ping came back with destination unreachable. Could this be NAT or routing related?


  • Netgate Administrator

    If the tunnel is up at phase 1 and phase 2 and the SAs show correctly for subnets you are trying to ping between the IPSec daemon should always grab that traffic.
    If it doesn't it would try to send it to the default gateway which is probably your WAN.
    The IPSec daemon can be by-passed if you have policy routing rules in place, with gateways defined.

    Steve



  • We don't use policy routing rules as of yet - not on smaller appliances.

    WAN shows the echo request but it never makes it back to IPsec.

    FYI. As a temp measure I set up backup tunnels with openvpn that seems the easiest and quest solution. That works but I do prefer IPsec over Openvpn for static tunnels.


  • Netgate Administrator

    Hmm, you see the encrypted packets leaving the WAN but nothing on the IPSec interface?

    Or am I reading that wrong?

    Steve



  • There is no traffic on the IPSec interface but I do see the echo request on the WAN.

    Steve, I have to assume you're the same fellow who helped me out a few times before netgate time. Then netgate took over and my support didn't get carried over to netgate at least that's what it seems. Anyways thank you for the attention now on this matter.


  • Netgate Administrator

    I could be. 😉 Though I have only ever worked for Netgate doing pfSense support (while getting paid!).

    If you're seeing the unencrypted ping request leaving the WAN then the IPSec daemon is not seeing that as interesting traffic.

    Check in Status > IPSec on the SPDs tab that the traffic selectors are there covering the source and destinations you are pinging.

    Steve



  • SPD tabs show both remote and local networks for inbound and outbound.


  • Netgate Administrator

    Hmm, can we see the output of ipsec statusall and the details of the ping that's failing?

    Steve



  • I have a similar problem with pfSense 2.4.4.

    LAN interface stops routing traffic, tunnels stop working after some minutes or sometimes hours

    I have multiple Phase 2 tunnels, if i restart a computer in local network or restart IPsec service multiple times and try to ping tunnel remote IP then tunnels start to get packets but after a while traffic stops again.

    People had the same problem before : https://forum.netgate.com/topic/98893/pfsense-2-3-lan-interface-stops-routing-traffic-stops-working-after-2-or-3-day

    And it was fixed with 2.3.1, i think it was a solution by disabling all but 1 CPU. May it be the same?


  • Netgate Administrator

    If it works at all on upgraded devices then you have a different issue. Please start a new thread to address that.

    Steve



  • Sorry. Still having this issue I'll try to get some ipsec status printed out this week. It's been busy.
    Thank you.


  • Netgate Administrator

    Just to confirm; the traffic counters on the phase 2 status shows 0 at both ends and in both directions?

    Steve



  • You just pointed out to something I never paid attention to.
    In lack of time I looked at one of the setups and Connection is established but phase to status is not available on the connection that's not working.


  • Netgate Administrator

    Ah, phase 1 comes up but not phase 2? Then check for a mismatch there. The IPSec logs should show an error.

    Steve



  • No intention of hijacking the thread.
    I'm using 2.4.4-RELEASE (amd64)
    built on Thu Sep 20 09:33:19 EDT 2018
    FreeBSD 11.2-RELEASE-p3

    I create an ipSec tunnel with identical configuration of others created on a previous rev (2.4.3p1) and the tunnel itself establishes, but no traffic passes through. On the phase 2 items, they're configured in a fashion similar to the other working tunnels. One thing I noticed that was very strange. In Static -> IPsec -> SADs, the whole section on this 2.4.4 instance is EMPTY, but on the others, it's populated with two entries per phase 2 item.


  • Netgate Administrator

    The SADs will only appear once the tunnel is up. The SPDs should be be there whether it is up or not though.

    If you see no SADs it's not establishing. Check the ipsec logs.

    Steve



  • con2000: #3 toOffice pu.bl.ic.ip 10.xx.xx.xx NAT-T re.mo.te.ip re.mo.te.ip IKEv2
    initiator 23725 seconds (06:35:25) AES_CBC
    HMAC_SHA1_96
    PRF_HMAC_SHA1
    MODP_1024 ESTABLISHED
    4026 seconds (01:07:06) ago

    Status shows as "ESTABLISHED" for the main tunnel.

    log snippet:
    Nov 8 18:28:26 charon 09[NET] <con2000|3> received packet: from re.mo.te.ip[4500] to 10.xx.xx.1[4500] (76 bytes)
    Nov 8 18:28:26 charon 09[ENC] <con2000|3> parsed INFORMATIONAL request 252 [ ]
    Nov 8 18:28:26 charon 09[ENC] <con2000|3> generating INFORMATIONAL response 252 [ ]
    Nov 8 18:28:26 charon 09[NET] <con2000|3> sending packet: from 10.xx.xx.1[4500] to re.mo.te.ip[4500] (76 bytes)
    Nov 8 18:28:30 charon 12[CFG] vici client 574 connected
    Nov 8 18:28:30 charon 16[CFG] vici client 574 registered for: list-sa
    Nov 8 18:28:30 charon 16[CFG] vici client 574 requests: list-sas
    Nov 8 18:28:30 charon 16[CFG] vici client 574 disconnected


  • Netgate

    You want to look for log messages detailing bringing up the ESP inner tunnels based on the traffic selectors.



  • @derelict How do I do that? I'm in Status -> System Logs -> System -> General
    what would I filter by?


  • Netgate Administrator

    Look on the IPSec tab there. Errors will likely be evident just by reading it unless you have a few IPSec tunnels in which case they may be lost in the logging from that. Try restarting the problem tunnel and then immediately checking the IPSec log again.

    Look for phase 2 issues such as those shown here:
    https://www.netgate.com/docs/pfsense/vpn/ipsec/ipsec-troubleshooting.html#phase-2-network-mismatch

    Steve



  • Thanks, it was a phase 2 no acceptable ENCRYPTION_ALGORITHM found message
    Mismatched AES128 on one side and AES256 on the other.

    Sorry for the thread-jack, I'll go away now.



  • I'm not seeing phase2 related errors.
    1_1541785064291_2018-11-09 12_34_21-GW1- Status_ System Logs_ IPsec.png 0_1541785064290_2018-11-09 12_28_11-GW1- Status_ IPsec_ Overview.png


  • Netgate

    That is all dead peer detection.

    Your Phase 2 looks like it is coming up.

    Obfuscating the private addresses makes it almost impossible to help you. Nobody cares what your private addresses are.



  • Phase 2 been coming up for weeks now.
    0_1541786318274_2018-11-09 12_56_22-GW1- Status_ IPsec_ Overview.png


  • Netgate Administrator

    Ok, so where are you actually sending traffic from?

    Does pfSense have an IP in 192.168.27.0/24? If so try sending some pings to any address in 192.168.97.0/24 from Diag > Ping using the 192.168.27.0/24 interfaces as the source. Those should definitely appear in the traffic counter as outbound packets even if there are no replies.

    Steve



  • from 192.168.27.1 to 192.168.97.1 No traffic is showing/passing on IPSEC.

    PING 192.168.97.1 (192.168.97.1): 56 data bytes

    --- 192.168.97.1 ping statistics ---
    10 packets transmitted, 0 packets received, 100.0% packet loss


  • Netgate Administrator

    You didn't select a source interface as I said otherwise it would show that. That traffic won't go over the VPN unless the source IP matching the local subnet selector.

    Steve



  • Sorry Steve. I did overlook that request.

    PING 192.168.97.50 (192.168.97.50) from 192.168.27.1: 56 data bytes

    --- 192.168.97.50 ping statistics ---
    10 packets transmitted, 0 packets received, 100.0% packet loss



  • Might be a shot into the dark, but what are your firewall rules for IPsec? Did you add rules for the new tunnels?


  • Netgate Administrator

    Ok so that target may not respond to pings or is blocked etc but did you see 10 packets on the Phase 2 status outbound?

    Traffic from the firewall itself should always be allowed out across the tunnel. Unless you have specific blocking 'OUT' floating rules.

    Steve



  • at this point for testing I have a very open rule set on both ends.
    As for traffic It doesn't even display the phase2 for ipsec tunnels


  • Netgate Administrator

    Your screenshot above shows the Phase 2 as established but the traffic counters show 0 packets in or out. After running that ping it should show 10 packets out.

    Steve



  • I know but it doesn't.


  • Netgate

    Please run the ping test again and post another shot of Status > IPsec with the phase 2 expanded. Thanks.



  • @derelict
    another thread-hijacker here:
    I see issues on a tunnel between my 2.4.4 SG-1000 and a remote SG-3100 on 2.4.3-p1 still.

    tunnel comes up, no traffic goes through .. no ping via shell, nothing seen in Status page.

    disabled that new async-option, checked and upgraded strongswan (on SG-1000), re-saved tunnel configs, restarted IPSEC ... I will disable other tunnels and check back with a screenshot or so.


  • Netgate

    If you know you're hijacking why not just start another thread?