IPSec tunnel problems after pfSense 2.2.3 upgrade



  • Hi,

    I have upgraded my pfSense full-installation on my SG-2440 unit from version 2.2.2 to version 2.2.3.
    The upgrade was successful, but now my two IPSec LAN-to-LAN tunnels are not working anymore.

    When I log with "tcpdump -i enc0" I can see the traffic going into the tunnel, but no traffic is returning anymore.

    The System/IPSec logs are without errors or warnings and the tunnels are up and running.
    Both sides do not show IPSec errors, so it looks like the firewall is blocking the incomming traffic thru the tunnel.
    I have tried allow-any-any rules in the IPSec rule segment but that doesn't solve anything either.

    I have also deleted the tunnels and recreated them, but without any success.

    Does anyone have a suggestion how to solve this?

    pfinfo shows:

    enc0
    Cleared:    Thu Jun 25 12:07:39 2015
    References:  6               
    In4/Pass:    [ Packets: 0                  Bytes: 0                  ]
    In4/Block:  [ Packets: 0                  Bytes: 0                  ]
    Out4/Pass:  [ Packets: 833                Bytes: 69592              ]
    Out4/Block:  [ Packets: 0                  Bytes: 0                  ]
    In6/Pass:    [ Packets: 0                  Bytes: 0                  ]
    In6/Block:  [ Packets: 0                  Bytes: 0                  ]
    Out6/Pass:  [ Packets: 0                  Bytes: 0                  ]
    Out6/Block:  [ Packets: 0                  Bytes: 0                  ]

    ipsec.conf:

    config setup
    uniqueids = yes
    charondebug=""

    conn con2000
    fragmentation = yes
    keyexchange = ikev1
    reauth = yes
    forceencaps = no
    mobike = no
    rekey = yes
    installpolicy = yes
    type = tunnel
    dpdaction = restart
    dpddelay = 30s
    dpdtimeout = 180s
    auto = route
    left = XXX.XXX.XXX.XXX
    right = XXX.XXX.XXX.XXX
    leftid = XXX.XXX.XXX.XXX
    ikelifetime = 21600s
    lifetime = 3600s
    ike = aes128-sha1-modp1024!
    esp = aes256-sha1-modp1024,aes192-sha1-modp1024,aes128-sha1-modp1024!
    leftauth = psk
    rightauth = psk
    rightid = XXX.XXX.XXX.XXX
    aggressive = no
    rightsubnet = 192.168.0.0/24
    leftsubnet = 192.168.1.0/24

    conn con1000
    fragmentation = yes
    keyexchange = ikev1
    reauth = yes
    forceencaps = no
    mobike = no
    rekey = yes
    installpolicy = yes
    type = tunnel
    dpdaction = clear
    dpddelay = 10s
    dpdtimeout = 60s
    auto = add
    left = XXX.XXX.XXX.XXX
    right = XXX.XXX.XXX.XXX
    leftid = XXX.XXX.XXX.XXX
    ikelifetime = 21600s
    lifetime = 3600s
    ike = aes128-sha1-modp1024!
    esp = aes256-sha1-modp1024,aes192-sha1-modp1024,aes128-sha1-modp1024,aes256-sha1-modp1024,aes192-sha1-modp1024,aes128-sha1-modp1024!
    leftauth = psk
    rightauth = psk
    rightid = XXX.XXX.XXX.XXX
    aggressive = no
    rightsubnet = 192.168.101.0/24
    leftsubnet = 192.168.1.0/24

    conn con1001
    fragmentation = yes
    keyexchange = ikev1
    reauth = yes
    forceencaps = no
    mobike = no
    rekey = yes
    installpolicy = yes
    type = tunnel
    dpdaction = clear
    dpddelay = 10s
    dpdtimeout = 60s
    auto = add
    left = XXX.XXX.XXX.XXX
    right = XXX.XXX.XXX.XXX
    leftid = XXX.XXX.XXX.XXX
    ikelifetime = 21600s
    lifetime = 3600s
    ike = aes128-sha1-modp1024!
    esp = aes256-sha1-modp1024,aes192-sha1-modp1024,aes128-sha1-modp1024,aes256-sha1-modp1024,aes192-sha1-modp1024,aes128-sha1-modp1024!
    leftauth = psk
    rightauth = psk
    rightid = XXX.XXX.XXX.XXX
    aggressive = no
    rightsubnet = 192.168.101.0/24
    leftsubnet = 192.168.150.0/24



  • I was using certs on 2.2.2 but the tunnels wont come up on 2.2.3 .
    I will try PSK instead to see if I get the same problem as you before reinstalling 2.2.2 .



  • Are you seeing any errors in the IPSEC log?
    EDIT I should have asked if you have debugging enabled and if you are seeing errors in the logs with the debugging on. /EDIT

    IPSEC can be a temperamental beast.  What does the far end look like?  Does it still have security associations stuck from when the IPSEC tunnel was up prior to the upgrade?  Can you try to send traffic from the far end to a specific IP on the side you upgraded?  Often times IPSEC needs 2-way traffic to properly reinitialize the tunnels, specifically when one side is reset and the other side is not.



  • I don't see any IPSec errors logged, not on the pfSense side and not on the two other sides (Stormshield and Draytek).
    I control all these products and have restarted everything, without any luck.
    The tunnels are connected without problems, phase 1 and phase 2, but no traffic is passing thru the tunnel.
    With pfSense 2.2.2 this was working fine.
    Debug logging is not enabled yet….



  • You posted stats that show traffic is being sent out of the enc0 interface, but not being received back in.  Can your other firewalls produce similar statistics?

    If you are sending traffic out over the enc0 interface, the other sides should be receiving traffic.  It would be interesting to see if those IPSEC tunnels are sending traffic.  Since your PFSense doesn't show any blocked or passed traffic IN, then at least it would appear it isn't receiving any traffic at all.



  • I have checked with ssh on both sides and can see the trafic going into the tunnel on the pfSense but not coming out on the Stormshield and vice versa. So it seems to get lost in a big black hole :-)

    It seems that I am not alone;
    https://forum.pfsense.org/index.php?topic=95659.0

    There are a lot of IPSec related problems mentioned on this forum after upgrading to v2.2.3.



  • Yeah, seems you might be getting smacked with a bug since others are experiencing the same thing.

    You stated you rebuilt your tunnels… did you try rebuilding the rules?  Maybe the rules somehow became dis-associated with the IPSEC tunnel.



  • Hi,

    Having the same issue too. VPN up (to Amazon), but no traffic flowing.

    -=david=-



  • I have created some new allow-any-any rules above all existing rules to check if it was rule related, but that didn't help. So I think it is not rule related….
    OpenVPN is still running fine, it is purely IPSec related.



  • Hmm, gremlins perhaps…

    I have 2 PFSense units I just upgraded today to 2.2.3 (i386).  I have a tunnel between them.  I did have to "manually" start the tunnel, but these are units I use specifically for testing, so they don't really send traffic over the tunnel unless I create it.

    Anyway, I just manually started the tunnel and it came up and I am able to ping across the tunnel.  I am using fairly standard settings and seems pretty close to your build.

    
     config setup
            uniqueids = yes
            charondebug=""
    
     conn bypasslan
            leftsubnet = 192.168.1.11/32
            rightsubnet = 192.168.1.0/24
            authby = never
            type = passthrough
            auto = route
    
     conn con1000
            fragmentation = yes
            keyexchange = ikev1
            reauth = yes
            forceencaps = no
            mobike = no
            rekey = yes
            installpolicy = yes
            type = tunnel
            dpdaction = none
            auto = route
            left = x.x.x.x
            right = y.y.y.y
            leftid = x.x.x.x
            ikelifetime = 28800s
            lifetime = 28800s
            ike = aes256-sha256-modp1024!
            esp = aes256-sha256-modp1024!
            leftauth = psk
            rightauth = psk
            rightid = y.y.y.y
            aggressive = no
            rightsubnet = 192.168.230.0/24
            leftsubnet = 172.16.100.1|192.168.1.0/24
    
    

    Granted, this is between 2 PFS hosts.  I can't speak to how IPSEC is working to other firewalls under 2.2.3.



  • Silly question, but do you see any "blocked" packets the WAN of your firewall for packets coming from the far end firewalls (and vice versa)?  I don't mean IPSEC in particular, I am just curious since the enc0 interface isn't reporting any received packets, are any of those packets actually reaching the firewall and getting blocked.



  • Just want to chime in and say I am also experiencing this issue after a successful upgrade to 2.2.3 - IPSec working fine before the upgrade.

    Chris



  • Another "me too".  I've got a pair so I updated only the secondary.  The primary, still on 2.2.2, works fine for all tunnels.  The secondary on 2.2.3 has dead tunnels.  ESP traffic is going both directions, DPD packets exchanged, but nothing makes it to the remote LAN at either end.

    The remote endpoints are a Cisco (no access for me) and an OpenBSD host (full access).  I have taken a lot of notes, verbose outputs of "ipsec statusall" and "ipsecctl -sa -v" from each end, and traffic captures to try to compare the difference between the working 2.2.2.<->OpenBSD and the broken 2.2.3<->OpenBSD, but nothing jumps out to my relatively untrained eye.

    The Cisco tunnel is also dead when on 2.2.3, I just can't troubleshoot from the Cisco end.  It works great on 2.2.2 though.

    It's not well organized or anything but if it helps someone else help us all figure out what's broken, it's ZeroBin'ed at: http://sebsauvage.net/paste/?1659557931393c28#Rt+LFQvj5kP8L0Bhv5D/8N7ZNypDZ9Wz7y9fMPXnuZU=



  • Glad I am not the only one with this exactly same problem.  We use WatchGuard at the office and been working great right up to PfSense 2.2.2.  Soon as I upgraded it to 2.2.3 IPSec stopped working.  Far as I can tell Phase 1 and Phase 2 are working.  Just it's not passing traffic over the tunnel like it should.  On my PfSense I do have the pings to hit a sever at work to keep the tunnel alive.  That is not even working.

    Then I thought it's a firewall rules / routing issue since I actually don't have IPSec rules setup and didn't need to since I only want to access the servers from my house and not the other way around.  I created the pass rules anyway.  No luck.

    I don't have any packages installed that would account for this.  I may have to roll back to 2.2.2 until it's been fixed.  Bummer.  Love the added features in IPSec but at this point can't use it.



    • 1
      After upgrading from PFSense 2.2.2 to 2.2.3 (X64), my IPsec VPNs between PFSense and 2 Cisco RV042 (version 2 hardware) are now down.
      The IPsec status show up as:

    ADI_VPN  xxx.isp.com  188.27.x.x
    Port: 500  Any identifier  5.12.y.y
    Port: 500  IKEv1
    initiator  3DES_CBC:0
    HMAC_SHA1_96:0
    PRF_HMAC_SHA1
    MODP_768

    connecting

    Connect

    ADI_VPN  xxx.isp.com 188.27.x.x
    Port: 500  5.12.y.y  5.12.y.y
    Port: 500  IKEv1
    responder  47 minutes  3DES_CBC:0
    HMAC_SHA1_96:0
    PRF_HMAC_SHA1
    MODP_768

    established
    5 seconds ago

    Local LAN class: 172.77.17.0/24
    Remote LAN class: 172.24.96.0/24

    Thanks for any suggestion.
    I will be coming later with screenshots if required.



  • Same here with local ipsec tunnel between vmware horizon view connection and security server. pfsense somehow blocks ipsec traffic although the udb ports 500/4500 are open in nat.

    until this is fixed, i will use 2.2.2.

    :(


  • Netgate Administrator

    It appears as though this maybe an error decrypting traffic from the remote host. Though we have not replicated it here yet, we use IPSec extensively and have been running 2.2.3 for some time internally.
    Has anyone tried an encryption type other than AES?
    Where exactly is this traffic being lost and are you all seeing the same thing?
    Run a ping from behind pfSense 2.2.3 to to some host on the other side of the tunnel.
    Run a packet capture on the ipsec interface in pfSense. I expect to see only the leaving echo requests there if the ping is failing.
    If you can, run a packet capture at the other end. Do you see the ping requests and replies?
    Run a packet capture on the pfSense WAN interface. Do you see the ESP (or UDP 4500 if you're using NAT-T) traffic leaving and returning from the remote tunnel end IP?

    From the evidence we have gathered so far into this it seems as though the encrypted traffic is returning but not being decrypted correctly so never leaving the ipsec interface internally.
    If any of you can confirm or refute that it would be very helpful.

    Steve



  • I am using 3DES with Broadcom hardware crypto cards. On 2.2.2 everything is fine.
    Updating to 2.2.3 the phase 1 cannot be established as mutual RSA certificate authentication fails. After restoring 2.2.2, all of my tunnels authenticate again.



  • I just looked on the pfSense bug tracker and there are no issues yet for 2.2.3
    https://redmine.pfsense.org/projects/pfsense/issues?query_id=49
    Does this one qualify?


  • Netgate Administrator

    It definitely will when/if we have it narrowed down to something. Or eliminated it in some other way.

    Steve



  • I've just switched across to 3DES from AES and the issue is still present :/

    Shrewsoft VPN client throws a 'Gateway authentication error'

    Chris



  • If any of you can confirm or refute that it would be very helpful.

    Hi Steve,

    Did you take a look at the pastebin I posted?  I show captures both directions while on 2.2.3.  When pinging from the pfSense side, you do see encrypted pings leave pfSense and arrive at the far end (OpenBSD).  The OpenBSD end shows the encrypted packets arriving on the WAN interface, but never anything on enc0.  When you try the same test, but pinging from the far OpenBSD side back to pfSense, the symptoms are exactly the same.  You see encrypted pings leave OpenBSD's WAN, arrive at pfSense's WAN, but never anything at all on pfSense's enc0.

    On 2.2.2 the same setup works, the only thing changing is 2.2.2->2.2.3 on the pfSense side.  It's almost like on both ends they refuse to decrypt the packet from the other side?



  • I notice from a WAN-side capture of working pings on 2.2.2 vs non-working on 2.2.3 that the packets from the pfSense side are 4 bytes larger than expected?

    2.2.2 working, 132 bytes both directions:
    –---------------
    1.001710 esp 2.2.2.80 > 1.1.1.251 spi 0xad9827e8 seq 2 len 132
    0.002942 esp 1.1.1.251 > 2.2.2.80 spi 0xc5197ced seq 2 len 132
    0.998066 esp 2.2.2.80 > 1.1.1.251 spi 0xad9827e8 seq 3 len 132
    0.002869 esp 1.1.1.251 > 2.2.2.80 spi 0xc5197ced seq 3 len 132

    2.2.3 broken, 136 bytes when initiated from pfSense side, still 132 bytes trying from OpenBSD side

    1.003313 esp 2.2.3.80 > 1.1.1.251 spi 0xd258f6d0 seq 8 len 136
    1.001410 esp 2.2.3.80 > 1.1.1.251 spi 0xd258f6d0 seq 9 len 136
    1.002726 esp 2.2.3.80 > 1.1.1.251 spi 0xd258f6d0 seq 10 len 136
        ... obviously no replies from OpenBSD end, but I can initiate a ping from that LAN:
    0.066538 esp 1.1.1.251 > 2.2.3.80 spi 0xc2be6e0e seq 2 len 132 (DF)
    0.999772 esp 1.1.1.251 > 2.2.3.80 spi 0xc2be6e0e seq 3 len 132 (DF)
    1.000032 esp 1.1.1.251 > 2.2.3.80 spi 0xc2be6e0e seq 4 len 132 (DF)

    Is it a red herring, or unexpected for a known, fixed-length ping to be 4 bytes larger coming from 2.2.3?  Maybe this results in decryption failures, hence the lack of any traffic ever on enc0 from the respective remote LANs?


  • Netgate Administrator

    @yuljk What version of pfSense did you come from?
    Can you get any more info from the screwsoft debug console?

    @frmpf Checking that now. Thanks.

    Steve



  • seems like an MTU issue somehow ? packets get fragmentented ….

    could someone try: https://doc.pfsense.org/index.php/IPsec_Troubleshooting#Packet_Loss_with_Certain_Protocols



  • I reverted one of my 2.2.3 NanoBSD firewalls back to 2.2.2.  This gives me a setup similar to what some have described here… having one firewall on 2.2.3 and one on 2.2.2.  And my IPSEC... still works.  :-\

    I've hit the weird issues before so I can appreciate what you guys are dealing with... but I can't seem to replicate the issue.  If I have time today, I might try to fire up a tunnel on an ASA to a PFS 2.2.2 and 2.2.3 and see what results I get.



  • @heper:  Packets are making the trip successfully from end to end though, they just disappear after they hit the WAN-side interface of either endpoint.  And the pings I'm testing with only result in 132 byte packets, too small to need fragmentation (although they're 136 bytes from a 2.2.3 endpoint, so maybe something is overloading a header field or…?  No explanation for that one yet).

    EDIT:  OK, it's the Differentiated Services Field that grows by 4 when you're on 2.2.3 instead of 2.2.2, that's why the packet length goes from 132 -> 136



  • @stephenw10 - Here is an IKE debug from Shrewsoft VPN client

    (IP's removed for security)

    I upgraded from 2.2.2 to 2.2.3 on an ALIX 2C2.

    15/06/26 16:20:45 ## : IKE Daemon, ver 2.2.2
    15/06/26 16:20:45 ## : Copyright 2013 Shrew Soft Inc.
    15/06/26 16:20:45 ## : This product linked OpenSSL 1.0.1c 10 May 2012
    15/06/26 16:20:45 ii : opened 'C:\Program Files\ShrewSoft\VPN Client\debug\iked.log'
    15/06/26 16:20:45 ii : opened 'C:\Program Files\ShrewSoft\VPN Client/debug/dump-ike-decrypt.cap'
    15/06/26 16:20:45 ii : opened 'C:\Program Files\ShrewSoft\VPN Client/debug/dump-ike-encrypt.cap'
    15/06/26 16:20:45 ii : rebuilding vnet device list …
    15/06/26 16:20:45 ii : device ROOT\VNET\0000 disabled
    15/06/26 16:20:45 ii : network process thread begin ...
    15/06/26 16:20:45 ii : pfkey process thread begin ...
    15/06/26 16:20:45 ii : ipc server process thread begin ...
    15/06/26 16:22:22 ii : ipc client process thread begin ...
    15/06/26 16:22:22 <a :="" peer="" config="" add="" message<br="">15/06/26 16:22:22</a> <a :="" proposal="" config="" message<br="">15/06/26 16:22:22</a> <a :="" proposal="" config="" message<br="">15/06/26 16:22:22</a> <a :="" client="" config="" message<br="">15/06/26 16:22:22</a> <a :="" xauth="" username="" message<br="">15/06/26 16:22:22</a> <a :="" xauth="" password="" message<br="">15/06/26 16:22:22</a> <a :="" local="" id="" 'blah@blah.com'="" message<br="">15/06/26 16:22:22</a> <a :="" preshared="" key="" message<br="">15/06/26 16:22:22</a> <a :="" peer="" tunnel="" enable="" message<br="">15/06/26 16:22:22 DB : peer ref increment ( ref count = 1, obj count = 0 )
    15/06/26 16:22:22 DB : peer added ( obj count = 1 )
    15/06/26 16:22:22 ii : local address blah blah selected for peer
    15/06/26 16:22:22 DB : peer ref increment ( ref count = 2, obj count = 1 )
    15/06/26 16:22:22 DB : tunnel ref increment ( ref count = 1, obj count = 0 )
    15/06/26 16:22:22 DB : tunnel added ( obj count = 1 )
    15/06/26 16:22:22 DB : tunnel ref increment ( ref count = 2, obj count = 1 )
    15/06/26 16:22:22 DB : new phase1 ( ISAKMP initiator )
    15/06/26 16:22:22 DB : exchange type is aggressive
    15/06/26 16:22:22 DB : blah blah:500 <-> blah blah:500
    15/06/26 16:22:22 DB : 36e8a65f4ceb8136:0000000000000000
    15/06/26 16:22:22 DB : phase1 ref increment ( ref count = 1, obj count = 0 )
    15/06/26 16:22:22 DB : phase1 added ( obj count = 1 )
    15/06/26 16:22:22 >> : security association payload
    15/06/26 16:22:22 >> : - proposal #1 payload
    15/06/26 16:22:22 >> : -- transform #1 payload
    15/06/26 16:22:22 >> : -- transform #2 payload
    15/06/26 16:22:22 >> : -- transform #3 payload
    15/06/26 16:22:22 >> : -- transform #4 payload
    15/06/26 16:22:22 >> : -- transform #5 payload
    15/06/26 16:22:22 >> : -- transform #6 payload
    15/06/26 16:22:22 >> : key exchange payload
    15/06/26 16:22:22 >> : nonce payload
    15/06/26 16:22:22 >> : identification payload
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local supports XAUTH
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local supports nat-t ( draft v00 )
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local supports nat-t ( draft v01 )
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local supports nat-t ( draft v02 )
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local supports nat-t ( draft v03 )
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local supports nat-t ( rfc )
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local supports FRAGMENTATION
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local supports DPDv1
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local is SHREW SOFT compatible
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local is NETSCREEN compatible
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local is SIDEWINDER compatible
    15/06/26 16:22:22 >> : vendor id payload
    15/06/26 16:22:22 ii : local is CISCO UNITY compatible
    15/06/26 16:22:22 >= : cookies 36e8a65f4ceb8136:0000000000000000
    15/06/26 16:22:22 >= : message 00000000
    15/06/26 16:22:22 -> : send IKE packet blah blah:500 -> blah blah:500 ( 774 bytes )
    15/06/26 16:22:22 DB : phase1 resend event scheduled ( ref count = 2 )
    15/06/26 16:22:22 DB : phase1 ref decrement ( ref count = 1, obj count = 1 )
    15/06/26 16:22:22 <- : recv IKE packet blah blah:500 -> blah blah:500 ( 436 bytes )
    15/06/26 16:22:22 DB : phase1 found
    15/06/26 16:22:22 DB : phase1 ref increment ( ref count = 2, obj count = 1 )
    15/06/26 16:22:22 ii : processing phase1 packet ( 436 bytes )
    15/06/26 16:22:22 =< : cookies 36e8a65f4ceb8136:76baa49b68ee1f4d
    15/06/26 16:22:22 =< : message 00000000
    15/06/26 16:22:22 << : security association payload
    15/06/26 16:22:22 << : - propsal #1 payload
    15/06/26 16:22:22 << : -- transform #1 payload
    15/06/26 16:22:22 ii : unmatched isakmp proposal/transform
    15/06/26 16:22:22 ii : hash type ( hmac-sha1 != hmac-md5 )
    15/06/26 16:22:22 !! : peer violates RFC, transform number mismatch ( 1 != 2 )
    15/06/26 16:22:22 ii : matched isakmp proposal #1 transform #1
    15/06/26 16:22:22 ii : - transform    = ike
    15/06/26 16:22:22 ii : - cipher type  = aes
    15/06/26 16:22:22 ii : - key length  = 256 bits
    15/06/26 16:22:22 ii : - hash type    = sha1
    15/06/26 16:22:22 ii : - dh group    = group2 ( modp-1024 )
    15/06/26 16:22:22 ii : - auth type    = xauth-initiator-psk
    15/06/26 16:22:22 ii : - life seconds = 86400
    15/06/26 16:22:22 ii : - life kbytes  = 0
    15/06/26 16:22:22 << : key exchange payload
    15/06/26 16:22:22 << : nonce payload
    15/06/26 16:22:22 << : identification payload
    15/06/26 16:22:22 ii : phase1 id match ( natt prevents ip match )
    15/06/26 16:22:22 ii : received = ipv4-host blah blah
    15/06/26 16:22:22 << : nat discovery payload
    15/06/26 16:22:22 << : nat discovery payload
    15/06/26 16:22:22 << : hash payload
    15/06/26 16:22:22 << : vendor id payload
    15/06/26 16:22:22 ii : peer supports XAUTH
    15/06/26 16:22:22 << : vendor id payload
    15/06/26 16:22:22 ii : peer supports DPDv1
    15/06/26 16:22:22 << : vendor id payload
    15/06/26 16:22:22 ii : peer is CISCO UNITY compatible
    15/06/26 16:22:22 << : vendor id payload
    15/06/26 16:22:22 ii : peer supports FRAGMENTATION
    15/06/26 16:22:22 << : vendor id payload
    15/06/26 16:22:22 ii : peer supports nat-t ( rfc )
    15/06/26 16:22:22 ii : nat discovery - remote address is translated
    15/06/26 16:22:22 ii : switching to src nat-t udp port 4500
    15/06/26 16:22:22 ii : switching to dst nat-t udp port 4500
    15/06/26 16:22:22 == : DH shared secret ( 128 bytes )
    15/06/26 16:22:22 == : SETKEYID ( 20 bytes )
    15/06/26 16:22:22 == : SETKEYID_d ( 20 bytes )
    15/06/26 16:22:22 == : SETKEYID_a ( 20 bytes )
    15/06/26 16:22:22 == : SETKEYID_e ( 20 bytes )
    15/06/26 16:22:22 == : cipher key ( 32 bytes )
    15/06/26 16:22:22 == : cipher iv ( 16 bytes )
    15/06/26 16:22:22 == : phase1 hash_i ( computed ) ( 20 bytes )
    15/06/26 16:22:22 >> : hash payload
    15/06/26 16:22:22 >> : nat discovery payload
    15/06/26 16:22:22 >> : nat discovery payload
    15/06/26 16:22:22 >= : cookies 36e8a65f4ceb8136:76baa49b68ee1f4d
    15/06/26 16:22:22 >= : message 00000000
    15/06/26 16:22:22 >= : encrypt iv ( 16 bytes )
    15/06/26 16:22:22 == : encrypt packet ( 100 bytes )
    15/06/26 16:22:22 == : stored iv ( 16 bytes )
    15/06/26 16:22:22 DB : phase1 resend event canceled ( ref count = 1 )
    15/06/26 16:22:22 -> : send NAT-T:IKE packet blah blah:4500 -> blah blah:4500 ( 140 bytes )
    15/06/26 16:22:22 == : phase1 hash_r ( computed ) ( 20 bytes )
    15/06/26 16:22:22 == : phase1 hash_r ( received ) ( 20 bytes )
    15/06/26 16:22:22 !! : phase1 sa rejected, invalid auth data
    15/06/26 16:22:22 !! : blah blah:4500 <-> blah blah:4500
    15/06/26 16:22:22 !! : 36e8a65f4ceb8136:76baa49b68ee1f4d
    15/06/26 16:22:22 ii : sending peer DELETE message
    15/06/26 16:22:22 ii : - blah blah:4500 -> blah blah:4500
    15/06/26 16:22:22 ii : - isakmp spi = 36e8a65f4ceb8136:76baa49b68ee1f4d
    15/06/26 16:22:22 ii : - data size 0
    15/06/26 16:22:22 >> : hash payload
    15/06/26 16:22:22 >> : delete payload
    15/06/26 16:22:22 == : new informational hash ( 20 bytes )
    15/06/26 16:22:22 == : new informational iv ( 16 bytes )
    15/06/26 16:22:22 >= : cookies 36e8a65f4ceb8136:76baa49b68ee1f4d
    15/06/26 16:22:22 >= : message adcf7119
    15/06/26 16:22:22 >= : encrypt iv ( 16 bytes )
    15/06/26 16:22:22 == : encrypt packet ( 80 bytes )
    15/06/26 16:22:22 == : stored iv ( 16 bytes )
    15/06/26 16:22:22 -> : send NAT-T:IKE packet blah blah:4500 -> blah blah:4500 ( 124 bytes )
    15/06/26 16:22:22 ii : phase1 removal before expire time
    15/06/26 16:22:22 DB : phase1 deleted ( obj count = 0 )
    15/06/26 16:22:22 DB : tunnel ref decrement ( ref count = 1, obj count = 1 )
    15/06/26 16:22:22 DB : policy not found
    15/06/26 16:22:22 DB : policy not found
    15/06/26 16:22:22 DB : policy not found
    15/06/26 16:22:22 DB : policy not found
    15/06/26 16:22:22 <- : recv IKE packet blah blah:500 -> blah blah:500 ( 76 bytes )
    15/06/26 16:22:22 DB : phase1 not found
    15/06/26 16:22:22 ww : ike packet from blah blah ignored, unknown phase1 sa for peer
    15/06/26 16:22:22 ww : 36e8a65f4ceb8136:76baa49b68ee1f4d
    15/06/26 16:22:22 DB : removing tunnel config references
    15/06/26 16:22:22 DB : removing tunnel phase2 references
    15/06/26 16:22:22 DB : removing tunnel phase1 references
    15/06/26 16:22:22 DB : tunnel deleted ( obj count = 0 )
    15/06/26 16:22:22 DB : peer ref decrement ( ref count = 1, obj count = 1 )
    15/06/26 16:22:22 DB : removing all peer tunnel references
    15/06/26 16:22:22 DB : peer deleted ( obj count = 0 )
    15/06/26 16:22:22 ii : ipc client process thread exit ...

    Cheers</a>



  • We had same issue, and had to revert to 2.2.2.  All IPSEC tunnels not routed.  On boot we saw that it was REALLY slow reinstalling all packages, and it also displayed in the log, IPSEC con2000  unrouted . or something to that effect.  Seems like all connections P1 and p2 go through, but nothing is routed….


  • Netgate Administrator

    Ok, this looks like a problem with AES-CBC crypto at Phase2.

    Try changing your Phase2 to anything other than AES-CBC (the default value). We are using AES-GCM which is working fine but anything Blowfish also works.

    Steve



  • @jdp0418:

    I reverted one of my 2.2.3 NanoBSD firewalls back to 2.2.2.  This gives me a setup similar to what some have described here… having one firewall on 2.2.3 and one on 2.2.2.  And my IPSEC... still works.  :-\

    I've hit the weird issues before so I can appreciate what you guys are dealing with... but I can't seem to replicate the issue.  If I have time today, I might try to fire up a tunnel on an ASA to a PFS 2.2.2 and 2.2.3 and see what results I get.

    I've reinstalled 2.2.2 to get things working again but I'm wondering if the problem is an upgrade issue on a 2.2.2 installation and not a fresh new install of 2.2.3 . Installing 2.2.2 or 2.2.3 on NanoBSD isn't that really a fresh install.

    I have just downloaded the 2.2.3 64-bit CD Installer and will do the following:-

    On working 2.2.2 remote host…
    1. Disable pfBlockerNG.
    2. Backup config and save on laptop.
    3. Re-enable pfBlockerNG.
    4. Shutdown, disconnect and keep ready for reinstatement.

    On identical hardware test box...
    5. Install pfSense 2.2.3 from CD
    6. Login from laptop and restore backup config from working 2.2.2 host
    7. When restore is complete, reboot.
    8. Enable pfBlocker.
    9. Check IPsec 3DES tunnels from 2.2.3 remote host to 2.2.2 central host are up and passing packets.



  • Hi,

    Bug created for further investigation.

    https://redmine.pfsense.org/issues/4791

    -=david=-


  • Netgate Administrator

    Any of you confirm that an alternative Phase2 encryption type fixes this?

    Steve



  • @stephenw10:

    Ok, this looks like a problem with AES-CBC crypto at Phase2.

    Try changing your Phase2 to anything other than AES-CBC (the default value). We are using AES-GCM which is working fine but anything Blowfish also works.

    I will try this shortly and report back (these units aren't production yet so I can test at will)



  • Ok, I setup my phase 2 as AES-128 - AES-XCBC (only; no other options selected).  It now appears I have a phase 1 tunnel up, but no phase 2 connection exists between the sites.  If I set it as one of several options under hash, phase 2 comes up.  I noticed that several of the configs posted here appear to have SHA1 selected as the hash algorithm though.

    @vbentley
    I am not 100% clear on how nanoBSD updates, but I am pretty sure its just writes a new image on one of the boot slices of the flash drive.  So yes, that is probably more akin to a fresh install than an upgrade.  That definitely is a big difference!



  • Specifying AES-256-GCM only for phase2 results in the same error for me in the Shrewsoft client.



  • Changed to 3DES and the tunnel works fine, I'll try the other AES modes shortly but tight on time right now…  on the OpenBSD side it didn't like me simply changing "aes" to "aes-128-gcm", so I'll have to dig a bit more.



  • frmpf - Which hash algorithm are you using with 3DES?

    Cheers


  • Netgate Administrator

    The Shrewsoft client is extremely picky.

    Just to sure you use 3DES in the Phase2 config frmpf?

    Steve



  • I've left the hash as SHA1 in all trials…  I had success with 3DES for quick mode, left main mode as AES:

    On the pfSense side, I only changed the enc for phase 2 from aes to 3des, everything else the same.

    Similarly on OpenBSD, just a single quick mode change from aes to 3des

    Doesn't work with 2.2.3

    ike dynamic esp from 192.168.87.0/24 to { 172.29.200.0/24, 192.168.77.0/24 } peer 2.2.2.80
      main auth hmac-sha1 enc aes group modp1024 lifetime 28800
      quick auth hmac-sha1 enc aes group modp1024 lifetime 3600
      srcid 1.1.1.251 dstid 2.2.2.80 psk xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

    Works with 2.2.3

    ike dynamic esp from 192.168.87.0/24 to { 172.29.200.0/24, 192.168.77.0/24 } peer 2.2.2.80
      main auth hmac-sha1 enc aes group modp1024 lifetime 28800
      quick auth hmac-sha1 enc 3des group modp1024 lifetime 3600
      srcid 1.1.1.251 dstid 2.2.2.80 psk xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx