Cannot route across IPSec VPN?



  • We have some new servers at Rackspace. With the setup they provided a Vyatta image for us to setup a VPN between our sites.

    At Rackspace we have two networks, 192.168.30.0/24 and 10.188.66.121/32.
    At our colo facility our internal network is 192.168.200.0/24.
    At our main office the internal network is 192.168.0.0/23.
    The connection between the main office subnet and colo subnet is an MPLS with a bare bones Linux router on each end. No NAT between those subnets. No iptables firewall.

    The connection from our pfSense to the Vyatta is an IPSec VPN, that appears to be working correctly. We can ping, SSH, and everything else between the two networks at Rackspace and the network at the colo without any problem.  We can even ping and SSH  networks all we want.  We can even SSH from a system at Rackspace, to a system on our main office network.

    What we cannot do is anything other than ping from out main office subnet to the Rackspace network. But we can ping, which doesn't make any sense to me.

    IPSec phase 1 configuration:
    protocol ipv4
    interface wan
    remote gateway (public IP for Vyatta)
    authentication method mutal psk
    negotiation mode main
    my identifier (my public IP)
    peer identified (public IP for Vyatta)
    pre-shared key xxxxxxxxxxxxxxx
    encryption algorith AES 256
    hash algorith SHA1
    DH key group 2
    Lifetime 86400

    phase 2 -1
    mode tunnel ipv4
    local network type lan
    remote network 192.168.30.0/24
    protocol esp
    encryption algorithms AES 256 bits
    has algorithms sha1
    pfs key group 2
    lifetime 3600

    phase 2-2
    same except
    remote network 10.188.66.121/32

    phase 2-3
    same except
    local network type network, address 192.168.0.0/23
    remote network 192.168.30.0/24

    phase 2-4
    same except
    local network type network, address 192.168.0.0/23
    remote network 192.168.30.0/24

    Firewall rules:
    Floating - nothing
    WAN - nothing
    LAN - anti-lockout rule, and IPv4****** basically wide open
    IPsec - IPv4****** basically wide open

    Like I said, I can SSH from 192.168.30.1 to 192.168.200.x. I can SSH from 192.168.200.x to 192.168.30.x. I can initiate an SSH connection from 192.168.30.2 to 192.168.0.187. I can connect to the database at 10.188.66.121 from the systems on 192.168.200.x. I cannot initiate an SSH session or database connection from 192.168.0.187 to 192.168.30.1, but I can ping it. Same usernames, same passwords, same SSH key.

    Taking a look at the connections on Vyatta I don't even see the connection being attempted from 192.168.0.187 to 192.168.30.1. Looking at the filter.log on pfSense shows the following when I SSH from 192.168.200.42 to 192.168.30.2:

    Jul 13 16:47:09 rackspacevpn-colov filterlog: 84,16777216,,1000104255,em0,match,pass,out,4,0x0,,64,47585,0,none,50,esp,120,97.65.238.10,104.239.231.219,datalength=100

    Attempting to SSH into the same system from 192.168.0.187 generates the following entries, and fails:

    ul 13 16:48:00 rackspacevpn-colov filterlog: 71,16777216,,1000004121,em1,match,pass,in,4,0x60,,59,49500,0,DF,6,tcp,60,192.168.0.187,192.168.200.41,49402,22,0,S,1614338842,,29200,,mss;sackOK;TS;nop;wscale
    Jul 13 16:48:01 rackspacevpn-colov filterlog: 71,16777216,,1000004121,em1,match,pass,in,4,0x60,,59,49501,0,DF,6,tcp,60,192.168.0.187,192.168.200.41,49402,22,0,S,1614338842,,29200,,mss;sackOK;TS;nop;wscale
    Jul 13 16:48:03 rackspacevpn-colov filterlog: 71,16777216,,1000004121,em1,match,pass,in,4,0x60,,59,49502,0,DF,6,tcp,60,192.168.0.187,192.168.200.41,49402,22,0,S,1614338842,,29200,,mss;sackOK;TS;nop;wscale
    Jul 13 16:48:04 rackspacevpn-colov filterlog: 67,16777216,,1000003811,em0,match,pass,out,4,0xb8,,64,14482,0,none,17,udp,76,97.65.238.10,208.75.88.4,123,123,56

    Mtr shows it is connects in exactly the path expected, but I can do nothing other than ping. What am I missing here?



  • It's passing inbound at least, from those logs. Packet capture on IPsec, you see the TCP traffic leaving there?



  • Ran a quick capture on IPsec.  Once doing a ping for a few responses, then ssh to the same host.  The packet capture shows the ICMP but nothing else even though I was capturing any protocol.  Started over, just SSH this time, and the packet capture was completely empty.

    21:06:57.186096 (authentic,confidential): SPI 0xc9bf72fe: IP (tos 0x0, ttl 58, id 32441, offset 0, flags [DF], proto ICMP (1), length 84, bad cksum 20e3 (->21e3)!)
        192.168.0.187 > 192.168.30.1: ICMP echo request, id 20392, seq 1, length 64
    21:06:57.225725 (authentic,confidential): SPI 0xca814bc8: IP (tos 0x0, ttl 64, id 37729, offset 0, flags [none], proto ICMP (1), length 84)
        192.168.30.1 > 192.168.0.187: ICMP echo reply, id 20392, seq 1, length 64
    21:06:58.187242 (authentic,confidential): SPI 0xc9bf72fe: IP (tos 0x0, ttl 58, id 32593, offset 0, flags [DF], proto ICMP (1), length 84, bad cksum 204b (->214b)!)
        192.168.0.187 > 192.168.30.1: ICMP echo request, id 20392, seq 2, length 64
    21:06:58.226716 (authentic,confidential): SPI 0xca814bc8: IP (tos 0x0, ttl 64, id 37775, offset 0, flags [none], proto ICMP (1), length 84)
        192.168.30.1 > 192.168.0.187: ICMP echo reply, id 20392, seq 2, length 64
    21:06:59.189348 (authentic,confidential): SPI 0xc9bf72fe: IP (tos 0x0, ttl 58, id 32709, offset 0, flags [DF], proto ICMP (1), length 84, bad cksum 1fd7 (->20d7)!)
        192.168.0.187 > 192.168.30.1: ICMP echo request, id 20392, seq 3, length 64
    21:06:59.228169 (authentic,confidential): SPI 0xca814bc8: IP (tos 0x0, ttl 64, id 37788, offset 0, flags [none], proto ICMP (1), length 84)
        192.168.30.1 > 192.168.0.187: ICMP echo reply, id 20392, seq 3, length 64

    Did it again, but set the interface to LAN instead of IPsec.

    21:11:57.443179 00:15:17:90:09:04 > 52:54:00:15:3f:7f, ethertype IPv4 (0x0800), length 74: (tos 0x60, ttl 59, id 44548, offset 0, flags [DF], proto TCP (6), length 60)
        192.168.0.187.36582 > 192.168.200.41.22: Flags ~~, cksum 0x91e7 (correct), seq 3040750311, win 29200, options [mss 1460,sackOK,TS val 97354979 ecr 0,nop,wscale 7], length 0
    21:11:57.449071 00:15:17:90:09:04 > 52:54:00:15:3f:7f, ethertype IPv4 (0x0800), length 60: (tos 0xa0, ttl 59, id 46834, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.0.187.36582 > 192.168.200.41.22: Flags [R], cksum 0xf687 (correct), seq 3040750312, win 0, length 0
    21:11:58.440038 00:15:17:90:09:04 > 52:54:00:15:3f:7f, ethertype IPv4 (0x0800), length 74: (tos 0x60, ttl 59, id 44549, offset 0, flags [DF], proto TCP (6), length 60)
        192.168.0.187.36582 > 192.168.200.41.22: Flags ~~, cksum 0x90ed (correct), seq 3040750311, win 29200, options [mss 1460,sackOK,TS val 97355229 ecr 0,nop,wscale 7], length 0
    21:11:58.445045 00:15:17:90:09:04 > 52:54:00:15:3f:7f, ethertype IPv4 (0x0800), length 60: (tos 0xa0, ttl 59, id 47039, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.0.187.36582 > 192.168.200.41.22: Flags [R], cksum 0xf687 (correct), seq 3040750312, win 0, length 0
    21:12:00.444029 00:15:17:90:09:04 > 52:54:00:15:3f:7f, ethertype IPv4 (0x0800), length 74: (tos 0x60, ttl 59, id 44550, offset 0, flags [DF], proto TCP (6), length 60)
        192.168.0.187.36582 > 192.168.200.41.22: Flags ~~, cksum 0x8ef8 (correct), seq 3040750311, win 29200, options [mss 1460,sackOK,TS val 97355730 ecr 0,nop,wscale 7], length 0
    21:12:00.451737 00:15:17:90:09:04 > 52:54:00:15:3f:7f, ethertype IPv4 (0x0800), length 60: (tos 0xa0, ttl 59, id 47187, offset 0, flags [DF], proto TCP (6), length 40)
        192.168.0.187.36582 > 192.168.200.41.22: Flags [R], cksum 0xf687 (correct), seq 3040750312, win 0, length 0

    I am at a loss.  I even added some routes just to see what happened but it didn't have any effect.  No idea where to go from here.  I may have to figure out how to create a MIP and just treat the 192.168.0.0/23 traffic as if it were on 192.168.200.0/24.~~~~~~



  • That's a different destination, thought you meant you could ping but not SSH to and from the same iP. Probably missing a matching P2 for that source and destination network.



  • Nope, same destination.  The pfsense box is 192.168.200.41, Vyatta on the other end is 192.168.30.1.  My desktop in this test was 192.168.0.187. I was able to ping, but not SSH. First capture was from IPSec, second was from LAN.  Both of them same test.  ICMP shows up on the IPsec capture, nothing else. LAN capture is just showing the traffic coming from the desktop to the pfsense system on the LAN, nothing on the IPsec capture.

    I even fired up another system with 2.2.2 just now to see if the problem was being caused by the bug in 2.2.3 people have been reporting with IPsec. Same problem.

    Could you elaborate on the "missing a matching P2 for that source and destination network"? Thanks for all your help.



  • Are you running pfSense on bare hardware or virtualized? I have seen similar "non-senses" with the vmxnet driver supplied by VMware



  • We've actually tried it both ways and are getting the same result.  Using Intel Server adapters on the hardware version, e1000 virtualized adapters on the VM (KVM). They Vyatta is virtuallized, but Rackspace and other people use them all the time for similar configurations without a problem.


Log in to reply