[resolved]IPSec Site-to-Site VPN passes only some Traffic

[mhecker]

Hello pfSense community,

we are currently implementing a new firewall system based on pfSense. Routing and firewall rules to control access between our local VLANs is working fine - we passed our test schedule implementing restrictions between guest, employee and management VLANs. We have configured QoS / Traffic Shaping for our VoIP phones already.

So far we are quite happy with our project going on at a good pace, but the last two days we focused on Site to Site VPNs using IPSec and ran into the following problems:

Our new pfSense at Main-Office:
pfSense Version: 2.2.2-Release (amd64)
Equipped with 3 Dual-Port Intel NICs (igb0, igb1, igb2, igb3, em0, em1)

Current Network Configuration
igb3: LAN 10.100.10.254, Network 10.100.10.0/24
igb2: WAN with pppoe0 dial-in / fixed IPv4 address assigned by ISP
igb0 + em0 bonded as lagg0 in Failover-mode attached to 2 different Switches

We have several VLANs attached to lagg0, for IPSec testing we used two of those networks:
lagg0_vlan10: 10.100.10.253, Network: 10.100.10.0/24 (same Network as LAN / Management Network)
lagg0_vlan110: 10.100.110.254, Network: 10.100.110.0/24 (Employee Network)
etc.

Two Branch-Offices running Funkwerk / bintec rs120 respectively rs353jw devices
Branch-Office 1:
bintec rs120
Router IP: 192.168.2.1
Local subnet: 192.168.2.0/24

Branch-Office 2:
bintec rs353jw
Router IP: 192.168.102.1
Local subnet: 192.168.102.0/24

What works:
Scenario 1: VPN configured to use 10.100.10.0/24 as local subnet

• IPSec tunnels are working
• Inbound traffic from both branch offices to our main office is always working
  – 192.168.2.0/24 -> 10.100.10.0/24
  -- 192.168.102.0/24 -> 10.100.10.0/24
• Outbound traffic originating from LAN-Interface to the branch-offices is working if the LAN network is configured in the phase 2 profile.
  -- 10.100.10.0/24 -> 192.168.2.0/24
  -- 10.100.10.0/24 -> 192.168.102.0/24

Scenario 2: VPN configured to use 10.100.110.0/24 as local subnet

• IPSec tunnels are working
• Inbound traffic from both branch offices to our main office is always working
  -- 192.168.2.0/24 -> 10.100.110.0/24
  -- 192.168.102.0/24 -> 10.100.110.0/24
• Outbound traffic originating from LAN-Interface to the branch-offices is working if the LAN network is configured in the phase 2 profile.
  -- 10.100.110.0/24 -> 192.168.102.0/24

However, in this scenario the following way does not work:
-- 10.100.110.0/24 -> 192.168.2.0/24
No packets appear on tcpdump -i enc0 or blocked by the firewall.

We added a third tunnel to scenario 2 with an additional remote network 192.168.0.0/24 that showed the same behaviour as the 192.168.2.0/24 network. The 192.168.102.0/24 tunnel is still working both ways here, both other tunnels only work one-way.

We added a wildcard rule (pass, source: *, destination: , protocol: ipv4) to the IPSec interface in the firewall settings.

ipsec statusall shows all IP Addresses configured as listening for every configured IPSec connection.

Output of setkey -DP

192.168.0.0/24[any] 10.100.110.0/24[any] any
        in ipsec
        esp/tunnel/<branch3-remote>-

<main office="">/unique:3
        created: Jun 17 14:03:27 2015  lastused: Jun 17 14:03:27 2015
        lifetime: 9223372036854775807(s) validtime: 0(s)
        spid=522 seq=5 pid=85741
        refcnt=1
192.168.102.0/24[any] 10.100.110.0/24[any] any
        in ipsec
        esp/tunnel/<branch2-remote>-

<main office="">/unique:2
        created: Jun 17 14:03:40 2015  lastused: Jun 17 14:24:05 2015
        lifetime: 9223372036854775807(s) validtime: 0(s)
        spid=526 seq=4 pid=85741
        refcnt=1
192.168.2.0/24[any] 10.100.10.0/24[any] any
        in ipsec
        esp/tunnel/<branch1-remote>-

<main office="">/unique:1
        created: Jun 17 14:12:28 2015  lastused: Jun 17 14:14:07 2015
        lifetime: 9223372036854775807(s) validtime: 0(s)
        spid=532 seq=3 pid=85741
        refcnt=1
10.100.110.0/24[any] 192.168.0.0/24[any] any
        out ipsec
        esp/tunnel/

<main office="">-<branch3-remote>/unique:3
        created: Jun 17 14:03:27 2015  lastused: Jun 17 14:03:27 2015
        lifetime: 9223372036854775807(s) validtime: 0(s)
        spid=521 seq=2 pid=85741
        refcnt=1
10.100.110.0/24[any] 192.168.102.0/24[any] any
        out ipsec
        esp/tunnel/

<main office="">-<branch2-remote>/unique:2
        created: Jun 17 14:03:40 2015  lastused: Jun 17 14:24:05 2015
        lifetime: 9223372036854775807(s) validtime: 0(s)
        spid=525 seq=1 pid=85741
        refcnt=1
10.100.10.0/24[any] 192.168.2.0/24[any] any
        out ipsec
        esp/tunnel/

<main office="">-<branch1-remote>/unique:1
        created: Jun 17 14:12:28 2015  lastused: Jun 17 14:14:07 2015
        lifetime: 9223372036854775807(s) validtime: 0(s)
        spid=531 seq=0 pid=85741
       refcnt=1

We appreciate your help!</branch1-remote></main></branch2-remote> </main></branch3-remote> </main>

</main></branch1-remote> </main></branch2-remote> </main></branch3-remote> 

[mhecker]

Hello community,

we resolved this issue with help from the pfSense support.

First of Steve pointed out that our LAN and VLAN10 interfaces were on the same subnet which may cause problems, thus we removed the VLAN10 from our bonded interface to be on the safe side.

The actual problem was caused by firewall rules blocking access to RFC1918 subnets from the local VLANs to our remote networks.

We had a pass rule for the remote subnets, but this rule was on the wrong interface group. We enabled logging on every block/reject rule that we had in place and those packets appeared as rejected by another interface group's reject-rule. Moving the pass-rules to the correct interface group fixed the issue.

Kind regards