OpenVPN server by Virtual pfSense to Community-PVLAN issue



  • pfSense experts - please help!

    In our Hyper-V deployment there is a very strange issue with a virtual pfSense FW when its LAN port is on a Community PVLAN and is the default gateway for all hosts on the same Community PVLAN.
    All PVLANs are provided by Cisco Nexus 1000V switch. The regular VLANs are on the Hyper-V switch.

    Everything else works as expected, except the OpenVPN site-to-site or RA client tunnels where all packets coming from the OpenVPN tunnel interface to LAN interface simply vanish…

    Note:
    1. The problem goes away if the pfSense's LAN interface and nodes behind it (Hyper-V VMs) are moved to any regular VLAN.
    2. The IPSec site-to-site tunnels do work as expected in this Hyper-V/Cisco PVLAN environment.
    3. The pfSense is unaware of any VLANs

    Here is OpenVPN server config:

    
    dev ovpns2
    verb 1
    dev-type tun
    dev-node /dev/tun2
    writepid /var/run/openvpn_server2.pid
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp
    cipher AES-128-CBC
    auth SHA256
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    local *.*.*.19
    ifconfig 10.1.8.1 10.1.8.2
    lport 1195
    management /var/etc/openvpn/server2.sock unix
    route 192.168.0.0 255.255.0.0
    route 10.0.0.0 255.0.0.0
    route 172.16.0.0 255.240.0.0
    secret /var/etc/openvpn/server2.secret 
    comp-lzo adaptive
    push "route 172.31.5.0 255.255.255.0"
    push "route 10.131.0.0 255.255.0.0"
    
    

    … here is client config:

    
    dev tun
    persist-tun
    persist-key
    proto udp
    cipher AES-128-CBC
    auth SHA256
    pull
    resolv-retry infinite
    remote *.*.*.19 1195
    route 10.131.0.0 255.255.0.0
    ifconfig 10.1.8.2 10.1.8.1
    keepalive 10 60
    ping-timer-rem
    secret pfSense-udp-1195.secret
    comp-lzo
    
    

    **We have a ticket open with Cisco but they are pointing to the pfSense being the culprit…

    According to the pfSense packet capture on the LAN port, the ICMP packets sent from a VM (10.131.102.17) on the PVLAN are sent over the S2S OpenVPN tunnel to remote client (192.168.1.182) and the replays are sent back, but the VM never receives them:**

    13:15:03.330330 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18320, offset 0, flags [none], proto ICMP (1), length 60)
        10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1765, length 40
    13:15:03.336943 00:1d:d8:b7:1e:20 > 00:1d:d8:b7:1e:7c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20219, offset 0, flags [none], proto ICMP (1), length 60)
        192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1765, length 40
    13:15:08.315107 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18321, offset 0, flags [none], proto ICMP (1), length 60)
        10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1766, length 40
    13:15:08.321571 00:1d:d8:b7:1e:20 > 00:1d:d8:b7:1e:7c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20220, offset 0, flags [none], proto ICMP (1), length 60)
        192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1766, length 40
    13:15:13.315482 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18322, offset 0, flags [none], proto ICMP (1), length 60)

    However no reply packets are received at the Cisco N1KV switch port (LTL=53) where the pfSense is connected to.

    nex1# module vem 3 execute vempkt show capture info
    Stage : Ingress
        LTL : 53
        VLAN : Unspecified
        Filter : ip proto 1
    Stage : Egress
        LTL : 53
        VLAN : Unspecified
        Filter : ip proto 1
    Stage : Drop
        LTL : 53
        VLAN : Unspecified
        Filter : Unspecified

    nex1#
    nex1# module vem 3 execute vempkt start
    nex1# module vem 3 execute vempkt stop
    Will suspend log after next 0 entries
    nex1# module vem 3 execute vempkt stop
    Suspended log
    nex1#
    nex1# module vem 3 execute vempkt display detail all
    ********************** Entry 1 *************************

    –----Packet Entry Information------
                  Timestamp  :  Apr 06 13:15:02.602002
                Packet Entry  :  1
                        CPU  :  2
              Bytes Captured  :  74

    ------Packet Length Information------
              Packet Length  : 74
        Packet Buffer Length  : 74
        Packet Mapped Length  : 74

    ------SF Packet Information------
              Capture Stage  : Egress
            SF Packet Flags  : Original
                  Source LTL  : 49
            Destination LTL  : 53
                        HWBD  : 19
                  Vlan/SegID  : 1102

    ------Packet L3 Header Information------
            Source IP Address  : 10.131.102.17
      Destination IP Address  : 192.168.1.182
            IP Protocol Type:    1

    ------Packet L2 Header Information------
          Source MAC Address  : 00:1d:d8:b7:1e:7c
    Destination MAC Address  : 00:1d:d8:b7:1e:20
                      Type    : 2048

    ------Packet Platform Information------
                          NBL  : 0xFFFFE0015B08BB60
                      Source  : 2 - 0
                  Send Flags  : 0x0
                VMQ Queue ID  : 0
                    NBL Type  : 0
            NBL Checksum Info  : 0x220011
              NBL 802.1Q Info  : 0x0
            Native Forwarding  : 0x0
            Virtual Subnet Id  : 0x0
            NBL Direction      : Ingress
                  Dest Count  : 1
                Current Dest  : 10 - 0
                  Dest Flags  : 0

    Payload :
        00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
        00016: 00 3c 47 8f 00 00 80 01 00 00 0a 83 66 11 c0 a8
        00032: 01 b6 08 00 46 77 00 01 06 e4 61 62 63 64 65 66
        00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
        00064: 77 61 62 63 64 65 66 67 68 69


    ********************** Entry 2 *************************

    ------Packet Entry Information------
                  Timestamp  :  Apr 06 13:15:03.617679
                Packet Entry  :  2
                        CPU  :  7
              Bytes Captured  :  74

    ------Packet Length Information------
              Packet Length  : 74
        Packet Buffer Length  : 74
        Packet Mapped Length  : 74

    ------SF Packet Information------
              Capture Stage  : Egress
            SF Packet Flags  : Original
                  Source LTL  : 49
            Destination LTL  : 53
                        HWBD  : 19
                  Vlan/SegID  : 1102

    ------Packet L3 Header Information------
            Source IP Address  : 10.131.102.17
      Destination IP Address  : 192.168.1.182
            IP Protocol Type:    1

    ------Packet L2 Header Information------
          Source MAC Address  : 00:1d:d8:b7:1e:7c
    Destination MAC Address  : 00:1d:d8:b7:1e:20
                      Type    : 2048

    ------Packet Platform Information------
                          NBL  : 0xFFFFE001539B2B60
                      Source  : 2 - 0
                  Send Flags  : 0x0
                VMQ Queue ID  : 0
                    NBL Type  : 0
            NBL Checksum Info  : 0x220011
              NBL 802.1Q Info  : 0x0
            Native Forwarding  : 0x0
            Virtual Subnet Id  : 0x0
            NBL Direction      : Ingress
                  Dest Count  : 1
                Current Dest  : 10 - 0
                  Dest Flags  : 0

    Payload :
        00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
        00016: 00 3c 47 90 00 00 80 01 00 00 0a 83 66 11 c0 a8
        00032: 01 b6 08 00 46 76 00 01 06 e5 61 62 63 64 65 66
        00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
        00064: 77 61 62 63 64 65 66 67 68 69


    ********************** Entry 3 *************************

    ------Packet Entry Information------
                  Timestamp  :  Apr 06 13:15:08.602318
                Packet Entry  :  3
                        CPU  :  2
              Bytes Captured  :  74

    ------Packet Length Information------
              Packet Length  : 74
        Packet Buffer Length  : 74
        Packet Mapped Length  : 74

    ------SF Packet Information------
              Capture Stage  : Egress
            SF Packet Flags  : Original
                  Source LTL  : 49
            Destination LTL  : 53
                        HWBD  : 19
                  Vlan/SegID  : 1102

    ------Packet L3 Header Information------
            Source IP Address  : 10.131.102.17
      Destination IP Address  : 192.168.1.182
            IP Protocol Type:    1

    ------Packet L2 Header Information------
          Source MAC Address  : 00:1d:d8:b7:1e:7c
    Destination MAC Address  : 00:1d:d8:b7:1e:20
                      Type    : 2048

    ------Packet Platform Information------
                          NBL  : 0xFFFFE001539B2B60
                      Source  : 2 - 0
                  Send Flags  : 0x0
                VMQ Queue ID  : 0
                    NBL Type  : 0
            NBL Checksum Info  : 0x220011
              NBL 802.1Q Info  : 0x0
            Native Forwarding  : 0x0
            Virtual Subnet Id  : 0x0
            NBL Direction      : Ingress
                  Dest Count  : 1
                Current Dest  : 10 - 0
                  Dest Flags  : 0

    Payload :
        00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
        00016: 00 3c 47 91 00 00 80 01 00 00 0a 83 66 11 c0 a8
        00032: 01 b6 08 00 46 75 00 01 06 e6 61 62 63 64 65 66
        00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
        00064: 77 61 62 63 64 65 66 67 68 69


    ********************** Entry 4 *************************

    ------Packet Entry Information------
                  Timestamp  :  Apr 06 13:15:13.602580
                Packet Entry  :  4
                        CPU  :  3
              Bytes Captured  :  74

    ------Packet Length Information------
              Packet Length  : 74
        Packet Buffer Length  : 74
        Packet Mapped Length  : 74

    ------SF Packet Information------
              Capture Stage  : Egress
            SF Packet Flags  : Original
                  Source LTL  : 49
            Destination LTL  : 53
                        HWBD  : 19
                  Vlan/SegID  : 1102

    ------Packet L3 Header Information------
            Source IP Address  : 10.131.102.17
      Destination IP Address  : 192.168.1.182
            IP Protocol Type:    1

    ------Packet L2 Header Information------
          Source MAC Address  : 00:1d:d8:b7:1e:7c
    Destination MAC Address  : 00:1d:d8:b7:1e:20
                      Type    : 2048

    ------Packet Platform Information------
                          NBL  : 0xFFFFE0015B08BB60
                      Source  : 2 - 0
                  Send Flags  : 0x0
                VMQ Queue ID  : 0
                    NBL Type  : 0
            NBL Checksum Info  : 0x220011
              NBL 802.1Q Info  : 0x0
            Native Forwarding  : 0x0
            Virtual Subnet Id  : 0x0
            NBL Direction      : Ingress
                  Dest Count  : 1
                Current Dest  : 10 - 0
                  Dest Flags  : 0

    Payload :
        00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
        00016: 00 3c 47 92 00 00 80 01 00 00 0a 83 66 11 c0 a8
        00032: 01 b6 08 00 46 74 00 01 06 e7 61 62 63 64 65 66
        00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
        00064: 77 61 62 63 64 65 66 67 68 69




  • assuming this part is taken from the LAN NIC (or whichever interface is the one where the traffic apparently isn't reaching the switch):

    @bk:

    13:15:03.330330 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18320, offset 0, flags [none], proto ICMP (1), length 60)
        10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1765, length 40
    13:15:03.336943 00:1d:d8:b7:1e:20 > 00:1d:d8:b7:1e:7c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20219, offset 0, flags [none], proto ICMP (1), length 60)
        192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1765, length 40

    It's being sent out that NIC, and either disappearing at the Hyper-V level or on the switch.

    The fact it works on a non-PVLAN shows your VM's config is fine.



  • Thanks for chiming in CMB.

    Note that the packet captures at the latter part of my previous post were done on the Nexus 1000V switch port #53 where pfSense FW's LAN interface is connected to.  So the generic [packets are]

    either disappearing at the Hyper-V level or on the switch.

    does not help much.

    I wish it was that simple… Consider this:

    1.  Why IPSec tunnel in exactly the same PVLAN scenario works fine (the VMs behind the LAN port are reachable)?
    2.  Why Port Forward on the outside interface to a VM in exactly the same PVLAN scenario works fine?
    3.  Why only OpenVPN does NOT work with the PVLANs?

    There must be something special about how the OVPN packets come from the LAN interface...

    I am new to pfSense but I think it has something to do with the NAT masquerading and/or the proxy arp. Does that make sense?

    How these are utilized and controlled in the pfSense FreeBSD OS?

    Any help tweaking those (or other relevant) settings would be appreciated!



  • BUMP

    Anyone who has something useful to say on the matter?

    Can't believe no one set up pfSense on a Private VLAN (PVLAN)?!



  • After several months of troubleshooting work with Cisco engineers and even escalating to their Nexus developers the culprit could not be found…

    However, upgrading the pfSense to the latest 2.3.1 version SOLVED the problem!  :o
    I hope someone could explain what was changed in the 2.3.1-RELEASE (amd64), built on Tue May 17 18:46:53 CDT 2016 in regards to the OpenVPN code to make it work.


Log in to reply