OpenVPN server by Virtual pfSense to Community-PVLAN issue
-
pfSense experts - please help!
In our Hyper-V deployment there is a very strange issue with a virtual pfSense FW when its LAN port is on a Community PVLAN and is the default gateway for all hosts on the same Community PVLAN.
All PVLANs are provided by Cisco Nexus 1000V switch. The regular VLANs are on the Hyper-V switch.Everything else works as expected, except the OpenVPN site-to-site or RA client tunnels where all packets coming from the OpenVPN tunnel interface to LAN interface simply vanish…
Note:
1. The problem goes away if the pfSense's LAN interface and nodes behind it (Hyper-V VMs) are moved to any regular VLAN.
2. The IPSec site-to-site tunnels do work as expected in this Hyper-V/Cisco PVLAN environment.
3. The pfSense is unaware of any VLANsHere is OpenVPN server config:
dev ovpns2 verb 1 dev-type tun dev-node /dev/tun2 writepid /var/run/openvpn_server2.pid script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp cipher AES-128-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local *.*.*.19 ifconfig 10.1.8.1 10.1.8.2 lport 1195 management /var/etc/openvpn/server2.sock unix route 192.168.0.0 255.255.0.0 route 10.0.0.0 255.0.0.0 route 172.16.0.0 255.240.0.0 secret /var/etc/openvpn/server2.secret comp-lzo adaptive push "route 172.31.5.0 255.255.255.0" push "route 10.131.0.0 255.255.0.0"
… here is client config:
dev tun persist-tun persist-key proto udp cipher AES-128-CBC auth SHA256 pull resolv-retry infinite remote *.*.*.19 1195 route 10.131.0.0 255.255.0.0 ifconfig 10.1.8.2 10.1.8.1 keepalive 10 60 ping-timer-rem secret pfSense-udp-1195.secret comp-lzo
**We have a ticket open with Cisco but they are pointing to the pfSense being the culprit…
According to the pfSense packet capture on the LAN port, the ICMP packets sent from a VM (10.131.102.17) on the PVLAN are sent over the S2S OpenVPN tunnel to remote client (192.168.1.182) and the replays are sent back, but the VM never receives them:**
13:15:03.330330 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18320, offset 0, flags [none], proto ICMP (1), length 60)
10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1765, length 40
13:15:03.336943 00:1d:d8:b7:1e:20 > 00:1d:d8:b7:1e:7c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20219, offset 0, flags [none], proto ICMP (1), length 60)
192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1765, length 40
13:15:08.315107 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18321, offset 0, flags [none], proto ICMP (1), length 60)
10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1766, length 40
13:15:08.321571 00:1d:d8:b7:1e:20 > 00:1d:d8:b7:1e:7c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20220, offset 0, flags [none], proto ICMP (1), length 60)
192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1766, length 40
13:15:13.315482 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18322, offset 0, flags [none], proto ICMP (1), length 60)
…However no reply packets are received at the Cisco N1KV switch port (LTL=53) where the pfSense is connected to.
nex1# module vem 3 execute vempkt show capture info
Stage : Ingress
LTL : 53
VLAN : Unspecified
Filter : ip proto 1
Stage : Egress
LTL : 53
VLAN : Unspecified
Filter : ip proto 1
Stage : Drop
LTL : 53
VLAN : Unspecified
Filter : Unspecifiednex1#
nex1# module vem 3 execute vempkt start
nex1# module vem 3 execute vempkt stop
Will suspend log after next 0 entries
nex1# module vem 3 execute vempkt stop
Suspended log
nex1#
nex1# module vem 3 execute vempkt display detail all
********************** Entry 1 *************************–----Packet Entry Information------
Timestamp : Apr 06 13:15:02.602002
Packet Entry : 1
CPU : 2
Bytes Captured : 74------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1------Packet L2 Header Information------
Source MAC Address : 00:1d:d8:b7:1e:7c
Destination MAC Address : 00:1d:d8:b7:1e:20
Type : 2048------Packet Platform Information------
NBL : 0xFFFFE0015B08BB60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0Payload :
00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
00016: 00 3c 47 8f 00 00 80 01 00 00 0a 83 66 11 c0 a8
00032: 01 b6 08 00 46 77 00 01 06 e4 61 62 63 64 65 66
00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
00064: 77 61 62 63 64 65 66 67 68 69
********************** Entry 2 *************************
------Packet Entry Information------
Timestamp : Apr 06 13:15:03.617679
Packet Entry : 2
CPU : 7
Bytes Captured : 74------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1------Packet L2 Header Information------
Source MAC Address : 00:1d:d8:b7:1e:7c
Destination MAC Address : 00:1d:d8:b7:1e:20
Type : 2048------Packet Platform Information------
NBL : 0xFFFFE001539B2B60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0Payload :
00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
00016: 00 3c 47 90 00 00 80 01 00 00 0a 83 66 11 c0 a8
00032: 01 b6 08 00 46 76 00 01 06 e5 61 62 63 64 65 66
00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
00064: 77 61 62 63 64 65 66 67 68 69
********************** Entry 3 *************************
------Packet Entry Information------
Timestamp : Apr 06 13:15:08.602318
Packet Entry : 3
CPU : 2
Bytes Captured : 74------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1------Packet L2 Header Information------
Source MAC Address : 00:1d:d8:b7:1e:7c
Destination MAC Address : 00:1d:d8:b7:1e:20
Type : 2048------Packet Platform Information------
NBL : 0xFFFFE001539B2B60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0Payload :
00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
00016: 00 3c 47 91 00 00 80 01 00 00 0a 83 66 11 c0 a8
00032: 01 b6 08 00 46 75 00 01 06 e6 61 62 63 64 65 66
00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
00064: 77 61 62 63 64 65 66 67 68 69
********************** Entry 4 *************************
------Packet Entry Information------
Timestamp : Apr 06 13:15:13.602580
Packet Entry : 4
CPU : 3
Bytes Captured : 74------Packet Length Information------
Packet Length : 74
Packet Buffer Length : 74
Packet Mapped Length : 74------SF Packet Information------
Capture Stage : Egress
SF Packet Flags : Original
Source LTL : 49
Destination LTL : 53
HWBD : 19
Vlan/SegID : 1102------Packet L3 Header Information------
Source IP Address : 10.131.102.17
Destination IP Address : 192.168.1.182
IP Protocol Type: 1------Packet L2 Header Information------
Source MAC Address : 00:1d:d8:b7:1e:7c
Destination MAC Address : 00:1d:d8:b7:1e:20
Type : 2048------Packet Platform Information------
NBL : 0xFFFFE0015B08BB60
Source : 2 - 0
Send Flags : 0x0
VMQ Queue ID : 0
NBL Type : 0
NBL Checksum Info : 0x220011
NBL 802.1Q Info : 0x0
Native Forwarding : 0x0
Virtual Subnet Id : 0x0
NBL Direction : Ingress
Dest Count : 1
Current Dest : 10 - 0
Dest Flags : 0Payload :
00000: 00 1d d8 b7 1e 20 00 1d d8 b7 1e 7c 08 00 45 00
00016: 00 3c 47 92 00 00 80 01 00 00 0a 83 66 11 c0 a8
00032: 01 b6 08 00 46 74 00 01 06 e7 61 62 63 64 65 66
00048: 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 75 76
00064: 77 61 62 63 64 65 66 67 68 69
-
assuming this part is taken from the LAN NIC (or whichever interface is the one where the traffic apparently isn't reaching the switch):
@bk:
13:15:03.330330 00:1d:d8:b7:1e:7c > 00:1d:d8:b7:1e:20, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 18320, offset 0, flags [none], proto ICMP (1), length 60)
10.131.102.17 > 192.168.1.182: ICMP echo request, id 1, seq 1765, length 40
13:15:03.336943 00:1d:d8:b7:1e:20 > 00:1d:d8:b7:1e:7c, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20219, offset 0, flags [none], proto ICMP (1), length 60)
192.168.1.182 > 10.131.102.17: ICMP echo reply, id 1, seq 1765, length 40It's being sent out that NIC, and either disappearing at the Hyper-V level or on the switch.
The fact it works on a non-PVLAN shows your VM's config is fine.
-
Thanks for chiming in CMB.
Note that the packet captures at the latter part of my previous post were done on the Nexus 1000V switch port #53 where pfSense FW's LAN interface is connected to. So the generic [packets are]
either disappearing at the Hyper-V level or on the switch.
does not help much.
I wish it was that simple… Consider this:
1. Why IPSec tunnel in exactly the same PVLAN scenario works fine (the VMs behind the LAN port are reachable)?
2. Why Port Forward on the outside interface to a VM in exactly the same PVLAN scenario works fine?
3. Why only OpenVPN does NOT work with the PVLANs?There must be something special about how the OVPN packets come from the LAN interface...
I am new to pfSense but I think it has something to do with the NAT masquerading and/or the proxy arp. Does that make sense?
How these are utilized and controlled in the pfSense FreeBSD OS?
Any help tweaking those (or other relevant) settings would be appreciated!
-
BUMP
Anyone who has something useful to say on the matter?
Can't believe no one set up pfSense on a Private VLAN (PVLAN)?!
-
After several months of troubleshooting work with Cisco engineers and even escalating to their Nexus developers the culprit could not be found…
However, upgrading the pfSense to the latest 2.3.1 version SOLVED the problem! :o
I hope someone could explain what was changed in the 2.3.1-RELEASE (amd64), built on Tue May 17 18:46:53 CDT 2016 in regards to the OpenVPN code to make it work.