OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2)
-
I have a setup with my pfSense router - that has been working for a long time (a known good configuration). OpenVPN connects to AirVPN without issue, and sure enough - no issues outbound. The inbound port forwarding is simply not working. Firewalls are good, the NAT rules appear fine as well.
What changed? My ISP is now Bell FTTH (Fiber 1 Gbps service). I use a media converter SFP/RJ45 (To avoid using their Home Hub 3000). On the pfSense side of things, a new interface was created to do VLAN 35, and PPPoE. Everything connects without issue. Default gateway is set, outbound NAT works, Firewall rules were adjusted. I also updated from 2.4.? to 2.4.4 recently (without paying attention if port forwarding worked or not)
I did some packet captures, and see some packets come in, but nothing else. Strangely enough, I see port forward packets hitting the BELL_WAN interface with the port forward testing in the firewall logs - which shouldn't happen (should be the AirVPN interface). It is as if pfSense or OpenVPN is misclassifying inbound packets, if the firewall logs are right, or else normal behavior as they traverse the BELL_WAN PPPoE interface?
My instinct tells me it is not a bug with the new version of pfSense, but rather the need for a vlan tagged / pppoe interface that is causing a routing/forwarding issue with pfSense. I can reconnect the Bell home hub 3000 in bridge mode and re-test it all. Is there a way, GUI or SSH wise to really determine the root cause?
I relied heavily on this how-to for the pfSense / AirVPN setup: https://nguvu.org/pfsense/pfsense-baseline-setup/
I expect others have a similar setup at home for privacy. I have not been able to find any listed bug or issues.
Any suggestions? I'll gladly share any config to get this working.
-
When I run a packet capture, I noticed that the inbound connection attempt shows the BELL WAN IP address, and yet the test was absolutely done within the OpenVPN tunnel from AirVPN.
Capture from Bell WAN (VLAN35 tagged, PPPoE interface)
11:14:19.654648 IP 188.166.175.60.32814 > 69.159.103.171.63995: tcp 0
11:14:20.667177 IP 188.166.175.60.32814 > 69.159.103.171.63995: tcp 0
11:14:22.683177 IP 188.166.175.60.32814 > 69.159.103.171.63995: tcp 0
11:14:24.748651 IP 10.11.8.9.63995 > 188.166.175.60.52876: tcp 0
11:14:27.749209 IP 10.11.8.9.63995 > 188.166.175.60.52876: tcp 0
11:14:33.750196 IP 10.11.8.9.63995 > 188.166.175.60.52876: tcp 0Bell WAN IP is: 69.159.103.171
AirVPN Server: 188.166.175.60
OpenVPN pfSense IP: 10.11.8.9Can anyone explain to me why or how this is happening? I know its the clue to solving my issue, but I don't understand why pfSense is handling the OpenVPN packet on the public WAN IP.
-
Because that is where the packet is arriving.
-
@derelict Please tell me how I can determine that via capture or logs. The AirVPN test is clearly to the internal OpenVPN IP, and yet pfSense indicated otherwise.
-
Probably be better if you more completely described exactly what you're trying to do.
What kind of VPN?
Who is forwarding the port?
What port?
Where is it supposed to be arriving?
How are you testing?
-
@derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):
Probably be better if you more completely described exactly what you're trying to do.
What kind of VPN?
AirVPN, openvpn. pfSense is the client for the whole network
Who is forwarding the port?
AirVPN forwarding service.
What port?
TCP and UDP 63995Where is it supposed to be arriving?
QBittorrent client. AirVPN provides a public IP based on an exit node, and forwards a specific port TCP or UDP to the internally routed openvpn client IP.How are you testing?
Testing is done via AirVPN client area. I attached a screen capture,
-
There is no way that that test could result in the connection to 63995 arriving on your WAN unless the AirVPN tester is completely broken and they are connecting to your WAN address instead.
Or you are doing something strange like forwarding that inbound connection out your WAN and misreading the WAN pcap source address as the destination.
-
@derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):
There is no way that that test could result in the connection to 63995 arriving on your WAN unless the AirVPN tester is completely broken and they are connecting to your WAN address instead.
I hear you. But every test I run through it, indicates that pfSense is doing something unusual with packet headers. This is a new situation - pppoe with vlan tagging. I suspect a bug. I opened a ticket with AirVPN, and through their own testing, this is what I see packet wise. pfSense / openvpn on my side is doing something unexpected. What do you suggest I try to diagnose this correctly?
Or you are doing something strange like forwarding that inbound connection out your WAN and misreading the WAN pcap source address as the destination.
Ok, that is also a theory, but I need some help to track / log this behavior. I know the NAT and firewall rules are sane, and known to work. The only difference is the PPPOE / VLAN WAN interface configuration
-
PPPoE and VLAN cannot make traffic arrive to that destination address.
What does a packet capture for TCP port 63995 on the OpenVPN interface look like?
Please understand we are not AirVPN support. What do they have to say?
-
@derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):
PPPoE and VLAN cannot make traffic arrive to that destination address.
What does a packet capture for TCP port 63995 on the OpenVPN interface look like?
Please understand we are not AirVPN support. What do they have to say?
The packet capture falsely indicates the interface inbound. It is impossible for it to originate on the WAN. pfSense is doing something unusual here.
I don't expect AirVPN support here. They diagnosed, and I verified a few scenarios to confirm that. The issue isn't with them.
If I revert to using the Bell home hub 3000 in bridge mode, the WAN with PPPOE on it, all without the VLAN 35, it works.
I would share access to firewall for you to see for yourself.
-
The packet capture never lies. If you are capturing on WAN and seeing that, that is what is coming into WAN the firewall cannot generate traffic that comes into an interface. You should probably test again, tell us exactly what you are doing, and use full output from the packet capture page and post it up.
Again, capturing on the OpenVPN interface will show you what is coming over OpenVPN.
-
@derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):
The packet capture never lies. If you are capturing on WAN and seeing that, that is what is coming into WAN the firewall cannot generate traffic that comes into an interface. You should probably test again, tell us exactly what you are doing, and use full output from the packet capture page and post it up.
Again, capturing on the OpenVPN interface will show you what is coming over OpenVPN.
I agree, it shouldn't be wrong.
Here is a capture going to the OpenVPN interface:
21:06:02.209139 IP 188.166.175.60.40277 > 10.11.8.9.63995: UDP, length 48
21:06:07.494078 IP 188.166.175.60.52338 > 10.11.8.9.63995: tcp 0
21:06:08.504706 IP 188.166.175.60.52338 > 10.11.8.9.63995: tcp 0
21:06:10.520798 IP 188.166.175.60.52338 > 10.11.8.9.63995: tcp 0I do another test, capturing on the BELL_WAN interface (not the WAN interface, but the virtual interface with PPPOE and VLAN35).
21:08:04.389707 IP 188.166.175.60.38336 > 69.159.103.101.63995: UDP, length 48
21:08:04.789811 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
21:08:05.799557 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
21:08:07.815525 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
21:08:09.860945 IP 10.11.8.9.63995 > 188.166.175.60.52446: tcp 0
21:08:12.860513 IP 10.11.8.9.63995 > 188.166.175.60.52446: tcp 0The AirVPN test is strictly being sent to the OpenVPN interface IP of 10.11.8.9 in both cases. What explains this?
-
@steven-sl said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):
21:08:04.389707 IP 188.166.175.60.38336 > 69.159.103.101.63995: UDP, length 48
21:08:04.789811 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
21:08:05.799557 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
21:08:07.815525 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0It could be that your bittorrent client is actually making connections out WAN and asking for connections back on 63995 I suppose.
-
And if you would be so kind as to use full output so we can see what kind of TCP packets those are that'd be great.
-
@derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):
And if you would be so kind as to use full output so we can see what kind of TCP packets those are that'd be great.
I have multiple devices that are routed through the openvpn connection. In the case of my QBittorrent, it is on my laptop, and by all indications I am going through an AirVPN exit node in Montreal.
My captures are done via pfSense - diagnostics, packet capture. Apart from the interface choice, only port 63995 is set. Let me know if there is a better way.
0_1548641933198_AIRVPN-packetcapture.zip
0_1548641940993_BELLWAN-packetcapture.zip -
Post your OpenVPN firewall rules and your AIRVPN firewall rules.
-
It looks to me like AIRVPN's tester is attempting connections to both the tunnel address AND the address you are connecting from. They might use this to catch cases where people have passed/forwarded traffic on WAN but not OpenVPN. Just a guess since I am not AIRVPN.
There is really no other explanation for SYN traffic arriving on both interfaces. As I said, pcaps don't just conjure up phantom traffic and present it.
Your replies are likely being sent out WAN because your OpenVPN rules don't take into account what needs to happen when you receive connections from arbitrary addresses. You probably want to delete/disable all of the rules on the OpenVPN tab and pass the traffic on the AIRVPN tab instead. This will get reply-to on the states and force reply traffic from the bittorrent client back out the VPN where the connection came from instead of obeying the routing table and sending it out WAN.
-
@derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):
Post your OpenVPN firewall rules and your AIRVPN firewall rules.
Outbound from LANGroup to OpenVPN interface
Inbound firewall rule on AirVPN
Port forward NAT rule
Outbound NAT rule
OpenVPN client status
-
Great. And the rules on the OpenVPN tab?
-
@derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):
Great. And the rules on the OpenVPN tab?
OpenVPN (Although I believe this was created for my road warrior setup - OpenVPN server)
OpenVPN Server (if that matters)
