OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2)



  • I have a setup with my pfSense router - that has been working for a long time (a known good configuration). OpenVPN connects to AirVPN without issue, and sure enough - no issues outbound. The inbound port forwarding is simply not working. Firewalls are good, the NAT rules appear fine as well.

    What changed? My ISP is now Bell FTTH (Fiber 1 Gbps service). I use a media converter SFP/RJ45 (To avoid using their Home Hub 3000). On the pfSense side of things, a new interface was created to do VLAN 35, and PPPoE. Everything connects without issue. Default gateway is set, outbound NAT works, Firewall rules were adjusted. I also updated from 2.4.? to 2.4.4 recently (without paying attention if port forwarding worked or not)

    I did some packet captures, and see some packets come in, but nothing else. Strangely enough, I see port forward packets hitting the BELL_WAN interface with the port forward testing in the firewall logs - which shouldn't happen (should be the AirVPN interface). It is as if pfSense or OpenVPN is misclassifying inbound packets, if the firewall logs are right, or else normal behavior as they traverse the BELL_WAN PPPoE interface?

    My instinct tells me it is not a bug with the new version of pfSense, but rather the need for a vlan tagged / pppoe interface that is causing a routing/forwarding issue with pfSense. I can reconnect the Bell home hub 3000 in bridge mode and re-test it all. Is there a way, GUI or SSH wise to really determine the root cause?

    I relied heavily on this how-to for the pfSense / AirVPN setup: https://nguvu.org/pfsense/pfsense-baseline-setup/

    I expect others have a similar setup at home for privacy. I have not been able to find any listed bug or issues.

    Any suggestions? I'll gladly share any config to get this working.



  • When I run a packet capture, I noticed that the inbound connection attempt shows the BELL WAN IP address, and yet the test was absolutely done within the OpenVPN tunnel from AirVPN.

    Capture from Bell WAN (VLAN35 tagged, PPPoE interface)

    11:14:19.654648 IP 188.166.175.60.32814 > 69.159.103.171.63995: tcp 0
    11:14:20.667177 IP 188.166.175.60.32814 > 69.159.103.171.63995: tcp 0
    11:14:22.683177 IP 188.166.175.60.32814 > 69.159.103.171.63995: tcp 0
    11:14:24.748651 IP 10.11.8.9.63995 > 188.166.175.60.52876: tcp 0
    11:14:27.749209 IP 10.11.8.9.63995 > 188.166.175.60.52876: tcp 0
    11:14:33.750196 IP 10.11.8.9.63995 > 188.166.175.60.52876: tcp 0

    Bell WAN IP is: 69.159.103.171
    AirVPN Server: 188.166.175.60
    OpenVPN pfSense IP: 10.11.8.9

    Can anyone explain to me why or how this is happening? I know its the clue to solving my issue, but I don't understand why pfSense is handling the OpenVPN packet on the public WAN IP.


  • LAYER 8 Netgate

    Because that is where the packet is arriving.



  • @derelict Please tell me how I can determine that via capture or logs. The AirVPN test is clearly to the internal OpenVPN IP, and yet pfSense indicated otherwise.


  • LAYER 8 Netgate

    Probably be better if you more completely described exactly what you're trying to do.

    What kind of VPN?

    Who is forwarding the port?

    What port?

    Where is it supposed to be arriving?

    How are you testing?



  • @derelict

    @derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    Probably be better if you more completely described exactly what you're trying to do.

    What kind of VPN?

    AirVPN, openvpn. pfSense is the client for the whole network

    Who is forwarding the port?

    AirVPN forwarding service.

    What port?
    TCP and UDP 63995

    Where is it supposed to be arriving?
    QBittorrent client. AirVPN provides a public IP based on an exit node, and forwards a specific port TCP or UDP to the internally routed openvpn client IP.

    How are you testing?
    Testing is done via AirVPN client area. I attached a screen capture,

    0_1548632416174_e6b6db32-281f-4f01-b5e2-108310518f19-image.png


  • LAYER 8 Netgate

    There is no way that that test could result in the connection to 63995 arriving on your WAN unless the AirVPN tester is completely broken and they are connecting to your WAN address instead.

    Or you are doing something strange like forwarding that inbound connection out your WAN and misreading the WAN pcap source address as the destination.



  • @derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    There is no way that that test could result in the connection to 63995 arriving on your WAN unless the AirVPN tester is completely broken and they are connecting to your WAN address instead.

    I hear you. But every test I run through it, indicates that pfSense is doing something unusual with packet headers. This is a new situation - pppoe with vlan tagging. I suspect a bug. I opened a ticket with AirVPN, and through their own testing, this is what I see packet wise. pfSense / openvpn on my side is doing something unexpected. What do you suggest I try to diagnose this correctly?

    Or you are doing something strange like forwarding that inbound connection out your WAN and misreading the WAN pcap source address as the destination.

    Ok, that is also a theory, but I need some help to track / log this behavior. I know the NAT and firewall rules are sane, and known to work. The only difference is the PPPOE / VLAN WAN interface configuration


  • LAYER 8 Netgate

    PPPoE and VLAN cannot make traffic arrive to that destination address.

    What does a packet capture for TCP port 63995 on the OpenVPN interface look like?

    Please understand we are not AirVPN support. What do they have to say?



  • @derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    PPPoE and VLAN cannot make traffic arrive to that destination address.

    What does a packet capture for TCP port 63995 on the OpenVPN interface look like?

    Please understand we are not AirVPN support. What do they have to say?

    The packet capture falsely indicates the interface inbound. It is impossible for it to originate on the WAN. pfSense is doing something unusual here.

    I don't expect AirVPN support here. They diagnosed, and I verified a few scenarios to confirm that. The issue isn't with them.

    If I revert to using the Bell home hub 3000 in bridge mode, the WAN with PPPOE on it, all without the VLAN 35, it works.

    I would share access to firewall for you to see for yourself.


  • LAYER 8 Netgate

    The packet capture never lies. If you are capturing on WAN and seeing that, that is what is coming into WAN the firewall cannot generate traffic that comes into an interface. You should probably test again, tell us exactly what you are doing, and use full output from the packet capture page and post it up.

    Again, capturing on the OpenVPN interface will show you what is coming over OpenVPN.



  • @derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    The packet capture never lies. If you are capturing on WAN and seeing that, that is what is coming into WAN the firewall cannot generate traffic that comes into an interface. You should probably test again, tell us exactly what you are doing, and use full output from the packet capture page and post it up.

    Again, capturing on the OpenVPN interface will show you what is coming over OpenVPN.

    I agree, it shouldn't be wrong.

    Here is a capture going to the OpenVPN interface:
    21:06:02.209139 IP 188.166.175.60.40277 > 10.11.8.9.63995: UDP, length 48
    21:06:07.494078 IP 188.166.175.60.52338 > 10.11.8.9.63995: tcp 0
    21:06:08.504706 IP 188.166.175.60.52338 > 10.11.8.9.63995: tcp 0
    21:06:10.520798 IP 188.166.175.60.52338 > 10.11.8.9.63995: tcp 0

    I do another test, capturing on the BELL_WAN interface (not the WAN interface, but the virtual interface with PPPOE and VLAN35).

    21:08:04.389707 IP 188.166.175.60.38336 > 69.159.103.101.63995: UDP, length 48
    21:08:04.789811 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
    21:08:05.799557 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
    21:08:07.815525 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
    21:08:09.860945 IP 10.11.8.9.63995 > 188.166.175.60.52446: tcp 0
    21:08:12.860513 IP 10.11.8.9.63995 > 188.166.175.60.52446: tcp 0

    The AirVPN test is strictly being sent to the OpenVPN interface IP of 10.11.8.9 in both cases. What explains this?


  • LAYER 8 Netgate

    @steven-sl said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    21:08:04.389707 IP 188.166.175.60.38336 > 69.159.103.101.63995: UDP, length 48
    21:08:04.789811 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
    21:08:05.799557 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0
    21:08:07.815525 IP 188.166.175.60.60580 > 69.159.103.101.63995: tcp 0

    It could be that your bittorrent client is actually making connections out WAN and asking for connections back on 63995 I suppose.


  • LAYER 8 Netgate

    And if you would be so kind as to use full output so we can see what kind of TCP packets those are that'd be great.



  • @derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    And if you would be so kind as to use full output so we can see what kind of TCP packets those are that'd be great.

    I have multiple devices that are routed through the openvpn connection. In the case of my QBittorrent, it is on my laptop, and by all indications I am going through an AirVPN exit node in Montreal.

    My captures are done via pfSense - diagnostics, packet capture. Apart from the interface choice, only port 63995 is set. Let me know if there is a better way.

    0_1548641933198_AIRVPN-packetcapture.zip
    0_1548641940993_BELLWAN-packetcapture.zip


  • LAYER 8 Netgate

    Post your OpenVPN firewall rules and your AIRVPN firewall rules.


  • LAYER 8 Netgate

    It looks to me like AIRVPN's tester is attempting connections to both the tunnel address AND the address you are connecting from. They might use this to catch cases where people have passed/forwarded traffic on WAN but not OpenVPN. Just a guess since I am not AIRVPN.

    There is really no other explanation for SYN traffic arriving on both interfaces. As I said, pcaps don't just conjure up phantom traffic and present it.

    Your replies are likely being sent out WAN because your OpenVPN rules don't take into account what needs to happen when you receive connections from arbitrary addresses. You probably want to delete/disable all of the rules on the OpenVPN tab and pass the traffic on the AIRVPN tab instead. This will get reply-to on the states and force reply traffic from the bittorrent client back out the VPN where the connection came from instead of obeying the routing table and sending it out WAN.



  • @derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    Post your OpenVPN firewall rules and your AIRVPN firewall rules.

    Outbound from LANGroup to OpenVPN interface
    0_1548642405211_c64be4f3-d3ed-4ad0-9c8f-0bcc5fa59194-image.png

    Inbound firewall rule on AirVPN
    0_1548642513226_d37815de-89ba-453e-8585-e4d843a62b80-image.png

    Port forward NAT rule
    0_1548642597086_bf2436a9-0d26-49e0-aacb-fda69f40614a-image.png

    Outbound NAT rule
    0_1548642660348_066acf09-ac01-445e-b437-4608d6f072b5-image.png

    OpenVPN client status
    0_1548642721767_7c2a2bc6-448e-4c7a-8031-9bb95f8c1097-image.png


  • LAYER 8 Netgate

    Great. And the rules on the OpenVPN tab?



  • @derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    Great. And the rules on the OpenVPN tab?

    OpenVPN (Although I believe this was created for my road warrior setup - OpenVPN server)
    0_1548642939091_d212dfeb-baee-475f-8918-9454c1176471-image.png

    OpenVPN Server (if that matters)
    0_1548642994240_c75c900b-bcf8-4e2b-b36e-c77afdfee486-image.png


  • LAYER 8 Netgate

    Yeah. The port forward traffic cannot match that, which it does.

    That means you do not get reply-to on the states.

    I would assign an interface to the road warrior instance, put the pass any rule there, and put no rules on the OpenVPN tab and it will work.



  • @derelict said in OpenVPN, AirVPN and port forwarding no longer works (2.4.4relp2):

    Yeah. The port forward traffic cannot match that, which it does.

    That means you do not get reply-to on the states.

    I would assign an interface to the road warrior instance, put the pass any rule there, and put no rules on the OpenVPN tab and it will work.

    Thank you so much, I am i awe that you figured this out with the little bit I gave you. It is working now! I am so impressed!

    I don't fully understand why, but I did as you said - I created a new interface for what my setup has as "ovpns2", removed the allow all rule under OpenVPN and tested again. Success. Why would the default OpenVPN interface interfere?

    New Interface
    0_1548643703519_877877de-4f1b-4d70-88e1-2919b21dcd05-image.png

    Road warrior rules:
    0_1548643777793_a20b41dd-b62a-48ec-b856-323df7149a36-image.png

    OpenVPN (removed the only rule)
    0_1548643823964_ae44ef42-2e01-437b-af16-d0c5e30401ed-image.png


  • LAYER 8 Netgate

    Because the OpenVPN tab really represents an interface group of all OpenVPN instances both server and client.

    When a connection arrives into that it is impossible for pf to know which one it will arrive on so it cannot apply reply-to to the rules there. When you are accepting connections in from locations that your local OpenVPN knows about (has specific routes to), like in a remote access or typical site-to-site configuration, this does not matter because the routing table (and openvpn) will route the reply traffic where it needs to go.

    When you are accepting connections from arbitrary source addresses, you need reply-to on the rules or the routing table will direct the reply traffic which, to an arbitrary destination on the internet, will probably mean it tries to send it out to its default gateway.

    And, like most everywhere else in pfSense, first match wins and stops rule processing and interface group rules are processed before interface rules so traffic that needs reply-to cannot match rules there.


Log in to reply