Traffic Blocked, Expect Pass



  • Hi,

    OK, this one is driving me nuts - may be me, but seems very odd.

    I have an OpenVPN subnet set up, and I want to get traffic routed on to my LAN. The routing looks to be working, but pfSense is blocking some of the traffic. I have added a LAN rule to allow all traffic destined for my VPN subnet (192.168.254.0/24), but I specifically see the default rule (Default deny rule IPv4 (1000000103)) => and nothing hitting my added LAN rule, even though it's for this subnet.

    Any thoughts? It's very odd, but seems to be matching exactly … so not sure why the default rule is blocking.

    Thanks!


  • LAYER 8 Global Moderator

    "I have an OpenVPN subnet set up, and I want to get traffic routed on to my LAN."
    "and nothing hitting my added LAN rule, even though it's for this subnet."

    So your traffic is coming from your openvpn client?  Why would it hit your lan rules?  Rules are evaluated as they enter an interface, not as they leave.

    If you want to filter traffic coming from your vpn then you would allow the traffic on your openvpn interface.  Pretty sure out of the box its an any any rule. Did you edit this rule or remove it?

    Or is this openvpn subnet as you call it just another subnet on your pfsense?

    Again rules are evaluated as they enter an interface towards pfsense.  Rules are evaluated top down, first rule to trigger wins, no other rules are evaluated.



  • Let me try to explain - I know that wasn't real clear … :(

    I have an OpenVPN server on the LAN (not using pfSense for that, security concern), and I am forwarding traffic, routing it through pfSense (the gateway). Some traffic gets through (ICMP), but I see in the firewall logs that other traffic is being blocked (e.g. SSH, HTTP). What's odd is that I see the block in the firewall log, and it's blocked by the default rule ... but I have a rule on the LAN list that should be hit first ... right? Really, as long as it's on my list, it comes before the default block rule, correct?

    Thanks!


  • LAYER 8 Global Moderator

    "I have an OpenVPN server on the LAN (not using pfSense for that, security concern)"

    Unwarranted.  And now you have placed your server behind, and your going to run into asymmetrical routing issues most likely, and you most likely will have a 1 arm bandit sort of setup with hairpinned traffic flow even if you get rid of the asymmetrical routing issue via a transit network you setup.

    "but I have a rule on the LAN list that should be hit first "

    When would forwarded traffic (from the internet?) hit the LAN interface?

    So your vpn server is sitting on your lan??  Why don't you draw up your network and we can discuss.  But I can tell you the quick easy, secure best way to do this if you want to allow vpn traffic into your network is to just run openvpn on your pfsense which is the edge of your network, and now you remove any asymmetrical routing problems, and hairpin traffic.

    So you forwarded ICMP?  If you want the vpn port or ssh to get from the internet to some server sitting behind pfsense, then you need to forward those ports.  There would be no rules needed on the lan.  You would create the forward, it would auto create the firewall rules on your wan for you to allow the forwarded traffic.



  • You bet, let me try to explain. Here is the network setup,

    ************************                        ****************************
    * pfSense              *                        * OpenVPN Server           * 
    *                      * ====================== *                          *
    * LAN = 192.168.2.0/24 *                        * LAN IP = 192.168.2.64    *
    * LAN IP = 192.168.2.1 *                        * OVPN IP = 192.168.254.1  *
    ************************                        ****************************
                                                                  \
                                                                   \
                                                                    \
                                                                     \
                                                              ****************************
                                                              * OpenVPN Client           *
                                                              * OVPN IP = 192.168.254.6  *
                                                              ****************************
    

    Trying to get from the OpenVPN client, to another machine on the LAN (192.168.2.65), to it's web server … here is what I see, tcpdump on pfSense (the gateway),
    18:23:12.388447 IP 192.168.2.65.80 > 192.168.254.6.60327: Flags [S.], seq 1602483301, ack 3438329450, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
    18:23:12.577005 IP 192.168.2.65.80 > 192.168.254.6.60328: Flags [S.], seq 187999887, ack 1312274101, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0

    So, it's getting to pfSense. And I have created a (LAN) rule to pass this traffic (attached). So I believe it should get to the destination, right? But also attached, pfSense is blocking it.

    Could it be that the LAN check is not based on interface, but rather the expected subnet of the LAN? That would explain this, correct?

    Thanks!

    ![LAN Rule.PNG](/public/imported_attachments/1/LAN Rule.PNG)
    ![LAN Rule.PNG_thumb](/public/imported_attachments/1/LAN Rule.PNG_thumb)
    ![Firewall Block.PNG](/public/imported_attachments/1/Firewall Block.PNG)
    ![Firewall Block.PNG_thumb](/public/imported_attachments/1/Firewall Block.PNG_thumb)


  • LAYER 8 Global Moderator

    So is your openvpn server natting this 192.168.254 network to your lan network?  If not then you have to setup a gateway in pfsense to tell it how to get to this openvpn net.  And you run into asymmetrical routing since your connecting to what amounts to a downstream router to your lan and not connected via a transit network.

    Those blocks are out of state traffic notice the SA, which as told you in first post is going to be related to asymmetrical routing.

    What exactly do you expect this vpn server to do setup like you have.. Seems completely pointless!  If your going to set it up like that you either need to nat at your openvpn server to your 192.168.2 network or you need to setup pfsense to understand downstream network via a gateway and then route.  If you don't nat then you need to use a transit network.

    What exactly are you wanting to accomplish - such a setup seems utterly pointless.

    Your seeing the SA blocked because it is out of state.. Since pfsense never saw the SYN from the client trying to talk to your web server.

    So your client says hey I want to go to ip on 192.168.2 - openvpn says oh directly connected sends the traffic to the 192.168.2.x IP (red arrow) SYN.. But this 192.168.2.x devices says hey I want to talk to 192.168.254 so it sends it to its gateway pfsense (green arrow).  Pfsense sees this return traffic SYN,ACK - hey I don't have a state for this traffic, its not SYN so not going to open a state - DENY.

    If you explain to me what your wanting to accomplish exactly - be happy to walk you through how to correctly set it up.




  • Hi,

    First of all - thanks! I really appreciate you taking the time to respond to this in such detail. And what you say make sense. Sorry if I'm thick, just trying to figure this out … :).

    What am I trying to do? Really, just have an OpenVPN server inside the network (that also port shares with an HTTPS server => forward non-OpenVPN traffic to Apache), and allow client access to the machines on the LAN (subnet / network). I was using bridging, but that was up and down terribly ... and the folks on the OpenVPN mailing list recommended to get rid of bridging, move to a routed approach. I admit, it is much more reliable now (with routing instead of bridging), but now I'm having this nagging issue ... and what you say makes sense.

    Clear as mud?

    I'm definitely open to suggestions - fire away! ... :).

    Thanks again.


  • LAYER 8 Netgate

    Asymmetric routing is bad m'kay?



  • LAYER 8 Global Moderator

    "OpenVPN server inside the network"

    Why??  Pfsense comes with openvpn - click your openvpn server is up and running!

    Putting your openvpn server inside your edge like that is nothing but problems.  As already mentioned you have asymmetrical routing to deal with, and or hairpins.

    Do you have more than just lan on pfsense you can work with?  Do you have switch that supports vlans?  No matter how you look at it putting the server inside your network is ugly!!



  • No argument here … ;). I'm OK with moving to pfSense (I lose load balancing, but not a biggie) ... but - if I do, is a TUN interface supported, but still full access / routing to the LAN (subnet)? If so, I'm game! Is there any info on setting it up this way (to allow LAN access, using TUN and routing not bridging)?

    Thanks!


  • LAYER 8 Global Moderator

    How do you lose load balancing? Your drawing shows no load balancing happening.

    Bridging of tap is normally bad idea, and should only be used when you REQUIRE layer 2 connectivity through the vpn.. ie broadcast or multicast to work through it.

    As to access into your lan from a vpn, using tun - yes this is very simple.  And can be controlled with simple firewall rules, I vpn into all my local segments when I vpn into my network.



  • Excellent - I'll switch over then! And sorry to confuse you, load balancing was not part of the diagram, but it was part of the reason I was hoping to have the server(s) internal. But I'm OK to let that go.

    So if I set up OpenVPN on pfSense, you mention firewall rules to be able to access the subnet / LAN. Any info you can provide on this?

    Thanks!


  • LAYER 8 Global Moderator

    out of the box when you create the vpn connection using the wizard the rules on the openvpn interface will be any any.  If you want to limit or control them then you just put the rules on your openvpn interface.

    So you wanted to bring 2 servers up behind?  do you have 2 internet connections - if you explain what your wanting to accomplish a load balancing point of view we can discuss if can be accomplished and how, etc.

    edit:  So here is simple example.  So see how I can ping 192.168.9.100, this is box on my home network.  I am currently at work via vpn.  I then created a rule to block icmp to 192.168.9.100 from my vpn (2nd part of ping pic showing timeout).  And set this rule to log.  See it logged in my firewall - but dns query from my same vpn client was allowed through.

    btw - the ping times are bit high because only way I can vpn out at work is via a proxy, so when I vpn to home which is only few miles from work.  I have to go all the way to houston, then all the way back to here.  So latency is way higher than normal in my case - sucks but works just fine..




  • Hi,

    Nope, the load balancing was due to using flaky old HW for the server, so some redundancy. But not a biggie.

    Is there a way to "import" my legacy config? I have keys, etc. all set up … would be nice to start from that, given the key checks, etc. that are in place. Also, I'm remote, so if I mess the setup up I'll be down for a while ... ;-).

    Thanks!


  • LAYER 8 Global Moderator

    You want to use the config from your current openvpn in the pfsense openvpn..  While sure it would be possible, its prob much easier to just run through the wizard.. It really is all of like 1 minute to fire up remote vpn into pfsense.  If you have the keys and certs you could still use those.. Just import those into pfsense and then change your openvpn you setup with the wizard to use those certs and keys.

    But setup of openvpn with pfsense is much easier with the gui and wizard then doing the conf files for openvpn ;)



  • Yep, you're right - easy setup, i went ahead and set it up … now to get it working ... LMAO.

    A couple questions,

    • why is HW acceleration disabled? Would be nice to make use of that.
    • trying to use the "Client Export Utility" to get the info for the remote ... but I don't see an Next / Save type button. Likely me, but how to get the output? ... ;)

    Thanks!


  • LAYER 8 Global Moderator

    What hardware are you running on?  Does it support AES-NI?

    Not sure what you mean about next/save on the export.  Just pick your instance at the top, what address to use, etc. Then other options you might want to pick and then scroll to the bottom and you will see your different users that you have created in the user manager..  Click what you want to export, either just file, inline, installer even.




  • Thanks! That's what I was missing - I was creating a User Certificate, but it seems you need to do this by adding as a user. Much appreciated! The other secret is that I needed to add a Firewall Rule, to pass the OpenVPN port … correct? I assumed this would happen as part of the server setup, seems I was wrong.

    OK, I'm connected - but can't get to any machines on the LAN, or even to the OpenVPN server (ping even fails). Suggestions?

    As for the HW acceleration - it's not giving me an option, but does show this on the dashboard,

    CPU Type ...
    Intel(R) Core(TM) i5-2450M CPU @ 2.50GHz
    Current: 2500 MHz, Max: 2501 MHz
    4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads
    AES-NI CPU Crypto: Yes (active)
    
    Hardware crypto
    AES-CBC,AES-XTS,AES-GCM,AES-ICM
    


  • OK, got it up and running - thanks for all the help! Just a few minor things left - please let me know if you have any thoughts,

    • the trick was to add a firewall rule for the OpenVPN interface, allow all traffic there … is that the right answer though? ...  :).
    • actually, to the point above ... I added 2 rules => 1 to allow OpenVPN traffic through, the other to open the WAN to the OpenVPN port ... correct? Any other rules needed?
    • DNS back to the pfSense box (from the OpenVPN client) is being rejected (not failing, actually rejected). Do I need to do anything to allow the DNS server to reply. Still digging on this one.
    • HW accel is still not working, which is very odd. Any suggestions appreciated.

    Thanks again!!!


  • LAYER 8 Global Moderator

    What crypto did you set..
    AES-CBC,AES-XTS,AES-GCM,AES-ICM

    Did you run through the wizard for the server, it would of auto added your firewall rules you need.  Yes need WAN rule to allow the connection in.  And yes you would need rule on the openvpn interface.. Already went over that - when I blocked ping.

    If your dns is being rejected, is your tunnel network in your ACL for unbound?



  • Hi,

    A few thoughts / answers - thanks for all the pointers!

    What crypto did you set..
    AES-CBC,AES-XTS,AES-GCM,AES-ICM

    Correct - I left the default that pfSense set up … AES-256-GCM, AES-128-GCM. Perhaps a logic error in the check (with the number in the middle)?

    Did you run through the wizard for the server, it would of auto added your firewall rules you need.  Yes need WAN rule to allow the connection in.  And yes you would need rule on the openvpn interface.. Already went over that - when I blocked ping.

    Yep, sorry - long and winding path, I got sidetracked … ;). But that said, nope - no auto-created rules (from the Wizard ... yes, used that).

    If your dns is being rejected, is your tunnel network in your ACL for unbound?

    Yep, added that. After a reboot though, it's happy.

    And one more now it seems …  :(. When I try to connect from and Android client (using the Client Export, to OpenVPN Connect), I get the error,

    Unknown OpenVPN event occurred: Transport error on 'mydomain.com: NETWORK_EOF_ERROR
    
    

    Seen this one before?

    Thanks again!


  • LAYER 8 Global Moderator

    Openvpn connect for android and ios does not support the new option tls encryption and auth, need to set it to just tls auth..  I ran into that myself, took me a bit to figure out what was different between 2 different instances had running one worked, other didnt ;)

    As to the rules for wan – yeah they are created by the wizard..  I have them on my own setup, the comment says created by wizard.  If I bring up a new instance - it adds a rule.




  • Thanks for the info on TLS - much appreciated! Sorry you ran into it (can be painful), but sort of glad you did … ;).

    Odd on the wizard, no rules here - just the ones I created manually. That said, I may have done something wrong in the wizard (i.e. I'm guessing it's an operator error, not the tool).

    Still a bit confused about the lack of HW accel - would like to offload the CPU if possible. Do you know if there is a way to check it from the command line? And I guess either way - is it worth posting as a potential bug? Just thinking I can try to help others, but don't want to cause grief either.

    Thanks!


  • LAYER 8 Global Moderator

    You might want to post another thread about the HW thing.  I run my home pfsense on vm so no hardware for crypto.

    I have a sg-2440 at work, I could look into on monday about the hardware accel for crypto.



  • Cool, sounds good - thanks again for all your help. Really appreciated!

    Yep, posted another question about that. If it's a bug, want to be helpful, let folks know.

    Have a nice weekend!


Log in to reply