I can't get *OFF* my VPN anymore…:-)



  • Stupid situation  ;D

    G'morning all  :P

    I am running into a most strange situation:
    1. I have some traffic from a dedicated server running over PIA-VPN (firewall rule -> specific gateway assigned).
    2. This firewall rule is on the LAN tab, and one of the top most rules (so other LAN-clients come after that, and they go via the default gateway, which is WAN, which also is the default gateway in system/routing).
    3. For testing I had my own desktop also mandatory go through the VPN (also firewall rule with VPN-gateway assigned). 't Works.
    4. But: when I remove the firewall rule for my own desktop, reset the states, even reboot pfSense: I keep on going over the VPN-gateway  ::) :o ???

    How is this remotely possible? I mean, by removing rule/reset states/reboot I completely went to the original situation of going through the default WAN gateway (just as I was doing three days ago), but pfSense refuses to do so :(

    At the mean time now Gmail keeps refusing to let me pop3 it, since I am now accessing it from Germany which makes me a hacker.

    Would anybody know what is going on here, and: how to solve this problem? I am on 2.1.2 AMD.

    Thank you in advance,

    Bye,

    PS And Yahoo webmail does the same and now suddenly wants my GSM-number 'for verification' (what a crap, btw: anybody can use any prepaid mobile number to get into anybodies mail account this way. It is not really about 'verification', it is about knowing who you are. Time to get rid of gmail and yahoo and alikes).



  • My first guess is it's not happening :D

    At least it's not happening the way you think it's happening…....

    Any chance you could post your firewall rules for LAN and OPENVPN?

    When I hit these ones (especially if it survives a reboot), I have to assume I have something wrong in my rules.
    Personally I usually stop at some point and have a Scotch (you may substitute Beer, Wine, coffee, tea etc.) and give it a break.

    Perhaps a fresh set of eyes would help?



  • @divsys:

    My first guess is it's not happening :D

    At least it's not happening the way you think it's happening…....

    Any chance you could post your firewall rules for LAN and OPENVPN?

    When I hit these ones (especially if it survives a reboot), I have to assume I have something wrong in my rules.
    Personally I usually stop at some point and have a Scotch (you may substitute Beer, Wine, coffee, tea etc.) and give it a break.

    Perhaps a fresh set of eyes would help?

    ;D ;D ;D

    (It is too early to drink beer for me).

    I did attach a screenshot of my LAN rules.

    I did find a workaround (which I expected): disable the VPN-client, reset the states, et voila, I am off the VPN client and on the normal WAN again.

    So it appears some sort of bug that even resists a reboot/reset states  :P

    ![normal traffic not off VPN.jpg](/public/imported_attachments/1/normal traffic not off VPN.jpg)
    ![normal traffic not off VPN.jpg_thumb](/public/imported_attachments/1/normal traffic not off VPN.jpg_thumb)



  • :o :o :o :o :o

    So I disabled the VPN client, and my LAN client (my desktop) was off the German VPN. So I enabled the VPN-client again so my NAS (not my LAN-client) can go on the VPN again, reset states:

    The LAN-client is on the VPN again, even 'though there currently isn't a rule to direct this desktop over the VPN  :o

    This is not the first time I am thinking that the firewall rules don't get processed right (the rule descriptions in the firewall log often don't match either).

    I attach my system routing too.

    What might be the problem this time  :-[

    ![normal traffic not off VPN-2.jpg](/public/imported_attachments/1/normal traffic not off VPN-2.jpg)
    ![normal traffic not off VPN-2.jpg_thumb](/public/imported_attachments/1/normal traffic not off VPN-2.jpg_thumb)



  • Did you also check your Firewall/NAT rules?  I seem to remember months ago when I first set it up I had a similar problem and there were rules in the NAT section that seemed to be causing it.

    In the end I actually like how it works.  I put a second older wireless router between the pfSense and the WAN so that I can easily bypass the firewall and therefore the vpn whenever I need to.



  • @pfSensible:

    Did you also check your Firewall/NAT rules?  I seem to remember months ago when I first set it up I had a similar problem and there were rules in the NAT section that seemed to be causing it.

    In the end I actually like how it works.  I put a second older wireless router between the pfSense and the WAN so that I can easily bypass the firewall and therefore the vpn whenever I need to.

    Thank you for your reply  ;D

    What should I do with the NAT-rules? I had to create these manually after setting up OpenVPN (which I found stupid already, excuse me for saying  :-[ ), they are needed for the NAS to go over openvpn, so if I decide one single desktop client should no longer go via VPN, there isn't anything to do for the NAT-rules(?)

    It gets more buggy by the second, btw: I [b]disabled the OpenVPN-client again, yet this time in the dashboard the VPN-gateway stays up and connected, I still have an OpenVPN-IP from PIA (10.x), and … I am still appearing to come from Germany (so: via the VPN) according to dnsstuff.com (shows my external IP). On top of that: in Status/Services there is no openvpn-service.

    Oh, and I forgot: my wireless no longer works, as my smartphones are no longer able to access the internet - in the firewall log I see they don't even try to.

    Byggy crap  :'(



  • @Hollander:

    @pfSensible:

    Did you also check your Firewall/NAT rules?  I seem to remember months ago when I first set it up I had a similar problem and there were rules in the NAT section that seemed to be causing it.

    In the end I actually like how it works.  I put a second older wireless router between the pfSense and the WAN so that I can easily bypass the firewall and therefore the vpn whenever I need to.

    Thank you for your reply  ;D

    What should I do with the NAT-rules? I had to create these manually after setting up OpenVPN (which I found stupid already, excuse me for saying  :-[ ), they are needed for the NAS to go over openvpn, so if I decide one single desktop client should no longer go via VPN, there isn't anything to do for the NAT-rules(?)

    It gets more buggy by the second, btw: I [b]disabled the OpenVPN-client again, yet this time in the dashboard the VPN-gateway stays up and connected, I still have an OpenVPN-IP from PIA (10.x), and … I am still appearing to come from Germany (so: via the VPN) according to dnsstuff.com (shows my external IP). On top of that: in Status/Services there is no openvpn-service.

    Oh, and I forgot: my wireless no longer works, as my smartphones are no longer able to access the internet - in the firewall log I see they don't even try to.

    Byggy crap  :'(

    Yeah, ain't pfSense fun?

    Anyway, there is a NAT outbound rule that was created with the vpn on my pfSense (attached).

    Maybe if you just change your vpn to a different location it will work.  BTW, I configured every possible PIA vpn connection on my pfSense by exporting the configuration, editing the xml file and then reimporting the changed xml config.  This is much easier than creating each one manually within pfSense, now I can disable all but the vpn connection I want.  This makes it very easy to change locations.

    We had wireless issues at one point as well.  It turns out that our cheap p.o.s. edimax wireless card was the problem, it only worked for about 10 minutes after each reboot.  So be bought an Intel card that was on the FreeBSD support list.  But it turns out that pfSense did not support it.  So we bought an Atheros card that all the developers use.  That works.

    Atheros AR5004G AR5213A Super G 108 Mbps Mini PCI Card






  • Raaah  >:( :( :'( :-\

    I am getting depressed to the level of wanting to jump of a bridge. I wasted yet another half a day to get this to work. Some things in pfSense are great, but some things are buggy to the level of being unbearable  >:(

    _(Try to debug, look in firewall logs: rule description: "block unauthorised DNS". Port? 137. Which is a different rule to block netbios..).

    (Completely delete the OpenVPN client interface to PIA; suddenly the VLAN40 is assigned to the WAN-interface, which was the reason my Wifi didn't work anymore. Also took me 2 hours to discover that).

    (Try to upgrade from a minor to another minor: it keeps installing and re-installing packages for hours and hours, then mixes up interfaces and firewall rules, and you end up doing a fresh install for every minor upgrade; which is why I avoid upgrading whenever possible. Because a fresh install doesn't take long, but customizing the system and packages afterwards easily takes a whole day).

    (And things like these have happened for over the year that I am using pfSense now)._

    This is what I did:

    • I completely deleted the OpenVPN client, the settings in the firewall to direct the NAS via openVPN, the entry in system/routing, the OpenVPN-interface, the NAT (I deleted all NAT and put it back to automatic). I rebooted the box for the 47th time today.
    • So at that point I was at a virgin state, before I ever introduced VPN to my box. I thought.
    • I started out fresh creating the OpenVPN-client. I added the NAT, the routing, and this time only added firewall rules for the NAS to go via VPN. I did not create a firewall rule to have my desktop client go via VPN.
    • Reset states, reboot: et voila, the desktop client (so, not the NAS) is going via VPN.
    • Somewhere changes are not pruned the way it should be.

    I am sorry to sound pissed off, but: I am  >:(

    Yet another 6 hours wasted on this crap.

    I have donated via paypal, I have bought the Gold subscription, everything to support this project. I do understand that I have to invest personal time in dealing with this, but I don't have these kind of problems with PC-BSD nor with Mint or Debian.

    Is there a tutorial somewhere on how to report bugs? I do understand it is impossible for the admins/mods to see every thread on this forum.

    Sorry for expressing how pissed I am, but I am  :-[

    Bye,



  • @pfSensible:

    Yeah, ain't pfSense fun?

    Anyway, there is a NAT outbound rule that was created with the vpn on my pfSense (attached).

    Maybe if you just change your vpn to a different location it will work.  BTW, I configured every possible PIA vpn connection on my pfSense by exporting the configuration, editing the xml file and then reimporting the changed xml config.  This is much easier than creating each one manually within pfSense, now I can disable all but the vpn connection I want.  This makes it very easy to change locations.

    We had wireless issues at one point as well.  It turns out that our cheap p.o.s. edimax wireless card was the problem, it only worked for about 10 minutes after each reboot.  So be bought an Intel card that was on the FreeBSD support list.  But it turns out that pfSense did not support it.  So we bought an Atheros card that all the developers use.  That works.

    Atheros AR5004G AR5213A Super G 108 Mbps Mini PCI Card

    It is very nice of you to respond and also post the screenshots; thank you very much  ;D

    The wifi: in my case it turned out pfSense completely messed up interface assignments (see my post just below yours; you were typing when i posted it  :P ).

    I like your idea of adding all the PIA-servers to the list, as per your screenshot. I was already thinking about that, but it would take me quite some time. Your editing XML's is a nice shortcut. I will try that; thank you  ;D

    By the way: that NAT-rule, how did that get there? Because I had to create two rules manually, and you are using a PIA-DHCP'd 10.x address. But that address changes regularly, so that means your NAT won't work anymore?

    However, currently, I have to get rid of this crap I am in in which gmail and yahoo block me because 'I don't access it from my 'normal location' so 'I need to verify by phone'.

    What I am expected to do? Completely delete my OpenVPN, reboot, then log in to yahoo, then set up everything OpenVPN again, and this 600 times a day because the system pushes me over OpenVPN even 'though there is no rule telling it to do so?

    Buggy crap  >:(

    I might as well buy a Zyxel Walmart router. Takes less time. And time is more valuable than money, in a human's life.

    :-[



  • Nothing described there appears to be a bug. OpenVPN client will accept pushed routes unless you tell it not to. Most VPN providers push a default route, which will send all your traffic out the VPN. Add a route-nopull to not do that.

    Interface assignments breaking on upgrade? No, they don't. They will cause issues on reboot if you remove an interface and don't remove its assignment. Then you'll have to re-assign interfaces since your config doesn't match the interfaces you actually have anymore, which leaves it potentially unsafe to continue.

    If packages get stuck saying "reinstalling…" they're not reinstalling over and over, some package hung up the reinstall process and it's stuck there. If the system log doesn't show the package reinstall progressing, go to Diag>Backup/restore, clear package lock, click the reinstall packages button.

    There's no need for any NAT config on OpenVPN that specifies a particular IP. "Interface address" is there for that purpose.



  • @Hollander:

    _(Try to debug, look in firewall logs: rule description: "block unauthorised DNS". Port? 137. Which is a different rule to block netbios..).

    Logs strictly have rule numbers. Rule numbers change if you change your ruleset. There is no record of what the rule numbers used to be, so that's the expected behavior since you've changed your rules between when the log happened and when you clicked to see what rule hit it._



  • @Hollander:

    then mixes up interfaces and firewall rules, and you end up doing a fresh install for every minor upgrade;

    If that were remotely true there would be a massive abandonment of the project, tons of people complaining here, and we'd have a mess with our 500+ active commercial support customers. None of that's the case. Tens of thousands of systems have upgraded via auto-update alone in the past month or so, through 2.1.1, 2.1.2, 2.1.3. We've handled several dozen cases of "I upgraded and now <something>is broken" with support customers. The only such issues that were actually caused by the upgrade were those who didn't upgrade the AutoConfigBackup package before upgrading to 2.1.2 (though we sent out an email to all gold and support subscribers with a "HEADS UP", in addition to mentioning it in the release announcement and elsewhere). One of the pluses of having to do a 2.1.3 is it actually isn't impacted by that ACB bug.

    That was a minority of the cases though, as usual, the majority of "upgrade problems" are problems that would have happened just by rebooting the system. "I upgraded and the system isn't booting anymore" turns out to be the hard drive bit the dust, the BIOS no longer sees it at all post-upgrade. A variety of hardware failures that didn't exhibit themselves until a reboot is one of the more common. Dead drives, systems that can no longer get through POST to even attempt booting from anything. Those are most all the scenarios where people are using ages-old hardware, at times literally pulled out of a dumpster. A small number of cases where people loaded up way too much for 256 MB RAM on an ALIX, where they're running with maybe 10 MB free RAM, and are left with a system that can't boot without running out of RAM, causing the dying of random processes that are part of the boot process. Especially with several OpenVPN instances because it uses more RAM during its immediate start up than it does during normal operation. In those cases, the upgrade aborts and it just reboots into the same version, except you have a config that can't necessarily successfully boot because you've overloaded things. A number of other scenarios along those lines, where it's not the upgrade, it's the reboot.</something>



  • @cmb:

    Nothing described there appears to be a bug. OpenVPN client will accept pushed routes unless you tell it not to. Most VPN providers push a default route, which will send all your traffic out the VPN. Add a route-nopull to not do that.

    Thank you Cmb, and sorry for my rant; I was fed up.

    Unfortunately, I fail to understand what you write. If I tell the firewall to send only specific traffic through the VPN-gateway (via the gateway settings at the bottom of the screen), and it doesn't care about that and sends all traffic through that gateway (which is not the default gateway so it can't be caused by that), how can this be 'the system works as designed'?(??).

    Everybody who helped me in this forum setting VPN up says the way I did it is the way it should be done, even your handbook says that. Or, to put it otherwise: then what is the use of the gateway-assignment in the firewall rule if the system doesn't care about it in the first place?

    As it is, it is not working for me: with a firewall rule to send only one server in my LAN (192.168.2.21) via the OpenVPN-gateway, it ignores that and sends all LAN-clients through the VPN-gateway.

    If you say that this is not a bug but a feature, then I'm losing you. Which most likely is my fault, and not yours, I might add. these faults come with my eternal noob status  :P

    EDIT: I am trying to relate the current experience to your explanation. Which is this:

    For the record: in the Firewall LAN rules you have some general rules (DNS, block netbios, etc), and then next we have the specific rule (say rule A) to direct a server on LAN (192.168.2.21) via the PIA gateway. After that, the default LAN-to-any comes (say rule B), which goes via the default gateway (so, normal VDSL, no PIAVPN).

    1. IF I put a firewall rule (say rule C) to the very top of the LAN-rules, so before all the other rules, to tell a specific LAN client (192.168.2.41) to go via the default gateway (so basically the same as rule B), it will do that. It will not go via the VPN.
    2. IF I remove that rule, the LAN-client is supposed to use the default rule B. But it doesn't do that: it goes via rule A, so via the VPN.
    3. Even IF I put the same rule C right after rule A (so I am explicitly telling the LAN-client to use the default gateway and not the VPN), it still goes via the VPN, so it uses the rule A, not the rule C and not the rule B.

    I am trying to relate this to your 'it works as designed', as it now seems the only way I can get the normal LAN client not to go via the VPN is to put an explicit 'use the default gateway' on the top of the LAN firewall rules. Which is the only place where this works.

    Would you mind sharing your thoughts on this?

    Thank you very much  ;D



  • @cmb:

    @Hollander:

    then mixes up interfaces and firewall rules, and you end up doing a fresh install for every minor upgrade;

    If that were remotely true there would be a massive abandonment of the project, tons of people complaining here, and we'd have a mess with our 500+ active commercial support customers. None of that's the case. Tens of thousands of systems have upgraded via auto-update alone in the past month or so[]

    I of course will not debate that, because what you say makes sense. However, I started with 2.0, and not a single upgrade since then (doing every upgrade since then up to 2.1.2) worked for me. Packages installing, deinstalling, reinstalling, deinstalling, and that for hours. And then WAN that suddenly had become LAN, and more of these matters. For that reason I don't upgrade anymore. Every upgrade means a fresh install for me. I don't know why it works for other people but not for me (like said, I am not disputing you), but for 4 or 5 upgrades it didn't work. As you know in science it is not the 99% that is interesting, it is the 1% that doesn't fit the model  ;)

    _PS About buggy: this morning I had another thing which I consider buggy; my VDSL broke down. As I am awaiting new, better, network cards, I currently have no fail over with my CABLE. So I simply switched the CABLE/VDSL-modem cables, so now the CABLE is in WAN. For that I had to change PPPoE to DHCP in the WAN-interface. The system first nagged me about 'you have to reassign the interfaces'.

    Huh, really? For what? For changing a network cable? Huh?

    And next refused to set the connection type for this CABLE-WAN connection to DHCP, complaining: 'a DHCP server is active on this interface, disable this first'.

    Huh, really? Where? The pfSense DHCP-server runs only on LAN and VLAN, not on WAN.

    It may be the supposed design, but you perhaps will agree with me that, for noobs (and perhaps also others) this is confusing to say the least._



  • @cmb:

    @Hollander:

    (Try to debug, look in firewall logs: rule description: "block unauthorised DNS". Port? 137. Which is a different rule to block netbios..).

    Logs strictly have rule numbers. Rule numbers change if you change your ruleset. There is no record of what the rule numbers used to be, so that's the expected behavior since you've changed your rules between when the log happened and when you clicked to see what rule hit it.

    If you allow me: this is a most strange design. Because with any one change in the firewall rule set the whole firewall log becomes useless.I have little to no serious knowledge about network security, which is why I declare myself the eternal noob in this area, but it seems, at least so they say, I have 'some experience' in huge ERP-systems (I even have an old badge here that says I am a so called 'platinum architect' (yes, I got to fly business class and the clients paid  ;D )), and if the ERP systems I helped implementing for years in my previous career would have a logic like this, none of the multinationals that use this ERP would even remotely consider a global roll out and deployment.

    Unless I am misunderstanding you of course, which is not to be ruled out for the eternal noob that I am _;D

    EDIT: to explain what I mean, from the area of ERP:

    Suppose, in MDM (Master Data Management), I have a Material Master 1 (meaning: a database record for a material, may that be a raw product, a semi-finished product or a finished product). The master has around 400 fields, one of them being 'description'. Now suppose I add a new Material Master 2, and because of this simple act, in the BOM-explosion (BOM = Bill Of Materials), suddenly the description of Material Master 1 has changed to be that of Material Master 2. Our factories would break down, because our factory workers can't seem to get a car tyre assembled into the Ipad they are currently working on ( ;D ).

    Does this help clarifying the point I am trying to make?_


  • Moderator

    I agree with you Hollander, the logs should remain in-sync after rule modifications; however the way it is now you just have to review your logs before you make rule changes. Using things like "aliases" as placeholders in rules help as you can edit the alias information without causing the logs to go out of sync.

    I believe that 2.2 will have a significant change in how the logs function.



  • This still is a problem  :-[ :'(

    It happens to appear randomly. Summary:

    [b]LAN-rules:

    • First rule: send traffic from 192.168.2.47, destination ALIAS_VPN, out via VPN-gateway.
    • Second rule: send all other traffic via the default gateway (DSL, failover with cable, so not via VPN-gateway).

    So: only a few specific sites should go via the VPN, all others should take the normal, not-VPN-way.
    What happens?
    Traffic from the 2.47 that should not go via the VPN goes via the VPN randomly. I notice that, since gmail (pop3) is not to go via the VPN, since gmail then complains that log in is not from my normal country and blocks all. I get frequent, yet random, emails that gmail has blocked my access since I appear to come from 'abroad'. So, with pop3 checking every five minutes, at least 2 or three times a day I get these gmail blocks, so at least 2-3 times a day, without me changing anything in pfSense, suddenly traffic is sent over the VPN.

    Step-2:
    So I put an explicit rule in front of all the others, telling pfSense to route all traffic from 2.47 through the VDSL line, so nothing via VPN.

    Result: everything goes via the VPN.

    Could perhaps anybody, an admin/developer, help me debug and solve this? Because this is useless: there is a reason why I have a VPN set up. If the system decides randomly on what it is doing I have no way of knowing if it is actually sending traffic that I do want sent through the VPN will actually go through the VPN.

    Please help (peep :-[ )?

    Thank you in advance very much,

    Bye,



  • Did you add the "route-nopull" I mentioned earlier? Traffic goes out via a VPN for one of two reasons - the routing table tells it to, or the firewall rules are configured as such that policy routing forces the traffic out that way. Traffic that doesn't match a firewall rule specifying the VPN gateway going across the VPN would have to be because of the routing table (or maybe it matches the policy routing rule and you don't realize it).



  • Thank you very much for replying, Sir Cmb  ;D

    (or maybe it matches the policy routing rule and you don't realize it).

    I am most sure that is not the case; the firewall rule uses an alias, and for example gmail.com is not in in that alias.

    @cmb:

    Did you add the "route-nopull" I mentioned earlier?

    Well, I tried, but it doesn't appear to help me. I addition to which I think I get an error:

    openvpn[51037]: Options error: option 'route' cannot be used in this context ([PUSH-OPTIONS])
    ```]
    
    This is what I have in the 'advanced options' in the OpenVPN-client config in the GUI:
    
    

    auth-user-pass /etc/openvpn-password.txt;
    remote-cert-tls server;
    verb 5;
    keepalive 5 30;
    route-nopull;

    
    The log is rather long, so I have pasted it in a text file and attached it to this post (I replaced my IP with [REMOVED] in that log.
    
    I'd be in much debt for help; thank you in advance very much  ;D ( :-* )
    
    Bye,
    
    [openvpnlog.txt](/public/_imported_attachments_/1/openvpnlog.txt)


  • The "Options error: option 'route' cannot be used in this context ([PUSH-OPTIONS])" log is normal when you have route-nopull, that's just OpenVPN telling you it's ignoring the pushed route(s), albeit in a confusing sounding log.

    Is there something you can do to replicate the issue? I'm curious what your routes look like under Diag>Routes at the time, and a copy of your /tmp/rules.debug file from when it's not working might be useful as well.



  • 2.2.6, still problems.

    FW rules, on top of the screen of course:

    1. 192.168.2.40 goes to geenstijl.nl -> use gateway PIA VPN.
    2. System/advanced: skip rules when gateway is down: checked.
    3. Disable PIA VPN: 2.40 happily goes to geenstijl.nl.

    The opposite is also true: all kinds of sites that are NOT in an alias to catch it via policy routing, simply go via the VPN they should not go through.

    route-nopull was added 18 months ago.

    It still seems pfsense is randomly deciding whether or not it will send traffic via the VPN; sometimes it sends traffic it shouldn't send, and vice versa.



  • Created new screen shots.

    I'm sure it is not a bug, but if somebody can explain to me why the top 1 rule in FW is not followed, I'd be obliged.












  • Have you enabled logging for each of the rules with a meaningful description so you can diagnose?

    There can be several reasons why imo.

    The route no-pull is not working. The syntax for that command can be variable so try with the space rather than hyphen.

    Also whether you are double NATing somehow. I had a replay situation at one point which meant the packets went  through the firewall twice with unexpect d results. The logging will diagnose that. My state table was in the thousands also.

    Also whether your netmasks are correct. You might intend one host but by /24 masking it you may let the whole subnet through etc. just do an audit of your netmasks for any mismatches.

    Finally that you have ipv4/v6 attention and protocols set correctly.



  • I cant be bothered to read a zillion lines of text.

    1. assign an interface to ovpn if you havent already.
    2. activate route-no-pull checkbox. If the checkbox is not there due to the ancient release you are running: enter it in adv field.

    If 1&2 dont help then post a screenshot of the routing table.

    Veel plezier.  ;)