need help fixing why i have no wan internet but have vpn internet. i buggered something up

[comet424]

@stephenw10
so i did a couple google search's for the nordvpn issue i found you had mention they did a change apparently they made it so you can only have 1 vpn tunnel at 1 time.. as i found when i was un highting and re highted the 2 vpns the dns resolved through me error 10.100.0.2 that you cant bind 2 to an interface or something... so i disabled it... and then i found my version of pfsense is not up to date even though it says it is 2.7.0 found a google search i had to run "certctl rehash" and i was able to update my pfsense as my availiable packages wouldnt show up as i still have that haproxy error ...
i had to re install ha proxy but i got error from the crash report

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec  6 20:45:47 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/F

Crash report details:

PHP Errors:
[16-Sep-2024 10:48:52 America/Toronto] PHP Fatal error:  Uncaught Error: Failed opening required 'haproxy/haproxy.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in Standard input code:4
Stack trace:
#0 {main}
  thrown in Standard input code on line 4

and the ha proxy page when i try to save it doesnt like

as for the dns lookups
it can find google and yahoo but it cant find my site to site ip
like 192.168.1.1 (pfsense box on other network) and 192.168.1.8 a server but i can do like home assistant which 192.168.1.12:8123 it will load but the page will time out and sometimes it will work the page... same with unraid it will work but the pages dont generate the information comes up like a blank page... and Pfsense itself doesnt ever load up anymore at 192.168.1.1 for whatever reason
but here some screen shots of the dns lookup

[comet424]

and i cant still ping from ping page using WAN connection
say google.. i can go through the vpn and it can resolve www.google.ca

but both WAN and VPN connection can ping 1.1.1.1 so it can go out

[comet424]

cant seem to post my reply says spam wish it would tell you what word is considered spam
and here i updated my other pfsense box the Host site of the Site To Site which i having issues accessing.. it had the same issues it was stuck on 2.7.0 so its up to date to 2.7.2 but here screen shots.. maybe something there is also conflicting i think it happened when i started doing the Bypass Policy and maybe i configured something wrong but the 192.168.1.1 is the host site for site to site and 192.168.0.1 is the client side of the site to site.. and it used to work fine in past but i think maybe sometime after the bypass policy it slowly stopped working is my guess.. its probably a nat or firewall rule...

and also on my 192.168.1.1 pfsense pinging google on the WAN doesnt work

so whatever i messed up on my 1 side i duplicated the messed up on the other pfsense box too i see

[stephenw10]

HAProxy is failing to start there because it cannot resolve homeassistant.home. Is that supposed to be an alias that is missing? If not does it resolve directly?

[comet424]

@stephenw10
i cant remember if you helped me or the john
set it up you type in homeassistant.home on the local network
and it points to 2 IP address's if 1 fails it flips over to the other one

it used to work on 2.7.0 but doesnt wanna work on this 2.7.2

and if i adjust the source ip matchs ip or alias to match host

and save and apply . it actually deletes it.. and then you dont have like 20 options to choose from for Expression you only get 4 options.. and then you still get a HA proxy error for a stats file directory not found or something

it also had worked a while ago.. i just havent noticed issues till the site to site issues i having not able to access the network like i used to be able to like i was right there.. and then i seen multiple issues as i usually just set it and forget it kinda thing for pfsense as it works and i dont adjust it

[stephenw10]

I would expect the destination to be homeassistant.home if that ACL is matching incomig traffic for load-balancing. The source could be anything.

[comet424]

@stephenw10 all i know is its for fail over not for load balancing
so when 192.168.0.12 goes down 192.168.0.10 is then accessable at home on the network but for whatever reason thats an issue now too lol seems
its spiraled into 3 issues now right..
1 is the site to site issue
2. ha proxy not running right anymore
3rd wan cant dns'
4th was the nordvpn tunnel vpn issues
5th pfsense was stuck on 2.7.0 and never updated

so i guess thats 5 issues not 3... so far only numbers 4 and 5 i fixed well 4th i just went to 1 vpn instead of a fail over since nordvpn no longer can do fail over
so numbers 1-3 still issues and i guess all that happened in one shot i have no idea...

is the Haproxy easy to fix as i cant use my 192.168.0.1:8123 i cant remember if i could access it as homeassistant.home anymore which would go directly to 192.168.0.12 but when i took it offline then 192.168.0.10 took over
how can i fix it for 2.7.2 what be the proper way least i can fix that and get that outta of the way to get back on track with this wan and site to site issue

[comet424]

oh ya and the here is the front end... i used to access home assistant by
192.168.0.1:8123

[stephenw10]

To be able to access it by homeassistant.home that needs to resolve to the front end IP which appears to be the LAN IP. So you must have added that somewhere. Probably as a host override in the DNS resolver.

That ACL makes no sense. It cannot match traffic as nothing will be coming from homeassistant.home. Unless you have an alias using that name containing your clients IP, which I assume is also in the LAN.

[comet424]

@stephenw10 ok i fixed it i checked the host override.. for whatever reason the host part went missing all i had was domain home and pointed to 192.168.0.1 forwhatever reason the homeassistant part was gone..
i fixed that and the ha proxy back up and running so that solves that issue now those 3 out of 5..

now still cant seem to solve the issue no internet on WAN well no dns resolving unless i need to highlight it too in the resolver but then doesnt that leak dns?

and with the site to site where pages are not loading right from the Host pfsense site to site can you see anything set up wrong.. in the nats that or rules that could be blocking

like i can get get access to like a Docker of Firefox off unraid so 192.168.1.8:3000 is KVM of firefox and i can access like the pfsense but i cant access 192.168.1.8:3000 always or it will time out.. too
is there a test to see if i loosing packets or if dns is conflicting or is it a NAT issue
least i getting this issues solved i appreciate the help so far.. cuz i would been stuck why the ha was still not working so i appreciate that always learning

[stephenw10]

Ok one thing at a time.

What exactly is not resolving via WAN? The pfSense box itself? Clients that are policy routed via WAN?

Your DNS config for pfSense is setup to only use the VPN but that should work as long as those DNS servers are responding.

Try manually querying those DNS servers from a client on the LAN. Try from a client that is routed over the VPN and one that isn't policy routed.

If the VPN DNS servers are not responding to pfSense you might be able to browse from a client still if they are being passed the DNS servers to use directly via DHCP. That would fail for clients routed via WAN.

[comet424]

@stephenw10 ok sorry

so right now vpn works

but the WAN doesnt work through pfsense ping like ping google.ca doesnt work but ping 1.1.1.1 works

what also doesnt work is my dmz which is 192.168.40.x it uses the WAN
ppoe for whatever reason it uses the nordvpn dns servers i did a dnsleaktest... and when i disconnect from the openvpn for nordvpn.. my WAN connection on the DMZ goes down yet it should use the Bell Canada Internet dns or like 1.1.1.1 as a dns so i not sure where to fix that

so here is i turned off my vpn and i just took 2 screen shots but here its not working the internet on the WAN side clients
the dnsleaktest screen shot is showing its using nordvpn dns on the wanppoe connection and the dmz clients point to 192.168.40.1
so i not sure if something buggered up thinking about how i read packages didnt update and pfsense stuck on 2.7.0 and i had to run that command to make it updateable and my host override was missing homeassistant in the hostname i wonder if it got corrupted... some how the config file...

but ya here is some screen shots maybe u can tell me where to look to try to fix
sorry if im rambling

[comet424]

so what i want is
WAN to use like 1.1.1.1 as a dns or the bell modem
VPN to use the 2 DNS's it has
and i dont want the WAN dns leaking into the VPN

but if i add like 1.1.1.1 to pppoe wan under the General Tab dns
it always leaks that dns cloudfare into the VPN so then its not a secure vpn tunnel

so i wanna make sure they go out there seperate ways and secure... unless you gotta run like 2 pfsense boxs really 1 for WAN and 1 for vpn to really accomplish it

[stephenw10]

Do you want pfSense itself to use only the VPN? That is how you have it setup currently. For DNS at least.

What DNS servers are you passing to clients via DHCP in each interface?

What you probably want to do here is pass the Nord DNS servers directly to clients on the interfaces that need to use it then force all that traffic via the VPN. Then you can set the DNS in pfSense back to the default values so it works as expected.

[comet424]

@stephenw10
what i thought i had and i saw another glitch in the NAT the "openvpn address " gone missing its now goes under openvpnip as an alias. i not sure if that command i ran to make pfsense upgradable able to 2.7.2 buggered it all up

but what i thought i had was
WAN for Pfsense so it will always work. and update itself. and for
DMZ and the small amount of NoVPN (alias) ips that would use the WAN connection

the rest uses VPN thats why i did the Bypass policy its supposed to seperate so NoVPN ips including Pfsense itself its supposed to use WAN and use the WAN dns and the VPNips alias you see that uses the NordVPN gateway... so its supposed to fork itself and stay seperated no no leakage over and then all the lan network can access the OpenVPN connection...

thats what i thought i had but i see its glitched more in NAT ...
always something lol

but ya Pfsense and NoVPN Ips supposed to use the WAN and Wan DNS
VPN Ip Clients use the Nord VPN and Its DNS

but for whatever reason WAN wont work for Pfsense firewall to ping gooogle but can ping 1.1.1.1 but WAN clients can ping and goto www.google.ca but for some reason its using the nordvpn dns's i dont know how that works or where it wan is using the nordvpns dns not its own

its too bad pfsense cant have multiple dns resolvers.. like 1 for your specficc VPN and 1 for the WAN and then it could keep it seperate... or least in a wizard if its already possible..

sorry i rambling and hope i didnt confuse you.. makes sense to me so bear with me

[stephenw10]

Ok so an important thing here is that the route traffic takes and the route DNS requests take are potentially controlled by different things.

Traffic from pfSense itself always follows the system routing table. That means most traffic uses the default route / gateway. Currently you have that set to the WAN.

DNS queries can be from clients directly, in which case they follow, policy routing rules. Or if clients are using pfSense for DNS then it depends how the DNS resolver is configured. In your setup it is set to forwarding mode so it uses the DNS server defined in System > General Setup. Those are set to the NordVPN servers and with static routes via the VPN gateways.

DNS for pfSense itself is set to use the resolver and fall back to the defined servers but in forwarding that's the same. So there are no DNS requests going out of the WAN from pfSense itself.

What DNS servers are you passing to clients via DHCP?

[comet424]

@stephenw10 ok i think i understand.. i had to re read it a few times.. and how do you make it work for WAN for pfsense itself?

so ive tried a couple ways
first all the LAN, IoT,Cameras,DMZ,
they all point to pfsense for dns
so 192.168.0.1 for LAN
192.168.10.1 for Cameras
192.168.20.1 for Iot
192.168.40.1 for DMZ

and some ips on the LAN went to WAN and rest went to VPN
i do this so i could also use PFBlocker... but i find the Host override isnt always working right for Lancache.

so what ive also tried is dns
192.168.0.33
that is the dns server for the Lancache Server
and then it points to 192.168.0.32 which is lancache server itself
and the person that made that lancache said you cant use 192.168.0.1 as the uplink dns as it gets stuck in a loop i guess???
so ive used 1.1.1.1 and the nordvpn dns to try

reason i use the lancache server is my internet is only like 3megabit i live in rural and pay 100 bucks a month for it slow ass.. so i use it so it can backup my other comps or when i need to re install a game.. so it saves the windows updates.. or its supposed to when the host over ride but for some reason i dunno just doesnt work 100% when it points 192.168.0.32 lancache server ive never figured that issue

so i go back and forth from 192.168.0.1 and 192.168.0.33 just to try what works better etc. i never figured how to make it work perfect and just point to 192.168.0.1 or i dont know what the Best Practice what you professionals would do in a business deployment would do how you would plumb it or i mean route things

[stephenw10]

pfSense should be able to use Nords DNS server(s). That that seems to be failing is probably the issue here.

What results do you see when you try to resolve pfsense.org in Diag > DNS Lookup ?

If any are errors what are they?

[comet424]

@stephenw10
i find nordvpn goes down on me either its due to my 3megabit internet connection that its just slow as mollases ... so thats why i wanted pfsense itself to use the pppoe wan connection least it bypass's the vpn
so i not sure what best practices is and with what i mentioned with the lancache is there is a better way for my to set it up like professional done so it works smoother.. id take all the help... and then if my NAT is messed up id need help fixing it too and toss out what i dont need

but here is the dnslookup of pfsense.org

i know i have it set to default to use ppoe gateway.. but shows its using the nordvpn i get confused why its not taking the bell internet dns?

i was thinking also since there has been a corruption between 2.7.0 2.7.2 where i noticed some things are outta place and nat is address messed up i might import 1 piece at a time and adjust slightly since things i found werent pointing right direction things missing like that homessasistant missing from the host override part of it was gone

and here is the routing... screen shot i enabled the site to site not sure why it was disabled... and then it re enabled itself on NAT side... i think its halloween ghosts in my pfsense lol

[comet424]

found another ghost i having or gremlin bet you never seen this before


i might try tomorrow what i was mentioning start fresh and import some of the things and enter manually others
i have a lot more of those i see those just some