need help fixing why i have no wan internet but have vpn internet. i buggered something up

[comet424]

[stephenw10]

Is the Resolver set in forwarding mode? Is DNSSec enabled?

As I understand it Nord recently changed the config or all their server instances to use the same config. I'm not a Nord customer though.
If that is the case then it's not possible to route between them. Otherwise try a different server until you find one using a different server IP and gateway.

That would not prevent the firewall resolving though.

[comet424]

@stephenw10
so this is what i got set. its set from the nordvpn settings.. i havent changed things in about a year.. i only change the servers keys when they stop working so i dont know if things changed..

as what i want is wan for dmz, like gaming and xbox to use wan no restrictions and then the vpn for rest of the network where its secure and no dns leakage... and fail over to work properly so if 1 vpn doesnt work it goes to the to the other one... load balancing doesnt help for me i only have like 3megabit internet

and id like my site to site openvpn to work again it works partially but then pages dont work right or times out for i guess dns issue???
so what does it mean when both vpns get the same ips... isnt that a conflict like when you give 2 computers on the same network a same static ip and just doesnt work?

this is the nordvpn how i set it up last pfsense update... and here is screen shot of that
and i find my internet doesnt always work right.. is that the ip address's the same too a clash sometimes?
[link text]https://support.nordvpn.com/hc/en-us/articles/20382523899281-pfSense-2-5-Setup-with-NordVPN(link url)

[comet424]

i think things went down hill when i tried to make everything work with the bypass policy earlier this year i think it was... and it worked for a while but i think things stopped working and then i tried to fix it and not sure if its a NAT issue or dns stuff etc as i try to fix it so vpn is secure and wan is wide open and openvpn site to site works flawless

i was thinking of starting over from scratch but i figured its gotta be like 1 check box that is probably causing the issue?

[stephenw10]

Ok so you are forwarding DNS requests which means you are only using the DNS servers configuired in General Setup which are the Nord DNS servers.

But you have as an outgoing interface in the resolver setup LAN which would not get routed over the VPN as the default route is WAN so it fails.

What do you have set in General Setup for DNS Resolution Behavior ?

[comet424]

@stephenw10
ya so the lan goes out openvpn site to site wan and vpn
the 192.168.0.x is basiclly all vpn
192.168.40.x is the dmz or wan ips
and i remember you mentioned i couldnt add the dns server 1.1.1.1 to go out the wan pppoe on the general page as it doesnt work the way i thought it worked and that caues dns leakage..

and i bet its just some simple high light or check box that makes something work or not and i didnt notice as it doesnt happen right away that i probably broke t away ago and didnt notice

as for general page here is the pics

[comet424]

and i cant remember why i highlighted the LAN i not sure if thats for site to site or for the host override page that it goes to the lancache dns server 192.168.0.33 and lancache server 192.168.0.32

[stephenw10]

Ok so what happens if you test the DNS in Diag > DNS Lookup?

I assume it fails but what error is shown? Which servers fail?

[comet424]

@stephenw10
so i did a couple google search's for the nordvpn issue i found you had mention they did a change apparently they made it so you can only have 1 vpn tunnel at 1 time.. as i found when i was un highting and re highted the 2 vpns the dns resolved through me error 10.100.0.2 that you cant bind 2 to an interface or something... so i disabled it... and then i found my version of pfsense is not up to date even though it says it is 2.7.0 found a google search i had to run "certctl rehash" and i was able to update my pfsense as my availiable packages wouldnt show up as i still have that haproxy error ...
i had to re install ha proxy but i got error from the crash report

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec  6 20:45:47 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/F

Crash report details:

PHP Errors:
[16-Sep-2024 10:48:52 America/Toronto] PHP Fatal error:  Uncaught Error: Failed opening required 'haproxy/haproxy.inc' (include_path='.:/etc/inc:/usr/local/pfSense/include:/usr/local/pfSense/include/www:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form:/usr/local/share/pear:/usr/local/share/openssl_x509_crl/') in Standard input code:4
Stack trace:
#0 {main}
  thrown in Standard input code on line 4

and the ha proxy page when i try to save it doesnt like

as for the dns lookups
it can find google and yahoo but it cant find my site to site ip
like 192.168.1.1 (pfsense box on other network) and 192.168.1.8 a server but i can do like home assistant which 192.168.1.12:8123 it will load but the page will time out and sometimes it will work the page... same with unraid it will work but the pages dont generate the information comes up like a blank page... and Pfsense itself doesnt ever load up anymore at 192.168.1.1 for whatever reason
but here some screen shots of the dns lookup

[comet424]

and i cant still ping from ping page using WAN connection
say google.. i can go through the vpn and it can resolve www.google.ca

but both WAN and VPN connection can ping 1.1.1.1 so it can go out

[comet424]

cant seem to post my reply says spam wish it would tell you what word is considered spam
and here i updated my other pfsense box the Host site of the Site To Site which i having issues accessing.. it had the same issues it was stuck on 2.7.0 so its up to date to 2.7.2 but here screen shots.. maybe something there is also conflicting i think it happened when i started doing the Bypass Policy and maybe i configured something wrong but the 192.168.1.1 is the host site for site to site and 192.168.0.1 is the client side of the site to site.. and it used to work fine in past but i think maybe sometime after the bypass policy it slowly stopped working is my guess.. its probably a nat or firewall rule...

and also on my 192.168.1.1 pfsense pinging google on the WAN doesnt work

so whatever i messed up on my 1 side i duplicated the messed up on the other pfsense box too i see

[stephenw10]

HAProxy is failing to start there because it cannot resolve homeassistant.home. Is that supposed to be an alias that is missing? If not does it resolve directly?

[comet424]

@stephenw10
i cant remember if you helped me or the john
set it up you type in homeassistant.home on the local network
and it points to 2 IP address's if 1 fails it flips over to the other one

it used to work on 2.7.0 but doesnt wanna work on this 2.7.2

and if i adjust the source ip matchs ip or alias to match host

and save and apply . it actually deletes it.. and then you dont have like 20 options to choose from for Expression you only get 4 options.. and then you still get a HA proxy error for a stats file directory not found or something

it also had worked a while ago.. i just havent noticed issues till the site to site issues i having not able to access the network like i used to be able to like i was right there.. and then i seen multiple issues as i usually just set it and forget it kinda thing for pfsense as it works and i dont adjust it

[stephenw10]

I would expect the destination to be homeassistant.home if that ACL is matching incomig traffic for load-balancing. The source could be anything.

[comet424]

@stephenw10 all i know is its for fail over not for load balancing
so when 192.168.0.12 goes down 192.168.0.10 is then accessable at home on the network but for whatever reason thats an issue now too lol seems
its spiraled into 3 issues now right..
1 is the site to site issue
2. ha proxy not running right anymore
3rd wan cant dns'
4th was the nordvpn tunnel vpn issues
5th pfsense was stuck on 2.7.0 and never updated

so i guess thats 5 issues not 3... so far only numbers 4 and 5 i fixed well 4th i just went to 1 vpn instead of a fail over since nordvpn no longer can do fail over
so numbers 1-3 still issues and i guess all that happened in one shot i have no idea...

is the Haproxy easy to fix as i cant use my 192.168.0.1:8123 i cant remember if i could access it as homeassistant.home anymore which would go directly to 192.168.0.12 but when i took it offline then 192.168.0.10 took over
how can i fix it for 2.7.2 what be the proper way least i can fix that and get that outta of the way to get back on track with this wan and site to site issue

[comet424]

oh ya and the here is the front end... i used to access home assistant by
192.168.0.1:8123

[stephenw10]

To be able to access it by homeassistant.home that needs to resolve to the front end IP which appears to be the LAN IP. So you must have added that somewhere. Probably as a host override in the DNS resolver.

That ACL makes no sense. It cannot match traffic as nothing will be coming from homeassistant.home. Unless you have an alias using that name containing your clients IP, which I assume is also in the LAN.

[comet424]

@stephenw10 ok i fixed it i checked the host override.. for whatever reason the host part went missing all i had was domain home and pointed to 192.168.0.1 forwhatever reason the homeassistant part was gone..
i fixed that and the ha proxy back up and running so that solves that issue now those 3 out of 5..

now still cant seem to solve the issue no internet on WAN well no dns resolving unless i need to highlight it too in the resolver but then doesnt that leak dns?

and with the site to site where pages are not loading right from the Host pfsense site to site can you see anything set up wrong.. in the nats that or rules that could be blocking

like i can get get access to like a Docker of Firefox off unraid so 192.168.1.8:3000 is KVM of firefox and i can access like the pfsense but i cant access 192.168.1.8:3000 always or it will time out.. too
is there a test to see if i loosing packets or if dns is conflicting or is it a NAT issue
least i getting this issues solved i appreciate the help so far.. cuz i would been stuck why the ha was still not working so i appreciate that always learning

[stephenw10]

Ok one thing at a time.

What exactly is not resolving via WAN? The pfSense box itself? Clients that are policy routed via WAN?

Your DNS config for pfSense is setup to only use the VPN but that should work as long as those DNS servers are responding.

Try manually querying those DNS servers from a client on the LAN. Try from a client that is routed over the VPN and one that isn't policy routed.

If the VPN DNS servers are not responding to pfSense you might be able to browse from a client still if they are being passed the DNS servers to use directly via DHCP. That would fail for clients routed via WAN.

[comet424]

@stephenw10 ok sorry

so right now vpn works

but the WAN doesnt work through pfsense ping like ping google.ca doesnt work but ping 1.1.1.1 works

what also doesnt work is my dmz which is 192.168.40.x it uses the WAN
ppoe for whatever reason it uses the nordvpn dns servers i did a dnsleaktest... and when i disconnect from the openvpn for nordvpn.. my WAN connection on the DMZ goes down yet it should use the Bell Canada Internet dns or like 1.1.1.1 as a dns so i not sure where to fix that

so here is i turned off my vpn and i just took 2 screen shots but here its not working the internet on the WAN side clients
the dnsleaktest screen shot is showing its using nordvpn dns on the wanppoe connection and the dmz clients point to 192.168.40.1
so i not sure if something buggered up thinking about how i read packages didnt update and pfsense stuck on 2.7.0 and i had to run that command to make it updateable and my host override was missing homeassistant in the hostname i wonder if it got corrupted... some how the config file...

but ya here is some screen shots maybe u can tell me where to look to try to fix
sorry if im rambling