multiple machines some sites won't load first time especially google sites

rterren

Here's my setup and a little background. I'm new to learning firewalls, pfsense is my first one, as a consequence I keep blowing up the home fios connection and my family has threatened to kill me, so what i've done is leave the fios router plugged in and i've plugged my pfsense in as a dmz'd dhcp client on the fios router.

Using two separate win 10 machines plugged in directly to the fios router, I successfully tested their connectivity against various sites websites

when I plug into my pfsense and try to access those same sites it seems most work 99% except for google sites. google.com, mail.google.com, gmail.com, drive.google.com, docs.google.com. The google sites consistently fail to load until i refresh the browsers (chrome and firefox), sometimes I need to refresh 2 and 3 times.

iv'e tried leaving the pfsense wan dns server entries blank, I've also tried setting them to several different dns addresses including 8.8.8.8, 8.8.4.4, 1.1.1.1, 198.153.194.1, 204.117.214.10, 129.250.35.250.

In chrome I've tried clearing the cache, clearing the HSTS cache, i've reset my network connection by running

arp -d *
netsh int ip reset
netsh winsock reset
ipconfig /flushdns
ipconfig /release
ipconfig /renew

and rebooting both the pfsense and computers multiple times. I've restored to factory defaults, restored backups, nothing seems to work.

nslookup yields odd results, right now its failing to resolve google sites

nslookup
Default Server: pflookup.local
Address: 192.168.35.1

food.net
Server: pflookup.local
Address: 192.168.35.1

Non-authoritative answer:
Name: food.net
Address: 3.235.229.168

food.com
Server: pflookup.local
Address: 192.168.35.1

Non-authoritative answer:
Name: food.com
Addresses: 52.20.42.213
35.175.52.53

google.com
Server: pflookup.local
Address: 192.168.35.1

*** pflookup.local can't find google.com: Server failed

docs.google.com
Server: pflookup.local
Address: 192.168.35.1

*** pflookup.local can't find docs.google.com: Server failed

drive.google.com
Server: pflookup.local
Address: 192.168.35.1

*** pflookup.local can't find drive.google.com: Server failed

task.com
Server: pflookup.local
Address: 192.168.35.1

Non-authoritative answer:
Name: task.com
Addresses: 165.160.15.20
165.160.13.20

cnn.com
Server: pflookup.local
Address: 192.168.35.1

Non-authoritative answer:
Name: cnn.com
Addresses: 2a04:4e42::323
2a04:4e42:400::323
2a04:4e42:600::323
2a04:4e42:200::323
151.101.1.67
151.101.129.67
151.101.193.67
151.101.65.67

but 30 seconds later it works

google.com
Server: pflookup.local
Address: 192.168.35.1

Non-authoritative answer:
Name: google.com
Addresses: 2607:f8b0:4006:806::200e
142.250.64.78

drive.google.com
Server: pflookup.local
Address: 192.168.35.1

Non-authoritative answer:
Name: drive.google.com
Addresses: 2607:f8b0:4006:810::200e
172.217.10.78

docs.google.com
Server: pflookup.local
Address: 192.168.35.1

Non-authoritative answer:
Name: docs.google.com
Addresses: 2607:f8b0:4006:818::200e
172.217.3.110

I've changed my dhcp range, i briefly tried dns forwarding but i screwed that all up and took down the whole network in the house, even things not connected to the pfsense

in the fios router here are my wan dns servers

71.243.0.12
68.237.161.12

I'm running 2.4.5-RELEASE-p1 (amd64)
built on Tue Jun 02 17:51:54 EDT 2020
FreeBSD 11.3-STABLE

any clue what im doing wrong?

thank you
Ron

rterren

I have 2 theories on my problem.

1> it appears as though my fios is not passing ipv6 traffic and i believe the google dns prefers ipv6

2> double nat causing issues

Gertjan

@rterren said in multiple machines some sites won't load first time especially google sites:

1> it appears as though my fios is not passing ipv6 traffic and i believe the google dns prefers ipv6

nslookup will ask the DNS (your pfSense = 192.168.35.1) both an A and AAAA record for a zone.
This will work fine over a IPv4 and/or a IPv6.

@rterren said in multiple machines some sites won't load first time especially google sites:

2> double nat causing issues

You NATted some ports(s) in the FIOS and pfSense.
These NAT ruls only get used for incoming traffic, does not impact outgoing DNS traffic.

By default, pfSense will resolve. This means it doesn't use upstream DNS resolvers or forwards like the ones from your ISP, or even commercial " 8.8.8.8, 8.8.4.4, 1.1.1.1, 198.153.194.1, 204.117.214.10, 129.250.35.250" (they use/sell your request data).
The resolver contacts directly the main Root name servers. These servers know all about everybody, because they are the "name part" of the domains names on the Internet. The nice thing about them is : no set-up is needed, it will work out of the box. I advise you strongly not to change any DNS settings, neither entering any "DNS" IP manually.
So :
@rterren said in multiple machines some sites won't load first time especially google sites:

iv'e tried leaving the pfsense wan dns server entries blank,

doesn't need any effort. Just do nothing, and you'll be fine.

On the General Setup page, these are the perfect settings :

Even when when you hook up pfSense behind some other (ISP) router, your FIOS, it will obtain an IP on it's WAN interface as any other device (printer, PC, etc) you hooked up on the FIOS LAN. It will be an RFC1918 of course, an IP, from the LAN of the FIOS router..
Just be sure that that WAN (pfSEnse) IP - network doesn't conflict with the pfSense LAN network.

You got this wrong :

@rterren said in multiple machines some sites won't load first time especially google sites:

as a consequence I keep blowing up the home fios connection

normally, you should apply @home settings that are known to work.
And when you make changes, be ready to
Test - be able to ask the "what if ?" question and go back if needed.
Use a backup of your config if needed.

Fooling around with pfSense should be done @work. When things go down @work, you're not risking your live. It's the other way around : they'll pay you more when things go wrong.

edit : I forgot to mention the reason why sometimes resolving works, and sometimes it doesn't.
When multiple DNS servers are entered on the General settings page, they are used on a round robin base.
If one of them doesn't work out, there will be no answer. The next request will use the next DNS in line, and have an answer.
When you enter nothing here, the default 127.0.0.1 will get used (by pfSEnse itself == the resolver and the same resolver will also serve all pfSense LAN based devices). That will always work. That is, as long as the resolver has a free access to the main 13 Internet roots servers.

Btw : your LAN devices should always stay in the "network" mode as they were when you bought them : DHCP activated, no static settings. A PC today, with a default W10, never needs any user intervention. If it does, you already have network issues.
If you prefer that some LAN based devices always have the same IP, set up a static MAC lease for that PC on pfSense.

Last but not least : pfSense is a router / firewall , pretty identical to any other router. There is no "Networking the pfSense way". All the SOHO firewall routers are the same. pfSense has more functionalities, which could make you think it is more complex. That's not true : just don't use the features you don't need / don't understand, and you'll be fine.

rterren

@Gertjan Wow thats an awesome write up thanks for all of it. I decided to just completly remove the FIOS router and everything is working great now. I would have done it @work but I don't have a job, thanks covid.. lol