pfsense blocking certain/some sites

stephenw10

Mmm, it's just this one VM.

Still does it with DNSSec disabled...

johnpoz

@stephenw10 what I can tell what is wrong with their ns, is they do not answer via tcp

So normal via udp works fine.

So like to see a +trace from pfsense

; <<>> DiG 9.16.26 <<>> portal.bsnl.in +trace
;; global options: +cmd
.                       45761   IN      NS      j.root-servers.net.
.                       45761   IN      NS      k.root-servers.net.
.                       45761   IN      NS      l.root-servers.net.
.                       45761   IN      NS      m.root-servers.net.
.                       45761   IN      NS      a.root-servers.net.
.                       45761   IN      NS      b.root-servers.net.
.                       45761   IN      NS      c.root-servers.net.
.                       45761   IN      NS      d.root-servers.net.
.                       45761   IN      NS      e.root-servers.net.
.                       45761   IN      NS      f.root-servers.net.
.                       45761   IN      NS      g.root-servers.net.
.                       45761   IN      NS      h.root-servers.net.
.                       45761   IN      NS      i.root-servers.net.
.                       45761   IN      RRSIG   NS 8 0 518400 20221016050000 20221003040000 18733 . YIXaa/EBSQVICUNPRhTRK21PwpQy6pk6zgrYeokFCUG6pPKmfn+7gOiq k12OWXOTYRguXIWv0YauJlYZlRJFOucvxIWI2hE8oeppc5bCDBXUwZ2V 6GDOEYnCkk/8Bh7QgaAGpBYeNbuPj2TD1bDX1dHKOZ/PIOoXeSxAOuAi xkZzEi4/zXqDWmeDA7CVq74qNvVgfkVg0NXDxqFtmJH/cXwvdGsWbeaZ gu95le0xD12RbYGoxfzM06DT4YLJMPJ4evH26D2xnUolBqZ9tbqjAxcv AdnAllbVw5AcuaYQMCqn3qy/x+M4rJKmExFughKCvnZWXxTlGcZDRDt1 0VFw0g==
;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

in.                     172800  IN      NS      ns1.registry.in.
in.                     172800  IN      NS      ns2.registry.in.
in.                     172800  IN      NS      ns3.registry.in.
in.                     172800  IN      NS      ns4.registry.in.
in.                     172800  IN      NS      ns5.registry.in.
in.                     172800  IN      NS      ns6.registry.in.
in.                     86400   IN      DS      54739 8 1 2B5CA455A0E65769FF9DF9E75EC40EE1EC1CDCA9
in.                     86400   IN      DS      54739 8 2 9F122CFD6604AE6DEDA0FE09F27BE340A318F06AFAC11714A73409D4 3136472C
in.                     86400   IN      RRSIG   DS 8 1 86400 20221016170000 20221003160000 18733 . MH2IwInVoatMPeKOq084SdgHwlSAZxwSKLePZKNixFq/k5B9sjPwTPg2 sD9QebL9yV/nXQQkwouIpWrIk825ZZYSu+jqfPqX+orjMzlD1Md1EVZc TqWf+JqTTmMzGGnocx7ZswBFhTAXn5/g3enPXZqUyyvaxTVJ3QpWe7TQ ZAvK0hVSWRqcYaCJTyblVRB7X64DgiTuU5JBRVSVqcsqGtN2YIPZETlQ Y2deLx2TsaiDhF1YMKUfGVrji9/N3wGn90FGKNXPEOuLxmf4n/tshoaK 0CzachAt5++rERjalNoZjKCBmFF1o2eRi8DCD5Uqi4+qyeHvRTtJrr6d 48Txwg==
;; Received 795 bytes from 198.97.190.53#53(h.root-servers.net) in 58 ms

bsnl.in.                86400   IN      NS      ns11.bsnl.in.
bsnl.in.                86400   IN      NS      ns12.bsnl.in.
u7smslveus494o8dr4h483un5spuc1tu.in. 1800 IN NSEC3 1 1 0 - U7T80A19T7AQCC0P8AMD1AC4SCNB2DG5 NS SOA RRSIG DNSKEY NSEC3PARAM TYPE65534
u7smslveus494o8dr4h483un5spuc1tu.in. 1800 IN RRSIG NSEC3 8 2 1800 20221029024156 20220929023509 65169 in. OhLJIY+hNU2Iba31vmFAZmg83NwqnSy5kTbfU8cZYFG663HzbGHhdv/K GuGaRoYkqyEPpWfBF/VbAKHWi6F9fIGPR2+P2rKgD2eCzcuttKmq9bhX 4uHehoh+Qr06klPyF+TGp/iQvxyKJIMX0c/AFM2bbG4y/D7qO/5j0cK8 qheSA/XC8aOj/yRrY23Q84506B9plijHJfG3M+/T5qBjCA==
cpcirneso3q726baurorn492qjc704f7.in. 1800 IN NSEC3 1 1 0 - CPDC4IU515A25D00VQOT9RS8DOGC39NO NS DS RRSIG
cpcirneso3q726baurorn492qjc704f7.in. 1800 IN RRSIG NSEC3 8 2 1800 20221029024559 20220929023325 65169 in. d+tE+NTWj1j/jbF2vO1vjtwPcxNDdJFFk2VWc3ijj6q/utOfqL/wtZUv tZd6ofRu+M0SHvxGzjJcZpiqMf9HaMOkGKLXfXO1sohlJLqNuQgs4RTr 9VjO1qnfnXZNkSP2aDP9KdnKcwcHHQv4cR6J5hPi7XOaURTIM3kI5YkC yq4rdXQIxtkWC0D+aOUP+mpHrm4+27qbSbYoqOCDDRE9+Q==
;; Received 724 bytes from 156.154.100.20#53(ns5.registry.in) in 58 ms

portal.bsnl.in.         10800   IN      A       117.255.216.68
portal.bsnl.in.         10800   IN      RRSIG   A 7 3 10800 20221010221608 20221003211608 51428 bsnl.in. Vlc2csKOp69KSqiKUQl6iIAzgycNTMj1Oj+84dyYtjatWlBHMvtjkUMK XjhfLoI4RVZkaZgd20KNKddNKwId8Qs+kOH0fYSS4jAkEB+llzt5pOdN 8jYweG5dLFjgZmH67oUDLEjemO7PQiWduPOB7tXU5NukoKqjpD1HtL7m 8qI=
bsnl.in.                10800   IN      NS      ns12.bsnl.in.
bsnl.in.                10800   IN      NS      ns11.bsnl.in.
bsnl.in.                10800   IN      RRSIG   NS 7 2 10800 20221010221602 20221003211602 51428 bsnl.in. XGHAXve5mEGouSP3gISD3XJp3lQnQsk+qSdzm2UHsOlEcvNj0kyNwRl/ 1etqIKNnzByXhh3spngJdOlyMvsrlZfodsviJ/6v3VzlmJoawlUZuLov UddqQmq15Xnj7S3Hi5xPq8rTXIAXvqGSpjUifZDCFlUcmY89iTwpI9Sb FAo=
;; Received 797 bytes from 218.248.240.209#53(ns12.bsnl.in) in 334 ms

But notice when you try it via tcp


[22.05-RELEASE][admin@sg4860.local.lan]/root: dig portal.bsnl.in +trace +tcp

; <<>> DiG 9.16.26 <<>> portal.bsnl.in +trace +tcp
;; global options: +cmd
.                       45742   IN      NS      l.root-servers.net.
.                       45742   IN      NS      m.root-servers.net.
.                       45742   IN      NS      a.root-servers.net.
.                       45742   IN      NS      b.root-servers.net.
.                       45742   IN      NS      c.root-servers.net.
.                       45742   IN      NS      d.root-servers.net.
.                       45742   IN      NS      e.root-servers.net.
.                       45742   IN      NS      f.root-servers.net.
.                       45742   IN      NS      g.root-servers.net.
.                       45742   IN      NS      h.root-servers.net.
.                       45742   IN      NS      i.root-servers.net.
.                       45742   IN      NS      j.root-servers.net.
.                       45742   IN      NS      k.root-servers.net.
.                       45742   IN      RRSIG   NS 8 0 518400 20221016050000 20221003040000 18733 . YIXaa/EBSQVICUNPRhTRK21PwpQy6pk6zgrYeokFCUG6pPKmfn+7gOiq k12OWXOTYRguXIWv0YauJlYZlRJFOucvxIWI2hE8oeppc5bCDBXUwZ2V 6GDOEYnCkk/8Bh7QgaAGpBYeNbuPj2TD1bDX1dHKOZ/PIOoXeSxAOuAi xkZzEi4/zXqDWmeDA7CVq74qNvVgfkVg0NXDxqFtmJH/cXwvdGsWbeaZ gu95le0xD12RbYGoxfzM06DT4YLJMPJ4evH26D2xnUolBqZ9tbqjAxcv AdnAllbVw5AcuaYQMCqn3qy/x+M4rJKmExFughKCvnZWXxTlGcZDRDt1 0VFw0g==
;; Received 525 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

in.                     172800  IN      NS      ns1.registry.in.
in.                     172800  IN      NS      ns2.registry.in.
in.                     172800  IN      NS      ns3.registry.in.
in.                     172800  IN      NS      ns4.registry.in.
in.                     172800  IN      NS      ns5.registry.in.
in.                     172800  IN      NS      ns6.registry.in.
in.                     86400   IN      DS      54739 8 1 2B5CA455A0E65769FF9DF9E75EC40EE1EC1CDCA9
in.                     86400   IN      DS      54739 8 2 9F122CFD6604AE6DEDA0FE09F27BE340A318F06AFAC11714A73409D4 3136472C
in.                     86400   IN      RRSIG   DS 8 1 86400 20221016170000 20221003160000 18733 . MH2IwInVoatMPeKOq084SdgHwlSAZxwSKLePZKNixFq/k5B9sjPwTPg2 sD9QebL9yV/nXQQkwouIpWrIk825ZZYSu+jqfPqX+orjMzlD1Md1EVZc TqWf+JqTTmMzGGnocx7ZswBFhTAXn5/g3enPXZqUyyvaxTVJ3QpWe7TQ ZAvK0hVSWRqcYaCJTyblVRB7X64DgiTuU5JBRVSVqcsqGtN2YIPZETlQ Y2deLx2TsaiDhF1YMKUfGVrji9/N3wGn90FGKNXPEOuLxmf4n/tshoaK 0CzachAt5++rERjalNoZjKCBmFF1o2eRi8DCD5Uqi4+qyeHvRTtJrr6d 48Txwg==
;; Received 795 bytes from 193.0.14.129#53(k.root-servers.net) in 42 ms

bsnl.in.                86400   IN      NS      ns12.bsnl.in.
bsnl.in.                86400   IN      NS      ns11.bsnl.in.
u7smslveus494o8dr4h483un5spuc1tu.in. 1800 IN NSEC3 1 1 0 - U7T80A19T7AQCC0P8AMD1AC4SCNB2DG5 NS SOA RRSIG DNSKEY NSEC3PARAM TYPE65534
u7smslveus494o8dr4h483un5spuc1tu.in. 1800 IN RRSIG NSEC3 8 2 1800 20221029024156 20220929023509 65169 in. OhLJIY+hNU2Iba31vmFAZmg83NwqnSy5kTbfU8cZYFG663HzbGHhdv/K GuGaRoYkqyEPpWfBF/VbAKHWi6F9fIGPR2+P2rKgD2eCzcuttKmq9bhX 4uHehoh+Qr06klPyF+TGp/iQvxyKJIMX0c/AFM2bbG4y/D7qO/5j0cK8 qheSA/XC8aOj/yRrY23Q84506B9plijHJfG3M+/T5qBjCA==
cpcirneso3q726baurorn492qjc704f7.in. 1800 IN NSEC3 1 1 0 - CPDC4IU515A25D00VQOT9RS8DOGC39NO NS DS RRSIG
cpcirneso3q726baurorn492qjc704f7.in. 1800 IN RRSIG NSEC3 8 2 1800 20221029024559 20220929023325 65169 in. d+tE+NTWj1j/jbF2vO1vjtwPcxNDdJFFk2VWc3ijj6q/utOfqL/wtZUv tZd6ofRu+M0SHvxGzjJcZpiqMf9HaMOkGKLXfXO1sohlJLqNuQgs4RTr 9VjO1qnfnXZNkSP2aDP9KdnKcwcHHQv4cR6J5hPi7XOaURTIM3kI5YkC yq4rdXQIxtkWC0D+aOUP+mpHrm4+27qbSbYoqOCDDRE9+Q==
;; Received 724 bytes from 2001:502:2eda::20#53(ns5.registry.in) in 54 ms

;; Connection to 218.248.240.178#53(218.248.240.178) for portal.bsnl.in failed: connection refused.
;; Connection to 218.248.240.209#53(218.248.240.209) for portal.bsnl.in failed: connection refused.
[22.05-RELEASE][admin@sg4860.local.lan]/root:

And you get the same warning here
https://dnsviz.net/d/portal.bsnl.in/dnssec/

bingo600

@johnpoz
Nice detective work

Now i have to read up on the various dig features

/Bingo

johnpoz

@bingo600 why he would be trying tcp and not udp, not sure. But not answering tcp can cause issues for sure..

Sometimes it is just crazy to me the lack of any understanding of dns for companies that provide servers like websites.. And control their own dns.

Notice the ns are ns11 and ns12 bsnl.in its not like they are having their dns hosted by someone - they have registered their own name servers..

If your going to to do - please have some clue to how to setup and manage dns ;) And if your going to use dnssec - read the rfcs for gosh sake ;) Notice the 11 warnings where they are using the the wrong algo.

Notice this sort of error, when set the limit to 512, which was old limit without edns etc. that would for move to tcp from udp.. You can for sure exceed 512 when doing dnssec

$ dig @ns11.bsnl.in portal2.bsnl.in +dnssec +bufsize=512
;; Truncated, retrying in TCP mode.
;; Connection to 218.248.240.178#53(218.248.240.178) for portal2.bsnl.in failed: connection refused.

see msg size when I ask for dnssec info

;; Query time: 309 msec
;; SERVER: 218.248.240.178#53(218.248.240.178)
;; WHEN: Mon Oct 03 23:08:30 Central Daylight Time 2022
;; MSG SIZE  rcvd: 798

If I set bufsize to 798 it works, if set to 797 is fails because its trying to fall back to tcp, which isn't working.. So yeah I could see lots of problems with this domain for sure. If for whatever reason the connection to the NSers are being limited to the udp msg size. below the size of what that server is sending back.

As a hack/workaround to getting this to work, you could forward to say one of the big boys, google, quad9, 1111 etc.. You could setup a domain override for this specific domain, so that when you ask unbound for this specific domain, it forwards to say 8.8.8.8

The correct thing to do would be to figure out why this site isn't answering on tcp and have them fix that, or figure out exactly why there is a limit in the udp size going on..

Maybe his isp is limiting it? And works on other sites because they answer on the tcp fallback, but this site fails because its its not answering on tcp..

edit: see here when I set udp 512 limit, in my query.. It sends back hey this is Truncated the answer will not fit in that size. So the client asks via tcp, but the server not answering on tcp.

I don't know why pfsense would be limiting its size in its query, not exactly sure why client would be doing that either, so the limit has to be happening somewhere between I would think.

Would be interesting to see the query that is leaving pfsense.. Sniff on the wan while your client asks for the portal.bsnl.in would be good info. See in my sniff, I can tell if a payload size is limited by pfsense. Notice when I just ask, does it ask if a directed query from the client asks ns11 or ns12 directly?

For pfsense to ask, you might have to flush cache of pfsense between queries for it, etc.

Normally should look like this.

edit2:

Has the advanced edns options in unbound been messed with?

bingo600

@johnpoz said in pfsense blocking certain/some sites:

Has the advanced edns options in unbound been messed with?

I doubt it had been messed with by OP

Nice debugging

I'm forwarding pfSense (unbound) DNS queries to my local bind9 server(s).
Primarily because i already had that in place, with working DDNS registration for my home domain.

And i made dual secondary DNS servers in the summerhouse (blush), that my summerhouse pfS is using as forwarders. Summerhouse DNS'es are also registering DDNS on the primary DNS, making all DDNS to replicate via primary to secondaries , and available on both sites.

So any queries from me would be done by bind9.

Btw:
That https://dnsviz.net/ is quite revealing.
Now i have to look into the possibility of RR sigs on my "public domain" (hosted) - Thanx or ..... Now i can't get that out of my head

/Bingo

Gurveer

@johnpoz how to forward this portal2.bsnl.in or portal.bsnl.in to 8.8.8.8 or 1.1.1.1? Please help

Gertjan

@gurveer

First :

Then :

Save and then Apply.

Gurveer

@johnpoz @bingo600 @Gertjan @rcoleman-netgate @stephenw10 @viragomann thanks mine was set to automatic value based interface mtu changed to unbound default sites started working flawlessly after which i enabled dns forwarding too. Huge thanks you guys

bingo600

@gurveer
Why do you use forward, and not just let pfSense resolve ???
Any specific reason or ???

Another +1 for @johnpoz

Gertjan

@bingo600 said in pfsense blocking certain/some sites:

Any specific reason or ???

Something like : The name servers involved being unable to communicate over TCP (so these names servers 'break' DNS) + a total fxckxd up DNSSEC = his unbound resolver refuses to work.

All this said, something amazing is happening : I can look at the site :

and I'm not forwarding, I'm using the pfSense resolver unbound with default settings + dnssec hardening.
So, I tend to think 'something' is not good on @Gurveer's side.
If my pfSense + unbound resolvers works
It should also work for @Gurveer.

stephenw10

I think Johnpoz meant to add a domain override for bsnl.in to 8.8.8.8 or similar so everything else is still resolved directly rather than use forwarding mode.

johnpoz

@stephenw10 exactly..

But did he mess with his udp size? If he did he prob doesn't need to forward and this domain would work.

Gurveer

@bingo600 nothing just did on whim it is fine to run forwarder ?

johnpoz

@bingo600 said in pfsense blocking certain/some sites:

I doubt it had been messed with by OP

You would think that huh - but I find that quite often users click on shit all the time.. Until actually see screenshot or a sniff showing it using 4096, etc. You have no idea what is actually happening other than that ns isn't answering on tcp, and it sends back large info that could exceed 512, etc.

stephenw10

Well I have two machines here hitting this and I'm pretty certain I never changed the buffer size. Both are set to the default (auto based on MTU). Both have 1500B WANs.

Just digging to see what the actual buffer size is but I don't know where that is...yet

stephenw10

Looks to be 512 in unbound.conf...

stephenw10

Mmm, and on working devices:

edns-buffer-size: 1432

That's in 22.11 so newer Unbound version

johnpoz

@stephenw10 that was changed from default, see my screenshot shows 4096 as default.. on 22.05

I had not read that unbound was changing their default settings. But sure pfsense could, default for unbound I think is 1232, I think that came out of a flag day for dns back a couple of years ago.

But that can be overridden with that actual bufsize setting, etc. But the max-udp-size: I believe defaults to 4096.

But if your having issues with a NS that is not answering on tcp, it would behoove you to validate what your settings are, etc. To make sure they are appropriate for your network connection.

stephenw10

That isn't the pfSense default config though. At least not any longer. It should be Auto based on MTU there since 2.5 https://redmine.pfsense.org/issues/10293
If that's a machine that was upgraded from before that it might still be 'Unbound default'. Several of mine are which is why they weren't hitting it.
Still can;t see why these two machines I have ended up at 512 when all their interfaces are 1500B MTU. However I'm not seeing it on any 22.11 test box so it might be moot.

Steve

johnpoz

@stephenw10 I agree with a setting based on mtu is prob the best scenario..

IMHO the problem here is that the NS for this domain doesn't answer on tcp - which is going to cause problems for sure..

I see a few options here, make sure unbound is using an appropriate size for your network and its connection. If settings are correct, and still having issue then the hack/workaround for such a domain would be to forward for that "specific" domain to remove your connections issue with large udp.

The nuclear option sure is to forward all your queries and let them worry about it ;) But to me that is burning down the house to kill a spider..