Weirdest issue ever! Seems to be DNS problem but only on Twitter.

meda

Hey People!

I have this weird problem with Twitter, it seems to have difficulties with loading the page, especially images and some css. I have tried changing DNS with one to another, even disabling it but no luck so far. Well, it sometimes load okay but only sometimes.
I have 3 wans loadbalanced and that's all, pretty much basic installation, no extra package installed.

So I'm out of ideas. What possibly causes this?

twt.png_thumb

johnpoz

what is the fqdn trying to be resolved? can not tell with the twitter core.bundle.css lines

but abs.twimg.com

resolves just fine

; QUESTION SECTION:
;abs.twimg.com. IN A

;; ANSWER SECTION:
abs.twimg.com. 3600 IN CNAME wildcard.twimg.com.
wildcard.twimg.com. 3600 IN A 104.244.46.167
wildcard.twimg.com. 3600 IN A 104.244.46.39

;; AUTHORITY SECTION:
twimg.com. 13999 IN NS d01-01.ns.twtrdns.net.
twimg.com. 13999 IN NS r01-01.ns.twtrdns.net.
twimg.com. 13999 IN NS ns2.p34.dynect.net.
twimg.com. 13999 IN NS r01-02.ns.twtrdns.net.
twimg.com. 13999 IN NS d01-02.ns.twtrdns.net.
twimg.com. 13999 IN NS ns4.p34.dynect.net.
twimg.com. 13999 IN NS ns3.p34.dynect.net.
twimg.com. 13999 IN NS ns1.p34.dynect.net.

What aare you using for dns? Resolver (unbound)… Forwarder (dnsmasq)?

This means nothing without more detail

"I have tried changing DNS with one to another"

meda

Dns forwarder is disabled. Dns resolver is enabled and with default configuration. I've tried Google, OpenDns, Level3 and some others.

I don't think something's wrong with the twitter. It seems my pfsense box can't resolve nor load these certain source but I don't know why and how?

johnpoz

well you need to trouble the why.. Resolver would need to talk to the authoritative name servers. But if you just asking say google for it - you just need to be able to ask google for it..

dig @8.8.8.8 abs.twimg.com

; <<>> DiG 9.11.2 <<>> @8.8.8.8 abs.twimg.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26325
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;abs.twimg.com. IN A

;; ANSWER SECTION:
abs.twimg.com. 271 IN CNAME wildcard.twimg.com.
wildcard.twimg.com. 31 IN A 104.244.46.103
wildcard.twimg.com. 31 IN A 104.244.46.71

;; Query time: 15 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Tue Nov 14 06:15:58 Central Standard Time 2017
;; MSG SIZE rcvd: 97

Aurelux

@meda I have the exact same problem for months! Do you have problem with www.w3schools.com too? I've looked up many things but it's just weird. I hope someone came up with a clever solution :)

Unbound can't resolve abs.twimg.com and pbs.twimg.com (*.twimg.com) and www.w3schools.com.

Service restart sometimes fixes issue for short time. Sometimes even restart don't work.

johnpoz

Can you not talk to the authoritative ns for twing.com?

Stewart

Is it possible you have a package blocking it by chance? pfBlockerNG? Suricata? Squid? I know you said no extra packages installed but it's the only thing I can think of at the moment. I've seen ISPs have routing issues as well. Look up what you can about those domains through a whois and checking whatsmydns.net and test direct communication with them. For example, the name servers for twimg.com is: d01-01.ns.twtrdns.net
d01-02.ns.twtrdns.net
ns1.p34.dynect.net
ns2.p34.dynect.net
ns3.p34.dynect.net
ns4.p34.dynect.net
r01-01.ns.twtrdns.net
r01-02.ns.twtrdns.net

Testing communication with them would be my next step.

johnpoz

I am showing the domain having a bit of an issue with their NS as well

"The following NS name(s) were found in the authoritative NS RRset, but not in the delegation NS RRset (i.e., in the com zone): ns1.p34.dynect.net, ns2.p34.dynect.net"

You need to check if you can talk to all of the NS listed as authoritative for that domain. Stewart listed them above, and I did as well in my dig output.

What you can do for a work around if you have problems talking to specific NS, is you could create a domain override for that domain and point it only to the NS you can talk to.. Having issues with talking to some of the NS could explain why sometimes it works and sometimes it does not, etc.

You can use your fav tool for this, dig, nslookup, host to directly query those NS to validate you can talk to them and they answer for the fqdn in their domain your wanting to find.

Aurelux

I can ping and query all the nameservers for twimg.com. Entering manual domain override for twimg.com domain apparently fixes the problem. But of course it's not an optimal solution.

johnpoz

So you did a query to all of those NS? And they answered?

Did you notice they answer differently? To be honest this points to possible problem with their dns.. They have a dnssec problem with their delegation and RRset.. See my above posts.. And when I look their NS answer with different responses.. This is normally BAD!! Not the round robin, that is fine.. The completely different cname you get back from some that does not hand back A records and you have to follow a long chain.. Why are they handing off different? All of their NS should hand back the same thing.. either the wildcard.twimg.com cname or the other way, etc. shouldn't be different like that.

dig @d01-02.ns.twtrdns.net abs.twimg.com +short
wildcard.twimg.com.
104.244.46.135
104.244.46.71

dig @ns1.p34.dynect.net abs.twimg.com +short
cs196.wac.edgecastcdn.net.

dig @ns2.p34.dynect.net abs.twimg.com +short
wildcard.twimg.com.
104.244.46.167
104.244.46.135

dig @ns3.p34.dynect.net abs.twimg.com +short
cs196.wac.edgecastcdn.net.

dig @ns4.p34.dynect.net abs.twimg.com +short
wildcard.twimg.com.
104.244.46.71
104.244.46.7

dig @r01-01.twtrdns.net abs.twimg.com +short
dig: couldn't get address for 'r01-01.twtrdns.net': not found

dig @r01-01.ns.twtrdns.net abs.twimg.com +short
wildcard.twimg.com.
104.244.46.199

dig @r01-02.ns.twtrdns.net abs.twimg.com +short
cs196.wac.edgecastcdn.net.

See how when you qeury some you get back the cname and then the A record that change - for that.. They are doing a roundrobin.. But some only answer back the cname with no A record.. Wow what a nasty cname chain… Which is really really BAD Practice to do this, cname to a cname to a cname..

What is also funny is they hand off 60 second TTL for the wildcard record when you get the cname back for abs.. But the abs cname as ttl of 300, but then when you go to lookup the wildcard direct its got a ttl of 3600...

If you ask me their dns for this just plain borked! ;)

;; QUESTION SECTION:
;cs196.wac.edgecastcdn.net. IN A

;; ANSWER SECTION:
cs196.wac.edgecastcdn.net. 3600 IN CNAME cs2-wac.apr-8315.edgecastdns.net.
cs2-wac.apr-8315.edgecastdns.net. 3600 IN CNAME cs2-wac-us.8315.ecdns.net.
cs2-wac-us.8315.ecdns.net. 3600 IN CNAME cs45.wac.edgecastcdn.net.
cs45.wac.edgecastcdn.net. 3600 IN A 72.21.91.70

Following such a cname change could lead to problems with resolving if any of the NS fail in the chain, and for sure such a thing would take longer to resolve than normal.. Since when they change domains you have to resolve the NS for that domain, etc. So this could cause delays in the response, etc.. Again doing such a thing is bad practice.. 1 cname chained ok bend best practice a bit - but 4 actually.. Pretty stupid setup just asking for problems..

Its possible if your trying to resolve that long chain - this is where your having problems, you would have to follow the tree to see if having issue talking to any of those NS.. Or possible somewhere in that change dnssec is broke and then you wouldn't get any answer back if using dnssec.. So since they have dns pointing to different ways to get to the same final answer you could get lucky one time and work, and the next time and fail.. And since they really sort ttls on some 60, and max 1 hour.. your going to be doing a lot of queries for this, etc.

Gertjan

@Aurelux:

… nameservers for twimg.com. ....

Run some basic DNS tests on this domain and you will see that ….. it's messy at best.
I guess they are in the middle of some sort of maintenance and resolving suffers right now.

http://dnscheck.iis.se/
http://www.dnsstuff.com/

edit : johnpoz beated me on this one :)

johnpoz

"I guess they are in the middle of some sort of maintenance and resolving suffers right now."

Its quite possible they are in the middle of migration/change.. But Messy would be nice word for it.. Such a company and such a change should be a planned cut that should only take minutes.. Not days.. Maybe part of their migration failed and they didn't catch it ;)

meda

I've assigned different DNSs for every single WAN, it seemed to solve problem for 3 days but when I had changed to "none", it didn't feel anything changed, still working as it must be. So I'm lost here.

d01-01.ns.twtrdns.net
d01-02.ns.twtrdns.net
ns1.p34.dynect.net
ns2.p34.dynect.net
ns3.p34.dynect.net
ns4.p34.dynect.net
r01-01.ns.twtrdns.net
r01-02.ns.twtrdns.net

I've tried these, got reply for a while, then I did not, then I did. Now they're replying again.

By the way, I only use ftp client proxy, but don't think it's relevant.

pfs.png_thumb

johnpoz

"I've tried these, got reply for a while, then I did not, then I did. Now they're replying again."

If they do not respond then yeah your going to have problems.. Just use the work around point the domain to an override until such time as you don't have issues talking to them.. As stated it seems their dns is pretty messed up currently.. Maybe they are working on migration or problem?

Not sure what your doing there with your dns.. If you were using a forwarder like you have setup then not resolving something is on the forwarder not giving you an answer or your isp intercepting the traffic. Out of the box pfsense use unbound in resolver mode.. And would walk down from roots to talk to the authoritative server directly.

Aurelux

Even manually defining the NS of the twimg.com did not fix the issue. I could replicate johnpoz's output.

These are my unbound config and nslookup query directly to 208.78.70.34 (Twitter's own NS)

https://ibb.co/dTrgNm
https://ibb.co/dLHz8R
https://ibb.co/eQ6sTR

It's weird that no one else is having this problem. I also don't have any packages installed which can interefere with DNS operation.

johnpoz

you didn't duplicate my test - everyone of those queries is to the same ns

My would you point to that one - its not the SOA

;; ANSWER SECTION:
twimg.com. 3600 IN SOA ns1.p26.dynect.net. ops.twitter.com. 214840 3600 600 604800 60

Their dns seems to be a MESS if you ask me.. Their SOA is not even listed as one of the NS for the domain..

Why would you pick that one to query? Is one of the ones missing from the RRset

ns1.p34.dynect.net

The following NS name(s) were found in the authoritative NS RRset, but not in the delegation NS RRset (i.e., in the com zone): ns1.p34.dynect.net, ns2.p34.dynect.net

I would just use a override pointing say google or something until they fix their mess.

Aurelux

Ok, so it's not a PFsense issue it's a DNS misconfiguration on Twitter's end (esp CDN). Another post about this: https://twittercommunity.com/t/pbs-twimg-com-cdn-problems/92623/2

johnpoz

Its never had anything to do with pfsense at all ;) dns problem give you that, but it was never something specific to pfsense.

Like I said something not right with their dns.. you shouldn't point to different stuff like that.. if you going to use a CDN fine, but it sholdn't point to different cnames and have ones that just do one and then others than have 3 in a daisy chain, etc..