Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Weirdest issue ever! Seems to be DNS problem but only on Twitter.

    Scheduled Pinned Locked Moved General pfSense Questions
    18 Posts 5 Posters 3.5k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      meda
      last edited by

      Hey People!

      I have this weird problem with Twitter, it seems to have difficulties with loading the page, especially images and some css. I have tried changing DNS with one to another, even disabling it but no luck so far. Well, it sometimes load okay but only sometimes.
      I have 3 wans loadbalanced and that's all, pretty much basic installation, no extra package installed.

      So I'm out of ideas. What possibly causes this?

      twt.png
      twt.png_thumb

      1 Reply Last reply Reply Quote 0
      • johnpozJ
        johnpoz LAYER 8 Global Moderator
        last edited by

        what is the fqdn trying to be resolved?  can not tell with the twitter core.bundle.css lines

        but abs.twimg.com

        resolves just fine

        ; QUESTION SECTION:
        ;abs.twimg.com.                IN      A

        ;; ANSWER SECTION:
        abs.twimg.com.          3600    IN      CNAME  wildcard.twimg.com.
        wildcard.twimg.com.    3600    IN      A      104.244.46.167
        wildcard.twimg.com.    3600    IN      A      104.244.46.39

        ;; AUTHORITY SECTION:
        twimg.com.              13999  IN      NS      d01-01.ns.twtrdns.net.
        twimg.com.              13999  IN      NS      r01-01.ns.twtrdns.net.
        twimg.com.              13999  IN      NS      ns2.p34.dynect.net.
        twimg.com.              13999  IN      NS      r01-02.ns.twtrdns.net.
        twimg.com.              13999  IN      NS      d01-02.ns.twtrdns.net.
        twimg.com.              13999  IN      NS      ns4.p34.dynect.net.
        twimg.com.              13999  IN      NS      ns3.p34.dynect.net.
        twimg.com.              13999  IN      NS      ns1.p34.dynect.net.

        What aare you using for dns?  Resolver (unbound)… Forwarder (dnsmasq)?

        This means nothing without more detail

        "I have tried changing DNS with one to another"

        An intelligent man is sometimes forced to be drunk to spend time with his fools
        If you get confused: Listen to the Music Play
        Please don't Chat/PM me for help, unless mod related
        SG-4860 24.11 | Lab VMs 2.8, 24.11

        1 Reply Last reply Reply Quote 0
        • M
          meda
          last edited by

          Dns forwarder is disabled. Dns resolver is enabled and with default configuration. I've tried Google, OpenDns, Level3 and some others.

          I don't think something's wrong with the twitter. It seems my pfsense box can't resolve nor load these certain source but I don't  know why and how?

          1 Reply Last reply Reply Quote 0
          • johnpozJ
            johnpoz LAYER 8 Global Moderator
            last edited by

            well you need to trouble the why.. Resolver would need to talk to the authoritative name servers.  But if you just asking say google for it - you just need to be able to ask google for it..

            dig @8.8.8.8 abs.twimg.com

            ; <<>> DiG 9.11.2 <<>> @8.8.8.8 abs.twimg.com
            ; (1 server found)
            ;; global options: +cmd
            ;; Got answer:
            ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26325
            ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

            ;; OPT PSEUDOSECTION:
            ; EDNS: version: 0, flags:; udp: 512
            ;; QUESTION SECTION:
            ;abs.twimg.com.                IN      A

            ;; ANSWER SECTION:
            abs.twimg.com.          271    IN      CNAME  wildcard.twimg.com.
            wildcard.twimg.com.    31      IN      A      104.244.46.103
            wildcard.twimg.com.    31      IN      A      104.244.46.71

            ;; Query time: 15 msec
            ;; SERVER: 8.8.8.8#53(8.8.8.8)
            ;; WHEN: Tue Nov 14 06:15:58 Central Standard Time 2017
            ;; MSG SIZE  rcvd: 97

            An intelligent man is sometimes forced to be drunk to spend time with his fools
            If you get confused: Listen to the Music Play
            Please don't Chat/PM me for help, unless mod related
            SG-4860 24.11 | Lab VMs 2.8, 24.11

            1 Reply Last reply Reply Quote 0
            • A
              Aurelux
              last edited by

              @meda I have the exact same problem for months! Do you have problem with www.w3schools.com too? I've looked up many things but it's just weird. I hope someone came up with a clever solution :)

              Unbound can't resolve abs.twimg.com and pbs.twimg.com (*.twimg.com) and www.w3schools.com.

              Service restart sometimes fixes issue for short time. Sometimes even restart don't work.

              1 Reply Last reply Reply Quote 0
              • johnpozJ
                johnpoz LAYER 8 Global Moderator
                last edited by

                Can you not talk to the authoritative ns for twing.com?

                An intelligent man is sometimes forced to be drunk to spend time with his fools
                If you get confused: Listen to the Music Play
                Please don't Chat/PM me for help, unless mod related
                SG-4860 24.11 | Lab VMs 2.8, 24.11

                1 Reply Last reply Reply Quote 0
                • S
                  Stewart
                  last edited by

                  Is it possible you have a package blocking it by chance?  pfBlockerNG?  Suricata?  Squid?  I know you said no extra packages installed but it's the only thing I can think of at the moment.  I've seen ISPs have routing issues as well.  Look up what you can about those domains through a whois and checking whatsmydns.net and test direct communication with them.  For example, the name servers for twimg.com is: d01-01.ns.twtrdns.net
                  d01-02.ns.twtrdns.net
                  ns1.p34.dynect.net
                  ns2.p34.dynect.net
                  ns3.p34.dynect.net
                  ns4.p34.dynect.net
                  r01-01.ns.twtrdns.net
                  r01-02.ns.twtrdns.net

                  Testing communication with them would be my next step.

                  1 Reply Last reply Reply Quote 0
                  • johnpozJ
                    johnpoz LAYER 8 Global Moderator
                    last edited by

                    I am showing the domain having a bit of an issue with their NS as well

                    "The following NS name(s) were found in the authoritative NS RRset, but not in the delegation NS RRset (i.e., in the com zone): ns1.p34.dynect.net, ns2.p34.dynect.net"

                    You need to check if you can talk to all of the NS listed as authoritative for that domain.  Stewart listed them above, and I did as well in my dig output.

                    What you can do for a work around if you have problems talking to specific NS, is you could create a domain override for that domain and point it only to the NS you can talk to..  Having issues with talking to some of the NS could explain why sometimes it works and sometimes it does not, etc.

                    You can use your fav tool for this, dig, nslookup, host to directly query those NS to validate you can talk to them and they answer for the fqdn in their domain your wanting to find.

                    An intelligent man is sometimes forced to be drunk to spend time with his fools
                    If you get confused: Listen to the Music Play
                    Please don't Chat/PM me for help, unless mod related
                    SG-4860 24.11 | Lab VMs 2.8, 24.11

                    1 Reply Last reply Reply Quote 0
                    • A
                      Aurelux
                      last edited by

                      I can ping and query all the nameservers for twimg.com. Entering manual domain override for twimg.com domain apparently fixes the problem. But of course it's not an optimal solution.

                      1 Reply Last reply Reply Quote 0
                      • johnpozJ
                        johnpoz LAYER 8 Global Moderator
                        last edited by

                        So you did a query to all of those NS? And they answered?

                        Did you notice they answer differently?  To be honest this points to possible problem with their dns.. They have a dnssec problem with their delegation and RRset.. See my above posts..  And when I look their NS answer with different responses.. This is normally BAD!!  Not the round robin, that is fine..  The completely different cname you get back from some that does not hand back A records and you have to follow a long chain.. Why are they handing off different?  All of their NS should hand back the same thing.. either the wildcard.twimg.com cname or the other way, etc.  shouldn't be different like that.

                        dig @d01-02.ns.twtrdns.net abs.twimg.com +short
                        wildcard.twimg.com.
                        104.244.46.135
                        104.244.46.71

                        dig @ns1.p34.dynect.net abs.twimg.com +short
                        cs196.wac.edgecastcdn.net.

                        dig @ns2.p34.dynect.net abs.twimg.com +short
                        wildcard.twimg.com.
                        104.244.46.167
                        104.244.46.135

                        dig @ns3.p34.dynect.net abs.twimg.com +short
                        cs196.wac.edgecastcdn.net.

                        dig @ns4.p34.dynect.net abs.twimg.com +short
                        wildcard.twimg.com.
                        104.244.46.71
                        104.244.46.7

                        dig @r01-01.twtrdns.net abs.twimg.com +short
                        dig: couldn't get address for 'r01-01.twtrdns.net': not found

                        dig @r01-01.ns.twtrdns.net abs.twimg.com +short
                        wildcard.twimg.com.
                        104.244.46.199

                        dig @r01-02.ns.twtrdns.net abs.twimg.com +short
                        cs196.wac.edgecastcdn.net.

                        See how when you qeury some you get back the cname and then the A record that change - for that.. They are doing a roundrobin.. But some only answer back the cname with no A record..  Wow what a nasty cname chain… Which is really really BAD Practice to do this, cname to a cname to a cname..

                        What is also funny is they hand off 60 second TTL for the wildcard record when you get the cname back for abs.. But the abs cname as ttl of 300, but then when you go to lookup the wildcard direct its got a ttl of 3600...

                        If you ask me their dns for this just plain borked! ;)

                        ;; QUESTION SECTION:
                        ;cs196.wac.edgecastcdn.net.    IN      A

                        ;; ANSWER SECTION:
                        cs196.wac.edgecastcdn.net. 3600 IN      CNAME  cs2-wac.apr-8315.edgecastdns.net.
                        cs2-wac.apr-8315.edgecastdns.net. 3600 IN CNAME cs2-wac-us.8315.ecdns.net.
                        cs2-wac-us.8315.ecdns.net. 3600 IN      CNAME  cs45.wac.edgecastcdn.net.
                        cs45.wac.edgecastcdn.net. 3600  IN      A      72.21.91.70

                        Following such a cname change could lead to problems with resolving if any of the NS fail in the chain, and for sure such a thing would take longer to resolve than normal.. Since when they change domains you have to resolve the NS for that domain, etc.  So this could cause delays in the response, etc..  Again doing such a thing is bad practice.. 1 cname chained ok bend best practice a bit - but 4 actually..  Pretty stupid setup just asking for problems..

                        Its possible if your trying to resolve that long chain - this is where your having problems, you would have to follow the tree to see if having issue talking to any of those NS..  Or possible somewhere in that change dnssec is broke and then you wouldn't get any answer back if using dnssec.. So since they have dns pointing to different ways to get to the same final answer you could get lucky one time and work, and the next time and fail.. And since they really sort ttls on some 60, and max 1 hour.. your going to be doing a lot of queries for this, etc.

                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                        If you get confused: Listen to the Music Play
                        Please don't Chat/PM me for help, unless mod related
                        SG-4860 24.11 | Lab VMs 2.8, 24.11

                        1 Reply Last reply Reply Quote 0
                        • GertjanG
                          Gertjan
                          last edited by

                          @Aurelux:

                          …  nameservers for twimg.com. ....

                          Run some basic DNS tests on this domain and you will see that ….. it's messy at best.
                          I guess they are in the middle of some sort of maintenance and resolving suffers right now.

                          http://dnscheck.iis.se/
                          http://www.dnsstuff.com/

                          edit :  johnpoz beated me on this one  :)

                          No "help me" PM's please. Use the forum, the community will thank you.
                          Edit : and where are the logs ??

                          1 Reply Last reply Reply Quote 0
                          • johnpozJ
                            johnpoz LAYER 8 Global Moderator
                            last edited by

                            "I guess they are in the middle of some sort of maintenance and resolving suffers right now."

                            Its quite possible they are in the middle of migration/change.. But Messy would be nice word for it.. Such a company and such a change should be a planned cut that should only take minutes.. Not days.. Maybe part of their migration failed and they didn't catch it ;)

                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                            If you get confused: Listen to the Music Play
                            Please don't Chat/PM me for help, unless mod related
                            SG-4860 24.11 | Lab VMs 2.8, 24.11

                            1 Reply Last reply Reply Quote 0
                            • M
                              meda
                              last edited by

                              I've assigned different DNSs for every single WAN, it seemed to solve problem for 3 days but when I had changed to "none", it didn't feel anything changed, still working as it must be. So I'm lost here.

                              d01-01.ns.twtrdns.net
                              d01-02.ns.twtrdns.net
                              ns1.p34.dynect.net
                              ns2.p34.dynect.net
                              ns3.p34.dynect.net
                              ns4.p34.dynect.net
                              r01-01.ns.twtrdns.net
                              r01-02.ns.twtrdns.net

                              I've tried these, got reply for a while, then I did not, then I did. Now they're replying again.

                              By the way, I only use ftp client proxy, but don't think it's relevant.

                              pfs.png
                              pfs.png_thumb

                              1 Reply Last reply Reply Quote 0
                              • johnpozJ
                                johnpoz LAYER 8 Global Moderator
                                last edited by

                                "I've tried these, got reply for a while, then I did not, then I did. Now they're replying again."

                                If they do not respond then yeah your going to have problems.. Just use the work around point the domain to an override until such time as you don't have issues talking to them..  As stated it seems their dns is pretty messed up currently.. Maybe they are working on migration or problem?

                                Not sure what your doing there with your dns.. If you were using a forwarder like you have setup then not resolving something is on the forwarder not giving you an answer or your isp intercepting the traffic.  Out of the box pfsense use unbound in resolver mode.. And would walk down from roots to talk to the authoritative server directly.

                                An intelligent man is sometimes forced to be drunk to spend time with his fools
                                If you get confused: Listen to the Music Play
                                Please don't Chat/PM me for help, unless mod related
                                SG-4860 24.11 | Lab VMs 2.8, 24.11

                                1 Reply Last reply Reply Quote 0
                                • A
                                  Aurelux
                                  last edited by

                                  Even manually defining the NS of the twimg.com did not fix the issue. I could replicate johnpoz's output.

                                  These are my unbound config and nslookup query directly to 208.78.70.34 (Twitter's own NS)

                                  https://ibb.co/dTrgNm
                                  https://ibb.co/dLHz8R
                                  https://ibb.co/eQ6sTR

                                  It's weird that no one else is having this problem. I also don't have any packages installed which can interefere with DNS operation.

                                  1 Reply Last reply Reply Quote 0
                                  • johnpozJ
                                    johnpoz LAYER 8 Global Moderator
                                    last edited by

                                    you didn't duplicate my test - everyone of those queries is to the same ns

                                    My would you point to that one - its not the SOA

                                    ;; ANSWER SECTION:
                                    twimg.com.              3600    IN      SOA    ns1.p26.dynect.net. ops.twitter.com. 214840 3600 600 604800 60

                                    Their dns seems to be a MESS if you ask me.. Their SOA is not even listed as one of the NS for the domain..

                                    Why would you pick that one to query?  Is one of the ones missing from the RRset

                                    ns1.p34.dynect.net

                                    The following NS name(s) were found in the authoritative NS RRset, but not in the delegation NS RRset (i.e., in the com zone): ns1.p34.dynect.net, ns2.p34.dynect.net

                                    I would just use a override pointing say google or something until they fix their mess.

                                    An intelligent man is sometimes forced to be drunk to spend time with his fools
                                    If you get confused: Listen to the Music Play
                                    Please don't Chat/PM me for help, unless mod related
                                    SG-4860 24.11 | Lab VMs 2.8, 24.11

                                    1 Reply Last reply Reply Quote 0
                                    • A
                                      Aurelux
                                      last edited by

                                      Ok, so it's not a PFsense issue it's a DNS misconfiguration on Twitter's end (esp CDN). Another post about this: https://twittercommunity.com/t/pbs-twimg-com-cdn-problems/92623/2

                                      1 Reply Last reply Reply Quote 0
                                      • johnpozJ
                                        johnpoz LAYER 8 Global Moderator
                                        last edited by

                                        Its never had anything to do with pfsense at all ;)  dns problem give you that, but it was never something specific to pfsense.

                                        Like I said something not right with their dns.. you shouldn't point to different stuff like that..  if you going to use a CDN fine, but it sholdn't point to different cnames and have ones that just do one and then others than have 3 in a daisy chain, etc..

                                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                                        If you get confused: Listen to the Music Play
                                        Please don't Chat/PM me for help, unless mod related
                                        SG-4860 24.11 | Lab VMs 2.8, 24.11

                                        1 Reply Last reply Reply Quote 0
                                        • First post
                                          Last post
                                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.