Dynamic DNS *NOT* Updating "Cached IP"

barnettd

@steveits Yes it used to work. All the devices it is failing on have 2 internet connections and a gateway group (Tiers) that is monitored by Dynamic DNS. No changes were made to the configs.

I have noticed that when the gateway changes due to a connection outage it seems the IP is updated at namecheap, but when the gateway changes back to the primary, the dynamic dns stays on the secondary connection. Manually clicking the "Save and Force Update" button seemed to resolve the issue, but thats not ideal.

I honestly don't know if the issue is on the pfsense side or if namecheap changed their response somehow. Since yours is still working it seems to point toward pfsense.

nikolaosinlight

@barnettd somewhat comforting (but not really) that it isn't just me. I too have a tiered gateway group and yes tested the failure of the primary and it didn't switch the DNS.

For myself as well Namecheap gets updated and like you said if it works when there is no tiered gateway group then likely not a Namecheap interface change.

How do we get this looked at and resolved? To not update the IP in pfsense and stay red is a problem for sure as we can not turn up our frequency of cron checks... BUT to have tiered gateway failover with 2 ISPs and not failover the IP when the Tier 1 fails is really bad as services will fail - essentially there is no failover unless I am monitoring and force it as you did.

nikolaosinlight

@steveits I tried manually correcting the cached entries and reloaded the page and got full green. Then I dropped the primary (Tier 1) ISP and it went red, updated Namecheap correctly, and continued to show the prior IP address in UI and in the cache files.

I think @barnettd may be on to something in that this could be a bug related to a Tiered Gateway Group (which also does not work as it should when ISP fails).

I am only about a month into pfsense. Is there a place to report a bug? What is the process?

SteveITS

Since you mention gateway groups I did notice this redmine - DDNS set to a gateway group does not update on WAN failover but Jim couldn't duplicate the issue.

That redmine site is the place to report bugs.

biggsy

I noticed the following in my syslog and suspected a temporary problem but found the lines were also there yesterday:

Dec 10 01:01:01 php[4009]: rc.dyndns.update: phpDynDNS (mail): PAYLOAD: <?xml version="1.0" encoding="utf-16"?>
Dec 10 01:01:01 php[4009]: <interface-response>
Dec 10 01:01:01 php[4009]:   <Command>SETDNSHOST</Command>
Dec 10 01:01:01 php[4009]:   <Language>eng</Language>
Dec 10 01:01:01 php[4009]:   <IP>[redacted]</IP>
Dec 10 01:01:01 php[4009]:   <ErrCount>0</ErrCount>
Dec 10 01:01:01 php[4009]:   <errors />
Dec 10 01:01:01 php[4009]:   <ResponseCount>0</ResponseCount>
Dec 10 01:01:01 php[4009]:   <responses />
Dec 10 01:01:01 php[4009]:   <Done>true</Done>
Dec 10 01:01:01 php[4009]:   <debug><![CDATA[]]></debug>
Dec 10 01:01:01 php[4009]: </interface-response>
Dec 10 01:01:01 php[4009]: rc.dyndns.update: phpDynDNS (mail): (Unknown Response)
Dec 10 01:01:02 php[4009]: rc.dyndns.update: phpDynDNS (@): PAYLOAD: <?xml version="1.0" encoding="utf-16"?>
Dec 10 01:01:02 php[4009]: <interface-response>
Dec 10 01:01:02 php[4009]:   <Command>SETDNSHOST</Command>
Dec 10 01:01:02 php[4009]:   <Language>eng</Language>
Dec 10 01:01:02 php[4009]:   <IP>[redacted]</IP>
Dec 10 01:01:02 php[4009]:   <ErrCount>0</ErrCount>
Dec 10 01:01:02 php[4009]:   <errors />
Dec 10 01:01:02 php[4009]:   <ResponseCount>0</ResponseCount>
Dec 10 01:01:02 php[4009]:   <responses />
Dec 10 01:01:02 php[4009]:   <Done>true</Done>
Dec 10 01:01:02 php[4009]:   <debug><![CDATA[]]></debug>
Dec 10 01:01:02 php[4009]: </interface-response>
Dec 10 01:01:02 php[4009]: rc.dyndns.update: phpDynDNS (@): (Unknown Response)

That brought me to this topic. I have only one WAN, so no gateway groups.

I updated from 2.4.5_P1 to 2.5.2 back in September and I've made no changes to the Dynamic DNS setup for well over a year.

Older syslog records show that the Namecheap DDNS records were successfully updated in Sep, Oct and Nov.

I tried editing the DynDNS settings and saving. Only the "mail" record was updated to my current public IP. The generic (@) record now shows as 0.0.0.0 in the GUI. (I can live without the generic, though, as long as the mail gets through.)

The only significant change to my pfSense since updating to 2.5.2 was to install the Wireguard package.

I haven't been able to find any indication that Namecheap have changed anything at their end.

Edit: Will be watching my friend's pfSense, which is due to update his DDNS records, also at Namecheap, tomorrow morning.

biggsy

Yep, same logs on my friend's machine.

bmeeks

While looking around for other stuff, I found this link that appears related to the issue posted in this thread: https://www.reddit.com/r/NameCheap/comments/qz1mjf/namecheap_dynamic_dns_returning_utf16_encoded/hljqzja/. This does in fact appear to perhaps be a Namecheap problem.

biggsy

@bmeeks

Many thanks for finding this. I see that the original post and the last response form Namecheap were 21 days ago.

I just hope they don't start blocking because of too-frequent updates, which are now occurring daily because of this problem.

bmeeks

@biggsy said in Dynamic DNS *NOT* Updating "Cached IP":

@bmeeks

Many thanks for finding this. I see that the original post and the last response form Namecheap were 21 days ago.

I just hope they don't start blocking because of too-frequent updates, which are now occurring daily because of this problem.

The guys over on the "other Sense" forum site are also reporting the same issue.

biggsy

@bmeeks

It's Namecheap so perhaps I shouldn't be surprised that I can't find any DDNS update "abuse" policy on their web site.

barnettd

@bmeeks That issue definitely matches what I'm seeing. Setting up a custom client using the namecheap update URL seems to be work for now.

Hopefully this is resolved soon, I have quite a few devices using Dynamic DNS from namecheap and it would be a pain to change.

nikolaosinlight

@barnettd

So setting up a custom Dynamic DNS for Namecheap only works for me if I leave Result Match blank - even if I modify the IP address and save the full response as the following it does not work - so if Namecheap fails to update I am completely hosed:

<?xml version="1.0" encoding="utf-16"?>
<interface-response>
  <Command>SETDNSHOST</Command>
  <Language>eng</Language>
  <IP>%IP%</IP>
  <ErrCount>0</ErrCount>
  <errors />
  <ResponseCount>0</ResponseCount>
  <responses />
  <Done>true</Done>
  <debug><![CDATA[]]></debug>
</interface-response>

I opened a ticket with Namecheap and this was their first response:

"You are right, unfortunately, we have such a glitch in our system currently. Our technical department is working on the resolution, however, we have no certain time frames when the issue is solved. Although, we have checked it from our side, and it does not affect the main functionality of the Dynamic DNS feature, and the IP address should be updated successfully. We see from your request, that you may face an issue with the regular propagation. The only case is to wait up to 30 minutes till the A record is propagated to a new one. Unfortunately, such a process cannot be sped up, as it is for a 100% automatic one and it is a regular procedure. "

When I responded that the above makes not sense as Namecheap fully updates but it is rather the response that is not properly formulated for pfsense to process and moreover pfsense will keep trying pointlessly they said:

"We perfectly understand your concern. The matter regarding the incorrect encoding format has been already escalated. To our regret, there is no ETA yet."

Changing the encoding of a response from UTF-16 to UTF-8 is not rocket science.

As such, I am not holding my breath and think it is time to look at other Registrars (we have been doing more and more with AWS recently so likely will go that route).

--Nikolaos

SteveITS

From the Reddit page, "They think a recent change in php started enforcing the UTF encoded verification, which means it's possible namecheap's xml response has been wrong for quite some time but not noticed until now." I have no idea but is there a way to turn that off in PHP? Not a great solution but could be a workaround. The odd thing is it doesn't look like PHP was updated in 21.05.x/2.5.x builds. There was the JIT/PCRE fix for the 3100 and 1000.

nikolaosinlight

@steveits
I pointed Namecheap to the Reddit post and this post and they say they have no ETA.

I get software interfaces are hard to update especially if they have worked well in the past so they definitely don't want to break existing functionality but seems like they will not fix is my guess. If they do... great... but from their response it does not seem likely it will be anytime soon or possibly ever.

--Nikolaos

Gertjan

@nikolaosinlight

Then the solution, even temporary, is rather easy :
change dyndns.class behavior so it accepts the 'new' answer as the 'all is good' answer.

This

... ($ncresponse['interface-response']['ErrCount'] === "0") ....

part should be 0 or '0' or - I don't know, some new UTF like 0 or '0'
Is it an integer value - a text value ? 8 bit or other type of encoding ?

Whatever, make the test 'succeed' and you will all be good.
A soon as it re breaks again, undo the edits and the story is over.

Btw : I can't play with this myself, as I'm not a namecheap customer.

edit : I hope for them that this is only a dyndns.class problem, because, if not, every customer that uses a dyndns client will hammer their Dyndns service constantly ....

SteveITS

@gertjan said in Dynamic DNS *NOT* Updating "Cached IP":

change dyndns.class behavior so it accepts the 'new' answer

I suspect the issue is that $ncresponse is considered empty since the response is invalid due to the incorrect encoding. (or something similar, I did not look into this at all, just reading posts)

Presumably from Namecheap's standpoint they want to send changes through QA/testing so as not to break things for everyone else.

bmeeks

Here is a link to the other site's issue on this Namecheap dynamic DNS problem: https://github.com/opnsense/plugins/issues/2666#issuecomment-979154701.

If you follow the comments and links, you may uncover a way to duplicate the workaround on pfSense. I highly doubt the source code nor specific file matches, but the overall gist of the fix should be applicable on pfSense. Note that the "fix" is really just a temp workaround until Namecheap fixes their core issue.

The root cause of the problem is incorrect type coding by Namecheap.

biggsy

I asked Namecheap whether there are maximum or minimum limits on update frequency for their dynamic DNS service that could cause blocking of updates or expiry of the mapping.

The answer was, "There are no limitation on the update frequency." However, Cloudflare is in the mix there, too, so I guess I'll find out.

Before getting that response, I decided on manually updating the unix timestamps in the /cf/conf/dyndns*.cache files to prevent needless (and possibly excessive) updates being generated.

Hopefully this will be fixed soon - one way or another.

nikolaosinlight

@bmeeks

Actually the source code aligned pretty closely to the patch. It was trivial to add but did not work:

                            case 'namecheap':
                                    // $tmp = str_replace("^M", "", $data);
                                    $tmp = preg_replace('/(\<\?xml[^?]+?encoding=.?)utf-16([^?]+?\?\>)/i', '$1utf-8$2', $data);
                                    $ncresponse = @xml2array($tmp);

This essentially simply provides a regex that translates the response XML to instead of saying UTF-16 it says UTF-8. It does nothing to deal with potentially receiving UTF-16 characters. I have no idea why Namecheap is even trying to use UTF-16... been coding in Java for 24 years and while yes UTF-16 is there and all almost every app still created today uses UTF-8 throughout.

@Gertjan your idea to change the return code may work but I think there are bigger issues with the response since if I include in a Custom Namecheap client a Result Match that is exactly as expected with IP made as variable %IP% it does not work. I need Result Match blank so I do not think that I am even going to get to the point of that line of code anywhere near matching.

I have the Custom "Dumb" Namecheap clients working so I will stick to that... and I will look to move to another registrar. Getting tired with a number of things with Namecheap lately.

SteveITS

@nikolaosinlight Possibly something with mb_convert_encoding?

$tmp = mb_convert_encoding($data, "UTF-8", "UTF-16");

Bit of a hack, but might work until Namecheap fixes their end.