Issues with Cloudflare Dynamic DNS

4pack · Dec 15, 2016, 10:57 AM

I have been getting the following error for maybe the last month in the 2.3.3 Development Snapshots as well as the 2.4 Snapshots. I tried tracking the issues down and from what I could find, it seems like it isn't parsing the JSON received back from CloudFlare correctly. When this full JSON string (not contained in the logs unfortunately) is put into a JSON syntax checker, it comes out just fine, but in dyndns.class, json_last_error_msg() returns syntax error right after it tries to get the domain ID. This, of course, prevents the array from even being created, hence the unknown error.

Anyone have any ideas as I have hit a brick wall? My log is below (with hostnames changed).

Dec 15 01:01:05 php-cgi rc.dyndns.update: Dynamic DNS: updatedns() starting
Dec 15 01:01:05 php-cgi rc.dyndns.update: Dynamic DNS cloudflare (3g.hostname.org): <currentnicip>extracted from local system.
Dec 15 01:01:05 php-cgi rc.dyndns.update: Dynamic DNS (3g.hostname.org): running get_failover_interface for opt2. found ppp0
Dec 15 01:01:05 php-cgi rc.dyndns.update: Dynamic DNS cloudflare (3g.hostname.org): <currentnicip>extracted from local system.
Dec 15 01:01:05 php-cgi rc.dyndns.update: Dynamic Dns (3g.hostname.org): Current WAN IP: <currentnicip>Cached IP: <oldnicip>Dec 15 01:01:05 php-cgi rc.dyndns.update: DynDns (3g.hostname.org): Dynamic Dns: cacheIP != wan_ip. Updating. Cached IP: <oldnicip>WAN IP: <currentnicip>Dec 15 01:01:05 php-cgi rc.dyndns.update: Dynamic DNS cloudflare (3g.hostname.org): _update() starting.
Dec 15 01:01:06 php-cgi rc.dyndns.update: Dynamic DNS cloudflare (3g.hostname.org): _checkStatus() starting.
Dec 15 01:01:06 php-cgi rc.dyndns.update: phpDynDNS (3g): PAYLOAD: {"result":[{"id":"CLOUDFLAREIDAWHOLEBUNCHOFCHARACTERS","name":"hostname.org","status":"active","paused":false,"type":"full","development_mode":-803711,"name_servers":["cleo.ns.cloudflare.com","roxy.ns.cloudflare.com"],"original_name_servers":["ns-cloud-a1.googledomains.com","ns-cloud-a2.googledomains.com","ns-cloud-a3.googledomains.com","ns-cloud-a4.googledomains.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2016-12-07T11:13:00.513004Z","created_on":"2016-10-12T20:05:42.101483Z","meta":{"step":4,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"type":"user","id":"THISISMYUSERIDDONTLOOK","email":"MYEMAIL@EMAIL.COM"},"permissions":["#analytics:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#logs:read","#organization:edit","#organization:read","#ss
Dec 15 01:01:06 php-cgi rc.dyndns.update: phpDynDNS (3g): UNKNOWN ERROR - [/quote]</currentnicip></oldnicip></oldnicip></currentnicip></currentnicip></currentnicip>

PiBa · Dec 27, 2016, 12:13 AM

Add a little php code that writes the response to a file? Check if thats 'complete' and checks as valid json. And matches at least one of the 3 criteria.. content/errors[0]/success

file_put_contents("/tmp/cf-response.json", $data);

cozkramer · Dec 27, 2016, 2:31 AM

Maybe your going at it the wrong way? I found this: https://www.google.com/url?sa=t&source=web&rct=j&url=/amp/s/scotthelme.co.uk/replacing-dyndns-cloudflare-ddns/amp/&ved=0ahUKEwizjuKYqJPRAhXFgVQKHW6JC6wQFgg6MAI&usg=AFQjCNGEPBEtX_kd8XCL-4CjUhqcBkNDLQ

Looks like he made a simple solution and who doesn't love cron..

4pack · Dec 27, 2016, 11:46 AM

@cozkramer:

Maybe your going at it the wrong way? I found this: https://www.google.com/url?sa=t&source=web&rct=j&url=/amp/s/scotthelme.co.uk/replacing-dyndns-cloudflare-ddns/amp/&ved=0ahUKEwizjuKYqJPRAhXFgVQKHW6JC6wQFgg6MAI&usg=AFQjCNGEPBEtX_kd8XCL-4CjUhqcBkNDLQ

Looks like he made a simple solution and who doesn't love cron..

I would use this, but I might as well fix the functionality in pfSense so people with the same issue can fix it as well.

@PiBa:

Add a little php code that writes the response to a file? Check if thats 'complete' and checks as valid json. And matches at least one of the 3 criteria.. content/errors[0]/success
file_put_contents("/tmp/cf-response.json", $data);

I'll check this out and get back to you.

EDIT: I found the issue. For some reason, curl_exec($ch) is returning header contents as well and it seems to be messing up the JSON parsing. See below. I'll definitely need to modify the curl query in order for it to only return the body of the response.


HTTP/1.1 200 OK
Date: Tue, 27 Dec 2016 11:39:59 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=SOMEIDTHATYOUSHOULDNTSEE; expires=Wed, 27-Dec-17 11:39:59 GMT; path=/; domain=.cloudflare.com; HttpOnly
Expires: Sun, 25 Jan 1981 05:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Server: cloudflare-nginx
CF-RAY: NOOOOOPPPPEE-DFW

{"result":[{"id":"MYCLOUDFLAREID","name":"hostname.org","status":"active","paused":false,"type":"full","development_mode":-1857245,"name_servers":["cleo.ns.cloudflare.com","roxy.ns.cloudflare.com"],"original_name_servers":["ns-cloud-a1.googledomains.com","ns-cloud-a2.googledomains.com","ns-cloud-a3.googledomains.com","ns-cloud-a4.googledomains.com"],"original_registrar":null,"original_dnshost":null,"modified_on":"2016-12-07T11:13:00.513004Z","created_on":"2016-10-12T20:05:42.101483Z","meta":{"step":4,"wildcard_proxiable":false,"custom_certificate_quota":0,"page_rule_quota":3,"phishing_detected":false,"multiple_railguns_allowed":false},"owner":{"type":"user","id":"MYUSERID","email":"MYEMAILDONTTOUCH"},"permissions":["#analytics:read","#billing:edit","#billing:read","#cache_purge:edit","#dns_records:edit","#dns_records:read","#lb:edit","#lb:read","#logs:read","#organization:edit","#organization:read","#ssl:edit","#ssl:read","#waf:edit","#waf:read","#zone:edit","#zone:read","#zone_settings:edit","#zone_settings:read"],"plan":{"id":"0feeeeeeeeeeeeeeeeeeeeeeeeeeeeee","name":"Free Website","price":0,"currency":"USD","frequency":"","is_subscribed":true,"can_subscribe":false,"legacy_id":"free","legacy_discount":false,"externally_managed":false}}],"result_info":{"page":1,"per_page":20,"total_pages":1,"count":1,"total_count":1},"success":true,"errors":[],"messages":[]}

PiBa · Dec 27, 2016, 6:22 PM

The problem was 'probably' introduced with/around this commit: https://github.com/pfsense/pfsense/commit/0e0f580d3905a228f5d0f11629b68d5605e3f071
However there does seem to be code in place to cut off the header returned by curl before entering the _checkStatus function:

				$response = curl_exec($ch);
				$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
				$header = substr($response, 0, $header_size);
				$data = substr($response, $header_size);
				$this->_checkStatus($ch, $data, $header);

Would be nice to know what header_size and header it finds there.. Perhaps something aint working like it should?

p.s. your running latest snapshot version of pfSense?

4pack · Dec 28, 2016, 11:16 AM

@PiBa:

The problem was 'probably' introduced with/around this commit: https://github.com/pfsense/pfsense/commit/0e0f580d3905a228f5d0f11629b68d5605e3f071
However there does seem to be code in place to cut off the header returned by curl before entering the _checkStatus function:
				$response = curl_exec($ch);
				$header_size = curl_getinfo($ch, CURLINFO_HEADER_SIZE);
				$header = substr($response, 0, $header_size);
				$data = substr($response, $header_size);
				$this->_checkStatus($ch, $data, $header);
Would be nice to know what header_size and header it finds there.. Perhaps something aint working like it should?

p.s. your running latest snapshot version of pfSense?

That would make sense. I'll have it save (or at least log) the header size.

p.s. Yes I am. (2.4.0.b.20161228.0040)

EDIT: Wait, I just realized it fails right before this in the CloudFlare case block. See below.


// This is line 698
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
	'X-Auth-Email: '.$this->_dnsUser.'',
	'X-Auth-Key: '.$this->_dnsPass.'',
	'Content-Type: application/json'
));

// Get zone ID
$getZoneId = "https://{$dnsServer}/client/v4/zones/?name={$this->_dnsDomain}";
curl_setopt($ch, CURLOPT_URL, $getZoneId);
$output = json_decode(curl_exec($ch)); <----- FAILS RIGHT HERE. We need to strip headers before this.
$zone = $output->result[0]->id;

PiBa · Dec 28, 2016, 11:55 AM

Could you try and apply this patch and see if that fixes the issue? You can use the pfSense SystemPatches package for it with a strip count of 2:
https://github.com/PiBa-NL/pfsense/commit/d21097526668ed010b0cbdab424a250211a151ab.patch

4pack · Dec 28, 2016, 7:43 PM

@PiBa:

Could you try and apply this patch and see if that fixes the issue? You can use the pfSense SystemPatches package for it with a strip count of 2:
https://github.com/PiBa-NL/pfsense/commit/d21097526668ed010b0cbdab424a250211a151ab.patch

I pretty much did the same thing except I used curl_setopt($ch, CURLOPT_HEADER, 0); instead of the header stripping code. At the end of the CloudFlare block it is set back to 1. It is finally working using these additions. See below. Yours would work the same I imagine, with an exception of a few more lines of code.


curl_setopt($ch, CURLOPT_HEADER, 0); // +++++++++++++++ ADDED LINE 699
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
	'X-Auth-Email: '.$this->_dnsUser.'',
	'X-Auth-Key: '.$this->_dnsPass.'',
	'Content-Type: application/json'
));
// Get zone ID
$getZoneId = "https://{$dnsServer}/client/v4/zones/?name={$this->_dnsDomain}";
curl_setopt($ch, CURLOPT_URL, $getZoneId);
$output = json_decode(curl_exec($ch));
$zone = $output->result[0]->id;
if ($zone) { // If zone ID was found get host ID
	$getHostId = "https://{$dnsServer}/client/v4/zones/{$zone}/dns_records?name={$this->_FQDN}&type={$recordType}";
	curl_setopt($ch, CURLOPT_URL, $getHostId);
	curl_setopt($ch, CURLOPT_HEADER, 0); //+++++++++++++++ ADDED LINE 715
	$output = json_decode(curl_exec($ch));
	$host = $output->result[0]->id;
	if ($host) { // If host ID was found update host
		$hostData = array(
			"content" => "{$this->_dnsIP}",
			"type" => "{$recordType}",
			"name" => "{$this->_dnsHost}"
		);
		$data_json = json_encode($hostData);
		$updateHostId = "https://{$dnsServer}/client/v4/zones/{$zone}/dns_records/{$host}";
		curl_setopt($ch, CURLOPT_HEADER, 1); // +++++++++++++++ ADDED LINE 726
		curl_setopt($ch, CURLOPT_URL, $updateHostId);
	        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT');
		curl_setopt($ch, CURLOPT_POSTFIELDS, $data_json);
	}
}

I could stuff this into a commit request on GitHub if that would be easier. Let me know if you have any questions.

P.S. On an unrelated note, would it be possible to add a checkbox or something to only use the domain part for updates, because the way it is now it is impossible to update the root domain A record (aka, hostname.org 's A record).

PiBa · Dec 28, 2016, 9:19 PM

What about this? I think that might be the better overall option: https://github.com/PiBa-NL/pfsense/commit/15dcf1320c08eb9339eda3e6fdf04599c51694b7.patch

4pack · Dec 29, 2016, 1:50 PM

@PiBa:

What about this? I think that might be the better overall option: https://github.com/PiBa-NL/pfsense/commit/15dcf1320c08eb9339eda3e6fdf04599c51694b7.patch

This looks like it would work. I'll have to apply it later to find out. That patch seems to just remove headers by default unless it's needed, right?

PiBa · Dec 29, 2016, 2:06 PM

Yes that is the general idea behind it, get the headers just before they are 'required', and quite a bit 'cleaner' than my previous solution :D. its even including some cleanup and a bugfix for another provider. If you could verify that it works properly for you as i don't use CloudFlare which seems to be the most curl involved option in that location.. Ill send a pullrequest to get it integrated.

4pack · Jan 2, 2017, 3:54 PM

@PiBa:

Yes that is the general idea behind it, get the headers just before they are 'required', and quite a bit 'cleaner' than my previous solution :D. its even including some cleanup and a bugfix for another provider. If you could verify that it works properly for you as i don't use CloudFlare which seems to be the most curl involved option in that location.. Ill send a pullrequest to get it integrated.

Sorry I didn't test this sooner but this worked! I'll have to make a new thread requesting to add a checkbox or something to only use the domain part of the update. I did fix it myself by adding a condition that if there is only an underscore in the hostname, then only use the domain part.