Unbound TCP buffer settings not sticky
-
After installing unbound 1.4.21_3 none of the custom options i am using seem to work anymore. The error is
php: config.inc: The command '/usr/pbi/unbound-amd64/sbin/unbound-control start' returned exit code '1', the output was '/usr/pbi/unbound-amd64/etc/unbound/unbound.conf:104: error: syntax error read /usr/pbi/unbound-amd64/etc/unbound/unbound.conf failed: 1 errors in configuration file [1393437916] unbound[79295:0] fatal error: Could not read config file: /usr/pbi/unbound-amd64/etc/unbound/unbound.conf'
When i went into the file it starts on the first line of the custom options, if i remove all the options i have it will boot fine.
"msg-cache-slabs: 1;rrset-cache-slabs: 1;infra-cache-slabs: 1;key-cache-slabs: 1;outgoing-range: 950;val-clean-additional: yes;harden-glue: yes;do-not-query-localhost: yes;do-ip6: no;use-caps-for-id: no;hide-identity: yes;hide-version: yes"
the custom options i was using. I even tried using them one at a time and it still give me the errors.
The options in the configuration file should be like below
option: blah option: blah option: blah
the gui should take the custom options then split them by the delimiter ; then it puts them on a new line for each option
# Handle custom options if (!empty($adv_config['custom_options'])) { $custom_options = explode(";", ($adv_config['custom_options'])); $unbound_conf .= "\n# Unbound Custom options\n"; foreach ($custom_options as $ent) $unbound_conf .= $ent."\n";
Is that the line you see exactly in your config file? What are you putting in the advanced options form in the gui?
I looked over the code and it is the same. In the advanced options in the gui i have
"msg-cache-slabs: 1;rrset-cache-slabs: 1;infra-cache-slabs: 1;key-cache-slabs: 1;outgoing-range: 950;val-clean-additional: yes;harden-glue: yes;do-not-query-localhost: yes;do-ip6: no;use-caps-for-id: no;hide-identity: yes;hide-version: yes" as the semi colon is used to make a new line.
when i go into the config file, it is being set correctly.
Unbound Custom options
msg-cache-slabs: 1
rrset-cache-slabs: 1
infra-cache-slabs: 1
key-cache-slabs: 1
outgoing-range: 950
val-clean-additional: yes
harden-glue: yes
do-not-query-localhost: yes
do-ip6: no
use-caps-for-id: no
hide-identity: yes
hide-version: yes -
@Shinzo,
What version package were you running prior? I can tell you outgoing-range as a custom setting, along with a few select others, has never worked in 2.0 or 2.1, see my post @ https://forum.pfsense.org/index.php/topic,64897.msg352131.html#msg352131 . I had to strip my custom options down to only forward zone and addresses. No way to detune unbound. So you cant really troubleshoot the package by probable trial'n'error if it acts quirky. Unbound reboots by itself a few times a day on my 64-bit install. And I've never read anywhere until today that the word "option:" needs to be placed before every custom option entry if I read this thread correctly. Where is this documented? I don't use them in my options and the ones I have entered show in the unbound.conf fine. Can't say if, for example, "option: outgoing-range: 950" would be accepted by unbound but imagine it would get hung up on the pair of colon syntax, although I haven't tried it. -
@Shinzo,
What version package were you running prior? I can tell you outgoing-range as a custom setting, along with a few select others, has never worked in 2.0 or 2.1, see my post @ https://forum.pfsense.org/index.php/topic,64897.msg352131.html#msg352131 . I had to strip my custom options down to only forward zone and addresses. No way to detune unbound. So you cant really troubleshoot the package by probable trial'n'error if it acts quirky. Unbound reboots by itself a few times a day on my 64-bit install. And I've never read anywhere until today that the word "option:" needs to be placed before every custom option entry if I read this thread correctly. Where is this documented? I don't use them in my options and the ones I have entered show in the unbound.conf fine. Can't say if, for example, "option: outgoing-range: 950" would be accepted by unbound but imagine it would get hung up on the pair of colon syntax, although I haven't tried it.The "option: blah" was used as a example of how the output should be. As i continue to troubleshoot my issue, i removed the 4 previous lines which had to do with the forward-zones and got it up and running, il keep looking
oh and i was using version 1.4.21_2 and it was on a 2.1.1 box
-
FILE FORMAT
There must be whitespace between keywords. Attribute keywords end with
a colon ':'. An attribute is followed by its containing attributes, or
a value.Files can be included using the include: directive. It can appear any-
where, it accepts a single file name as argument. Processing continues
as if the text from the included file was copied into the config file
at that point. If also using chroot, using full path names for the
included files works, relative pathnames for the included names work if
the directory where the daemon is started equals its chroot/working
directory. Wildcards can be used to include multiple files, see
glob(7).Server Options
These options are part of the server: clause.Looks like the configuration file should be laid out in clause sections full of options as below:
server: option: blah remote-control: option: blah stub-zone: option: blah forward-zone: option: blah
but it seems that the custom server options are being added after the forward-zone clause. I have moved the code in the unbound.inc file up above the foward-zone code and attached it. Try backing up the unbound.inc and placing the one I have attached in its place.
-
#########################
Unbound configuration
#########################
Server config
server:
chroot: ""
username: "unbound"
directory: "/usr/pbi/unbound-amd64/etc/unbound"
pidfile: "/var/run/unbound.pid"
harden-referral-path: no
prefetch: no
prefetch-key: no
use-syslog: yes
port: 53
verbosity: 0
do-ip4: yes
do-ip6: yes
do-udp: yes
do-tcp: yes
do-daemonize: yes
module-config: "validator iterator"
unwanted-reply-threshold: 5000000
num-queries-per-thread: 512
jostle-timeout: 200
infra-host-ttl: 900
prefetch: no
prefetch-key: no
infra-cache-numhosts: 15000
outgoing-num-tcp: 10
incoming-num-tcp: 10
edns-buffer-size: 4096
statistics-interval: 0
extended-statistics: no
statistics-cumulative: no
cache-max-ttl: 86400
cache-min-ttl: 0
harden-dnssec-stripped: yes
hide-identity: yes
hide-version: yes
harden-glue: yes
num-threads: 4
msg-cache-slabs: 4
rrset-cache-slabs: 4
infra-cache-slabs: 4
key-cache-slabs: 4
msg-cache-size: 10m
rrset-cache-size: 20m
outgoing-range: 8192
#so-rcvbuf: 4m
#so-sndbuf: 4mInterface IP(s) to bind to
interface: 192.168.1.1
Interfaces to query from
outgoing-interface: 192.168.1.1
auto-trust-anchor-file: /usr/pbi/unbound-amd64/etc/unbound/root-trust-anchor
Access Control
Local attached networks allowed to utilize service and any user added ACLs
access-control: 127.0.0.0/8 allow
access-control: ::1 allow
access-control: 192.168.1.0/27 allowFor DNS Rebinding prevention
private-address: 10.0.0.0/8
private-address: 172.16.0.0/12
private-address: 192.168.0.0/16
private-address: 169.254.0.0/16
private-address: fd00::/8
private-address: fe80::/10Set private domains in case authorative name server returns a RFC1918 IP address
Host entries
local-zone: "lan" transparent
local-data-ptr: "127.0.0.1 localhost"
local-data: "localhost A 127.0.0.1"
local-data: "localhost.lan A 127.0.0.1"
local-data-ptr: "::1 localhost"
local-data: "localhost AAAA ::1"
local-data: "localhost.lan AAAA ::1"
local-data-ptr: "192.168.1.1 peanuts.lan"
local-data: "peanuts.lan A 192.168.1.1"
local-data: "peanuts A 192.168.1.1"DHCP Reservations
/etc/hosts entries
Domain overrides
#forward-zone:
#name: "."
#forward-addr: 75.75.75.75
#forward-addr: 75.75.76.76Unbound Custom options
msg-cache-slabs: 1
rrset-cache-slabs: 1
infra-cache-slabs: 1
key-cache-slabs: 1
outgoing-range: 950
val-clean-additional: yes
harden-glue: yes
do-not-query-localhost: yes
do-ip6: no
use-caps-for-id: no
hide-identity: yes
hide-version: yesRemote Control Config
remote-control:
control-enable: yes
control-interface: 127.0.0.1
control-port: 953
server-key-file: "/usr/pbi/unbound-amd64/etc/unbound/unbound_server.key"
server-cert-file: "/usr/pbi/unbound-amd64/etc/unbound/unbound_server.pem"
control-key-file: "/usr/pbi/unbound-amd64/etc/unbound/unbound_control.key"
control-cert-file: "/usr/pbi/unbound-amd64/etc/unbound/unbound_control.pem"thats how my file looks like when its working. Some of those options are doubled because they weren't there before or i didn't want to manually change the values every time the package was updated. If i put them in the custom options window, then it will overwrite the default value to what ever i wanted since the custom options are read last
thats the output in the status tab when its running. or do you want the config file?
-
#########################
Unbound configuration
#########################
snip
thats how my file looks like when its working. Some of those options are doubled because they weren't there before or i didn't want to manually change the values every time the package was updated. If i put them in the custom options window, then it will overwrite the default value to what ever i wanted since the custom options are read last
Check my update attachement. I think it is because the custom options are getting put in after the server: clause has been finished because the forward-zones clause has started. Can you test it out to see if it works properly with my changes?
-
sure just need a minute
-
#########################
Unbound configuration
#########################
snip
thats how my file looks like when its working. Some of those options are doubled because they weren't there before or i didn't want to manually change the values every time the package was updated. If i put them in the custom options window, then it will overwrite the default value to what ever i wanted since the custom options are read last
Check my update attachement. I think it is because the custom options are getting put in after the server: clause has been finished because the forward-zones clause has started. Can you test it out to see if it works properly with my changes?
yeah that did fix it.
it looks like
Unbound Custom options
msg-cache-slabs: 1
rrset-cache-slabs: 1
infra-cache-slabs: 1
key-cache-slabs: 1
val-clean-additional: yes
harden-glue: yes
do-not-query-localhost: yes
do-ip6: no
use-caps-for-id: no
hide-identity: yes
hide-version: yes
forward-zone:
name: "."
forward-addr: 75.75.75.75
forward-addr: 75.75.76.76 -
So all the options work correctly now?
-
I've never had an issue with forward zones taking. Bryan, where did you get the syntax document? Is it for the unbound package or unbound stand-alone, two different animals. And your work begs another question. Do the custom options override the GUI settings or cause unbound to not start?
Guess I'll have to do some testing myself. Looks like forward zones no longer make it from the GUI to the config file. If so, the GUI notes below the custom options box is in need of editing to clarify the syntax required.
-
-
Yes that was a mistake. Thanks @bryan.paradis. Fix pushed.
-
I stand corrected, the old syntax is still putting forward zones in the config file. Options are pushed way down with the recent expansion of reserve leases and hostnames. Thanks for clarifying the syntax bryan.
-
I've never had an issue with forward zones taking. Bryan, where did you get the syntax document? Is it for the unbound package or unbound stand-alone, two different animals. And your work begs another question. Do the custom options override the GUI settings or cause unbound to not start?
Guess I'll have to do some testing myself. Looks like forward zones no longer make it from the GUI to the config file. If so, the GUI notes below the custom options box is in need of editing to clarify the syntax required.
Forward zones would take but the custom options are being put in the forward zones clause area and not in the server clause area causing errors if you add any.
I stand corrected, the old syntax is still putting forward zones in the config file. Options are pushed way down with the recent expansion of reserve leases and hostnames. Thanks for clarifying the syntax bryan.
GUI generates a config file in unbound.inc out of what you fill out which is then what unbound uses. Just like the standalone.
Yes that was a mistake. Thanks @bryan.paradis. Fix pushed.
:) Your welcome
-
Thanks for clarifying Bryan. Wagonza, either it's not installing correctly or the unbound fix broke the service. When I install the package it always fails the first time and shows installed but won't run. Reboot and reinstall pkg, then unbound starts but doesn't resolve 100%. Hit'n'miss on the queries. Some webpages load, via google links all connections would fail. Had to point router to backup Unbound service on another box.
When I finally get it to install it shows in GUI as 1.4.21_3 just as it does in the available packages list. When I uninstall it the script reports it's uninstalling 1.4.21_1. Leaving remnants behind? Also notice watchdog leaves unbound in it's list and fills the log. Maybe its normal the uninstall wouldn't remove a watchdog entry.
And I'm still not clear when it can't install the unbound pkg from pfsense server that it doesn't use a cached copy of the package from local drive like the pkg installer says it will do. When unbound acts up its very hard to fix PfSense since all web updates are reliant on it. I'm going to look into putting unbound on an embedded linux box and forget it as a PfSense package at least until such time as it gets a thorough looking over.
On the bright side, outgoing-num-tcp & incoming-num-tcp GUI entries are now sticky.
-
Interesting…looks like you may have remnants of an old package lying around which might be the cause of your problems.
The fact that Watchdog still has it in its list is a good indicator of this.Sorry you have had problems with the package its been kind(er) to others.
Its going into 2.2 as a replacement for DNSMasq so no longer will it be a package. Might be a better experience for you then. -
Just noting that I reinstalled unbound multiple times on multiple boxes without any problems whatsoever after the latest batch of commits… So yeah, you probably have some state junk leftovers on your box.
-
States get reset on reboot, no? Don't think it's states unless only a state reset will do. So I should console ipkg list then ipkg remove package.name? Or is more cleanup needed?
-
Check your config.xml for multiple entries for Unbound.
Look between <installedpackages>xml tags.</installedpackages> -
./conf.default/config.xml or ./cf/conf/config.xml?
-
The latter one.
-
./conf.default/config.xml or ./cf/conf/config.xml?
Same thing /conf is a link of /cf/conf
mark62 are you running pfsense 2.1 that has been upgraded? Likely you need to manually clean out unbound.
Uninstall unbound
Turn on rw
Find / -name *unbound*
Delete everything you can find
/conf/config.XML clear out all everything to do with unbound. Backup first maybe.
Once you are sure you have killed everything reinstall.
If you have problems with internet try adding a DNS to resolv.conf -
./conf.default/config.xml or ./cf/conf/config.xml?
Same thing /conf is a link of /cf/conf
Yeah but /conf.default is not the same thing thats used, as an example, for reverting to factory defaults.
-
./conf.default/config.xml or ./cf/conf/config.xml?
Same thing /conf is a link of /cf/conf
Yeah but /conf.default is not the same thing thats used for factory defaults.
Haha whoops Justin woke UP didn't notice default
-
Thanks guys. Where are the firmware upgrades and package upgrades hosted to do a DNS resolve on? Same for both? Do I just need pfsense.com. Suppose a netstat during a temp pkg install might answer for one of the two. Hope the host IP doesn't change often. Good suggestion to locally resolve should happen to scramble Unbound again.
I find one unbound install in the xml after I had uninstalled the pkg by GUI this morning. As well, two unbound files remain. I'll clean these up tonight, cron a reboot, and report back.
Really appreciate your help…
-
Thanks guys. Where are the firmware upgrades and package upgrades hosted to do a DNS resolve on? Same for both? Do I just need pfsense.com. Suppose a netstat during a temp pkg install might answer for one of the two. Hope the host IP doesn't change often. Good suggestion to locally resolve should happen to scramble Unbound again.
I find one unbound install in the xml after I had uninstalled the pkg by GUI this morning. As well, two unbound files remain. I'll clean these up tonight, cron a reboot, and report back.
Really appreciate your help…
files.pfsense.org - pbi
www.pfsense.com - config -
Cleaned the XML of remnant Unbound package info. The reinstall went well and seems to be working normally now. Odd though, the install reported 21_1 rather than 21_3. After install the "Installed Packages" reports 21_3. And the Unbound Status reports 21_3. Looks like outgoing-num-tcp: 0 & incoming-num-tcp: 0 are still making it from GUI to XML.
–-------------------
Checking for package installation...
Downloading http://files.pfsense.org/packages/amd64/8/All/unbound-1.4.21_1-amd64.pbi
version: 1.4.21
verbosity: 1
threads: 2
modules: 2 [ validator iterator ]
uptime: 51 seconds
unbound (pid 34067) is running…
Unbound Services 1.4.21_3
Unbound is a validating, recursive, and caching DNS resolver. This package is a drop in replacement for Services: DNS Forwarder and also supports DNSSEC extensions. Once installed please configure the Unbound service by visiting Services: Unbound DNS. -
Cleaned the XML of remnant Unbound package info. The reinstall went well and seems to be working normally now. Odd though, the install reported 21_1 rather than 21_3. After install the "Installed Packages" reports 21_3. And the Unbound Status reports 21_3. Looks like outgoing-num-tcp: 0 & incoming-num-tcp: 0 are still making it from GUI to XML.
–-------------------
Checking for package installation...
Downloading http://files.pfsense.org/packages/amd64/8/All/unbound-1.4.21_1-amd64.pbi
version: 1.4.21
verbosity: 1
threads: 2
modules: 2 [ validator iterator ]
uptime: 51 seconds
unbound (pid 34067) is running…
Unbound Services 1.4.21_3
Unbound is a validating, recursive, and caching DNS resolver. This package is a drop in replacement for Services: DNS Forwarder and also supports DNSSEC extensions. Once installed please configure the Unbound service by visiting Services: Unbound DNS.Great! Wagonza updated the file already so the configuration should be working properly as you see.
As far as I can see its just the packages list xml has the newer version number and the unbound package xml has the older version number still. Wagonza just needs to update that line.
-
Yup that's all. Package just needs a rebuild on the package servers as well. Glad to hear all ok markn62.
-
Not what I was hoping to report. Thought all was well after the reinstall. Watched for about an hour and everything appeared normal. Bout 30 minutes after I walked away Unbound died and would not restart. Pointed back to the stand-alone unbound service AGAIN. This is getting so frustrating!
-
Any logs (both system and package logs) giving an indication as to why?
-
Not what I was hoping to report. Thought all was well after the reinstall. Watched for about an hour and everything appeared normal. Bout 30 minutes after I walked away Unbound died and would not restart. Pointed back to the stand-alone unbound service AGAIN. This is getting so frustrating!
You will need to be much clearer and specific.
What do you mean it died? Still running but not responding? Crashed/No process?
What do you see in system.log? Or any other pertinent logs. Better yet pastebin the log so we can look.
Am I assuming correctly you mean a separate box running unbound when you say stand-alone unbound service?Is this a fresh 2.1 installation or a upgrade from prior versions to 2.1?
If an upgrade what packages do you have installed using pkg_info on the shell? -
By died I mean the unbound service stopped on it's own after about an hour of normal operation and the watchdog or manual restart wouldn't get it started again. Yes, stand-alone means a separate box. It is an upgrade from 2.0 to 2.1. Pkg_Info reports no packages installed even though there are. I'm not certain the name and location of the system and package log. I could binpaste the SNMP log if I knew how. Is that the insert code button? Bout 100 lines of text probably best not pasted in this post.
-
By died I mean the unbound service stopped on it's own after about an hour of normal operation and the watchdog or manual restart wouldn't get it started again. Yes, stand-alone means a separate box. It is an upgrade from 2.0 to 2.1. Pkg_Info reports no packages installed even though there are. I'm not certain the name and location of the system and package log. I could binpaste the SNMP log if I knew how. Is that the insert code button? Bout 100 lines of text probably best not pasted in this post.
/var/log
system.log
unbound.log? Is there one?
/tmp/php_errors.txtwhat happens when you try to start unbound manually?
-
Forum access is with a PC other than what has access to the router logs so it would take me a bit to transfer files around and post here. For now here is the SNMP; http://pastebin.com/embed_js.php?i=kTkuRnzX I hope I binpasted correctly using a hyperlink. As mentioned, a manual restart would do nothing. The service would remain unstarted.
A quick look via the GUI edit file I find php_errors.txt is empty even though the SNMP shows some. The router was rebooted early AM by cron. The unbound log only has CLOG and some odd symbol after the word. The system log has plenty and isn't cooperative by remote login. Large file I suppose. May have to post the system log this evening.
-
Forgot to mention installed packages. Installed and running fine for a few months other than Unbound are;
-
Cron
-
PhpSysInfo
-
Service Watchdog
-
SipProxD
-
Unbound
-
-
Just grab Winscp and use same credentials as ssh. Just like using a FTP server. Unbound should be complaining somewhere about something.
-
Oh it complained all right. I take it you didn't look at the SNMP file. First it installed, then ran normally for a bit, then started to lag on the median time, then started with PHP errors, warning about needing an increase in open files. Then finally that there was no such file or directory. So in short, the wheels eventually fell off. Never seen anything like this. I presume this slow failure occured as more demand hit the unbound package. I didn't get up at 4AM to see if Unbound would start again after the reboot. Cleaning up the XML required that I repopulate the Unbound settings via the GUI. I matched what was in the backup XML so it should have the same configuration as before the upgrade. The SNMP reports that version 1.4.21 installed, not 1.4.23. Presume the version reported isn't an issue.
-
check the unbound-control script and look where it is trying to start the binary from. Is the path correct? What is the path really?
I am going to be flying for some IT work tonight so I won't be around for a bit maybe! Darn this labor market.
-
What happens if you execute "/usr/pbi/unbound-amd64/sbin/unbound-control start" from the command line?
Do you get a warning on 'too many file descriptors requested'?The second issue is that there are two watchdog services running. The Unbound package has its own watchdog (log lines with Unbound_Alarm) called unbound_monitor.sh. So that starts it up Unbound and then so does the watchdog package. Hence why you seeing the "bind: address already in use" error message.
You should see unbound-1.4.21_1-amd64 using 'pbi_info'. Doesnt make a difference with _X bit, however I have bumped that on the package servers to avoid confusion in the future. Dont reinstall just yet as the package builders still need to build and create the updated version.