Unbound TCP buffer settings not sticky

bryan.paradis · Feb 28, 2014, 7:46 PM

./conf.default/config.xml or ./cf/conf/config.xml?

Same thing /conf is a link of /cf/conf

mark62 are you running pfsense 2.1 that has been upgraded? Likely you need to manually clean out unbound.

Uninstall unbound
Turn on rw
Find / -name *unbound*
Delete everything you can find
/conf/config.XML clear out all everything to do with unbound. Backup first maybe.
Once you are sure you have killed everything reinstall.
If you have problems with internet try adding a DNS to resolv.conf

wagonza · Feb 28, 2014, 8:17 PM

@bryan.paradis:

@markn62:

./conf.default/config.xml or ./cf/conf/config.xml?

Same thing /conf is a link of /cf/conf

Yeah but /conf.default is not the same thing thats used, as an example, for reverting to factory defaults.

bryan.paradis · Feb 28, 2014, 8:19 PM

@wagonza:

@bryan.paradis:

@markn62:

./conf.default/config.xml or ./cf/conf/config.xml?

Same thing /conf is a link of /cf/conf

Yeah but /conf.default is not the same thing thats used for factory defaults.

Haha whoops Justin woke UP didn't notice default

markn62 · Feb 28, 2014, 8:39 PM

Thanks guys. Where are the firmware upgrades and package upgrades hosted to do a DNS resolve on? Same for both? Do I just need pfsense.com. Suppose a netstat during a temp pkg install might answer for one of the two. Hope the host IP doesn't change often. Good suggestion to locally resolve should happen to scramble Unbound again.

I find one unbound install in the xml after I had uninstalled the pkg by GUI this morning. As well, two unbound files remain. I'll clean these up tonight, cron a reboot, and report back.

Really appreciate your help…

bryan.paradis · Feb 28, 2014, 9:23 PM

@markn62:

Thanks guys. Where are the firmware upgrades and package upgrades hosted to do a DNS resolve on? Same for both? Do I just need pfsense.com. Suppose a netstat during a temp pkg install might answer for one of the two. Hope the host IP doesn't change often. Good suggestion to locally resolve should happen to scramble Unbound again.

I find one unbound install in the xml after I had uninstalled the pkg by GUI this morning. As well, two unbound files remain. I'll clean these up tonight, cron a reboot, and report back.

Really appreciate your help…

files.pfsense.org - pbi
www.pfsense.com - config

markn62 · Mar 3, 2014, 1:59 AM

Cleaned the XML of remnant Unbound package info. The reinstall went well and seems to be working normally now. Odd though, the install reported 21_1 rather than 21_3. After install the "Installed Packages" reports 21_3. And the Unbound Status reports 21_3. Looks like outgoing-num-tcp: 0 & incoming-num-tcp: 0 are still making it from GUI to XML.

–-------------------

Checking for package installation...
Downloading http://files.pfsense.org/packages/amd64/8/All/unbound-1.4.21_1-amd64.pbi

version: 1.4.21
verbosity: 1
threads: 2
modules: 2 [ validator iterator ]
uptime: 51 seconds
unbound (pid 34067) is running…

Unbound Services 1.4.21_3
Unbound is a validating, recursive, and caching DNS resolver. This package is a drop in replacement for Services: DNS Forwarder and also supports DNSSEC extensions. Once installed please configure the Unbound service by visiting Services: Unbound DNS.

bryan.paradis · Mar 3, 2014, 2:05 AM

@markn62:

Cleaned the XML of remnant Unbound package info. The reinstall went well and seems to be working normally now. Odd though, the install reported 21_1 rather than 21_3. After install the "Installed Packages" reports 21_3. And the Unbound Status reports 21_3. Looks like outgoing-num-tcp: 0 & incoming-num-tcp: 0 are still making it from GUI to XML.

–-------------------

Checking for package installation...
Downloading http://files.pfsense.org/packages/amd64/8/All/unbound-1.4.21_1-amd64.pbi

version: 1.4.21
verbosity: 1
threads: 2
modules: 2 [ validator iterator ]
uptime: 51 seconds
unbound (pid 34067) is running…

Unbound Services 1.4.21_3
Unbound is a validating, recursive, and caching DNS resolver. This package is a drop in replacement for Services: DNS Forwarder and also supports DNSSEC extensions. Once installed please configure the Unbound service by visiting Services: Unbound DNS.

Great! Wagonza updated the file already so the configuration should be working properly as you see.

As far as I can see its just the packages list xml has the newer version number and the unbound package xml has the older version number still. Wagonza just needs to update that line.

wagonza · Mar 3, 2014, 4:49 AM

Yup that's all. Package just needs a rebuild on the package servers as well. Glad to hear all ok markn62.

markn62 · Mar 3, 2014, 2:14 PM

Not what I was hoping to report. Thought all was well after the reinstall. Watched for about an hour and everything appeared normal. Bout 30 minutes after I walked away Unbound died and would not restart. Pointed back to the stand-alone unbound service AGAIN. This is getting so frustrating!

wagonza · Mar 3, 2014, 3:40 PM

Any logs (both system and package logs) giving an indication as to why?

bryan.paradis · Mar 3, 2014, 3:51 PM

@markn62:

Not what I was hoping to report. Thought all was well after the reinstall. Watched for about an hour and everything appeared normal. Bout 30 minutes after I walked away Unbound died and would not restart. Pointed back to the stand-alone unbound service AGAIN. This is getting so frustrating!

You will need to be much clearer and specific.

What do you mean it died? Still running but not responding? Crashed/No process?
What do you see in system.log? Or any other pertinent logs. Better yet pastebin the log so we can look.
Am I assuming correctly you mean a separate box running unbound when you say stand-alone unbound service?

Is this a fresh 2.1 installation or a upgrade from prior versions to 2.1?
If an upgrade what packages do you have installed using pkg_info on the shell?

markn62 · Mar 3, 2014, 5:02 PM

By died I mean the unbound service stopped on it's own after about an hour of normal operation and the watchdog or manual restart wouldn't get it started again. Yes, stand-alone means a separate box. It is an upgrade from 2.0 to 2.1. Pkg_Info reports no packages installed even though there are. I'm not certain the name and location of the system and package log. I could binpaste the SNMP log if I knew how. Is that the insert code button? Bout 100 lines of text probably best not pasted in this post.

bryan.paradis · Mar 3, 2014, 5:04 PM

@markn62:

By died I mean the unbound service stopped on it's own after about an hour of normal operation and the watchdog or manual restart wouldn't get it started again. Yes, stand-alone means a separate box. It is an upgrade from 2.0 to 2.1. Pkg_Info reports no packages installed even though there are. I'm not certain the name and location of the system and package log. I could binpaste the SNMP log if I knew how. Is that the insert code button? Bout 100 lines of text probably best not pasted in this post.

http://pastebin.com/

/var/log

system.log
unbound.log? Is there one?
/tmp/php_errors.txt

what happens when you try to start unbound manually?

markn62 · Mar 3, 2014, 5:29 PM

Forum access is with a PC other than what has access to the router logs so it would take me a bit to transfer files around and post here. For now here is the SNMP; http://pastebin.com/embed_js.php?i=kTkuRnzX I hope I binpasted correctly using a hyperlink. As mentioned, a manual restart would do nothing. The service would remain unstarted.

A quick look via the GUI edit file I find php_errors.txt is empty even though the SNMP shows some. The router was rebooted early AM by cron. The unbound log only has CLOG and some odd symbol after the word. The system log has plenty and isn't cooperative by remote login. Large file I suppose. May have to post the system log this evening.

markn62 · Mar 3, 2014, 5:42 PM

Forgot to mention installed packages. Installed and running fine for a few months other than Unbound are;

Cron
PhpSysInfo
Service Watchdog
SipProxD
Unbound

bryan.paradis · Mar 3, 2014, 5:47 PM

Just grab Winscp and use same credentials as ssh. Just like using a FTP server. Unbound should be complaining somewhere about something.

markn62 · Mar 3, 2014, 6:08 PM

Oh it complained all right. I take it you didn't look at the SNMP file. First it installed, then ran normally for a bit, then started to lag on the median time, then started with PHP errors, warning about needing an increase in open files. Then finally that there was no such file or directory. So in short, the wheels eventually fell off. Never seen anything like this. I presume this slow failure occured as more demand hit the unbound package. I didn't get up at 4AM to see if Unbound would start again after the reboot. Cleaning up the XML required that I repopulate the Unbound settings via the GUI. I matched what was in the backup XML so it should have the same configuration as before the upgrade. The SNMP reports that version 1.4.21 installed, not 1.4.23. Presume the version reported isn't an issue.

bryan.paradis · Mar 3, 2014, 6:54 PM

check the unbound-control script and look where it is trying to start the binary from. Is the path correct? What is the path really?

I am going to be flying for some IT work tonight so I won't be around for a bit maybe! Darn this labor market.

wagonza · Mar 3, 2014, 7:15 PM

What happens if you execute "/usr/pbi/unbound-amd64/sbin/unbound-control start" from the command line?
Do you get a warning on 'too many file descriptors requested'?

The second issue is that there are two watchdog services running. The Unbound package has its own watchdog (log lines with Unbound_Alarm) called unbound_monitor.sh. So that starts it up Unbound and then so does the watchdog package. Hence why you seeing the "bind: address already in use" error message.

You should see unbound-1.4.21_1-amd64 using 'pbi_info'. Doesnt make a difference with _X bit, however I have bumped that on the package servers to avoid confusion in the future. Dont reinstall just yet as the package builders still need to build and create the updated version.

markn62 · Mar 3, 2014, 7:37 PM

Thanks guys for all your help on this, much appreciated!

Bryan, have a safe flight. I find;
/usr/local/sbin/unbound-control
/usr/pbi/unbound-amd64/sbin/unbound-control
/usr/pbi/unbound-amd64/.sbin/unbound-control

Presume it's the …amd64/sbin...

Within this I have;
unbound
unbound-anchor
unbound-checkconf
unbound-control
unbound-control-setup
unbound-host

Not sure which file is the script. The unbound-control file is not readable. The unbound-control-setup has a line;
DESTDIR=/usr/pbi/unbound-amd64/etc/unbound

Wagonza,
I didn't know Unbound had its own watchdog, explains a lot. Perhaps certain package choices could be removed from the Wachdog service app list for those that will cause a conflict. If I execute "unbound-control start" I imagine Unbound will halt due to the fact the DNS forwarder is currently enabled and functioning in it's place. The GUI barks if fwd'r is enabled when unbound is enabled. This box is in active service so I have to be careful not to execute anything that will dump states or crash services. Can you assure nothing unexpected will happen except a possible warning?

Thanks again,
Mark