PHP Fatal Error Kea2Unbound

ajama1

Kea and unbound still run, and I haven't noticed any adverse effects. Any ideas on this error

I am running Pfsense 2.8 Beta

marcosm

Please reproduce the issue then generate a status report by going to /status.php and downloading the file linked on the page. You may share the report for review here:
https://nc.netgate.com/nextcloud/s/YHMtNSm8cjEN6Ne

ajama1

@marcosm Uploaded

marcosm

Run the following command:
/usr/local/sbin/unbound-control -c /var/unbound/unbound.conf list_local_data > /tmp/_197325.txt
Then also share the file /tmp/_197325.txt - you can download it from Diagnostics > Command Prompt.

ajama1

@marcosm Done, and I can see the file is quite large. I am running PfBlocker-Deve,l and I didn't have this problem in pfSense 2.7.2

No changes to pfblocker settings. I did a fresh install of pfSense on a new firewall due to a hardware issue and restored the config.

cmcdonald

@ajama1 said in PHP Fatal Error Kea2Unbound:

@marcosm Done, and I can see the file is quite large. I am running PfBlocker-Deve,l and I didn't have this problem in pfSense 2.7.2

No changes to pfblocker settings. I did a fresh install of pfSense on a new firewall due to a hardware issue and restored the config.

that would explain it.

This functionality is new in 2.8 and clearly doesn't scale well when Unbound has millions of records in its local cache. We will continue to optimize this.

ajama1

@cmcdonald More than happy to test any optimisations.

cmcdonald

@ajama1 said in PHP Fatal Error Kea2Unbound:

@cmcdonald More than happy to test any optimisations.

Thanks will reach out when we've got something to test.

ajama1

@cmcdonald Hi,

Will this be resolved before 2.8 comes out of BETA?

Gertjan

@cmcdonald said in PHP Fatal Error Kea2Unbound:

when Unbound has millions of records in its local cache

Your "list_local_data" gave a "millions" lines :

[25.03-BETA][root@pfSense.bhf.tld]/root: /usr/local/sbin/unbound-control -c /var/unbound/unbound.conf list_local_data | wc -l
     718

?

as every line is followed by one empty line after it, my 'file' has 359 entries.
Among them, env 30 are the IP and revers e IP of all know LAN devices, and there are also some line like :

invalid.        10800   IN      SOA     localhost. nobody.invalid. 1 3600 1200 604800 10800

To have "millions" you have "millions" of LAN devices ?
"Kea2Unbound" is a script file which are 'slow' to execute. Even if it was a binary, it would take 'a long time' to parse a "1 million lines" file.
At its start, kea, the DHCP (and DHCPv6) server has to create, maintain, these million of leases, and then dump them to a file that will be parsed by Kea2Unbound so they get 'injected' into "unbound".
Even kea would be super slow to start ....

Without dumping it here (or on pastebin) : what did you find in your /tmp/_197325.txt that explains why it was that big ?

Btw : just asking out of cursorily.
With my 30+ LAN devices, and some guest networks (captive portal, etc) this "Kea2Unbound" solved a decade old 'problem' for me.

marcosm

@Gertjan It includes all unbound records not just LAN leases. It's a lot but it makes sense once you include multiple DNSBL sources in pfBlockerNG. Setting DNSBL Mode to python may help with the issue.

ajama1

@marcosm Just changed the DNSBL mode from unbound to Python. Will see if this works

cmcdonald

@marcosm said in PHP Fatal Error Kea2Unbound:

@Gertjan It includes all unbound records not just LAN leases. It's a lot but it makes sense once you include multiple DNSBL sources in pfBlockerNG. Setting DNSBL Mode to python may help with the issue.

Yea, the Python DNSBL mode should help in this case.

backstory:

Unbound is architected as a collection of middleware "modules". As a DNS query comes in, these modules can choose to either handle the query outright, modify the query, and/or punt it to the next module in the list. By default there are two modules: validator and iterator. validator does DNSEC and iterator does the actual recursive resolving that Unbound is known for.

The Python module can sit somewhere in this list of modules...hence the Pre validator or Post validator distinction in the GUI. The ordering of the modules is important as you'll see.

The idea behind the Python module is to wrap and expose the C interface (Unbound is written in C) to a Python script. However, this has the added overhead of calling into the Python interpreter for every DNS query that Unbound receives. There is also the memory overhead for allocating python objects, etc. Hence the warning about Python mode requiring more memory...it does.

I've always considered the Python module more as a development tool for prototyping and doing PoCs for exploring a problem space, not something you'd necessarily want to implement a production solution in.

The next version of the kea2unbound solution will probably look something like this:

Implemented as a true Unbound module (first prototyped in Python and then rewritten in C, or perhaps Go with a C shim).
When a DNS query comes in for a host with a domain that is considered to be a "local domain" from the perspective of Kea, this host will first get checked against the active lease database from Kea.
A matching lease record will be transformed into appropriate DNS records (A/AAAA + PTR records) and inserted into the Unbound cache with the TTL being one-third of the lease duration. These records will be served to clients. Unbound cache rotation will take over the lifecycle of these records.
The next time a request is made for said host, the Unbound cache will be inspected first, so no further communication with Kea is necessary so-long as the cache is warm.

This new module will naturally come first in the list of Unbound modules so that we can handle it early, and it has the benefit of being about as fast as you could imagine it can possibly go. There is no need for reconciling and synchronizing lists of records for Unbound configuration files, etc.

The current implementation inserts and removes DNS records in response to Kea events, but that might not be the best solution.

Gertjan

@marcosm said in PHP Fatal Error Kea2Unbound:

@Gertjan It includes all unbound records not just LAN leases.

Then I have an issue with this command.

/usr/local/sbin/unbound-control -c /var/unbound/unbound.conf list_local_data

Return only 720 lines for me.
Looking at them, I can see lines like this :

...
ricoh.brit-hotel-fumel.net. 3600 IN A 192.168.1.16
ricoh.brit-hotel-fumel.net. 7200 IN AAAA 2a01:cb19:907:xxxx::87
...

and the reverse :
...
7.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.x.x.x.x.7.0.9.0.9.1.b.c.1.0.a.2.ip6.arpa. 7200 IN PTR ricoh.bhf.tld.
16.1.168.192.in-addr.arpa. 3600 IN PTR ricoh.bhf.tld.
....

and a bunch of other 'stuff' :

...
onion.  10800   IN      NS      localhost.
onion.  10800   IN      SOA     localhost. nobody.invalid. 1 3600 1200 604800 10800
invalid.        10800   IN      NS      localhost.
invalid.        10800   IN      SOA     localhost. nobody.invalid. 1 3600 1200 604800 10800
localhost.      10800   IN      AAAA    ::1
localhost.      10800   IN      A       127.0.0.1
localhost.      10800   IN      SOA     localhost. nobody.invalid. 1 3600 1200 604800 10800
...

all this data looks very 'local' to me

When using this :

/usr/local/sbin/unbound-control -c /var/unbound/unbound.conf dump_cache

I see - not surprised - a lot more. Close to 20K lines of cache stuff.

Hence my initial reflection : why should "list_local_data" return 'a lot of data' ? Ok if there are a lot of LAN devices (like thousands).

It's a lot but it makes sense once you include multiple DNSBL sources in pfBlockerNG. Setting DNSBL Mode to python may help with the issue.

Of course.
But why do you mention "DNSBL Mode to python" here ?
Afaik, that's a pfBLockerng related setting and how is that related to the "Fatal Error Kea2Unbound" question ?
What's the link between the "DNSBL Mode to python" and Kea2Unbound ?
For me, Kea2Unbound is a PHP script, called by 'kea' upon an DHCP lease event, and it will use "unbound-control" so it can inform unbound about it.

@cmcdonald said in PHP Fatal Error Kea2Unbound:

The next version of the kea2unbound solution will probably look something like this:

Do I understand correctly that :

a unbound 'kea' aware module written in Python, or a native binary compiled from C or Go ... ok, but a kea side module, activated by "library": "/usr/local/lib/kea/hooks/libdhcp_run_script.so" etc is still needed.
I understand that 'injecting' all the Lease-to-DNS info using unbound-control can be slow if there are many leases to handle ....
ah, ok, so : instead of 'kea' telling 'unbound' about local DNS host info, it's unbound asking kea about potential host (DNS) info ... right ?
question : if DHCP lease info was a static setup, can this be enforced with an infinite cache dns TTL ?

Thanks for the details.
As I'm using Plus, I'm using "Kea2Unbound" since day 1.
It works flawlessly for me.

marcosm

It lists all local records, not just "LAN devices". When pfBlockerNG is in unbound mode the records it adds are considered local.

Yes.
Maybe but it's probably better to have some TTL to make sure the client checks the server in case of e.g. admin changes.

Gertjan

@marcosm said in PHP Fatal Error Kea2Unbound:

When pfBlockerNG is in unbound mode the records it adds are considered local.

Thanks for insisting on that one.
I had to switch back to the old "Unbound mode" = one big DNSBL file, read into unbound during startup, to see what the difference is.
All these DNSBL entries are now considered local .... and indeed, the file became pretty big.

I really thought that the 'Unbound mode' wasn't used anymore, as 'way better' = 'Python mode' exists.

Thanks for the clarification.