Problem with L3 adoption for Unifi gear in 2.3.3 / 2.4 beta?

luckman212

Came here to post about an issue that I have personally encountered twice in the field now. The problem has only so far manifested with Unifi equipment placed behind a pfSense running 2.3.3 or 2.4 beta. What happens is that the Unifi gear fails to provision itself - it seems to fail to honor the Host Override for "unifi" in DNS Resolver and so cannot be adopted or registered to the portal. I have been using pfSense + Unifi for years so am not a beginner to the configuration of those devices. Currently have 41 pfSense sites that use Unifi gear, with a total of about 150 access points, all were working fine until we updated a couple past 2.3.2_p1.

I submitted PR#3599 that does address it but came here to request additional feedback and to ask if others have run into this and have any ideas about it. I did see a post on /r/PFSENSE regarding the same issue so I know I am at least not 100% alone.

I will try to set up more labs and get some Wireshark captures to see what is happening, but for now the patch to add the unqualified hostname to Unbound's configuration has worked for me.

update: this did turn out to be a bug, but with the version of udhcpc that's compiled into Unifi equipment. The bug was fixed a year-and-a-half ago, but it needs to be compiled into their firmware. Thanks to dok for helping clarify this, I opened a thread over on the Ubnt forum, waiting for response. For now, this post further down in this thread has more details as well as workarounds.

update2: cmb announced that the bug was fixed by Ubiquiti today. Not sure what specific build of the firmware contains the fix, but it should be out very soon.

update3: Ubnt released a version of their firmware that contains the busybox fix for this issue. Release post here (it's ver 3.7.49.6201)

FreeMinded

Hi Luckman212

I just ran into this issue as well and I'm still looking for a real fix…
If I understand the things correctly unbound (DNS Resolver) since pfSense 2.3.3 does not resolve hostnames only anymore on purpose [1] [2]. According to the friendly doktornotor, setting the search domain should actually make it work [3].

Now that I have set the search domain, I can ping the hostname without FQDN again from my Linux Box, but the Unifi APs still fails to connect to the controller.

The following workaround proposed by you on reddit makes it work again [4]. Paste this into the "Custom Options" box in your Resolver config:```
local-data: "unifi A 4.5.6.7"


What is the correct way to fix this? Could you, doktornotor, shed some more light on the right way of doing the things? You seem to have it working for you.

1: https://redmine.pfsense.org/issues/6064
2: https://github.com/pfsense/pfsense/commit/0fa68840504f6866901e0d02819d43a3ce9f9578
3: https://github.com/pfsense/pfsense/pull/3599

johnpoz

If you ask me the correct way to fix it would be to use fqdn for your inform url

ie
Or, using FQDN for the controller inform URL, http://FQDN:8080/inform

from https://help.ubnt.com/hc/en-us/articles/204909754-UniFi-Layer-3-methods-for-UAP-adoption-and-management

doktornotor

Errr, guys, dunno what you are doing there really.

Set up a host override for unifi. <search.domain>pointing to the controller's IP
put <search.domain>to "Domain name" and "Domain search list" in DHCP server settings
put the interface IP as DNS server in DHCP server settings

Done. Works.

If it doesn't work for you, perhaps reset your APs/switch/whatever UniFi gear.</search.domain></search.domain>

FreeMinded

I didn't set the domain name (only the the domain search list). With the domain name set as well it works!
Why is the domain name required in this case? Should it really be this way or is this a bug on the Unifi side?

Thanks doktornotor!

@johnpoz doing as you say would need to configure every AP manually. This is not an option if you have many of them.

doktornotor

This is from /etc/udhcpc/udhcpc (FW v3.7.42)


configure_dns()
{
        # do not remove resolv.conf if no dns servers received from dhcp
        if [ ".$dns" = "." ]; then
                return 0
        fi

        echo -n > $RESOLV_CONF
        [ -n "$domain" ] && echo search $domain >> $RESOLV_CONF
        for i in $dns ; do
                echo adding dns $i
                echo nameserver $i >> $RESOLV_CONF
        done
        echo "127.0.0.1 localhost.localdomain   localhost" > $HOSTS_CONF
        [ -n "$unifi_addr" ] && echo adding unifi-server $unifi_addr
        [ -n "$unifi_addr" ] && echo "$unifi_addr       unifi" >> $HOSTS_CONF
}

Busybox needs to be compiled with FEATURE_UDHCP_RFC3397 for Option 119 to work. No idea what's UBNT doing with busybox, looks to me like they are using Option 015 instead. (Then again, MS doesn't support this either, so you really should set up both.)

luckman212

Never before had to set any options in DHCP for search domain etc. Defaults always worked up until 2.3.3 so something has changed, still haven't been able to run those Wireshark captures to figure out what exactly the reason is – but I hope to find time to do that later today. I imagine that dok might not have encountered this (yet) because I believe existing devices that have already been Adopted and have their set-inform URL in place will continue to work, but if the devices are factory reset or brand new out of the box unprovisioned devices will fail to reach the controller.

johnpoz

If you have a lot of them to setup And your controller is on a different network - can you not just use the discovery tool on that local network your AP are on and set them to use the FQDN for the inform.. That is what the discovery util is for is it not.

If your not on site, then I would admin the dhcp to give them the fqdn they should use..

Setting up a host only name to resolve would be my last choice. Devices should always be resolved via fqdn not some host name.. Even if your on the same layer 2 and can broadcast for a name - it is still better practice to always use a fqdn..

Last option would be to set your dns to resolve host only name without a domain.

That is my 2 cents on the matter.

doktornotor

@luckman212:

I imagine that dok might not have encountered this (yet) because I believe existing devices that have already been Adopted and have their set-inform URL in place will continue to work, but if the devices are factory reset or brand new out of the box unprovisioned devices will fail to reach the controller.

Erm… nope. The whole goal of the setup is that you plug in a never configured AP via a cable and it shows up in controller for adoption. So that when some gear dies, you can ship a replacement and all that's required is plugging a cable in to get things working again. Works just perfectly fine with setup as shown above. It works with the controller on LAN, it works with a remote site connected via IPsec, or another connected via site-to-site OpenVPN. It works over WAN as well as long as you forward 8080 to the controller.

(And sorry, I cannot comment on setups that rely on some confusing, badly described/documented behind-the-scenes logic looking for options configured in completely different places of pfSense and picking that up for unrelated stuff when people cannot be bothered to set things up. It takes about 60 seconds to set up things explicitly and avoid any unexpected behaviour or breakage.)

johnpoz

where are you coming up with option 15 or 119.. You only need to setup option 43 for unifi devices to find their L3 controller..

doktornotor

Talking about the DNS method. (Option 43 is something I'd avoid since it causes inexplicable issues, such as broken PXE boot with non-UBNT gear.)

luckman212

DHCP Opt 43 will only accept an IP address. Pretty bad way to do it if you ask me, since IP's can change and if you have 100 sites… man what a nightmare to update it. DNS method that has always worked for me (again stating this) is much more flexible since you can use an FQDN.

Dok I was not trying to challenge your knowledge or imply anything I am merely stating facts. No need for aggression. Not sure what you mean about "badly described/documented behind-the-scenes logic looking for options configured in completely different places of pfSense and picking that up for unrelated stuff" but nothing I've posted is particularly esoteric. In any case, I don't think I'm the only one experiencing this, and I will get to the bottom of it shortly when I can sit down with some gear and do some deeper testing w/ packet captures.

doktornotor

I mean the logic of "$foo config option setup in that place will be used for $bar option in another place if $baz in yet another place is unchecked or $zomg is not configured". I cannot see how this helps anyone.

Other than that, I don't know what still doesn't work for you. Configure the ""Domain name" in DHCP server configuration. The busybox DHCP client doesn't take option 119. Most things don't, as noted above.

https://forum.pfsense.org/index.php?topic=126354.msg698200#msg698200

johnpoz

Ok I just tested this - who says you can set an override for single label ie host name to resolve? I just created this entry and it seems to resolve just fine with just using unifi in a dns query.

hostnamequery.png_thumb

luckman212

you set 'unifi' in the 'domain'? and left 'hostname' blank? never seen it done that way before.

johnpoz

its single label… that is how you would do it ;) You need to understand what the different entries in a fqdn are.. A single label or host would end up actually just being the tld (top level domain). Like com or net or org, etc. When you talk about it as a fqdn.. Which is what the gui is asking for..

This seems pretty simple to do vs just doing the same thing in the options box..

luckman212

Ok guys, long update here. I have been doing lots of testing.

TL;DR — there is a bug in the version of busybox udhcpc (1.19.4) compiled into Unifi hardware that causes this. When the search domain begins with a numeric character, udhcpc barfs on it and sets the search domain to "bad" in /etc/resolv.conf (screenshot below). This causes the device to fail to register with the controller. The bug was fixed a year-and-a-half ago so I don't know why we are stuck with such an ancient version, but I opened a post over on the UBNT forums and am awaiting a response. For now the only simple fix that works in all test cases below is to use workaround #2 or #3 below. If you care, you can read more…

There are 3 known workarounds:

1. Don't use a system domain that starts with a number (may or may not be an option for you)
2. Use my patch (for 2.4b use this commit, for 2.3.3 use this commit) and tick the "add unqualified (short) hostname" checkbox on your unifi.whatever host override PR#3599
3. In Custom Options of DNS Resolver, add a line e.g.

local-data: "unifi A 4.5.6.7"

I built up a mini lab with a fresh install of 2.3.3 on APU2 hardware. Bog-standard out of the box config, I was focusing solely on isolating this issue and reproducing it. I believe I have uncovered something strange.

The ingredients for the test were:
• unprovisioned Unifi Access Point - running latest stable firmware which at the time was 3.7.40.6115*
• pfSense CE 2.3.3 on APU2 hardware - clean install
• 2 interfaces configured - WAN/LAN
• Unifi WAP plugged directly into LAN interface (POE injector)
• DNS handled by Unbound - default config options [Transparent/DNSSEC enabled]
• Single Host Override defined "unifi.system-domain" pointing to imaginary controller 1.2.3.4
• system-domain was alternated between 36hudson.lan and hudson36.lan
• also tested all of the above again with PR#3599 installed, with and without enabling the "add unqualified hostname" option

Steps
1. boot Unifi WAP from powered-off state
2. once it's booted, ssh in and run

cat /etc/resolv.conf
ping unifi (if that fails, ping unifi.fqdn)
nslookup unifi (if fail, nslookup unifi.fqdn)

The test results are below. While it's not a pfSense-specific issue, I believe that my patch handles this problem cleanly, and due to the popularity of Unifi + pfSense, it would be helpful to have it in there. It does fix the issue for me and until Ubiquiti resolves the matter, at least we have an easy & consistent way to patch any affected systems. I do have some .pcap packet captures if anyone needs those, but honestly once I found the issue it was pretty easy to reproduce and after inspecting them in Wireshark, I don't think this bug is a result of any malformed requests or responses on the wire.

*also tested with 2 older firmwares [3.4.14.3413, 3.7.39.6089] – same results

doktornotor

Linking the UBNT thread here, such bugs obviously need to fixed by Ubiquiti.

https://community.ubnt.com/t5/UniFi-Wireless/Unifi-DHCP-opt-15-search-domain-starts-with-a-number-BARF/m-p/1854593#M214919

EDIT: That is an outdated busybox version bug.

luckman212

I had already linked to it in my above post…

While the bug is Ubiquiti's, since the culprit lies in the out of date udhcpc, other gear will likely exhibit the problem (eg Linksys etc).

Aside from providing a viable workaround, the PR I submitted adds a couple of useful features as well as fixes one bug on recent 2.4 snaps with editing / saving existing overrides. So I still believe it merits being merged.

doktornotor

The offending code is linked on the UBNT thread.