[solved] 2.4 broke LDAP against Mac OS Server

SpaceBass

hello friends!

Over the weekend I upgraded all my boxes to 2.4 and am enjoying the many improvements.

However, I've run into one significant snag. In my set up, authenticating against LDAP, specifically Mac OS 10.11 servers, seems to be broken.

For 2.3, this guide was a life saver:
https://www.reddit.com/r/PFSENSE/comments/4u3bzu/authentication_servers_os_x_open_directory_working/

That configuration worked perfectly for me for authing in OpenVPN and elsewhere in PFsense. in 2.4, something seems to have changed.

If I leave SSL enabled, I get the following in the logs:

php-fpm	86214	/system_authservers.php: ERROR! ldap_get_user_ous() could not bind anonymously to server .

If I disable SSL I get nothing in the logs, but the authentication still fails.

If I re-enable SSL and add a bind auth name and password, it simple says:

could not bind to server

The SSL certificates on the servers are commercially signed and valid and I'm using FQDNs in the authentication set up. I have confirmed that I can ping the hostnames from the PF console.

In the PF console, this returns every CN on the LDAP server successfully:

ldapsearch -H ldaps://sever.foo.bar:636 -b "DC=server,DC=foo,DC=bar"

where server.foo.bar is actually my LDAP server's FQDN

Anyone have any troubleshooting tips?

tim.mcmanus

What do the logs on the OS X Server show? When pfSense is making its queries, are they being received? Do they fail?

SpaceBass

before I post the MacOS logs, when I logged back into PF tonight, this little crash report was waiting for me:

amd64
11.0-RELEASE-p12
FreeBSD 11.0-RELEASE-p12 #8 1032370b1f4(RELENG_2_4_0): Wed Aug 23 00:42:26 CDT 2017     root@buildbot2.netgate.com:/builder/factory-240/tmp/obj/builder/factory-240/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[27-Aug-2017 21:55:08 Etc/UTC] PHP Warning:  ldap_start_tls(): Unable to start TLS: Connect error in /etc/inc/auth.inc on line 1074
[27-Aug-2017 21:55:08 Etc/UTC] PHP Stack trace:
[27-Aug-2017 21:55:08 Etc/UTC] PHP   1\. {main}() /usr/local/www/system_authservers.php:0
[27-Aug-2017 21:55:08 Etc/UTC] PHP   2\. ldap_get_user_ous() /usr/local/www/system_authservers.php:52
[27-Aug-2017 21:55:08 Etc/UTC] PHP   3\. ldap_start_tls() /etc/inc/auth.inc:1074
[27-Aug-2017 21:55:20 Etc/UTC] PHP Warning:  ldap_start_tls(): Unable to start TLS: Connect error in /etc/inc/auth.inc on line 1074
[27-Aug-2017 21:55:20 Etc/UTC] PHP Stack trace:
[27-Aug-2017 21:55:20 Etc/UTC] PHP   1\. {main}() /usr/local/www/system_authservers.php:0
[27-Aug-2017 21:55:20 Etc/UTC] PHP   2\. ldap_get_user_ous() /usr/local/www/system_authservers.php:52
[27-Aug-2017 21:55:20 Etc/UTC] PHP   3\. ldap_start_tls() /etc/inc/auth.inc:1074

No FreeBSD crash data found.

SpaceBass

@tim.mcmanus:

What do the logs on the OS X Server show? When pfSense is making its queries, are they being received? Do they fail?

nada…zip ... they don't show a thing :/
if I do a tcpdump port 636 I can see the traffic hitting the Mac OS box.

but the plot thickens.

If I change to STARTLS I get the following "error"? at the top of the PF page:

Warning: ldap_start_tls(): Unable to start TLS: Connect error in /etc/inc/auth.inc on line 1342 Call Stack: 0.0001 233640 1\. {main}() /usr/local/www/diag_authentication.php:0 0.0580 1633400 2\. authenticate_user() /usr/local/www/diag_authentication.php:47 0.0581 1633640 3\. ldap_backed() /etc/inc/auth.inc:1748 0.0593 1635256 4\. ldap_start_tls() /etc/inc/auth.inc:1342

Authentication still fails.

If I change to TCP (no encryption) it works. (Which I share was different than yesterday…but maybe the Hawthorne effect with tcpdump is real :) ... needless to say, I'd prefer not to drop SSL.

Derelict

Is your LDAP server configured for TCP - Standard (on port 389), TCP - STARTTLS (on port 389), or SSL (on port 636)??

SpaceBass

@Derelict:

Is your LDAP server configured for TCP - Standard (on port 389), TCP - STARTTLS (on port 389), or SSL (on port 636)??

it responds on both TCP and SSL. I prefer SSL obviously and it worked with SSL prior to the upgrade from 2.3.x to 2.4

and on multiple Ubuntu installs, this line in ldap.conf is sufficient (along with the base)

uri ldaps://server.foo.bar
```with the server's commercial cert I don't even have to load the CA on the ubuntu boxes.

Derelict

OK. But that is not what I asked. I asked how you had it set.

SpaceBass

@Derelict:

OK. But that is not what I asked. I asked how you had it set.

Sorry, perhaps I didn't understand the question correctly. My bad!

The server itself is set to respond to TCP - Standard (on port 389) and SSL (on port 636)

In PF I have tried:

TCP - Standard (on port 389) Worked once - nice again
TCP - STARTTLS (on port 389) - does not work

[iii] SSL (on port 636) - does not work [/iii]

Derelict

And standard works and SSL does not?

SpaceBass

@Derelict:

And standard works and SSL does not?

Standard worked once. Once I try SSL and the error gets thrown, standard doesn't work again until I reboot PF or wait for something (?) to time out… EG it worked once today but not since.

One other note on my end - above I said it's Mac OS 10.11 which is incorrect. I'm running 10.12.5

Derelict

I am looking at this. I don't know if it is OS X Server or SSL/STARTTLS LDAP in general.

I do not see what you see.

Standard always works.

SSL fails with Authentication Failed

STARTTLS fails with Authentication failed and also throws this:

Warning: ldap_start_tls(): Unable to start TLS: Connect error in /etc/inc/auth.inc on line 1342 Call Stack: 0.0000 233808 1. {main}() /usr/local/www/diag_authentication.php:0 0.0244 1622264 2. authenticate_user() /usr/local/www/diag_authentication.php:47 0.0244 1622504 3. ldap_backed() /etc/inc/auth.inc:1748 0.0247 1624128 4. ldap_start_tls() /etc/inc/auth.inc:1342

I can change the Transport at will and the behavior is the same. No reboots or anything like that necessary.

There is an issue with certificate validation it seems. Perhaps something changed in openldap. Not sure yet.

2.4.0-RC (amd64)
built on Mon Aug 28 11:05:02 CDT 2017
FreeBSD 11.0-RELEASE-p12

GeoTrust RapidSSL Certificate

macOS Server 10.11.6

ETA: According to packet captures both SSL and STARTTLS are failing with TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Unknown CA)

tim.mcmanus

You have to look also at the server version, 5.3 or 5.4, depending on the Mac OS version. There are also CLI commands to increase logging. man serveradmin is a good place to start.

SpaceBass

Hey friends!
Thanks to this forum and this Reddit thread: https://www.reddit.com/r/PFSENSE/comments/6wm2m1/24_broke_ldap_auth_against_mac_os_servers/
i've found success!

It turns out —not sure why this wasn't the case in 2.3.x —that the entire chain is needed for commercial certs to work.

I had tried concatenating them by hand and exporting them from Keychain Access but it didn't work.

This command, on the other hand, generated a complete chain that works for commercial certs on macOS

openssl s_client -host <fqdn> -port 443 -prexit -showcerts</fqdn>

This, of course, assumes ones web server uses the same cert and chain as the LDAP server, but in most OS X Server deployments, that's probably the case. If not, it'd be trivial to set up a web server using the LDAP cert and pull it this way. One might also simply change the port (although I didn't try it) to 636.

That spits out an entire chain of certs in the right order (which apparently matters) which one can past into a CA in PF's CA screen.

Hope this helps someone else!

Thanks everyone for the troubleshooting tips and help to get this far!

jimp

I pushed a fix yesterday to make it build the entire chain automatically.

Assuming you have all of the CA certs and intermediates imported, you can select the bottom intermediate as the LDAP CA and it will figure out the rest.