Several issues upon 2.5.0 upgrade

jimp

Saying "LDAP auth is broken" means nothing. You have provided no details. It works for me here, with and without SSL.

What are your LDAP server settings? Any errors in the logs? What does a packet capture of the LDAP exchange show?

There were changes which could have affected LDAP but there was no negative feedback after waiting for months in snapshots, and it worked in our own testing.

CARP is also working fine here on 2.5.0, as is maintenance mode. The CARP status page sometimes refreshes before the VIPs switch so be sure the refresh the page (e.g. by clicking "CARP status" in the breadcrumb bar) a few moments after entering maintenance mode. That's not a new issue, though.

If you had an error about the demotion factor normally that means something happened which indicated a failure (e.g. a CARP-enabled interface is marked down) but it happened twice somehow so the demotion factor didn't make sense. That would be a different and separate issue.

maverickws

@jimp thanks for the feedback.
About the CARP I'm really not worried about it as I mentioned after forcing the upgrade on the primary we had a slight hiccup on our network due to the disagreement but now all seems to be working OK so I'm not worried about that anymore. We made a couple of tests and its working well so far so...

About LDAP auth:

We are using LDAP with SSL, not sure what you mean by "LDAP Server Settings" they're the same as before the 2.4.5 upgrade.

port 636 SSL/TLS Encrypted, the LDAP server Certificate is selected, same protocol version, etc.

As I mentioned when I go to User Manager > Settings and select the LDAP server and hit "Save & Test" there are NO ERRORS and returns all OK with the organization units found which means it connects fine and authenticates otherwise would return nothing.

RFC 2307 style group membership is enabled, as was before. Its three pfsense that were authenticating without any issues before, now none authenticates.

Didn't do packet captures can do that next week but anyway the logs on the GUI have no error. When I go to "Diagnostics" and test authentication it fails, but it returns ZERO useful information, and doesn't print anything to the logs.

jimp

Does it work if you disable TLS for LDAP?

You could also try editing the CA entry for the LDAP server and checking the box which adds it to the trust store. Then in the LDAP server, change it to use the global trust store as well, rather than that specific CA.

mjsengineer

I've had a similar issue with two authentication servers I have setup. The only difference between the two configurations is tha one server has nothing defined for the "client certificate" and the other has a certificate which was imported from Samba4.

The auth server with no client certificate works. The one with the client cert no longer authenticates, but it did work prior to the 21.02 upgrade.

maverickws

Ok, so I did the following:

Edited the CA entry for the LDAP server and checked "add to trust store", then tested on Diagnostics > Authentication - Failed;
Went to System > User Manager > Authentication Servers and changed "Transport" to "Standard TCP" (and port 389) failed as well.
Went to the same place as before changed "Peer Certificate Authority" to Global Root CA List (since the cert had been added to the store) failed as well.

Having issues about the certificate, shouldn't the "Save & Test" Option under User Manager > Settings fail as well? After all, it uses the same settings to test (and retrieve the information) than it would to authenticate, no?

jimp

The fact that it failed without TLS makes me think that it's something else in the LDAP query that's wrong.

Unless the server rejects queries without TLS, that is.

With TLS off you should be able to get a packet capture of the LDAP exchange and debug what's going wrong -- you can see exactly what query is being sent and the reply.

maverickws

@jimp please just amuse me and please reply to the following:
if the settings are wrong, why does it work under User Manager > Settings
selecting the LDAP server and hitting "Save & Exit"

the LDAP server (Red Hat IDM) only accepts authenticated queries, so it has to communicate somehow. This works with both TLS on or off.

jimp

Are you sure it's working and not falling back to local database authentication?

If that works and others don't, it still makes me think it's a problem in the LDAP settings. It's possible you are communicating with the LDAP server OK but something in your other settings is not making a proper query/search and isn't getting and results.

Packet capturing auth attempts with TLS disabled is the fastest way to diagnose that. Download the packet capture and load it in Wireshark and it can show you everything that's going on.

maverickws

@jimp thank you for the feedback.
Well I am sure because of the following:
Under User Manager > Settings I am not testing auth per se, so can't fall back to local database authentication because there's no authentication.
However what it does is to test the LDAP Settings. And it queries the LDAP server and it returns the Organisational Units, please check the image below.
User Manager Settings Test
This information can only be returned after connecting to the LDAP server AND using an authenticated system user. We don't allow anonymous queries. So I can't agree with the remark "It's possible you are communicating with the LDAP server OK but something in your other settings is not making a proper query/search and isn't getting and results" at least this far is making the proper queries.

If TLS failed, this would fail as well.
We've had certificates mismatching before and it would fail this test.

We haven't changed any settings from the previous version to this version.
The issue is occurring on 3 pfSense routers, one is our office router, which is connected via Site-to-Site IPSec VPN. But the other two are locally connected. The three worked before, the three are failing now, after the update to 2.5.0.
I must say I haven't done the packet capture yet as I got a lot on my plate and I'll have to spare some time to do that. So I was trying to see if this approach exposing the issue with maximum detail would take us somewhere.

jimp

Authenticated binds are much different that attempting to query for a user, which is affected by all the other settings on the page for the various containers/base dn/etc.

All that proves is you can communicate with the server, it doesn't mean your other settings are OK.

Turn off TLS, take a packet capture of some auth attempts. See what is happening. That's the only way forward.

LucSuryo

This post is deleted!

Polle

Got exactly the same issue: after upgrading to 2.5 LDAP authentication stopped working.

As jimp mentioned, when switching to LDAP under 'SystemUser/Manager/Settings' and hitting 'Save & Test', it succesfully connects to our LDAP server and lists the OUs. When I try authentication from the Diagnostics menu, it fails. Since we are (were ...) using LDAP authentication for our OVPN clients, that fails too.
We're using LDAPS on port 636 - I tried switching to port 389 with standard TCP but got the same results - authentication not working ...

One more related question about this note under 'Authentication' servers:
NOTE: When using SSL/TLS or STARTTLS, this hostname MUST match a Subject Alternative Name (SAN) or the Common Name (CN) of the LDAP server SSL/TLS Certificate.
Will this work with a wildcard certificate? It has *.<domain> and <domain> as SAN names whereas our server is ldap.<domain>. * should cover that but there is no exact match of course ...

Polle

Correction - it's maverickws who posted the issue - and he's right, LDAP authentication is broken - period.

BossaOps

@polle I've looked, it really depends on what RFC, usually your CN would be *.domain, and that is normally considered a match, and the same should apply to the SAN names..

maverickws

I still haven't done a tcpdump to see what's going on, but I would like to add to the comments regarding certificate, it DOES NOT WORK if connecting in plain, so that about the certificate seems like a mere detail that really doesn't influence here.

Thank god we did not have other services on the pfSense bound to LDAP auth as is @Polle's case. I feel whatever changes have been made here are poorly documented and this will keep happening to more people.

mjsengineer

As I noted above, I have a similar issue. However I do have a working LDAP/VPN authentication setup.

Both Authentication Server setups have the following:

Port: 636
Transport: SSL/TLS Encrypted
Peer Certificate Authority: Same_cert_on_both_setup
Protocol Version: 3
Server Timeout: 25
Search Scope: Entire Subtree
Base DN: Identical-on-both-setups
Authentication Containers: Identical-on-both-setups
Extended Query: Enabled
Query: Identical-on-both-setups
Bind Credentials: Identical-on-both-setups

The ONLY difference between the configuration of Server 1 and Server 2
is that Server 2 has a "Client Certificate" defined. Server 1 has the
"Client Certificate" set to None.

The Diagnostics/Authentication worked on both servers pre upgrade.
Post upgrade, Server 2 - the one with a "Client Certificate" - no longer
authenticates successfully.

maverickws

Hi @mjsengineer what do you mean by "client certificate" or you mean client certificate on the VPN authentication setup only? there's no client certificate to login on the pfsense iirc?

BossaOps

@maverickws I think he means pfsense and LDAP server use mutual certificate verification for the broken setup, but only bind credentials (both over TLS) for the functional one. That's what's broken with Cloud Identity LDAP, it uses client certificates, not credentials.

maverickws

@bossaops our LDAP server doesn't verify the client certificate, I mean many services that connect to it have self-signed certificate. Either way, that wouldn't apply to a plain auth setup, and as requested above I've disabled TLS/SSL and changed the port to 389 and tested, the issues persisted.

mjsengineer

@maverickws I am referring to the "Client Certificate" option under the LDAP Server Settings section of the Authentication Server configuration on pfsense (ie. System -> User Manager -> Authentication Servers).