Several issues upon 2.5.0 upgrade
I'd like to share a couple of issues I've had upon updating to 2.5.0:
All our pfSense have LDAP auth, I'll go to this later. The first thing to know is LDAP auth is broken in 2.5.0.
So you fallback to the local admin user, easy, right?
On two of our pfSense we had to access via console and reset to the default pfSense password, then change it back.
CARP / HA
following the usual routine, you upgrade the secondary first, then enable maintenance mode on the primary, then update the primary.
So what happened was after upgrade CARP is broken so you can't really fallback.
After upgrading the secondary we went to the primary and enabled Maintenance Mode. Even though it said maintenance mode was enabled, the Primary was still Master and nothing to the secondary.
Also the primary was failing pfsync and sending notifications.
So even tho the maintenance mode wasn't working (we had backups of both routers we could restore quickly anyway) but I forced an update on the primary while had traffic going. When it went for a reboot the traffic shifted to the secondary that assumed the Master role.
After finishing the upgrade and rebooting... the same authentication issue (we've upgraded 3 routers, 2 on HA and one standalone) we reset the password via console.
I was actually hoping the primary was still on maintenance mode... which was, so traffic kept going through the first, but going to Status > CARP we had a message saying the CARP detected an issue on the CARP demotion factor. There's some button there to fix it, so apparently it did.
CARP is working fine not, except for one thing: the IPSec connection sometimes doesn't shift correctly back to the primary router when this reassumes the Master role.
OK, so LDAP is simply broken. Period.
If I go to:
System > User Manager > Settings
Select the LDAP authentication server and hit "Save & Test"
All tests come out OK
Attempting connection to ..... OK
Attempting bid to ..... OK
Attempting to fetch Organizational Units from .... OK
Organisational units found ... what was supposed.
If I go to
Diagnostics > Authentication
Select the LDAP authentication server
The following errors were detected:
On all routers.
Saying "LDAP auth is broken" means nothing. You have provided no details. It works for me here, with and without SSL.
What are your LDAP server settings? Any errors in the logs? What does a packet capture of the LDAP exchange show?
There were changes which could have affected LDAP but there was no negative feedback after waiting for months in snapshots, and it worked in our own testing.
CARP is also working fine here on 2.5.0, as is maintenance mode. The CARP status page sometimes refreshes before the VIPs switch so be sure the refresh the page (e.g. by clicking "CARP status" in the breadcrumb bar) a few moments after entering maintenance mode. That's not a new issue, though.
If you had an error about the demotion factor normally that means something happened which indicated a failure (e.g. a CARP-enabled interface is marked down) but it happened twice somehow so the demotion factor didn't make sense. That would be a different and separate issue.
@jimp thanks for the feedback.
About the CARP I'm really not worried about it as I mentioned after forcing the upgrade on the primary we had a slight hiccup on our network due to the disagreement but now all seems to be working OK so I'm not worried about that anymore. We made a couple of tests and its working well so far so...
About LDAP auth:
We are using LDAP with SSL, not sure what you mean by "LDAP Server Settings" they're the same as before the 2.4.5 upgrade.
port 636 SSL/TLS Encrypted, the LDAP server Certificate is selected, same protocol version, etc.
As I mentioned when I go to User Manager > Settings and select the LDAP server and hit "Save & Test" there are NO ERRORS and returns all OK with the organization units found which means it connects fine and authenticates otherwise would return nothing.
RFC 2307 style group membership is enabled, as was before. Its three pfsense that were authenticating without any issues before, now none authenticates.
Didn't do packet captures can do that next week but anyway the logs on the GUI have no error. When I go to "Diagnostics" and test authentication it fails, but it returns ZERO useful information, and doesn't print anything to the logs.
Does it work if you disable TLS for LDAP?
You could also try editing the CA entry for the LDAP server and checking the box which adds it to the trust store. Then in the LDAP server, change it to use the global trust store as well, rather than that specific CA.
I've had a similar issue with two authentication servers I have setup. The only difference between the two configurations is tha one server has nothing defined for the "client certificate" and the other has a certificate which was imported from Samba4.
The auth server with no client certificate works. The one with the client cert no longer authenticates, but it did work prior to the 21.02 upgrade.
Ok, so I did the following:
- Edited the CA entry for the LDAP server and checked "add to trust store", then tested on Diagnostics > Authentication - Failed;
- Went to System > User Manager > Authentication Servers and changed "Transport" to "Standard TCP" (and port 389) failed as well.
- Went to the same place as before changed "Peer Certificate Authority" to Global Root CA List (since the cert had been added to the store) failed as well.
Having issues about the certificate, shouldn't the "Save & Test" Option under User Manager > Settings fail as well? After all, it uses the same settings to test (and retrieve the information) than it would to authenticate, no?
The fact that it failed without TLS makes me think that it's something else in the LDAP query that's wrong.
Unless the server rejects queries without TLS, that is.
With TLS off you should be able to get a packet capture of the LDAP exchange and debug what's going wrong -- you can see exactly what query is being sent and the reply.
@jimp please just amuse me and please reply to the following:
if the settings are wrong, why does it work under User Manager > Settings
selecting the LDAP server and hitting "Save & Exit"
the LDAP server (Red Hat IDM) only accepts authenticated queries, so it has to communicate somehow. This works with both TLS on or off.
Are you sure it's working and not falling back to local database authentication?
If that works and others don't, it still makes me think it's a problem in the LDAP settings. It's possible you are communicating with the LDAP server OK but something in your other settings is not making a proper query/search and isn't getting and results.
Packet capturing auth attempts with TLS disabled is the fastest way to diagnose that. Download the packet capture and load it in Wireshark and it can show you everything that's going on.
@jimp thank you for the feedback.
Well I am sure because of the following:
User Manager > SettingsI am not testing auth per se, so can't fall back to local database authentication because there's no authentication.
However what it does is to test the LDAP Settings. And it queries the LDAP server and it returns the Organisational Units, please check the image below.
This information can only be returned after connecting to the LDAP server AND using an authenticated system user. We don't allow anonymous queries. So I can't agree with the remark "It's possible you are communicating with the LDAP server OK but something in your other settings is not making a proper query/search and isn't getting and results" at least this far is making the proper queries.
If TLS failed, this would fail as well.
We've had certificates mismatching before and it would fail this test.
We haven't changed any settings from the previous version to this version.
The issue is occurring on 3 pfSense routers, one is our office router, which is connected via Site-to-Site IPSec VPN. But the other two are locally connected. The three worked before, the three are failing now, after the update to 2.5.0.
I must say I haven't done the packet capture yet as I got a lot on my plate and I'll have to spare some time to do that. So I was trying to see if this approach exposing the issue with maximum detail would take us somewhere.
Authenticated binds are much different that attempting to query for a user, which is affected by all the other settings on the page for the various containers/base dn/etc.
All that proves is you can communicate with the server, it doesn't mean your other settings are OK.
Turn off TLS, take a packet capture of some auth attempts. See what is happening. That's the only way forward.
This post is deleted!
Got exactly the same issue: after upgrading to 2.5 LDAP authentication stopped working.
As jimp mentioned, when switching to LDAP under 'SystemUser/Manager/Settings' and hitting 'Save & Test', it succesfully connects to our LDAP server and lists the OUs. When I try authentication from the Diagnostics menu, it fails. Since we are (were ...) using LDAP authentication for our OVPN clients, that fails too.
We're using LDAPS on port 636 - I tried switching to port 389 with standard TCP but got the same results - authentication not working ...
One more related question about this note under 'Authentication' servers:
NOTE: When using SSL/TLS or STARTTLS, this hostname MUST match a Subject Alternative Name (SAN) or the Common Name (CN) of the LDAP server SSL/TLS Certificate.
Will this work with a wildcard certificate? It has *.<domain> and <domain> as SAN names whereas our server is ldap.<domain>. * should cover that but there is no exact match of course ...
Correction - it's maverickws who posted the issue - and he's right, LDAP authentication is broken - period.
@polle I've looked, it really depends on what RFC, usually your CN would be *.domain, and that is normally considered a match, and the same should apply to the SAN names..
I still haven't done a tcpdump to see what's going on, but I would like to add to the comments regarding certificate, it DOES NOT WORK if connecting in plain, so that about the certificate seems like a mere detail that really doesn't influence here.
Thank god we did not have other services on the pfSense bound to LDAP auth as is @Polle's case. I feel whatever changes have been made here are poorly documented and this will keep happening to more people.
As I noted above, I have a similar issue. However I do have a working LDAP/VPN authentication setup.
Both Authentication Server setups have the following:
- Port: 636
- Transport: SSL/TLS Encrypted
- Peer Certificate Authority: Same_cert_on_both_setup
- Protocol Version: 3
- Server Timeout: 25
- Search Scope: Entire Subtree
- Base DN: Identical-on-both-setups
- Authentication Containers: Identical-on-both-setups
- Extended Query: Enabled
- Query: Identical-on-both-setups
- Bind Credentials: Identical-on-both-setups
The ONLY difference between the configuration of Server 1 and Server 2
is that Server 2 has a "Client Certificate" defined. Server 1 has the
"Client Certificate" set to None.
The Diagnostics/Authentication worked on both servers pre upgrade.
Post upgrade, Server 2 - the one with a "Client Certificate" - no longer
Hi @mjsengineer what do you mean by "client certificate" or you mean client certificate on the VPN authentication setup only? there's no client certificate to login on the pfsense iirc?
@maverickws I think he means pfsense and LDAP server use mutual certificate verification for the broken setup, but only bind credentials (both over TLS) for the functional one. That's what's broken with Cloud Identity LDAP, it uses client certificates, not credentials.
@bossaops our LDAP server doesn't verify the client certificate, I mean many services that connect to it have self-signed certificate. Either way, that wouldn't apply to a plain auth setup, and as requested above I've disabled TLS/SSL and changed the port to 389 and tested, the issues persisted.
@maverickws I am referring to the "Client Certificate" option under the LDAP Server Settings section of the Authentication Server configuration on pfsense (ie. System -> User Manager -> Authentication Servers).
@maverickws It would be interesting seeing what changed inside the pfsense LDAP code, though I think the issue myself and msjengineer have is actually the local (running onthe pfsense) code not dealing with the self signed certificate properly, with what you're demonstrating very possibly this isn't the only breaking change they made.
BTW mutual certificate verification can use self signed certificates, just like you can use self signed certificates on web servers, the verifier just needs to have a CA/intermediate certificate signed by the same CA (just like the OpenVPN client certs are signed by the pfsense CA, which is self signed).
I actually have an stunnel proxy running here locally, as I use Cloud Identity LDAP with an IDP service that doesn't support mutual certificate verification, I will try and see if I can't get auth working via that route as a test.
@maverickws this finally works for me
user => netgate
base dn => dc=forum,dc=netgate,dc=org
user dn => uid=netgate,ou=users,dc=forum,dc=netgate,dc=org
so note we use uid! since ldap is use to shell to linux servers too
on the openldap server
- create user : netgate
- generate password for this user
- add to group vpn (we use memberof !!)
(home made script)
on the pfense
User Manager / Authentication Servers
- Descriptive name => myldap
- Type => LDAP
- Hostname or IP address =>10.10.10.10
- Port value => 389
- Transport => Standard TCP
- Peer Certificate Authority => Global Root CA List
- Client Certificate => None
- Protocol version => 3
- Server Timeout => 25
- Search scope => Entire Subtree
- Base DN => dc=forum,dc=netgate,dc=org
- Authentication containers => ou=users,dc=forum,dc=netgate,dc=org;ou=groups,dc=forum,dc=netgate,dc=org
- Extended query => checked
- Query => memberOf=cn=vpn,ou=groups,dc=forum,dc=netgate,dc=org
- Bind credentials => <your admin dn and password>
- User naming attribute => uid
- Group naming attribute => cn
- Group member attribute => memberOf
- RFC 2307 Groups => uncheck
- Group Object Class => groupOfNames
- rest all unchecked!
on the pfense
User Manager / Settings
- Authentication Server => myldap
click save & test
make sure there are no error
on the pfense
Diagnostics / Authentication
- Authentication Server => myldap
- Username => netgate
- Password => <generated password>
make sure there is are errors
Switched to port 389 and 'Standard TCP' - then tried authentication from the diagnostics and did a packet capture, what I see is that:
- box contacts the LDAP server and performs a bind request using the bind credentials
- ldap returns bindresponse success
- box performs some search request - OUs I specified and finally the user (uid) for the user that is authenticating
- next a bind request for the user
- ldap again returns bindresponse success
- and finally the box issues an unbindrequest
with the usual load of ACKs in between
So all that looks pretty much OK but nevertheless it returns "The following input errors were detected: Authentication failed." ....
i generate a new CA with values that reflects our company
Name => My OpenVPN
ST=CA, OU=forum, O=The Netgate Forum , L=Lala City, CN=vpn-ca, C=US
next setup OpenVPN Server
- Server mode => Remote Access (SSL/TLS + User Auth)
- Backend for authentication => myladp + Database
- Protocal => UDP on IPv4 Only
- Device mode => tun - layer 3 Tunnel Mode
- Interface => WAN
- Local port => 7070 <--- we use not the default to port value
- Description OpenVPN Netgate Forum
- TLS Configuration => checked <--- TLS key will be generated after save
- TLS Key Usage Mode => TLS Authentication
- TLS keydir direction => use Default Direction
- Peer Certificate Authority => My OpenVPN
- DH Parameter Length => 2048
- ECDH Curve => Use Default
- Data Encryption NegotiationEnable Data Encryption Negotiation => checked
- Data Encryption Algorithm => <we use only the AES-256-x>
- Fallback Data Encryption Algorithm -> ES-256_CBC (256 bit key, 128 bit block)
- Auth digest algorithm => SHA256
- Hardware Crypto => <no hardware since we are on a VM>
- Certificate Depth : One (Client + Server)
- Strict User-CN Matching : checked (Enforce match)
next only the important ones
IPv4 Tunnel Network => <your choice, buy use a /24!)
IPv4 Local network(s) => 10.10.0.0/16
Concurrent connections => <we set it to 25 to match the vm, which has 2 core and 2gb memory>
Allow Compression => Decompress incoming, do not compress outgoing
Compression => Adaptive LZO Compression
Username as Common Name => checked Use the authenticated client username instead of the certificate common name (CN).
@mjsengineer You mean the "Peer Certificate Authority" then I assume? I have no option called "Client Certificate", so I'm guessing that must be it.
@BossaOps right what I meant is we haven't add trust for any extra CA most of the times they just get the plain old default cert so I meant that wouldn't be the case here.
Edit: While I was typing this I was looking at @LucSuryo's reply and he also mentions that Client Certificate option. I don't see that in mine.
Is that what you meant @mjsengineer too?
I checked two pfSense running 2.5.0, none has that "Client Certificate" option.
I'm reviewing @LucSuryo's settings to see if I find something that'll fix this.
My settings are a bit different from @LucSuryo's but what got it fixed was disabling the option
RFC 2307 Groups:
I mean, fixed it in the sense that the
Diagnostic > Authenticationauthenticates successfully. But then says:
This user is a member of groups:and it returns empty, not referring the group its a member of and set on the extended query.
Trying to login to the pfSense returns that error
No page assigned to this user! Click here to logout....
@maverickws I have something different. I'm running 21.02 on a netgate SG-3100:
@maverickws When I unselect "LDAP Server uses RFC 2307 style group membership", the authentication check under the Diagnostics returns "authenticated successfully." My OpenVPN connection also authenticates successfully.
Seems that clears the problem.
@mjsengineer I can confirm it's always been there on netgate hardware, I started doing LDAP authentication after I purchased it so I don't know if it's exclusive to their hardware images.
yes it will not return any group unless you have setup group on pfsense
im trying to post PART 3, but get flagged as spam :-(
I had the same group in pfsense as a remote group and with the same name
but afterward, i t was not needed
in part 3 is creating the a user cert for the user and then export the client configuration : mind you i use openvpn-client-export (look in packages)
I see. I don't even want to comment this new version divergence ... %$#%# ... but I guess the issue isn't there either.
I just wonder how much longer is it going to take for Netgate acknowledge they f***ed up something on LDAP Auth, or provide instructions to fix it. I've had RFC 2307 Groups enabled since forever, and as far as I know Red Hat IDM is RFC 2307 compliant so I'm still trying to find out why doesn't work with it enabled.
Also, realising this new bug of not identifying the group the user belongs to.
@lucsuryo looking forward for that part 3. I have the group setup at:
System > User Manager > Users > Groups
Group name: the same name as the group on the LDAP server
Scope: Local (it was already set like that from before)
Group membership: I only see the local user.
i use both 2.5.0 on a small box and 21.02 in AWS
both had the issue, but no longer
if everyone get their vpn + ldap working then there is no need for me to post PART 3 then :)
the only important part of the client export is
- Host Name Resolution -> other
- Host Name -> use a dns resolved entry for the vpn
- Legacy Client : checked, Do not include OpenVPN 2.5 settings in the client configuration.
we have some user on old tunnelblick
@maverickws My problem isn't related to the RFC 2307 Groups unfortunately, never had it checked :(
Just saw that this recipe does state "For users of pfSense factory software version 2.4.4-RELEASE-p1 or later"
group should be remote :) since the user is not in the local database
Do you need part 3? (user cert + client export)
it seem the forum guard thinks I am a spammer :-(
@LucSuryo I guess we do have different setups hehe but at least I've gotten a little further.
I'm going to review the part of that article where they mention the groups, maybe that has something to do with it. Hadn't checked there either because ... well... all was working before!
P.S. I've tried to make some posts before that also got flagged for spam. Sometimes still happens, never really understood why.
same here , everything worked just fine (been using Pfsense for 7+ years)
never had this issue...
and i pulled my hairs for 2 days!
on the group : for us it works with our without no issue
this is just simple : vpn + ldap and it got broken in 2.5.0 and 21.02...
we have a commercial openvn (latest version) and zero issue
the one important thing pfsense is missing a built-in 2fa like the commercial openvpn...
i know i need to do some reading how to get radius + google authenticator working, but that is for an other day and other beer :-)
still need part 3?
I'm not sure about part 3 really.
So I was talking with some folks that mentioned that Red Hat IDM uses RFC2 307bis so that may be part why my auth wasn't working.
Now my issue is to know why
Diagnostic > Authenticationreturns
This user is a member of groups:empty, and why when I login it says there are no pages assigned to the user.
I recreated the group, set as "remote" and added all privileges except "Deny config write".
Btw the other 2 pfSense that are on HA config didn't even had the groups and it worked.
So looking at this:
I don't really understand this "issue", as before having RFC2307 style membership WORKED JUST FINE, as others mentioned this has been working for a long time with extended query, querying the right containers. I really don't know what was the issue of the OP of that issue.
What I know is that now if I have RFC 2307 checked the authentication fails, and if I uncheck it it doesn't work as it returns no group.
@jimp I noticed you participated on that issue, could you please elucidate why was that "fix" merged, for a problem that honestly did not exist?
Doing an extended query memberOf=cn=fwadmin,cn=groups,cn=accounts,dc=domain,dc=io fails.
And Chris Linstruth said he tested against FreeIPA 2 months ago with what version?? what settings were those against what FreeIPA version?