google LDAP connection failed

assitenzatecnicaistituto · Mar 4, 2021, 9:11 AM

I have a problem after update my Netgate XG-7100 to the version 21.02-RELEASE-p1.
After the update my pfSense failed to bind to ldap.google.com.

Attempting connection to ldap.google.com OK
Attempting bind to ldap.google.com failed

i have renewed the user credentials but no success.

Any idea?

BossaOps · Mar 4, 2021, 11:21 AM

@assitenzatecnicaistituto Salve, I have the exact same problem, tried updating the bind credentials, checked the validity of the certificate, ma senza fortuna.

I've run connectivity tests, no issues there.. the audit logs from google show everything fine up until the time I updated, then literally no records at all, not even errors on google's side. I'd like to try running the ldpasearch command, but I'm not able to locate the certificate, and possibly it's not anywhere as this says: https://forum.netgate.com/topic/106475/where-is-the-location-of-the-pfsense-certificates/2

The certificate management page shows the certificate as not in use?

Going to try and create the certificates in the shell, I'll let you know what I get from that.

Si spera.

BossaOps · Mar 4, 2021, 12:06 PM

@bossaops So, I think I've figured it out, I created the files in the shell, linked them as env variables.

If you want to try, the command is this one (copy your certs into files):

env LDAPTLS_CERT=goog.crt LDAPTLS_KEY=goog.key ldapsearch -ZZ -H ldaps://ldap.google.com:636 -b dc=yourdomain,dc=com '(mail=some.user@yourdomain.com)'

and I get back this:

ldap_start_tls: Can't contact LDAP server (-1)
        additional info: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed (self signed certificate)

So, for some reason pfsense is no longer happy with the cert, and I'm not sure where to import the CA...

viktor_g · Mar 4, 2021, 12:21 PM

@bossaops Could you show cert info with openssl x509 -in goog.crt -text -noout ?
You can redact your private cert info.

BossaOps · Mar 4, 2021, 12:27 PM

@viktor_g Nothing private, it's the client cert that you generate when creating an LDAP client in the google admin console.. easy enough to recreate if it's a seecurity issue.

 openssl x509 -in goog.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1566223926543 (0x16caa38850f)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = Google Inc., L = Mountain View, CN = LDAP Client, OU = GSuite, C = US, ST = California
        Validity
            Not Before: Aug 19 14:12:05 2019 GMT
            Not After : Aug 18 14:12:05 2022 GMT
        Subject: O = Google Inc., L = Mountain View, CN = LDAP Client, OU = GSuite, C = US, ST = California
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:a9:e5:6f:1b:b5:7e:66:8e:d0:27:b5:a1:25:11:
                    1f:66:6c:48:1e:e5:42:94:2b:12:e0:68:b5:95:5c:
                    f3:22:bd:1e:40:e7:31:d0:57:8d:68:96:ce:75:e0:
                    4a:ce:7b:7b:fe:09:73:bf:c4:00:4c:7a:53:98:f4:
                    6b:86:f7:aa:c3:2d:e9:8e:24:84:64:cc:74:11:96:
                    56:b9:6c:0b:e9:e5:47:be:70:0c:0d:68:94:62:e3:
                    23:97:44:4e:6b:5a:27:67:85:1e:8e:1f:d5:bc:ae:
                    0e:2f:8c:1f:69:f3:31:ba:90:05:57:a9:c8:02:ff:
                    d6:d4:00:21:60:5b:6d:2e:cb:ea:6f:d6:d2:e7:bd:
                    6a:f9:bb:c5:03:2d:df:5f:83:56:78:5d:7d:1d:22:
                    42:5d:93:ef:f6:b2:f2:d1:db:9f:21:0b:6e:7a:b6:
                    6a:47:aa:66:d3:2e:fb:1d:28:29:82:02:07:1e:3c:
                    85:17:11:bc:e9:fb:ef:84:78:da:be:37:e3:41:5a:
                    5c:58:86:5d:43:4a:40:1d:c1:83:83:2f:9f:78:88:
                    8f:8f:45:dd:ba:03:b2:9d:17:93:98:9a:3e:bd:cf:
                    3b:7b:e5:02:9d:bc:ec:b4:61:13:78:31:45:0e:1f:
                    e8:48:82:4c:af:dc:8e:86:ef:02:dd:f2:5c:62:34:
                    70:4f
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         4f:a7:ea:60:ef:fb:94:ad:9b:3f:1c:5e:ec:fc:08:d8:8f:7b:
         98:02:68:f3:a0:f1:af:f0:69:9a:c2:ef:cd:77:3c:46:68:3c:
         e3:4e:81:f8:06:41:d8:3b:24:30:dc:2f:cc:cb:e2:86:1f:7f:
         8c:07:1a:89:29:b3:f5:6a:5b:29:7d:5f:3f:89:ef:c5:4d:05:
         53:93:c9:2e:38:69:81:ef:76:14:cb:97:29:a3:ce:8b:c9:fe:
         ab:06:b5:df:a3:de:e8:a2:63:66:70:8b:7f:2c:67:af:4c:f6:
         f8:b4:a5:ec:d9:db:2b:a2:d0:4e:10:c4:38:c9:e2:bf:6b:19:
         40:10:58:b6:cc:d2:24:e3:2c:1b:e9:cf:a4:6e:72:3d:c7:14:
         29:a6:59:84:0d:a8:5b:dc:d7:2f:20:96:77:e2:56:2c:7d:95:
         6a:dc:c3:d0:4c:af:0f:1d:2b:8f:0b:ba:68:aa:6c:35:b9:cd:
         9a:d5:70:c8:19:62:01:59:fa:30:a1:bd:01:9a:8d:d8:f1:a4:
         ca:16:8c:22:96:58:eb:62:ac:55:92:97:cf:c9:de:67:4f:3b:
         c7:40:14:20:48:b1:e0:c8:fc:ad:4c:d4:8e:47:b8:da:3c:1b:
         05:3a:f1:34:97:b9:d2:36:0e:bf:2d:8d:1f:d6:45:70:88:8d:
         dd:e6:29:56

BossaOps · Mar 4, 2021, 2:42 PM

@viktor_g So, one thing I'm wondering about is if the LDAP code in pfsense is compiled without SNI, as google specifically says :

OpenSSL client/library does not support SNI (Server Name Indication)

During the connectivity test, the following output might be returned:

Verify return code: 18 (self signed certificate)

The Secure LDAP service requires a TLS client that supports and initiates a TLS session using SNI (Server Name Indication). If the TLS client does not support SNI, then the TLS server (ldap.google.com) returns a self-signed certificate that will not pass CA validation checks, to indicate that SNI is required.

I get the same error if I run ldapsearch without the certificate, and with the certificate included.. however running openssl s_client I get a fully qualfied cert and intermediate cert.

 echo | openssl s_client -showcerts -connect ldap.google.com:636
CONNECTED(00000003)
depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
verify return:1
depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1
verify return:1
depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = ldap.google.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = Mountain View, O = Google LLC, CN = ldap.google.com
   i:C = US, O = Google Trust Services, CN = GTS CA 1O1
-----BEGIN CERTIFICATE-----
MIIFljCCBH6gAwIBAgIRAIoX7ziwr6xtAwAAAADLP4gwDQYJKoZIhvcNAQELBQAw
QjELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczET
MBEGA1UEAxMKR1RTIENBIDFPMTAeFw0yMTAxMjYwODUzNTlaFw0yMTA0MjAwODUz
NThaMGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlhMRYwFAYDVQQH
Ew1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgTExDMRgwFgYDVQQDEw9s
ZGFwLmdvb2dsZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDI
DsMlFHdqOF5oINuFWucHGB4tPn/e5+pCbILXIhTuysf6LX7/wx3UHgQFqk+xVF2D
9jr9w3/4TvPhYwpSGV9zriL3HhIXdI7EDJk+glkkPbeJTjbU3VFN5bPFlQj2PzOk
BhuPJYR21cOkAu/GrKcjY/CX/zNx2tyOs2vtmf/aJ3yI3J42lRemEc1VnFtM1BAl
tx3vQ1dxyslANeBkLu5jV19uOdVSv/5PxzFPxb7Wb7NqrizAcnP+ZID0tKVVhNkp
cu+iH9Eqt4zLyy2Day9XURLKVFacIbEzVuVRbvxtsFEICiUFiIxH7HGadVsWgP65
SQOpnEH6niRDrQLcWUOlAgMBAAGjggJeMIICWjAOBgNVHQ8BAf8EBAMCBaAwEwYD
VR0lBAwwCgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUuYu8j9uF
UDLGQiNrAlBYTB6ylX8wHwYDVR0jBBgwFoAUmNH4bhDrz5vsYJ8YkBug630J/Ssw
aAYIKwYBBQUHAQEEXDBaMCsGCCsGAQUFBzABhh9odHRwOi8vb2NzcC5wa2kuZ29v
Zy9ndHMxbzFjb3JlMCsGCCsGAQUFBzAChh9odHRwOi8vcGtpLmdvb2cvZ3NyMi9H
VFMxTzEuY3J0MBoGA1UdEQQTMBGCD2xkYXAuZ29vZ2xlLmNvbTAhBgNVHSAEGjAY
MAgGBmeBDAECAjAMBgorBgEEAdZ5AgUDMDMGA1UdHwQsMCowKKAmoCSGImh0dHA6
Ly9jcmwucGtpLmdvb2cvR1RTMU8xY29yZS5jcmwwggEFBgorBgEEAdZ5AgQCBIH2
BIHzAPEAdwD2XJQv0XcwIhRUGAgwlFaO400TGTO/3wwvIAvMTvFk4wAAAXc+HT/v
AAAEAwBIMEYCIQC7cv324GDRYdeRJ0Befkpg+HtRQliMsG5GwE20hRS/1wIhAJAc
knuz/JvQYkWoeZuNIN8nlyIJgihBhFmuamTfc8tXAHYARJRlLrDuzq/EQAfYqP4o
wNrmgr7YyzG1P9MzlrW2gagAAAF3Ph1AKwAABAMARzBFAiEAyMzXk/cGMvCDP44J
djHjmB31Xxk05WbTO43FXL2kS8oCIAzWIM2Ci0XSrAp8ggGcnvV6gWfk1MTwtdPx
sS9iet3FMA0GCSqGSIb3DQEBCwUAA4IBAQBA8iajNZTnWSG4sMaCwe7H5pIqXmdp
aPSB7O+kz47sgXmNvHxKDYKImRwNy0pOOozk7FoGclSn3uN4gez97cpKvmS3cigL
PcWI7yAr0AliiEryl+MG6K4dRgw3ndTMdPoRU7D+UScHxKdGXM4sp2+259gSRsFo
T89mXnjdNScYqxfbvC0eGOYZ3ubST8PxyvP7S+iRm4pyi7/47N6BW/HkPRfd8JOS
zuUrhPsvSE/CgNZBBpMLLQhk+HWStQz+2JaRUo7lYTiITz/YJUMeT2CjbI5UPFjG
WdB9MzyTemh66KlfM8HF6ZXyJqYW8AnfGi4xJ6/kA5+T/X0wA4uHxBsS
-----END CERTIFICATE-----
 1 s:C = US, O = Google Trust Services, CN = GTS CA 1O1
   i:OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign
-----BEGIN CERTIFICATE-----
MIIESjCCAzKgAwIBAgINAeO0mqGNiqmBJWlQuDANBgkqhkiG9w0BAQsFADBMMSAw
HgYDVQQLExdHbG9iYWxTaWduIFJvb3QgQ0EgLSBSMjETMBEGA1UEChMKR2xvYmFs
U2lnbjETMBEGA1UEAxMKR2xvYmFsU2lnbjAeFw0xNzA2MTUwMDAwNDJaFw0yMTEy
MTUwMDAwNDJaMEIxCzAJBgNVBAYTAlVTMR4wHAYDVQQKExVHb29nbGUgVHJ1c3Qg
U2VydmljZXMxEzARBgNVBAMTCkdUUyBDQSAxTzEwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQDQGM9F1IvN05zkQO9+tN1pIRvJzzyOTHW5DzEZhD2ePCnv
UA0Qk28FgICfKqC9EksC4T2fWBYk/jCfC3R3VZMdS/dN4ZKCEPZRrAzDsiKUDzRr
mBBJ5wudgzndIMYcLe/RGGFl5yODIKgjEv/SJH/UL+dEaltN11BmsK+eQmMF++Ac
xGNhr59qM/9il71I2dN8FGfcddwuaej4bXhp0LcQBbjxMcI7JP0aM3T4I+DsaxmK
FsbjzaTNC9uzpFlgOIg7rR25xoynUxv8vNmkq7zdPGHXkxWY7oG9j+JkRyBABk7X
rJfoucBZEqFJJSPk7XA0LKW0Y3z5oz2D0c1tJKwHAgMBAAGjggEzMIIBLzAOBgNV
HQ8BAf8EBAMCAYYwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMBIGA1Ud
EwEB/wQIMAYBAf8CAQAwHQYDVR0OBBYEFJjR+G4Q68+b7GCfGJAboOt9Cf0rMB8G
A1UdIwQYMBaAFJviB1dnHB7AagbeWbSaLd/cGYYuMDUGCCsGAQUFBwEBBCkwJzAl
BggrBgEFBQcwAYYZaHR0cDovL29jc3AucGtpLmdvb2cvZ3NyMjAyBgNVHR8EKzAp
MCegJaAjhiFodHRwOi8vY3JsLnBraS5nb29nL2dzcjIvZ3NyMi5jcmwwPwYDVR0g
BDgwNjA0BgZngQwBAgIwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly9wa2kuZ29vZy9y
ZXBvc2l0b3J5LzANBgkqhkiG9w0BAQsFAAOCAQEAGoA+Nnn78y6pRjd9XlQWNa7H
TgiZ/r3RNGkmUmYHPQq6Scti9PEajvwRT2iWTHQr02fesqOqBY2ETUwgZQ+lltoN
FvhsO9tvBCOIazpswWC9aJ9xju4tWDQH8NVU6YZZ/XteDSGU9YzJqPjY8q3MDxrz
mqepBCf5o8mw/wJ4a2G6xzUr6Fb6T8McDO22PLRL6u3M4Tzs3A2M1j6bykJYi8wW
IRdAvKLWZu/axBVbzYmqmwkm5zLSDW5nIAJbELCQCZwMH56t2Dvqofxs6BBcCFIZ
USpxu6x6td0V7SvJCCosirSmIatj/9dSSVDQibet8q/7UK4v4ZUN80atnZz1yg==
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = California, L = Mountain View, O = Google LLC, CN = ldap.google.com

issuer=C = US, O = Google Trust Services, CN = GTS CA 1O1

---
No client certificate CA names sent
Requested Signature Algorithms: ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:RSA-PSS+SHA512:RSA+SHA512:RSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA256:RSA-PSS+SHA256:RSA+SHA256:ECDSA+SHA384:RSA-PSS+SHA384:RSA+SHA384:RSA-PSS+SHA512:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3062 bytes and written 429 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

assitenzatecnicaistituto · Mar 5, 2021, 6:48 AM

Thanks BossaOps, I'm not exactly an expert, it was easy enough to do.
So there's no way to fix it?

viktor_g · Mar 5, 2021, 8:21 AM

Could you check /var/log/system.log for LDAP bind errors?

pfSense PHP version supports SNI:
OPENSSL_TLSEXT_SERVER_NAME = true
see https://www.php.net/manual/en/openssl.constsni.php
and ldapsearch is not used for it

Redmine issue created: https://redmine.pfsense.org/issues/11626

BossaOps · Mar 5, 2021, 10:00 AM

@assitenzatecnicaistituto I think it's going to take a little while to understand what changed, it should be solvable.

BossaOps · Mar 5, 2021, 10:12 AM

@viktor_g just that same old error I always get:

4 11:23:19 office-gateway php-fpm[35209]: /diag_authentication.php: ERROR! Could not bind to LDAP server Gsuite. Please check the bind credentials.

Same thing more or less in auth.log

Feb 24 17:12:58 office-gateway openvpn[345]: /openvpn.auth-user.php: ERROR! Could not bind to LDAP server Gsuite. Please check the bind credentials.
Mar  2 11:02:41 office-gateway php-fpm[96371]: /diag_authentication.php: ERROR! Could not bind to LDAP server Gsuite. Please check the bind credentials.

Is there any way to get a more complete error from that code? I had to add --ZZ to ldapsearch to get it to state what the error was, it would be nice to understand which of the many reasons the connection fails. As an aside, Cloud Identity LDAP does not actually use the bind credentials (you can connect without them). They will use them if the client insists on sending them.

Why do I need both a certificate and access credentials to authenticate LDAP clients?

Only the certificate authenticates the LDAP client. The access credentials only exist if the client insists upon also sending a username and password. On their own, the access credentials don’t confer any access to the LDAP server or user data, but they should be kept secret to prevent them from being used to log in to certain LDAP clients.

In the case where an LDAP client requires access credentials, we authenticate LDAP clients with both certificates and access credentials.

BossaOps · Mar 8, 2021, 9:30 AM

@viktor_g I guess your bug tracker is private, but I'd really like to inform Jim P that

Cloud Identity LDAP stopped working only when we upgraded to 2021.02
Services that are using connections to the same directory continue to run without issue so a google change is not the most likely culprit.
If you set up the stunnel part per this: https://docs.netgate.com/pfsense/en/latest/recipes/auth-google-gsuite.html it works.

BossaOps · Mar 8, 2021, 9:45 AM

@assitenzatecnicaistituto Actually, if you set up the stunnel package as per this page: https://docs.netgate.com/pfsense/en/latest/recipes/auth-google-gsuite.html#install-the-stunnel-pfsense-package-ce-or-2-4-4-release

and configure it as indicated, the LDAP works again.

assitenzatecnicaistituto · Mar 8, 2021, 3:33 PM

@bossaops thanks! You save me..

BossaOps · Mar 8, 2021, 4:28 PM

@assitenzatecnicaistituto Piacere mio!

racecarr · Apr 30, 2021, 7:54 PM

@bossaops Thank you for providing this as a work around. Worked for my organization, as well. Hard to believe that NetGate would release an update that breaks existing functionality and not provide some notice about the fix.
I upgraded yesterday from 2.4.5 to 21.02.2 and the LDAP connection to Google IdP that had been working for almost a year instantly broke.
Switched to using stunnel as you suggested and we're back in business.

Albertopfsense · Jul 20, 2021, 5:00 PM

good morning
i have the same problem
this is last row of error report
69034 /diag_authentication.php: ERROR! Could not bind to LDAP server Google. Please check the bind credentials.
Jul 20 18:51:20 stunnel 69347 LOG5[0]: Connection reset: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
Jul 20 18:51:20 stunnel 69347 LOG3[0]: SSL_accept: /build/ce-crossbuild-252/sources/FreeBSD-src/crypto/openssl/ssl/record/rec_layer_s3.c:1544: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
Jul 20 18:51:20 stunnel 69347 LOG6[0]: Peer certificate not required
Jul 20 18:51:20 stunnel 69347 LOG5[0]: Service [Google] accepted connection from 127.0.0.1:50399
Jul 20 18:50:45 stunnel 67696 LOG5[ui]: Switched to chroot directory: /var/tmp/stunnel
Jul 20 18:50:45 stunnel 67696 LOG5[ui]: Configuration successful

what can i try to test the ldap functionality ?
thanks Alberto