haproxy - not working with ProfileManager (certificate problem?)

regexaurus · Jul 28, 2018, 5:28 AM

I have haproxy 0.54_2 on pfSense 2.4.3, with a single/dynamic public/WAN IP address, to support a few servers running web (80/443) services on the LAN. 2 of the servers are working great with the haproxy setup. The third server (Apple Profile Manager) won't respond to external traffic, and haproxy stats always shows the HTTPs and HTTP backends for that server as being Down. We're using SNI/hostname detection/backend selection (not offloading SSL).
After (re)starting haproxy, I always see package log entries like:

Jul 28 00:56:06 <hostname> haproxy[3696]: Proxy FE1-HTTP started.
Jul 28 00:56:06 <hostname> haproxy[3696]: Proxy FE2-SNI started.
Jul 28 00:56:06 <hostname> haproxy[3696]: Proxy BE-AppleProfileManager-HTTP_http_ipvANY started.
Jul 28 00:56:06 <hostname> haproxy[3696]: Proxy BE-AppleProfileManager_https_ipvANY started.
Jul 28 00:56:06 <hostname> haproxy[3751]: Server BE-AppleProfileManager-HTTP_http_ipvANY/AppleProfileManager is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 28 00:56:06 <hostname> haproxy[3751]: backend BE-AppleProfileManager-HTTP_http_ipvANY has no server available!
Jul 28 00:56:06 <hostname> haproxy[3751]: Server BE-AppleProfileManager_https_ipvANY/AppleProfileManager is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 54ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul 28 00:56:06 <hostname> haproxy[3751]: backend BE-AppleProfileManager_https_ipvANY has no server available!

In MacOS Server, I see numerous log entries like this (corresponding to haproxy health checks, I think):

ww.example.com "-" 192.168.1.1 - - [28/Jul/2018:01:06:19 -0400] "OPTIONS / HTTP/1.0" 503 1001101 "-" "-"
www.example.com "-" 192.168.1.1 - - [28/Jul/2018:01:06:20 -0400] "\x16\x03\x01\x02" 400 226 "-" "-"

(where 192.168.1.1 is the internal/LAN ip address of the router)

I'm not sure, but I think the problem may be with the certificate/chain (PositiveSSL) we're using, but I'm not sure. At some point, I stumbled on the response by jimp (Jim Pingle, I assume :)) here: https://forum.netgate.com/topic/62744/haproxy-intermediate-certificates-unknown-issuer-missing-chain/2

I have root/intermediate certificates chained accordingly. It doesn't seem to make a difference. Suggestions? Thank you!

PiBa · Aug 5, 2018, 6:18 PM

@regexaurus
i dont know what profile manager is exactly but ill assume it kinda functions like a webserver .

For the \x16\x03\x01 log line it seems haproxy is perhaps re-encrypting already encrypted traffic. Or the webserver is expecting plain http traffic?
For the options 503 perhaps you need to add a host header to the healthcheck to make it request a domainname that is configured on the webserver? So it wont respond with a 'service not available' .?.

If you send a https request to the profile manager IP directly with a browser https://profilemanager-IP-address/ does that work? Does it reply with a '200 OK' http response?

p.s. can you include your haproxy.cfg file from the bottom of the settings tab?

config please :)

regexaurus · Aug 5, 2018, 6:19 PM

@piba 0_1533491385820_haproxy_cfg.txt
Here you go. I modified slightly (e.g. IP addresses, hostnames) to protect the not-so-innocent. ;-)
Apple Profile Manager is part of macOS Server and used for MDM (mobile device management). It listens on ports 80 and 443 (also port 1640, for macOS Server prior to v. 5.3). See https://support.apple.com/en-us/HT202487.
We're using macOS Server v. 5.6.1.

@piba asked:

If you send a https request to the profile manager IP directly with a browser https://profilemanager-IP-address/ does that work? Does it reply with a '200 OK' http response?

Internally, yes, I can access Profile Manager by IP address (e.g. https://192.168.1.54). I first see certificate warnings (hostname mismatch--of course, it won't match the IP), which I can bypass/make exception.

HAProxy shouldn't be re-encrypting and the front end doesn't offload SSL. It should only be using SNI to determine the intended destination (by hostname) and pass unmodified traffic to the appropriate internal destination.

PiBa · Aug 5, 2018, 6:38 PM

@regexaurus

server			AppleProfileManager 192.168.1.54:80 ssl check inter 1000  verify none

This looks a bit strange? The :80 with ssl encryption.

regexaurus · Aug 6, 2018, 6:44 PM

@piba
Agreed. I'm not sure if that was how I originally set it, or if I enabled that while trying to troubleshoot (dubious). Anyway, I disabled SSL on that backend. It made no apparent difference.

PiBa · Aug 6, 2018, 6:44 PM

@regexaurus
The

Jul 28 00:56:06 <hostname> haproxy[3751]: Server BE-AppleProfileManager-HTTP_http_ipvANY/AppleProfileManager is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 1ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

Should be 'fixed' / changed ?

I presume now you have 2 backends with a 503 ? Can you add a http-version to the health-check? Perhaps include a hostname with the version it to?

regexaurus · Aug 7, 2018, 1:53 AM

@PiBa
Yes, now after restarting haproxy, in package logs:

Aug 6 20:10:54 <hostname> haproxy[54284]: Server BE-AppleProfileManager-HTTP_http_ipvANY/AppleProfileManager is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Aug 6 20:10:54 <hostname> haproxy[54284]: backend BE-AppleProfileManager-HTTP_http_ipvANY has no server available!
Aug 6 20:10:55 <hostname> haproxy[54284]: Server BE-AppleProfileManager_https_ipvANY/AppleProfileManager is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 56ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Aug 6 20:10:55 <hostname> haproxy[54284]: backend BE-AppleProfileManager_https_ipvANY has no server available!

I wasn't sure what to set Http check version to and thought Chrome DevTools might help with this. Fired up a session and discovered that even though macOS Server / Profile Manager pages appear to load normally when accessed directly/internally, DevTools offered an interesting clue. Both document root (/) and /favicon.ico return 503 (Service Unavailable) status codes!

Odd!

After clicking around a bit and exploring macOS Server package contents, I updated the URLs used for check requests to:

http://apm.example.com/__collabd/coreclientbase/static/spacer.gif

and

https://apm.example.com/__collabd/coreclientbase/static/spacer.gif

for the http and https backends, respectively. Normally, these requests default to document root. DevTools shows the updated URLs return 200 status codes for internal access. After saving the changes and restarting haproxy, package logs and haproxy stats show those backends are UP / OK. I haven't yet tested externally.

PiBa · Aug 7, 2018, 5:20 PM

@regexaurus
Sounds good i guess..

regexaurus · Aug 9, 2018, 4:22 PM

@PiBa
Yes, I can at least access the macOS Server portal / Profile Manager externally now. SCEP device enrollment isn't working externally for me, though it is internally. I'm not sure how important that is--I think that's (mostly) an enroll once kind of deal. It looks like someone else beat me to experiencing this trouble, and found at least a sledge hammer style workaround. ;-)
Thanks for all your help!