Hi there,
I have an issue with HAProxy and Kubernetes Ingress Controller in Proxy Mode.
I have a Kubernetes cluster hosting all my applications, which is sitting behind a pfSense firewall with HAProxy. All incoming requests on TCP port 443 hitting the firewall are being forwarded in TCP mode to the Kubernetes NGINX Ingress Controller. The NGINX Ingress Controller is set in proxy mode, and HAProxy is configured to send traffic to the backend in proxy mode as well.
Problem:While my applications work perfectly in a web browser, I encounter a strange issue when trying to push Docker images to my private Harbor registry. The error I receive is:
ERROR: failed to solve: failed to push harbor.mr-elamin.com/library/redis-proxy:3.20.3: failed to do request: Head "https://harbor.mr-elamin.com/v2/library/redis-proxy/blobs/sha256:1e9f215cdf351e6ca07ba06d8f13ad228bc80c58c34e1d455a6335581e340466": EOF Debugging:I tested the connection using curl https://harbor.mr-elamin.com and observed the following behavior:
The connection is successful two to three times, returning status code 200. When successful, the curl command hits the access log of my NGINX Ingress Controller. Occasionally, the connection fails with an error, and no entry is recorded in the NGINX access log.Here is an example of the curl command output when it succeeds:
curl https://harbor.mr-elamin.com/ -v * Host harbor.mr-elamin.com:443 was resolved. * IPv6: (none) * IPv4: 93.128.34.35 * Trying 93.128.34.35:443... * Connected to harbor.mr-elamin.com (93.128.34.35) port 443 * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: /usr/lib/ssl/cert.pem * CApath: /usr/lib/ssl/certs * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS * ALPN: server accepted h2 * Server certificate: * subject: CN=harbor.mr-elamin.com * start date: Nov 30 23:30:54 2024 GMT * expire date: Feb 28 23:30:53 2025 GMT * subjectAltName: host "harbor.mr-elamin.com" matched cert's "harbor.mr-elamin.com" * issuer: C=US; O=Let's Encrypt; CN=R10 * SSL certificate verify ok. * Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption * Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption * Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption * using HTTP/2 * [HTTP/2] [1] OPENED stream for https://harbor.mr-elamin.com/ * [HTTP/2] [1] [:method: GET] * [HTTP/2] [1] [:scheme: https] * [HTTP/2] [1] [:authority: harbor.mr-elamin.com] * [HTTP/2] [1] [:path: /] * [HTTP/2] [1] [user-agent: curl/8.5.0] * [HTTP/2] [1] [accept: */*] > GET / HTTP/2 > Host: harbor.mr-elamin.com > User-Agent: curl/8.5.0 > Accept: */* > * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * old SSL session ID is stale, removing < HTTP/2 200 < date: Thu, 19 Dec 2024 22:17:25 GMT < content-type: text/html < content-length: 785 < last-modified: Mon, 11 Nov 2024 09:31:49 GMT < etag: "6731cf05-311" < cache-control: no-store, no-cache, must-revalidate < accept-ranges: bytes < strict-transport-security: max-age=31536000; includeSubDomains < <!DOCTYPE html> <html> <head> <meta charset="utf-8"/> <title>Harbor</title> <base href="/"/> <meta name="viewport" content="width=device-width, initial-scale=1"/> <link rel="icon" type="image/x-icon" href="favicon.ico?v=2"/> <link rel="stylesheet" href="styles.ac415221c96d2bef.css"></head> <body> <harbor-app> <div class="spinner spinner-lg app-loading app-loading-fixed"> Loading... </div> </harbor-app> <script src="runtime.91e26be486c2d932.js" type="module"></script><script src="polyfills.d87db3092ff69ed9.js" type="module"></script><script src="scripts.3846d86d42cdb753.js" defer></script><script src="main.b7158e4b1766e693.js" type="module"></script></body> </html> * Connection #0 to host harbor.mr-elamin.com left intactAnd here is an example of the curl command output when it fails:
curl https://harbor.mr-elamin.com/ -v * Host harbor.mr-elamin.com:443 was resolved. * IPv6: (none) * IPv4: 93.128.34.35 * Trying 93.128.34.35:443... * Connected to harbor.mr-elamin.com (93.128.34.35) port 443 * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * CAfile: /usr/lib/ssl/cert.pem * CApath: /usr/lib/ssl/certs * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to harbor.mr-elamin.com:443 * Closing connection curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to harbor.mr-elamin.com:443 Test:To isolate the issue, I removed the proxy mode setting from the NGINX Ingress Controller and connected my PC directly to the NGINX Controller. In this configuration, I was able to push Docker images and perform other operations without any problems.
Configuration:Here is the relevant HAProxy configuration on pfSense:
Frontend Configuration:
frontend k8s-nginx-ingress-ssl-pass-thru-custom mode tcp bind *:443 timeout client 30s use_backend k8s-ssl-pass-thruBackend Configuration:
backend k8s-ssl-pass-thru mode tcp timeout connect 30s timeout server 30s server k8s-server 10.18.0.180:443 send-proxy Observations: The issue seems to be related to the proxy mode configuration in HAProxy and the NGINX Ingress Controller. Direct access without proxy mode works perfectly, while indirect access with proxy mode results in intermittent failures.I would appreciate any insights or suggestions from the community to resolve this issue.
Thank you for your assistance!
here are some screenshots for more clarification:
f6145d19-1d63-42a5-bfd6-62535139b6f0-image.png
eee7469b-96b4-4d81-a469-c6b36e097e2e-image.png
5fe5241e-f633-422f-b657-a36c02ab4bcc-image.png
5472f361-c1d8-4193-9255-53bab761f3d6-image.png
cc881f50-d504-4de0-a8c8-094ce5c69ac9-image.png