Issue with HAProxy and Kubernetes Ingress Controller in Proxy Mode

mr-elamin2

Hi there,
I have an issue with HAProxy and Kubernetes Ingress Controller in Proxy Mode.

Setup:

I have a Kubernetes cluster hosting all my applications, which is sitting behind a pfSense firewall with HAProxy. All incoming requests on TCP port 443 hitting the firewall are being forwarded in TCP mode to the Kubernetes NGINX Ingress Controller. The NGINX Ingress Controller is set in proxy mode, and HAProxy is configured to send traffic to the backend in proxy mode as well.

Problem:

While my applications work perfectly in a web browser, I encounter a strange issue when trying to push Docker images to my private Harbor registry. The error I receive is:

ERROR: failed to solve: failed to push harbor.mr-elamin.com/library/redis-proxy:3.20.3: failed to do request: Head "https://harbor.mr-elamin.com/v2/library/redis-proxy/blobs/sha256:1e9f215cdf351e6ca07ba06d8f13ad228bc80c58c34e1d455a6335581e340466": EOF

Debugging:

I tested the connection using curl https://harbor.mr-elamin.com and observed the following behavior:

The connection is successful two to three times, returning status code 200.
When successful, the curl command hits the access log of my NGINX Ingress Controller.
Occasionally, the connection fails with an error, and no entry is recorded in the NGINX access log.

Here is an example of the curl command output when it succeeds:

curl https://harbor.mr-elamin.com/ -v
* Host harbor.mr-elamin.com:443 was resolved.
* IPv6: (none)
* IPv4: 93.128.34.35
*   Trying 93.128.34.35:443...
* Connected to harbor.mr-elamin.com (93.128.34.35) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /usr/lib/ssl/cert.pem
*  CApath: /usr/lib/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=harbor.mr-elamin.com
*  start date: Nov 30 23:30:54 2024 GMT
*  expire date: Feb 28 23:30:53 2025 GMT
*  subjectAltName: host "harbor.mr-elamin.com" matched cert's "harbor.mr-elamin.com"
*  issuer: C=US; O=Let's Encrypt; CN=R10
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://harbor.mr-elamin.com/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: harbor.mr-elamin.com]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: harbor.mr-elamin.com
> User-Agent: curl/8.5.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200 
< date: Thu, 19 Dec 2024 22:17:25 GMT
< content-type: text/html
< content-length: 785
< last-modified: Mon, 11 Nov 2024 09:31:49 GMT
< etag: "6731cf05-311"
< cache-control: no-store, no-cache, must-revalidate
< accept-ranges: bytes
< strict-transport-security: max-age=31536000; includeSubDomains
< 
<!DOCTYPE html>
<html>
    <head>
        <meta charset="utf-8"/>
        <title>Harbor</title>
        <base href="/"/>
        <meta name="viewport" content="width=device-width, initial-scale=1"/>
        <link rel="icon" type="image/x-icon" href="favicon.ico?v=2"/>
    <link rel="stylesheet" href="styles.ac415221c96d2bef.css"></head>
    <body>
        <harbor-app>
            <div class="spinner spinner-lg app-loading app-loading-fixed">
                Loading...
            </div>
        </harbor-app>
    <script src="runtime.91e26be486c2d932.js" type="module"></script><script src="polyfills.d87db3092ff69ed9.js" type="module"></script><script src="scripts.3846d86d42cdb753.js" defer></script><script src="main.b7158e4b1766e693.js" type="module"></script></body>
</html>
* Connection #0 to host harbor.mr-elamin.com left intact

And here is an example of the curl command output when it fails:

curl https://harbor.mr-elamin.com/ -v
* Host harbor.mr-elamin.com:443 was resolved.
* IPv6: (none)
* IPv4: 93.128.34.35
*   Trying 93.128.34.35:443...
* Connected to harbor.mr-elamin.com (93.128.34.35) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /usr/lib/ssl/cert.pem
*  CApath: /usr/lib/ssl/certs
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to harbor.mr-elamin.com:443 
* Closing connection
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to harbor.mr-elamin.com:443

Test:

To isolate the issue, I removed the proxy mode setting from the NGINX Ingress Controller and connected my PC directly to the NGINX Controller. In this configuration, I was able to push Docker images and perform other operations without any problems.

Configuration:

Here is the relevant HAProxy configuration on pfSense:

Frontend Configuration:

frontend k8s-nginx-ingress-ssl-pass-thru-custom
     mode tcp
     bind *:443
     timeout client 30s
     use_backend k8s-ssl-pass-thru

Backend Configuration:

backend k8s-ssl-pass-thru
     mode tcp
     timeout connect 30s
     timeout server 30s
     server k8s-server 10.18.0.180:443 send-proxy

Observations:

The issue seems to be related to the proxy mode configuration in HAProxy and the NGINX Ingress Controller.
Direct access without proxy mode works perfectly, while indirect access with proxy mode results in intermittent failures.

I would appreciate any insights or suggestions from the community to resolve this issue.

Thank you for your assistance!

here are some screenshots for more clarification:

mr-elamin2

I managed to resolve the problem by removing the frontend name. After making this change, everything started working normally.

Updated Frontend Configuration:

mode tcp
bind *:443
timeout client 30s
use_backend k8s-ssl-pass-thru

By simplifying the configuration and removing the unnecessary frontend name, the setup became functional. If anyone else is facing similar issues, I recommend checking if any redundant configuration elements can be removed.