[solved] backend server nginx down HAProxy

wesleylc1 · Sep 19, 2019, 7:03 PM

Hi guys!
We set up a new nginx web server to run the "NextCloud" application, server with subnet 192.168.40.xxx/22, "http and https" traffic redirection made by firewall pfsense 2.4.4 with subnet192.168. 43.xxx/22, I run the HAProxy service version 1.8.17 to direct external access; There are currently two front end configurations, one for port 80 and one for port 443, which work for other systems already deployed.

Below illustrates the settings of this new backend.

Configuration Applied to Backend

Configuration Applied to frontend

Default backend, access control lists and actions

Statistics Report

I have already performed the tests below,

root@server:/usr/share/ca-certificates# nc -zv 192.168.40.xx 80
Connection to 192.168.40.xx 80 port [tcp/http] succeeded!

root@server:/usr/share/ca-certificates# netstat -npa |grep 80
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1314/nginx: master  
tcp6       0      0 :::80                   :::*                    LISTEN      1314/nginx: master  
unix  3      [ ]         STREAM     CONNECTED     20809    1/init               /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     18026    1/init               /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     18680    712/systemd-timesyn  
unix  3      [ ]         STREAM     CONNECTED     20801    1/init               /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     19805    1/init               
unix  3      [ ]         STREAM     CONNECTED     20805    1/init               /run/systemd/journal/stdout

root@server:/usr/share/ca-certificates# nmap localhost

Starting Nmap 7.60 ( https://nmap.org ) at 2019-09-11 21:24 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000013s latency).
Not shown: 997 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
80/tcp   open  http
3306/tcp open  mysql

Nmap done: 1 IP address (1 host up) scanned in 1.71 seconds

best regards,
Wesley Santos

PiBa · Sep 11, 2019, 9:36 PM

@wesleylc1
The LastChk says the check result is: 'L7STS/405'. So it got a response from the webserver, in that regard the ports are open and accepting connections. This is likely due to the health-check method using the HEAD method. Try changing that to GET in the haproxy backend configuration?

wesleylc1 · Sep 11, 2019, 9:59 PM

hi @PiBa , it worked after changing the Http verification method option from OPTIONS to GET. Not finding because I was looking for this setting in the frontend.

According to the image, I can confirm that my backend is active

wesleylc1 · Sep 12, 2019, 7:53 PM

Hi @PiBa I am trying to configure this backend as https, but is in HAproxy presents the error "fL7STS / 404 in 2ms", which I can illustrate with an image, I can also validate that port 443 is already open on this server.

frontend HA_Sistemas-https
	bind			189.20.108.xx:443 name 189.20.108.90:443   ssl crt-list /var/etc/haproxy/HA_Sistemas-https.crt_list  
	bind			187.75.209.xx:443 name 187.75.209.246:443   ssl crt-list /var/etc/haproxy/HA_Sistemas-https.crt_list  
	mode			http
	log			global
	option			log-separate-errors
	option			httplog
	option			http-keep-alive
	timeout client		300000
	acl			rootrequested	var(txn.txnpath) -m str -i /
	acl			time-sheet	var(txn.txnhost) -m str -i time-sheet.projectus.com.br
	acl			qsms	var(txn.txnhost) -m str -i qsms.projectus.com.br
	acl			treinamento	var(txn.txnhost) -m str -i treinamento.projectus.com.br
	acl			proposta	var(txn.txnhost) -m str -i proposta.projectus.com.br
	acl			chamado	var(txn.txnhost) -m str -i chamado.projectus.com.br
	acl			pesquisa	var(txn.txnhost) -m str -i pesquisa.projectus.com.br
	acl			pesquisa-ce	var(txn.txnhost) -m str -i pesquisa-ce.projectus.com.br
	acl			nextcloud	var(txn.txnhost) -m str -i nextcloud.projectus.com.br
	http-request set-var(txn.txnpath) path
	http-request set-var(txn.txnhost) hdr(host)
	http-request redirect location /TimeSheet/faces/login.xhtml  if  rootrequested time-sheet 
	http-request redirect location /SistemaQSMS/  if  rootrequested qsms 
	http-request redirect location /Treinamento/  if  rootrequested treinamento 
	http-request redirect location /Proposta/  if  rootrequested proposta 
	http-request redirect location /Chamado/  if  rootrequested chamado 
	http-request redirect location /PesquisaClima/  if  rootrequested pesquisa 
	http-request redirect location /PesquisaClimaCE/  if  rootrequested pesquisa-ce 
	http-response add-header Content-Security-Policy upgrade-insecure-requests  if  !time-sheet !qsms !treinamento !proposta !chamado !pesquisa !pesquisa-ce 
	use_backend HA_Sistemas-45-xx_80-www_ipvANY  if  !time-sheet !qsms !treinamento !proposta !chamado !pesquisa !pesquisa-ce 
	use_backend HA_Sistemas_40-xx_443-nextcloud_ipvANY  if  !nextcloud 
	default_backend HA_Sistemas_43-xx_8443_ipvANY

backend HA_Sistemas_40-xx_443-nextcloud_ipvANY
	mode			http
	id			109
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			nextcloud 192.168.40.xx:443 id 108 ssl check inter 1000  verify none

best regards,
Wesley Santos

PiBa · Sep 12, 2019, 10:52 PM

@wesleylc1 said in backend server nginx down HAProxy:

I can also validate that port 443 is already open on this server

There is no need to validate this.. haproxy knows the port is 'open' if it wasnt you would get a L4 issue, now its showing a L7 issue.. which means that the webserver is not responding with a 'good' 200 / 2xx or 3xx code. So assumes it is not ready to serve actual requests.

So instead of using nmap to check for open ports, run a curl or wget or fetch to check what the actual response of the webserver is.? And if what request should be made to get a proper status back if possible.. or if its the proper response then tell haproxy to expect 404 as a valid response for the healthcheck.. but that would not be my first choice...

So try and figure out what request needs to be made to get a proper reply back, perhaps a different method, or a different url? or does the webserver require SNI to be send when serving requests?? Once you know the issue change configuration accordingly to accommodate that.

wesleylc1 · Sep 13, 2019, 6:59 AM

Hi @PiBa this error occurred after configuring nginx to respond on port 443, worked fine with port 80, after applying the port 443 settings, started not loading web application.

According to the waget output on nextcloud.projectus.com.br, it connects to ports 80 and 443, but there is an error "HTTP request sent, waiting for response ... 302 found".

root@server:~# wget nextcloud.projectus.com.br
--2019-09-13 00:07:23--  http://nextcloud.projectus.com.br/
Resolving nextcloud.projectus.com.br (nextcloud.projectus.com.br)... 189.20.108.90
Connecting to nextcloud.projectus.com.br (nextcloud.projectus.com.br)|189.20.108.xx|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://nextcloud.projectus.com.br/ [following]
--2019-09-13 00:07:23--  https://nextcloud.projectus.com.br/
Connecting to nextcloud.projectus.com.br (nextcloud.projectus.com.br)|189.20.108.xx|:443... connected.
ERROR: cannot verify nextcloud.projectus.com.br's certificate, issued by ‘CN=COMODO RSA Organization Validation Secure Server CA,O=COMODO CA Limited,L=Salford,ST=Greater Manchester,C=GB’:
  Unable to locally verify the issuer's authority.
To connect to nextcloud.projectus.com.br insecurely, use `--no-check-certificate'.

dragoangel · Sep 13, 2019, 7:05 AM

@wesleylc1 302 is redirect from nginx? You want server your backend (nginx) on https 443 port? - then haproxy backend MUST go to ip:443 of nginx with ssl checkbox enabled. Prepere for that you need clearly understand how PKI validation work. Or only do ssl offloading on haproxy? - then remove redirect to https on nginx at least for haproxy server ip and haproxy backend must speak to nginx 80 port...

Can you show nginx config? If you on nginx configure multiple site's they then haproxy backend must have action 'request header Host 'servername''.
Fyi: backend healthcheck issues will never be related to frontend configuration.

wesleylc1 · Sep 13, 2019, 8:56 PM

Hello everyone, I know this issue is more related to the nginx web server, not the HAproxy service. I did a few things, including deleting settings from the nginx configuration file and creating a new one based on official documentation, for example, the reported error became "502 Bad Gateway".

upstream php-handler {
    server 127.0.0.1:9000;
    #server unix:/var/run/php/php7.2-fpm.sock;
}

server {
    listen 80;
    listen [::]:80;
    server_name nextcloud.projectus.com.br;
    # enforce https
    return 301 https://nextcloud.projectus.com.br:443$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name nextcloud.projectus.com.br;

    # Use Mozilla's guidelines for SSL/TLS settings
    # https://mozilla.github.io/server-side-tls/ssl-config-generator/
    # NOTE: some settings below might be redundant
    ssl_certificate /root/certificado/2019-2020/ssl_bundle.crt;
    ssl_certificate_key /root/certificado/2019-2020/server.key;

    # Add headers to serve security related headers
    # Before enabling Strict-Transport-Security headers please read into this
    # topic first.
    #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
    #
    # WARNING: Only add the preload option once you read about
    # the consequences in https://hstspreload.org/. This option
    # will add the domain to a hardcoded list that is shipped
    # in all major browsers and getting removed from this list
    # could take several months.
    add_header X-Content-Type-Options nosniff;
    add_header X-XSS-Protection "1; mode=block";
    add_header X-Robots-Tag none;
    add_header X-Download-Options noopen;
    add_header X-Permitted-Cross-Domain-Policies none;
    add_header Referrer-Policy no-referrer;

    # Remove X-Powered-By, which is an information leak
    fastcgi_hide_header X-Powered-By;

    # Path to the root of your installation
    root /var/www/nextcloud;

    location = /robots.txt {
        allow all;
        log_not_found off;
        access_log off;
    }

    # The following 2 rules are only needed for the user_webfinger app.
    # Uncomment it if you're planning to use this app.
    #rewrite ^/.well-known/host-meta /public.php?service=host-meta last;
    #rewrite ^/.well-known/host-meta.json /public.php?service=host-meta-json last;

    # The following rule is only needed for the Social app.
    # Uncomment it if you're planning to use this app.
    #rewrite ^/.well-known/webfinger /public.php?service=webfinger last;

    location = /.well-known/carddav {
      return 301 $scheme://$host:$server_port/remote.php/dav;
    }
    location = /.well-known/caldav {
      return 301 $scheme://$host:$server_port/remote.php/dav;
    }

    # set max upload size
    client_max_body_size 512M;
    fastcgi_buffers 64 4K;

    # Enable gzip but do not remove ETag headers
    gzip on;
    gzip_vary on;
    gzip_comp_level 4;
    gzip_min_length 256;
    gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
    gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

    # Uncomment if your server is build with the ngx_pagespeed module
    # This module is currently not supported.
    #pagespeed off;

    location / {
        rewrite ^ /index.php$request_uri;
    }

    location ~ ^\/(?:build|tests|config|lib|3rdparty|templates|data)\/ {
        deny all;
    }
    location ~ ^\/(?:\.|autotest|occ|issue|indie|db_|console) {
        deny all;
    }

    location ~ ^\/(?:index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+)\.php(?:$|\/) {
        fastcgi_split_path_info ^(.+?\.php)(\/.*|)$;
        include fastcgi_params;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $fastcgi_path_info;
        fastcgi_param HTTPS on;
        # Avoid sending the security headers twice
        fastcgi_param modHeadersAvailable true;
        # Enable pretty urls
        fastcgi_param front_controller_active true;
        fastcgi_pass php-handler;
        fastcgi_intercept_errors on;
        fastcgi_request_buffering off;
    }

    location ~ ^\/(?:updater|oc[ms]-provider)(?:$|\/) {
        try_files $uri/ =404;
        index index.php;
    }

    # Adding the cache control header for js, css and map files
    # Make sure it is BELOW the PHP block
    location ~ \.(?:css|js|woff2?|svg|gif|map)$ {
        try_files $uri /index.php$request_uri;
        add_header Cache-Control "public, max-age=15778463";
        # Add headers to serve security related headers (It is intended to
        # have those duplicated to the ones above)
        # Before enabling Strict-Transport-Security headers please read into
        # this topic first.
        #add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;";
        #
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header X-Robots-Tag none;
        add_header X-Download-Options noopen;
        add_header X-Permitted-Cross-Domain-Policies none;
        add_header Referrer-Policy no-referrer;

        # Optional: Don't log access to assets
        access_log off;
    }

    location ~ \.(?:png|html|ttf|ico|jpg|jpeg|bcmap)$ {
        try_files $uri /index.php$request_uri;
        # Optional: Don't log access to other assets
        access_log off;
    }
}

I collected the nginx web server log, it is possible to identify a denial of connection by my pfsense firewall, which has subnet 192.168.43.xxx/22

2019/09/13 20:37:10 [error] 11415#11415: *23 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.43.xx, server: nextcloud.projectus.com.br, request: "OPTIONS / HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000"

Documentation Reference Link

best regards,
Wesley Santos

dragoangel · Sep 13, 2019, 9:04 PM

@wesleylc1 o_O, Really read previous message...
Redirect on backend to ssl, http2 on backend... Stop it... Think 5 mins.

wesleylc1 · Sep 13, 2019, 9:07 PM

@dragoangel do you think I should leave nginx as http and redirect ssl directly to the backend?

best regards,
Wesley Santos

dragoangel · Sep 13, 2019, 9:13 PM

@wesleylc1 it will at least help you understand how it works. And latter you can configure second test backend with ssl. But in any case: redirects to subdomain, another protocol better be done on proxy, not on backend. And http2 is useless on backend - haproxy will use http1.1 when speaking with backend but will give h2 to end clients if you configure it on Frontend binding.

wesleylc1 · Sep 13, 2019, 9:14 PM

@dragoangel As shown, my backend is already pointing to port 443 and selecting the SSL check box.

dragoangel · Sep 13, 2019, 9:18 PM

@wesleylc1 Lol red line on ipv4 192.168.x.x, funny. Why cares, you mentioned public domain (https://nextcloud.projectus.com.br), but masked private ip? Ok. And what haproxy status you now see on backend ?

wesleylc1 · Sep 13, 2019, 9:20 PM

@dragoangel lol lol 502 Bad Gateway

dragoangel · Sep 13, 2019, 9:24 PM

@wesleylc1 Why at bottom writed Ubuntu...? You go to haproxy or to nginx and see this?.. In any case: go to nginx straight and check you backend first. It eork? If yes - then fix your healthcheck. If it still not work add header host on backend config of haproxy in case you have in nginx multiple domains on same port-cert

wesleylc1 · Sep 16, 2019, 2:57 PM

Hi guys, I set up https internally on the nginx server, already responding to https requests on nextcloud.projectus.com.br, now I'm trying to apply backend settings to my front end, which is already configured to use port 443, but my The backend displays the error "L7STS / 400 in 46ms".

front end settings

frontend HA_Sistemas-https
	bind			189.20.108.xx:443 name 189.20.108.xx:443   ssl crt-list /var/etc/haproxy/HA_Sistemas-https.crt_list  
	bind			187.75.209.2xx:443 name 187.75.209.xx:443   ssl crt-list /var/etc/haproxy/HA_Sistemas-https.crt_list  
	mode			http
	log			global
	option			log-separate-errors
	option			httplog
	option			http-keep-alive
	timeout client		300000
	acl			rootrequested	var(txn.txnpath) -m str -i /
	acl			time-sheet	var(txn.txnhost) -m str -i time-sheet.projectus.com.br
	acl			qsms	var(txn.txnhost) -m str -i qsms.projectus.com.br
	acl			treinamento	var(txn.txnhost) -m str -i treinamento.projectus.com.br
	acl			proposta	var(txn.txnhost) -m str -i proposta.projectus.com.br
	acl			chamado	var(txn.txnhost) -m str -i chamado.projectus.com.br
	acl			pesquisa	var(txn.txnhost) -m str -i pesquisa.projectus.com.br
	acl			pesquisa-ce	var(txn.txnhost) -m str -i pesquisa-ce.projectus.com.br
	acl			nextcloud	var(txn.txnhost) -m str -i nextcloud.projectus.com.br
	http-request set-var(txn.txnpath) path
	http-request set-var(txn.txnhost) hdr(host)
	http-request redirect location /TimeSheet/faces/login.xhtml  if  rootrequested time-sheet 
	http-request redirect location /SistemaQSMS/  if  rootrequested qsms 
	http-request redirect location /Treinamento/  if  rootrequested treinamento 
	http-request redirect location /Proposta/  if  rootrequested proposta 
	http-request redirect location /Chamado/  if  rootrequested chamado 
	http-request redirect location /PesquisaClima/  if  rootrequested pesquisa 
	http-request redirect location /PesquisaClimaCE/  if  rootrequested pesquisa-ce 
	http-response add-header Content-Security-Policy upgrade-insecure-requests  if  !time-sheet !qsms !treinamento !proposta !chamado !pesquisa !pesquisa-ce 
	use_backend HA_Sistemas-45-xx_80-www_ipvANY  if  !time-sheet !qsms !treinamento !proposta !chamado !pesquisa !pesquisa-ce 
	default_backend HA_Sistemas_40-xxx_443-nextcloud_ipvANY
	default_backend HA_Sistemas_43-xxx_8443_ipvANY

backend settings

backend HA_Sistemas_40-xxx_443-nextcloud_ipvANY
	mode			http
	id			109
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk OPTIONS / 
	server			nextcloud 192.168.40.xxx:443 id 108 ssl check inter 1000  verify none

best regards,
Wesley Santos

dragoangel · Sep 16, 2019, 5:30 PM

@wesleylc1 Try disable healthchecks for beginning

wesleylc1 · Sep 16, 2019, 5:37 PM

@dragoangel How do I disable health checks?

Would it be as in the picture?

dragoangel · Sep 16, 2019, 5:43 PM

@wesleylc1 yes. That it. Try opening you frontend domain

wesleylc1 · Sep 16, 2019, 5:45 PM

@dragoangel No external access yet.