HAproxy 2.0 Prometheus Monitoring How-to

dragoangel

Yesterday HAproxy 2.0 (Devel version) of pfSense was been updated to support build-in Prometheus exporter.

Here is how to use it:

First you need install Prometheus and Grafana in any Unix system (if you not already have them).

There is plenty of ways how to do this, via Docker, Docker-Compose or on bare OS.

I will describe how to do it on RHEL/CentOS on bare OS.

Install Prometheus:

    PROM_RELEASE='2.17.2'
    wget https://github.com/prometheus/prometheus/releases/download/v${PROM_RELEASE}/prometheus-${PROM_RELEASE}.linux-amd64.tar.gz
    useradd --no-create-home --shell /bin/false prometheus
    mkdir /etc/prometheus
    mkdir /var/lib/prometheus
    tar -xvzf prometheus-${PROM_RELEASE}.linux-amd64.tar.gz
    rm -rf prometheus-${PROM_RELEASE}.linux-amd64.tar.gz
    mv -f prometheus-${PROM_RELEASE}.linux-amd64 prometheuspackage
    mv -f prometheuspackage/prometheus /usr/local/bin/
    chown root:root /usr/local/bin/prometheus
    chmod 755 /usr/local/bin/prometheus
    mv -f prometheuspackage/promtool /usr/local/bin/
    chown root:root /usr/local/bin/promtool
    chmod 755 /usr/local/bin/promtool
    cat <<EOF | sudo tee /etc/prometheus/prometheus.yml
    global:
      scrape_interval: 10s
     
    scrape_configs:
      - job_name: 'prometheus_master'
        scrape_interval: 5s
        static_configs:
          - targets: ['localhost:9091']
    EOF
    chown -R prometheus:prometheus /etc/prometheus
    chown -R prometheus:prometheus /var/lib/prometheus
    cat <<EOF | sudo tee /etc/systemd/system/prometheus.service
    [Unit]
    Description=Prometheus
    Wants=network-online.target
    After=network-online.target
     
    [Service]
    User=prometheus
    Group=prometheus
    Type=simple
    ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus/ \
    --storage.tsdb.retention.time=30d \
    --web.listen-address=0.0.0.0:9091 \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries
     
    [Install]
    WantedBy=multi-user.target
    EOF
    systemctl daemon-reload
    systemctl enable --now prometheus

Added Repository to OS for Grafana:

cat <<EOF | sudo tee /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

Created and enabled FirewallD permite rule for Grafana:

cat <<EOF | sudo tee /etc/firewalld/services/grafana.xml
<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>grafana</short>
  <description>Grafana is a multi-platform open source analytics and interactive visualization software.</description>
  <port protocol="tcp" port="3000"/>
</service>
EOF
firewall-cmd --reload
firewall-cmd --zone=public --add-service=grafana --permanent
firewall-cmd --reload

Configured Grafana with HTTPS and SMTP:

app_mode = production
instance_name = grafana.concosto.com
 
[paths]
data = /var/lib/grafana
temp_data_lifetime = 24h
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = conf/provisioning
 
[server]
protocol = https
http_addr =
http_port = 3000
domain = grafana.concosto.com
enforce_domain = false
root_url = %(protocol)s://%(domain)s/
serve_from_sub_path = false
router_logging = false
static_root_path = public
enable_gzip = false
cert_file = /etc/grafana/ssl/grafana.concosto.com.crt
cert_key = /etc/grafana/ssl/grafana.concosto.com.key
 
[dataproxy]
logging = false
timeout = 30
send_user_header = true
 
[analytics]
reporting_enabled = false
check_for_updates = true
 
[security]
disable_initial_admin_creation = true
admin_user = admin
admin_password = PLEASE-Сhange-Me!
disable_gravatar = false
disable_brute_force_login_protection = false
cookie_secure = true
cookie_samesite = lax
allow_embedding = false
 
[users]
allow_sign_up = false
allow_org_create = false
auto_assign_org = false
default_theme = dark
 
[auth.anonymous]
enabled = false
 
[smtp]
enabled = true
host = mail.concosto.com:465
user = grafana@concosto.com
password = PLEASE-Adjust-Me!
skip_verify = false
from_address = grafana@concosto.com
from_name = Concosto Grafana
 
[emails]
welcome_email_on_sign_up = true
 
[alerting]
enabled = true
execute_alerts = true

Note, that you need:

change concosto.com to your real domain
create SSL certificate for your domain. I use own internal CA for such staff and use HAproxy to validate its SSL cert and forward Grafana to world
change Grafana admin username/password
change SMTP settings

After we configre Grafana we can enable and start it:

systemctl daemon-reload
systemctl enable --now grafana

Now we can procced to HAproxy configuration of pfSense.

We will create new frontend with prometheus metrics:

Frontend name: https-promex

Description: [promex]

External address:

YOUR INTERNAL ADDRESS (OR PUBLIC IF PROMETHERUS SERVER NOT IN SAME LOCATION)
Port: 9001 (OR ANY OTHER THAT ISN'T USED)
SSL Offloading: Enabled
Advanced: process 1 (note that you can more proccessed, but better have only one proccess. This settings avaible on General Tab: Number of processes to start: set it to 1 and Number of theads to start per process: set it to count of your vCPUs)

Max connections: 10 (to prevent evil usage)

Don't log normal: enabled (to not load Haproxy by logging all this metrics requests every 10 secs)

Advanced pass thru:

option http-use-htx
http-request use-service prometheus-exporter if { path /metrics }
    stats enable
    stats uri /stats
    stats refresh 10s

Certificate: chose SSL certificate domain of which is binded to your ADDRESS and will be used in Prometheus server config

Add ACL for certificate Subject Alternative Names: Enabled

OCSP: Enable if your SSL certificate has Must-Staple option

Save your new frontend. And now we need configure Firewall to allow access to our HAproxy Metrics only from Prometheus Server IP.

Now you can add new backend for our Grafana:

Name: grafana

Server list:

Name: your server name
ForwartTo: server IP and port
Address: Set dns name or IP of Grafana server
Port: 3000
Encrypt(SSL): enabled

Click on [+] button under server to extend advanced properties.

Check certificate: enabled
Certificate check CN: grafana.concosto.com
CA: choose your internal CA that was been used to issue your Grafana SSL cert
Advanced: sni str(grafana.concosto.com) check-sni grafana.concosto.com

Access control lists and actions:

Actions:

http-request header set name: Host, fmt: grafana.concosto.com

Connection timeout: 5000

Server timeout: 30000

Health check method: Basic

Note, that you need adjust grafana.concosto.com to your real CN or SSL in HAproxy backend config.

Save your new backend.

Now we need create new shared frontend in your main HAproxy frontend to your new Grafana backend.

Name: grafana.concosto.com

Shared Frontend: enabled

Primary frontend: choose one you need

Access Control lists:

Name: grafana.concosto.com, Expression: Host matches, Value: grafana.concosto.com

Actions:

Action: Use Backend, Condition acl name: grafana.concosto.com, Backend: choose your Grafana backend

Certificate: choose your SSL for Grafana fronend, this can be SSL cert from Lets Encrypt for example. Google how get it via ACME plugin.

OCSP: enable it if your SSL had Must Staple or if your SSL CA support it atleast

Save your new shared frontend and apply changed to HAproxy.

Go to Firewall, choose Tab for Interfaces of ADDRESSes which you choose on HAproxy http-promex frontend and create new rule:

Action: Pass

Address Family: tune it to your needs

Protocol: TCP

Source: single host or alias - choose here alias for your Prometheus Server or write its IP. I recommend use pfSense aliases for such stuff

Destination: single host or alias - choose here alias for your HAproxy http-promex fornend or write its IP

Destination Port Range: 9001 (or whatever you was been use)

Description: Allow access to HAproxy Metrics from Prometheus Server

Save your Firewall rule.

Go back to your Unix server and open /etc/prometheus/prometheus.yml via your favorite text editor.

Paste to bottom of config:

- job_name: 'haproxy2.0'
  scheme: https
  static_configs:
  - targets:
    - 'pfsensen01.concosto.com:9001'

Note: adjust hostname and port to your actuall one and take care about tabs as in YAML they are important.

Go to your new Grafana

Login with your admin username and password

Add your Prometheus server: http://localhost:9091 if it was been installed on same server as Granafa.

Import HAproxy 2.0 Granafa Dashboard by putting ID: 10225 and click important. Home page: https://grafana.com/grafana/dashboards/10225

Now you must see your pfSense HAproxy monitoring up and working

Thats all. Hope you liked this how-to.

dragoangel

For those people who has HAproxy not on pfSense:

frontend http-promex
	bind			<%IP%>:9001 name <%IP%>:9001  ssl crt-list /var/etc/haproxy/http-promex.crt_list  process 1
	mode			http
	log				global
	option			dontlog-normal
	option			http-keep-alive
	maxconn			10
	timeout client		30000
	option http-use-htx
	http-request use-service prometheus-exporter if { path /metrics }
		stats enable
		stats uri /stats
		stats refresh 10s
	acl			aclcrt_http-promex	var(txn.txnhost) -m reg -i ^pfsensen01\.concosto\.com(:([0-9]){1,5})?$
	http-request set-var(txn.txnhost) hdr(host)

frontend https-f01
	bind			<%IP%>:443 name <%IP%>:443  ssl crt-list /var/etc/haproxy/https-f01.crt_list alpn h2,http/1.1 no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384 
	mode			http
	log				global
	option			httplog
	option			http-server-close
	option			forwardfor
	acl https ssl_fc
	http-request set-header		X-Forwarded-Proto http if !https
	http-request set-header		X-Forwarded-Proto https if https
	timeout client		30000
	acl			grafana.concosto.com	var(txn.txnhost) -m str -i grafana.concosto.com
	http-request set-var(txn.txnpath) path
	http-request set-var(txn.txnhost) hdr(host)
	capture request header X-Forwarded-For len 128
	capture request header Host len 32 
	capture request header User-Agent len 128
	use_backend grafana.concosto.com  if  grafana.concosto.com

backend grafana.concosto.com
	mode		http
	id			160
	log			global
	timeout connect		5000
	timeout server		30000
	retries			3
	http-request set-header Host grafana.concosto.com
	server			grafana.concosto.com <%IP%>:3000 id <%ID%> ssl check inter 1000  ca-file /var/etc/haproxy/internal-ca.pem verifyhost grafana.concosto.com resolvers globalresolvers sni str(grafana.concosto.com) check-sni grafana.concosto.com

Note: find change <%IP%> and grafana.concosto.com in config to your one