HAProxy - Backend behind Open-VPN

[mikem123]

Just a quick one,

I have 2 Pfsense servers working really well with an openvpn tunnel connecting the 2 sites.
All routes etc are working as expected, with VM's communicating over the tunnel. (mysql repl, ldsync etc)

I have however hit an issue with HAProxy being able to utilise a node on the second site as a backend server.

I can curl the site etc from PfSense but HAProxy is showing it as down. Is this a limitation i have hit, or is there some trickery i can do to make this work.

[PiBa]

"Why" does haproxy mark the backend as down? is it a L4 , L6 or L7 issue? The LastCkh column on the haproxy stats page should tell..
Basically if you can curl a webserver from the pfSense machine then haproxy should also be able to reach it without much issues..

[mikem123]

Layer 7 check (http), just on the non default web server port I use (8080). It used to work to the secondary server I had previously at site 1, just since moving it to the second DC behind the VPN it no longer works. Site 1 still works with the same config though this hits site 1, 10.219.0.10
Stats not enabled at the moment

Basic tests
pfSense / Diagnostic / Test Port
Port test to host: 10.219.1.10 Port: 8080 successful.

HAProxy
app03.XYZ.local
DOWN

Backend Settings
app03.XYZ.local
Address+Port:
10.219.1.10
8080

Which interface will HAProxy test from? the one bound to the front end or will it choose the best interface. My ADC knowledge is generally with F5's so just making sure I am not making assumptions based on previous knowledge.

Checking the httpd access log from site 1, I can see the health checks
10.219.0.1 10.219.0.1 - - [29/Sep/2020:09:50:15 +1000] "OPTIONS / HTTP/1.0" 200 7995 "-" "-"
10.219.0.1 10.219.0.1 - - [29/Sep/2020:09:50:16 +1000] "OPTIONS / HTTP/1.0" 200 7995 "-" "-"
10.219.0.1 10.219.0.1 - - [29/Sep/2020:09:50:18 +1000] "OPTIONS / HTTP/1.0" 200 7995 "-" "-"
10.219.0.1 10.219.0.1 - - [29/Sep/2020:09:50:19 +1000] "OPTIONS / HTTP/1.0" 200 7995 "-" "-"
10.219.0.1 10.219.0.1 - - [29/Sep/2020:09:50:20 +1000] "OPTIONS / HTTP/1.0" 200 7995 "-" "-"
10.219.0.1 10.219.0.1 - - [29/Sep/2020:09:50:21 +1000] "OPTIONS / HTTP/1.0" 200 7995 "-" "-"
10.219.0.1 10.219.0.1 - - [29/Sep/2020:09:50:22 +1000] "OPTIONS / HTTP/1.0" 200 7995 "-" "-"
10.219.0.1 10.219.0.1 - - [29/Sep/2020:09:50:23 +1000] "OPTIONS / HTTP/1.0" 200 7995 "-" "-"

Ahh sugar, looks to be a id10T error.
Sorry about this, i will work on the forbidden now

10.219.3.1 - - [29/Sep/2020:09:52:34 +1000] "OPTIONS / HTTP/1.0" 403 4897 "-" "-"
10.219.3.1 - - [29/Sep/2020:09:52:35 +1000] "OPTIONS / HTTP/1.0" 403 4897 "-" "-"
10.219.3.1 - - [29/Sep/2020:09:52:36 +1000] "OPTIONS / HTTP/1.0" 403 4897 "-" "-"
10.219.3.1 - - [29/Sep/2020:09:52:37 +1000] "OPTIONS / HTTP/1.0" 403 4897 "-" "-"
10.219.3.1 - - [29/Sep/2020:09:52:38 +1000] "OPTIONS / HTTP/1.0" 403 4897 "-" "-"
10.219.3.1 - - [29/Sep/2020:09:52:39 +1000] "OPTIONS / HTTP/1.0" 403 4897 "-" "-"

[PiBa]

@mikem123 said in HAProxy - Backend behind Open-VPN:

Which interface will HAProxy test from? the one bound to the front end or will it choose the best interface.

It would use the 'best' interface..

You could probably 'force' a client-ip for the connection if your dealing with a ipsec tunnel that doesn't fill the routing-table properly that might help. But with openvpn that shouldnt be needed. (In the ipsec case the client-ip would have to be set with a 'advanced text option' the webgui doesnt provide this choice.)

But anyhow looks like a L7 issue :), you could tell the healthcheck to expect the 403 as 'valid'.. Depends how precise you want/need the check to be.