Websockets configuration in HAProxy

[Peque]

Hi Forum
I having setup a HAproxy on Pfsense - for handling our incomming request to webpages.
One of the sites are running Qlik

I'm struggling since the server is using websockets.
I can find other guides on using HAProxy directly - as described Here

I'm not running as Loadbalancing since the setup is a testsetup - for development on our Qlik server.

But what I do not get and do not know where to configure - is these lines:

## routing based on Host header
  acl host_ws hdr_beg(Host) -i ws.
  use_backend bk_ws if host_ws
## routing based on websocket protocol header
  acl hdr_connection_upgrade hdr(Connection)  -i upgrade
  acl hdr_upgrade_websocket  hdr(Upgrade)     -i websocket
  use_backend bk_ws if hdr_connection_upgrade hdr_upgrade_websocket
  default_backend bk_web

I cannot find where and how to create these Actions since i cannot se the option for how to create these settings. Since Qlik is using websockets - we get a timeout - can refresh and the page are loading - But if I press refresh it'll load the page.
I've look in generel, frontends and backends but cannot find anywhere I can configure these lines

Buit I would like the setup to run as smooth as possible without these timeouts

[PiBa]

@peque
You should be able to create the acl's like this:

The 'hdr_beg' looks a bit different than what is generated in the config, but should have the same effect as the '-m beg'. As for the 'use-backend' actions, i guess you can figure that part out? If not ill make a little screenshot that part as well.

[clumbo]

@piba

Thankyou

This is just what I have been looking for I have been trying to workout how to do this with bitwarden_rs

Any chance you could share your a more full screenshot of the backend part too?

Thanks

[dfir_dk]

@clumbo Did you happen to make it work with bitwarden_rs? I'm here for the same reason struggling with this issue :)

[sgtpepperaut]

@dfir_dk said in Websockets configuration in HAProxy:

a

hey fellow bitwarder...looking for the same. i havent been able to get this to work :(

since im also ssl offloading i get a WSS connection failed in the frontend. :(

[dfir_dk]

@sgtpepperaut
Hey dude, I've found my solution here: https://github.com/dani-garcia/bitwarden_rs/wiki/Proxy-examples

I've started to use "HAproxy (by @williamdes)" and it looks like this:

Hope this helps..

[sgtpepperaut]

@dfir_dk

yes it does thanks!

but what I cant get my head around is why I need two different backends...

how do your backend configs look like? specifically the ws backend. Isnt it the same port?! or is there some magic going on ?

[dfir_dk]

@sgtpepperaut
Yes, it is built upon two different services. One that handles everything except notifications, which is handled by a different service on port 3012.

Take a look here:

https://github.com/dani-garcia/bitwarden_rs/wiki/Enabling-WebSocket-notifications

I have two backends that point to the same IP address but different port 80 and 3012.

If you do not enable notifications then you do not need the additional config :)

[Peque]

Well still struggling a whole lot with this websocket and how to configure my proxy for using the Qlik server
As configured Backend - it looks like this:

And the frontend are configured as this

But are not getting through to the server at all - just getting error 503
But am I doing this wrong - is the custom ACL placed under the backend or frontend ( The same input are available both places) - and since this is the first I have a site using websockets - so not sure if this is configured totally wrong ?
Please advise

P

[senseivita]

@peque if you haven't fixed this, I think I might be able to help. — or maybe help someody else. :)

According to Websockets Load Balancing with HAProxy all the connections can be made in HTTP mode just fine but when the 101 HTTP status code is passed the connection is "upgraded"* and from then on it's not processed anymore, much like TCP or SNI traffic. So it needs to be setup like that but at the same time offloading what needs a new certificate (I assume this is the reason). Leave your old config as is and first find what exactly needs to be passed through.

Find the switchover point

Connect directly to your backend using your browser:

• Modify it and your hostsfile/DNS temporarily so it thinks you're connecting through a proxy if need be
  • Remember to clear system's local DNS caches and your browsers' caches. Safari is the easiest one for this (⌘⌥e). Overall it's really annoying. As a refresher:
    • Windows: ipconfig /flushdns
    • macOS: sudo killall mDNSResponder mDNSResponderHelper
    • Linux: JK--You don't need my help. systemd-resolve --flush-caches maybe?
• Open a new browser window in private session mode
• If it loads anything automatically, go to "about:blank" no quotes, no scheme (i.e: http:// https:// ws://…)
• Open the dev tools and go to the Network tab:
  • Firefox has a direct shortcut: ⌘⌥e () ⌃⇧e (Windows)
  • Chromium-based/Safari: ⌘⌥i () ⌃⇧i (Windows) to open the dev tools, then click on Network.
    • Safari needs to have the Developer menu enabled first: ⌘, then Advanced then check Show Develop menu in menu bar
• If for some reason it's not blank, look for the icon that clears it. A little trash icon or something being crossed over…
• Look for the icon to disable the caches and enable it.
• Look for the option to persist the logs and enable it (in FF it's in the little ︎ icon)
• Load your site and scroll to the top of the entries and select it to view the headers, use the arrow keys to scroll down the list until you find the 101.

Divert the right traffic to the right place

As other mentioned earlier, your app is serving content from more than one server; if it doesn't use a another port for the other traffic, an internal proxy must be redirecting the websockets connection based on the URL's path.

Duplicate your existing backend for that server and on the server section uncheck the SSL box. Give it a new new (it will have "-copy" at the end if you duplicated it) by appending/prepending something like SNI or websockets, whatever matches your naming scheme if you have one.

With the information you learned using your browser, edit your frontend to add the rules (as separate rules, not replacing) you found before (or use the link tot the article in this post at the beginning) so it matches the traffic to be redirected to the new backend which is exactly as the one you had before except without the SSL checkbox –– because HAProxy won't process any of it, it'll only pass it through.

Position the rules and double check it "stuck"

According to the rule itself slide it into position. Let me elaborate:
Let's assume your app uses pathnames to separate the traffic, it's reachable at [https://…]/app and you found out the websocket connection is at the root /. Since you need to offload the /app connection the rule to divert the traffic for the / connection should then go after /app's because it's broader/less specific. Since you'd be appending preexisting frontends, it should be alright to just add it at the end.

Because HTTP is also served from the backend, it's more likely that the websocket connection it's served at a deeper path level though, so using the previous example /app continues to be your HTTP traffic and the websocket connection is served at /app/ws. In this case you would need to move the rule above /app's. (click its check box and click on the little anvil/anchor icon of the other rule).

︎, ︎, ︎ but still doesn't work? Double check your saved config

Recently I moved back to the built-in HAProxy on pfSense. I had moved out to a dedicated machine where I was forced to learn all of this and then some. Setting up the health checks on pfSense I noticed some wouldn't pass regardless they were configured correctly. I knew this I just went though hell learning about it.

Then one time when I hovered the cursor on the server name on the stats page the little tooltip (or whatever it's called) appeared with the server's IP address and the port, both of which weren't what I entered in the configuration. Adding new servers to the backend then deleting the older ones didn't work either. I needed to add a completely different backend then switch the frontend to this new backend and only then delete the broken backend. The config sometimes just won't stick, I have noticed this happens all over pfSense so you have to "nudge" it into working. It's not a big deal but it's not obvious either so it can go unnoticed long enough for you to screw up other parts of the system thinking they're at fault.

Strangely enough now that I have all greens the tooltip won't appear in any of the backends… That's another story though.

*: I use quotes because everywhere else in the HAProxy docs to upgrade a connection is the opposite, e.g; TCP → HTTP → H2 → QUIC. Here you're going backwards TCP ← HTTP/X.

[mthx]

@peque Got it working with the Help of this Guide ( https://github.com/dani-garcia/bitwarden_rs/wiki/Enabling-WebSocket-notifications ) , but wanted to say that i removed ACL1 & ACL 2 cause this guide assumes bitwarden is the only thing running on it and will lead to every Domain configured in Haproxy loading Bitwarden. since i have allready configured Bitwarden without WS and had it working, i left that backend as is and took those first 2 ACLs away

[mvern]

Thank you for this! Got my application to work. Much appreciated.