Matrix Synapse behind HAProxy on pfSense

OCT0PUSCRIME

@OCT0PUSCRIME said in Matrix Synapse behind HAProxy on pfSense:

@tiran Did you ever figure this out? I have got the URL: https://MYDOMAIN/.well-known/matrix/server resolving to the correct json file, but the federation tester is still giving me a timeout for some reason.

Following up: my issue was really dumb. If you feel you have everything set up correctly, but you are getting timeouts on the federation tester, make sure your GeoIP blocking isn't too restrictive with pfBlockerNG, if you have that set up.

sub2010

@tiran
Damn i didnt get it. I need the matrix well know configuration.

frontend https
  acl matrix-well-known-client-path path /.well-known/matrix/client
  acl matrix-well-known-server-path path /.well-known/matrix/server
  use_backend matrix-well-known-client if matrix-well-known-client-path
  use_backend matrix-well-known-server if matrix-well-known-server-path
 
backend matrix-well-known-client
  http-after-response set-header Access-Control-Allow-Origin "*"
  http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
  http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
  http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.example.com"},"m.identity_server":{"base_url":"https://identity.example.com"}}'

backend matrix-well-known-server
  http-after-response set-header Access-Control-Allow-Origin "*"
  http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
  http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
  http-request return status 200 content-type application/json string '{"m.server":"matrix.example.com:443"}'

I created a frontend

  acl matrix-well-known-client-path path /.well-known/matrix/client
  acl matrix-well-known-server-path path /.well-known/matrix/server
  use_backend matrix-well-known-client if matrix-well-known-client-path
  use_backend matrix-well-known-server if matrix-well-known-server-path

I created two backends

http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.ibinary.de"},"m.identity_server":{"base_url":"https://matrix.org"}}'

The Error Code is

Errors found while starting haproxy
[NOTICE] (59904) : haproxy version is 2.7.6-4dadaaa
[NOTICE] (59904) : path to executable is /usr/local/sbin/haproxy
[WARNING] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:71] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
[ALERT] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:72] : error detected while parsing switching rule : no such ACL : 'matrix-well-known-client'.
[ALERT] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:73] : error detected while parsing switching rule : no such ACL : 'matrix-well-known-server'.
[ALERT] (59904) : config : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg
[ALERT] (59904) : config : Fatal errors found in configuration.

OCT0PUSCRIME

@sub2010

Take away the advanced passthrough stuff. Create one frontend called WELL-KNOWN and 2 backends. WELL-KNOWN-MATRIX-CLIENT and WELL-KNOWN-MATRIX-SERVER.

In the front end create 2 Access Control Lists.

NAME EXPRESSION VALUE
SERVER path starts with /.well-known/matrix/server
CLIENT path starts with /.well-known/matrix/client

and 2 Actions

ACTION CONDITIONAL ACL NAME
Use Backend WELL-KNOWN-MATRIX-SERVER SERVER
Use Backend WELL-KNOWN-MATRIX-CLIENT CLIENT

For your backends do not put any configuration except in the advanced passthrough:

WELL-KNOWN-MATRIX-SERVER
http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.server":"[YOURDOMAIN]:[YOURPORT]"}'

WELL-KNOWN-MATRIX-CLIENT
http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://[YOURDOMAIN]"}}'

Note: the json response will vary depending on how you have your matrix server configured. I am using delegation on mine so my port is 443, but yours might be 8448 depending how you are serving.

Note 2: This is a completely separate frontend than my matrix frontend. It only handles the well-known requests. I have a separate frontend for my actual matrix server and you might also have a third frontend for your federation.

Note 3: I also notice on your matrix frontend you do not have use forward for checked. Check that. That takes away the need to have the X-Forwarded Proto in your advanced config. I don't think it's necessary on the well-known frontend, but I tend to just leave it checked on most of mine.

tiran

@sub2010

I serve my well-know from the matrix server. I just published the whole domain on port 443.

Anyway you error suggest that there is no 'matrix-well-known-server' and 'matrix-well-known-client'.
ACL rule and indeed you call your rules 'matrix-well-known-client-path' and 'matrix-well-known-server-path' and under actions you use the names without the -path suffix.

Also the actions are redundant and is where the error is coming from.

Use either actions and Access Control List where you set the matrix.ibi.. stuff.
Or remove the actions and use advanced pass thru option.

I would suggest remove the advanced pass thru on the frontend entirely.

Just add 2 more ACLs matching the actions you already added
Screenshot 2023-09-12 at 08.06.32.png

Don't forget to add -path at the end of you actions Condition acl names.

sub2010

@tiran
@OCT0PUSCRIME

I'm sorry I didn't respond earlier. I had to replace my Synology Reverse Proxy in the background. I implemented your settings as follows, but unfortunately, without success. I must be doing something wrong; could you please check my configuration?
In Matrix Federation, I'm getting an error.

Frontend:
Element-Matrix Frontend

Shared Frontend
Matrix-Synapse

Well-Kown Config

Backend

Well-Kown Config

tiran

Hi,
Looks fine to me. Except the frondend IPs

I don't know you setup with pfsense but mine is used as main router connected to a WAN interface.
Thats where the frontend has to listen.
Yours displaying a private 192.168... address.

You should be able to navigate to your site. matrix.[yourdomain].de/.well-known/matrix/server
And get the json response.

Thats only happening if you make your HAProxy listening on you WAN IP.
Also your DNS should point to you WAN IP.

These are my frontends

Screenshot 2023-09-19 at 11.01.16.png

One for Federation.
One for the web stuff. Which includes the /.well-known stuff and all clients like element and cinny I let the server handle that for me.

This are my backends
Screenshot 2023-09-19 at 11.25.07.png Screenshot 2023-09-19 at 11.27.57.png

I don't use the advanced configuration.

I also added two allow rules to the WAN interface on the firewall for port 443 and 8448 as I mentioned in an earlier post.

Not sure whats the problem is withs yours. I could be the wan configuration or the backend configuration.

Try debugging it with the hosts file. And add your domain to it with the pfsense local IP/matrix server IP and see if it works locally before testing it over the internet.

/etc/hosts

192.168.1.1 matrix.yourdomain.com

Then your computer will use your local ip for your domain.

BTW I use this to manage my server.
https://github.com/spantaleev/matrix-docker-ansible-deploy

Since matrix has a lot of moving parts its nice to have a consistent installation.

Hope you get it figured out.

sub2010

Damn it, i didnt get it run. I dont know where my error is.

I dont wont to give up. Can you further support me?

Lets start from beginning. My synapse config shows this configuration:

server_name: "domain.de"
public_baseurl: "https://matrix.domain.de"
serve_server_wellknown: true

DNS Settings
Host A Entry
domain.de -> IPv4 Adresse
Host A Entry
matrix.domain.de -> IPv4 Adresse

Front End Matrix
Listen addresse (Shared Frontend)
CARP WAN 1 Port 443 SSL Offloading
CARP WAN 2 Port 443SSL Offloading
Access Control lists
matrix Host Matches matrix.domain.de
Actions
use backend matrix matrix-synapse
Use "forwardfor" option.

Backend
Server list
192.168.8.8 8448

Front End Federation
Listen addresse
CARP WAN 1 Port 443 SSL Offloading
CARP WAN 1 Port 8448
Access Control lists

Actions

Use "forwardfor" option.
Server Certficate

Backend matrix-well-known-client
Advanced Settings
http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.domain.de"}}'

Backend matrix-well-known-server
Advanced Settings
http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.server":"matrix.domain.de:443"}'

Federation Tester failed

OCT0PUSCRIME

@sub2010

I mean, a lot of this is going to depend on your matrix config as well.

It looks mostly fine to me, but firstly:
serve_server_wellknown: true

Should not be set in your homeserver.yaml becuase you are serving the well-known from the reverse proxy. This setting makes it available to serve from your matrix host, which is not needed for you. This isn't your issue though.

I see your back end is forwarding to 8448. That requires TLS certs on your matrix server and your homeserver.yaml configured to use them. If that's the case, then I think your well-known/matrix/server should be set to port 8448 and not 443.

I would recommend configuring your matrix listener to listen on 8008 instead and your backend configured to go to 8008 without TLS. With this you dont need a frontend for 8448 and you can leave your well-known/matrix/server port at 443.

sub2010

@OCT0PUSCRIME

First, thank you very much for your support.

Done

It looks mostly fine to me, but firstly:
serve_server_wellknown: true

Done

I see your back end is forwarding to 8448. That requires TLS certs on your matrix server and your homeserver.yaml configured to use them. If that's the case, then I think your well-known/matrix/server should be set to port 8448 and not 443.

Done

I would recommend configuring your matrix listener to listen on 8008 instead and your backend configured to go to 8008 without TLS. With this you dont need a frontend for 8448 and you can leave your well-known/matrix/server port at 443.

The Federation Matrix Error Code

{
    "WellKnownResult": {
        "m.server": "",
        "result": "Get \"https://domain.de/.well-known/matrix/server\": x509: certificate has expired or is not yet valid: current time 2023-10-05T08:24:35Z is after 2023-06-14T15:01:22Z",
        "CacheExpiresAt": 0
    },
    "DNSResult": {
        "SRVSkipped": false,
        "SRVCName": "",
        "SRVRecords": null,
        "SRVError": {
            "Message": "lookup _matrix._tcp.domain.de on 8.8.8.8:53: no such host"
        },
        "Hosts": {
            "domain.de": {
                "CName": "domain.de.",
                "Addrs": [
                    "2a03:4000:61:5c1f::19:967",
                    "87.122.55.127"
                ],
                "Error": null
            }
        },
        "Addrs": [
            "[2a03:4000:61:5c1f::19:967]:8448",
            "87.122.55.127:8448"
        ]
    },
    "ConnectionReports": {},
    "ConnectionErrors": {
        "87.122.55.127:8448": {
            "Message": "Get \"https://87.122.55.127:8448/_matrix/key/v2/server\": EOF"
        },
        "[2a03:4000:61:5c1f::19:967]:8448": {
            "Message": "Get \"https://[2a03:4000:61:5c1f::19:967]:8448/_matrix/key/v2/server\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
        }
    },
    "Version": {
        "error": "Get \"matrix://domain.de/_matrix/federation/v1/version\": EOF"
    },
    "FederationOK": false
}

OCT0PUSCRIME

@sub2010
The error is telling you exactly what's wrong. You have an expired cert that expired on June 14th. Check the cert on your haproxy front end and the one on your matrix host if you have one.

sub2010

@OCT0PUSCRIME
I didnt get it. AGAIN ;-).

First i have a wildcard Certificate, this ist for matrix.domain.de
But my matrix Server runs on servername domain.de

How i can configure the cert for any the domain.de?

This is my frontend configuration

sub2010

@sub2010

Is this SRV Entry correct?

Same Error as well.

{
    "WellKnownResult": {
        "m.server": "",
        "result": "Get \"https://domain.de/.well-known/matrix/server\": x509: certificate has expired or is not yet valid: current time 2023-10-05T15:37:02Z is after 2023-06-14T15:01:22Z",
        "CacheExpiresAt": 0
    },
    "DNSResult": {
        "SRVSkipped": false,
        "SRVCName": "_matrix._tcp.domain.de.",
        "SRVRecords": [
            {
                "Target": "matrix.domain.de.",
                "Port": 443,
                "Priority": 1,
                "Weight": 1
            }
        ],
        "SRVError": null,
        "Hosts": {
            "matrix.domain.de.": {
                "CName": "matrix.domain.de.",
                "Addrs": [
                    "87.122.55.127"
                ],
                "Error": null
            }
        },
        "Addrs": [
            "87.122.55.127:443"
        ]
    },
    "ConnectionReports": {},
    "ConnectionErrors": {
        "87.122.55.127:443": {
            "Message": "Non-200 response 503 from remote server"
        }
    },
    "Version": {
        "error": "Get \"matrix://domain.de/_matrix/federation/v1/version\": x509: certificate is valid for *.domain.de, not domain.de"
    },
    "FederationOK": false
}

How can i change the certificate?

OCT0PUSCRIME

@sub2010
I use the same config. domain.tld and matrix.domain.tld. I'm not sure about your srv record, I dont use one.

For my certificate I use 1 certificate. In acme you can specify multiple domains for one certificate. Mine includes. *.domain.tld and domain.tld

Get a cert like that, put it on your haproxy frontend and also put it on your matrix host and point your homeserver.yaml to it and restart matrix. The error is still saying your cert is expired, so I am assuming the cert you have on your matrix host that your homeserver.yaml is pointing to is expired.