Matrix Synapse behind HAProxy on pfSense

frostys

Hello,
I am currently trying to deploy a Synapse instance behind pfSense with HAProxy.
Synapse itself works fine, but the federation part of Synapse is not working at all.
Response from federationtester.matrix.org:

{
  "WellKnownResult": {
    "m.server": "",
    "result": "Get \"https://matrix.domain.tld/.well-known/matrix/server\": dial tcp MYIP:443: i/o timeout",
    "CacheExpiresAt": 0
  },
  "DNSResult": {
    "SRVSkipped": false,
    "SRVCName": "",
    "SRVRecords": null,
    "SRVError": {
      "Message": "lookup _matrix._tcp.matrix.domain.tld on 8.8.8.8:53: no such host"
    },
    "Hosts": {
      "matrix.domain.tld": {
        "CName": "mycname.",
        "Addrs": [
          "MYIP"
        ],
        "Error": null
      }
    },
    "Addrs": [
      "MYIP:8448"
    ]
  },
  "ConnectionReports": {},
  "ConnectionErrors": {
    "MYIP8448": {
      "Message": "Get \"https://MYIP:8448/_matrix/key/v2/server\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
    }
  },
  "Version": {
    "error": "Get \"matrix://matrix.domain.tld/_matrix/federation/v1/version\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
  },
  "FederationOK": false
}

For setting up I used this Ansible playbook.

I also tried multiple different configurations for the federation HAProxy part, but to no avail.

I am hoping someone here did setup Synapse behind HAProxy on pfSense and can help me out here.

Baker0052

Got mine working here with Pfsense, HAProxy and the same Ansible script.
Matrix Federation Tester Output

{
  "WellKnownResult": {
    "m.server": "",
    "result": "Get \"https://MYDOMAIN/.well-known/matrix/server\": x509: certificate has expired or is not yet valid: current time 2021-12-30T22:11:39Z is after 2019-07-20T00:20:42Z",
    "CacheExpiresAt": 0
  },
  "DNSResult": {
    "SRVSkipped": false,
    "SRVCName": "_matrix._tcp.MYDOMAIN.",
    "SRVRecords": [
      {
        "Target": "matrix.MYDOMAIN.",
        "Port": 8448,
        "Priority": 10,
        "Weight": 0
      }
    ],
    "SRVError": null,
    "Hosts": {
      "matrix.MYDOMAIN.": {
        "CName": "matrix.MYDOMAIN.",
        "Addrs": [
          "MY.IP.Addr.Rss"
        ],
        "Error": null
      }
    },
    "Addrs": [
      "MY.IP.Addr.Rss:8448"
    ]
  },
  "ConnectionReports": {
    "MY.IP.Addr.Rss:8448": {
      "Certificates": [
        {
          "SubjectCommonName": "MYDOMAIN",
          "IssuerCommonName": "R3",
          "SHA256Fingerprint": "mNxQhNc5kh0y/m0M/lNmUT6tH/ZagjQ+yd/fHuKqwRA",
          "DNSNames": [
            "MYDOMAIN"
          ]
        },
        {
          "SubjectCommonName": "R3",
          "IssuerCommonName": "ISRG Root X1",
          "SHA256Fingerprint": "Z63RFmsCCuYbj1/JaBPATCqliZYHloZVcqPH5zdhPf0",
          "DNSNames": null
        },
        {
          "SubjectCommonName": "ISRG Root X1",
          "IssuerCommonName": "DST Root CA X3",
          "SHA256Fingerprint": "bZn7Jl6xxbN0R2X8vGSPPNjhv/r9xML5m51Hz3/xwk8",
          "DNSNames": null
        }
      ],
      "Cipher": {
        "Version": "TLS 1.3",
        "CipherSuite": "TLS_AES_256_GCM_SHA384"
      },
      "Checks": {
        "AllChecksOK": true,
        "MatchingServerName": true,
        "FutureValidUntilTS": true,
        "HasEd25519Key": true,
        "AllEd25519ChecksOK": true,
        "Ed25519Checks": {
          "ed25519:a_uphM": {
            "ValidEd25519": true,
            "MatchingSignature": true
          }
        },
        "ValidCertificates": true
      },
      "Errors": [],
      "Ed25519VerifyKeys": {
        "ed25519:a_uphM": "X9d+yyyMpzQ/KmWXvTScn13Iiki/k8H5tyxii9y64rw"
      },
      "Info": {},
      "Keys": {
        "old_verify_keys": {},
        "server_name": "MYDOMAIN",
        "signatures": {
          "MYDOMAIN": {
            "ed25519:a_uphM": "huZnEh+oLK2aKPspuQx5iq12e0QO3I1igbx2vZ513awgDHPieRuw1JUitm1z+kvWWFu6ZCT7W1dBFHyIann3Cg"
          }
        },
        "valid_until_ts": 1640988673800,
        "verify_keys": {
          "ed25519:a_uphM": {
            "key": "X9d+yyyMpzQ/KmWXvTScn13Iiki/k8H5tyxii9y64rw"
          }
        }
      }
    }
  },
  "ConnectionErrors": {},
  "Version": {
    "name": "Synapse",
    "version": "1.49.2"
  },
  "FederationOK": true
}

Your HAProxy Config would be helpfull

tiran

Hi @Baker0052 keen to share your haproxy conf. I have the same problem and cannot figure it out.

OCT0PUSCRIME

@tiran Did you ever figure this out? I have got the URL: https://MYDOMAIN/.well-known/matrix/server resolving to the correct json file, but the federation tester is still giving me a timeout for some reason.

sub2010

Hi there,

how you configure the matrix federation settings in the HAProxy GUI?
I didnt know how i configure the specification from synapse:

Can you help me?
frontend https
bind *:443,[::]:443 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
http-request set-header X-Forwarded-For %[src]

Matrix client traffic

acl matrix-host hdr(host) -i matrix.example.com matrix.example.com:443
acl matrix-path path_beg /_matrix
acl matrix-path path_beg /_synapse/client

use_backend matrix if matrix-host matrix-path

frontend matrix-federation
bind *:8448,[::]:8448 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
http-request set-header X-Forwarded-For %[src]

default_backend matrix

backend matrix
server matrix 127.0.0.1:8008

tiran

I created 2 seperate frontends and 2 backends for the matrix federation and for the matrix client.

Then set the default backend for each frontend.

Matrix Federation(Frontend) points to -> matrix federation (backend) server port 8448
Matrix Client(Frontend) points to -> matrix client backend server port 80

Under advanced settings in the frontend I put this stuff.
Screenshot 2023-09-11 at 19.42.20.png

Don't forget to add a firewall rule in the WAN section to allow the port 8448

Screenshot 2023-09-11 at 19.53.28.png

Hope that helps.

OCT0PUSCRIME

@OCT0PUSCRIME said in Matrix Synapse behind HAProxy on pfSense:

@tiran Did you ever figure this out? I have got the URL: https://MYDOMAIN/.well-known/matrix/server resolving to the correct json file, but the federation tester is still giving me a timeout for some reason.

Following up: my issue was really dumb. If you feel you have everything set up correctly, but you are getting timeouts on the federation tester, make sure your GeoIP blocking isn't too restrictive with pfBlockerNG, if you have that set up.

sub2010

@tiran
Damn i didnt get it. I need the matrix well know configuration.

frontend https
  acl matrix-well-known-client-path path /.well-known/matrix/client
  acl matrix-well-known-server-path path /.well-known/matrix/server
  use_backend matrix-well-known-client if matrix-well-known-client-path
  use_backend matrix-well-known-server if matrix-well-known-server-path
 
backend matrix-well-known-client
  http-after-response set-header Access-Control-Allow-Origin "*"
  http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
  http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
  http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.example.com"},"m.identity_server":{"base_url":"https://identity.example.com"}}'

backend matrix-well-known-server
  http-after-response set-header Access-Control-Allow-Origin "*"
  http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
  http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
  http-request return status 200 content-type application/json string '{"m.server":"matrix.example.com:443"}'

I created a frontend

  acl matrix-well-known-client-path path /.well-known/matrix/client
  acl matrix-well-known-server-path path /.well-known/matrix/server
  use_backend matrix-well-known-client if matrix-well-known-client-path
  use_backend matrix-well-known-server if matrix-well-known-server-path

I created two backends

http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.ibinary.de"},"m.identity_server":{"base_url":"https://matrix.org"}}'

The Error Code is

Errors found while starting haproxy
[NOTICE] (59904) : haproxy version is 2.7.6-4dadaaa
[NOTICE] (59904) : path to executable is /usr/local/sbin/haproxy
[WARNING] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:71] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
[ALERT] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:72] : error detected while parsing switching rule : no such ACL : 'matrix-well-known-client'.
[ALERT] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:73] : error detected while parsing switching rule : no such ACL : 'matrix-well-known-server'.
[ALERT] (59904) : config : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg
[ALERT] (59904) : config : Fatal errors found in configuration.

OCT0PUSCRIME

@sub2010

Take away the advanced passthrough stuff. Create one frontend called WELL-KNOWN and 2 backends. WELL-KNOWN-MATRIX-CLIENT and WELL-KNOWN-MATRIX-SERVER.

In the front end create 2 Access Control Lists.

NAME EXPRESSION VALUE
SERVER path starts with /.well-known/matrix/server
CLIENT path starts with /.well-known/matrix/client

and 2 Actions

ACTION CONDITIONAL ACL NAME
Use Backend WELL-KNOWN-MATRIX-SERVER SERVER
Use Backend WELL-KNOWN-MATRIX-CLIENT CLIENT

For your backends do not put any configuration except in the advanced passthrough:

WELL-KNOWN-MATRIX-SERVER
http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.server":"[YOURDOMAIN]:[YOURPORT]"}'

WELL-KNOWN-MATRIX-CLIENT
http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://[YOURDOMAIN]"}}'

Note: the json response will vary depending on how you have your matrix server configured. I am using delegation on mine so my port is 443, but yours might be 8448 depending how you are serving.

Note 2: This is a completely separate frontend than my matrix frontend. It only handles the well-known requests. I have a separate frontend for my actual matrix server and you might also have a third frontend for your federation.

Note 3: I also notice on your matrix frontend you do not have use forward for checked. Check that. That takes away the need to have the X-Forwarded Proto in your advanced config. I don't think it's necessary on the well-known frontend, but I tend to just leave it checked on most of mine.

tiran

@sub2010

I serve my well-know from the matrix server. I just published the whole domain on port 443.

Anyway you error suggest that there is no 'matrix-well-known-server' and 'matrix-well-known-client'.
ACL rule and indeed you call your rules 'matrix-well-known-client-path' and 'matrix-well-known-server-path' and under actions you use the names without the -path suffix.

Also the actions are redundant and is where the error is coming from.

Use either actions and Access Control List where you set the matrix.ibi.. stuff.
Or remove the actions and use advanced pass thru option.

I would suggest remove the advanced pass thru on the frontend entirely.

Just add 2 more ACLs matching the actions you already added
Screenshot 2023-09-12 at 08.06.32.png

Don't forget to add -path at the end of you actions Condition acl names.

sub2010

@tiran
@OCT0PUSCRIME

I'm sorry I didn't respond earlier. I had to replace my Synology Reverse Proxy in the background. I implemented your settings as follows, but unfortunately, without success. I must be doing something wrong; could you please check my configuration?
In Matrix Federation, I'm getting an error.

Frontend:
Element-Matrix Frontend

Shared Frontend
Matrix-Synapse

Well-Kown Config

Backend

Well-Kown Config

tiran

Hi,
Looks fine to me. Except the frondend IPs

I don't know you setup with pfsense but mine is used as main router connected to a WAN interface.
Thats where the frontend has to listen.
Yours displaying a private 192.168... address.

You should be able to navigate to your site. matrix.[yourdomain].de/.well-known/matrix/server
And get the json response.

Thats only happening if you make your HAProxy listening on you WAN IP.
Also your DNS should point to you WAN IP.

These are my frontends

Screenshot 2023-09-19 at 11.01.16.png

One for Federation.
One for the web stuff. Which includes the /.well-known stuff and all clients like element and cinny I let the server handle that for me.

This are my backends
Screenshot 2023-09-19 at 11.25.07.png Screenshot 2023-09-19 at 11.27.57.png

I don't use the advanced configuration.

I also added two allow rules to the WAN interface on the firewall for port 443 and 8448 as I mentioned in an earlier post.

Not sure whats the problem is withs yours. I could be the wan configuration or the backend configuration.

Try debugging it with the hosts file. And add your domain to it with the pfsense local IP/matrix server IP and see if it works locally before testing it over the internet.

/etc/hosts

192.168.1.1 matrix.yourdomain.com

Then your computer will use your local ip for your domain.

BTW I use this to manage my server.
https://github.com/spantaleev/matrix-docker-ansible-deploy

Since matrix has a lot of moving parts its nice to have a consistent installation.

Hope you get it figured out.

sub2010

Damn it, i didnt get it run. I dont know where my error is.

I dont wont to give up. Can you further support me?

Lets start from beginning. My synapse config shows this configuration:

server_name: "domain.de"
public_baseurl: "https://matrix.domain.de"
serve_server_wellknown: true

DNS Settings
Host A Entry
domain.de -> IPv4 Adresse
Host A Entry
matrix.domain.de -> IPv4 Adresse

Front End Matrix
Listen addresse (Shared Frontend)
CARP WAN 1 Port 443 SSL Offloading
CARP WAN 2 Port 443SSL Offloading
Access Control lists
matrix Host Matches matrix.domain.de
Actions
use backend matrix matrix-synapse
Use "forwardfor" option.

Backend
Server list
192.168.8.8 8448

Front End Federation
Listen addresse
CARP WAN 1 Port 443 SSL Offloading
CARP WAN 1 Port 8448
Access Control lists

Actions

Use "forwardfor" option.
Server Certficate

Backend matrix-well-known-client
Advanced Settings
http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.domain.de"}}'

Backend matrix-well-known-server
Advanced Settings
http-after-response set-header Access-Control-Allow-Origin "*"
http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
http-request return status 200 content-type application/json string '{"m.server":"matrix.domain.de:443"}'

Federation Tester failed

OCT0PUSCRIME

@sub2010

I mean, a lot of this is going to depend on your matrix config as well.

It looks mostly fine to me, but firstly:
serve_server_wellknown: true

Should not be set in your homeserver.yaml becuase you are serving the well-known from the reverse proxy. This setting makes it available to serve from your matrix host, which is not needed for you. This isn't your issue though.

I see your back end is forwarding to 8448. That requires TLS certs on your matrix server and your homeserver.yaml configured to use them. If that's the case, then I think your well-known/matrix/server should be set to port 8448 and not 443.

I would recommend configuring your matrix listener to listen on 8008 instead and your backend configured to go to 8008 without TLS. With this you dont need a frontend for 8448 and you can leave your well-known/matrix/server port at 443.

sub2010

@OCT0PUSCRIME

First, thank you very much for your support.

Done

It looks mostly fine to me, but firstly:
serve_server_wellknown: true

Done

I see your back end is forwarding to 8448. That requires TLS certs on your matrix server and your homeserver.yaml configured to use them. If that's the case, then I think your well-known/matrix/server should be set to port 8448 and not 443.

Done

I would recommend configuring your matrix listener to listen on 8008 instead and your backend configured to go to 8008 without TLS. With this you dont need a frontend for 8448 and you can leave your well-known/matrix/server port at 443.

The Federation Matrix Error Code

{
    "WellKnownResult": {
        "m.server": "",
        "result": "Get \"https://domain.de/.well-known/matrix/server\": x509: certificate has expired or is not yet valid: current time 2023-10-05T08:24:35Z is after 2023-06-14T15:01:22Z",
        "CacheExpiresAt": 0
    },
    "DNSResult": {
        "SRVSkipped": false,
        "SRVCName": "",
        "SRVRecords": null,
        "SRVError": {
            "Message": "lookup _matrix._tcp.domain.de on 8.8.8.8:53: no such host"
        },
        "Hosts": {
            "domain.de": {
                "CName": "domain.de.",
                "Addrs": [
                    "2a03:4000:61:5c1f::19:967",
                    "87.122.55.127"
                ],
                "Error": null
            }
        },
        "Addrs": [
            "[2a03:4000:61:5c1f::19:967]:8448",
            "87.122.55.127:8448"
        ]
    },
    "ConnectionReports": {},
    "ConnectionErrors": {
        "87.122.55.127:8448": {
            "Message": "Get \"https://87.122.55.127:8448/_matrix/key/v2/server\": EOF"
        },
        "[2a03:4000:61:5c1f::19:967]:8448": {
            "Message": "Get \"https://[2a03:4000:61:5c1f::19:967]:8448/_matrix/key/v2/server\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
        }
    },
    "Version": {
        "error": "Get \"matrix://domain.de/_matrix/federation/v1/version\": EOF"
    },
    "FederationOK": false
}

OCT0PUSCRIME

@sub2010
The error is telling you exactly what's wrong. You have an expired cert that expired on June 14th. Check the cert on your haproxy front end and the one on your matrix host if you have one.

sub2010

@OCT0PUSCRIME
I didnt get it. AGAIN ;-).

First i have a wildcard Certificate, this ist for matrix.domain.de
But my matrix Server runs on servername domain.de

How i can configure the cert for any the domain.de?

This is my frontend configuration

sub2010

@sub2010

Is this SRV Entry correct?

Same Error as well.

{
    "WellKnownResult": {
        "m.server": "",
        "result": "Get \"https://domain.de/.well-known/matrix/server\": x509: certificate has expired or is not yet valid: current time 2023-10-05T15:37:02Z is after 2023-06-14T15:01:22Z",
        "CacheExpiresAt": 0
    },
    "DNSResult": {
        "SRVSkipped": false,
        "SRVCName": "_matrix._tcp.domain.de.",
        "SRVRecords": [
            {
                "Target": "matrix.domain.de.",
                "Port": 443,
                "Priority": 1,
                "Weight": 1
            }
        ],
        "SRVError": null,
        "Hosts": {
            "matrix.domain.de.": {
                "CName": "matrix.domain.de.",
                "Addrs": [
                    "87.122.55.127"
                ],
                "Error": null
            }
        },
        "Addrs": [
            "87.122.55.127:443"
        ]
    },
    "ConnectionReports": {},
    "ConnectionErrors": {
        "87.122.55.127:443": {
            "Message": "Non-200 response 503 from remote server"
        }
    },
    "Version": {
        "error": "Get \"matrix://domain.de/_matrix/federation/v1/version\": x509: certificate is valid for *.domain.de, not domain.de"
    },
    "FederationOK": false
}

How can i change the certificate?

OCT0PUSCRIME

@sub2010
I use the same config. domain.tld and matrix.domain.tld. I'm not sure about your srv record, I dont use one.

For my certificate I use 1 certificate. In acme you can specify multiple domains for one certificate. Mine includes. *.domain.tld and domain.tld

Get a cert like that, put it on your haproxy frontend and also put it on your matrix host and point your homeserver.yaml to it and restart matrix. The error is still saying your cert is expired, so I am assuming the cert you have on your matrix host that your homeserver.yaml is pointing to is expired.