Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Matrix Synapse behind HAProxy on pfSense

    Scheduled Pinned Locked Moved Cache/Proxy
    haproxymatrixsynapsefirewallrules
    19 Posts 5 Posters 5.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      sub2010
      last edited by

      Hi there,

      how you configure the matrix federation settings in the HAProxy GUI?
      I didnt know how i configure the specification from synapse:

      Can you help me?
      frontend https
      bind *:443,[::]:443 ssl crt /etc/ssl/haproxy/ strict-sni alpn h2,http/1.1
      http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
      http-request set-header X-Forwarded-For %[src]

      Matrix client traffic

      acl matrix-host hdr(host) -i matrix.example.com matrix.example.com:443
      acl matrix-path path_beg /_matrix
      acl matrix-path path_beg /_synapse/client

      use_backend matrix if matrix-host matrix-path

      frontend matrix-federation
      bind *:8448,[::]:8448 ssl crt /etc/ssl/haproxy/synapse.pem alpn h2,http/1.1
      http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
      http-request set-header X-Forwarded-For %[src]

      default_backend matrix

      backend matrix
      server matrix 127.0.0.1:8008

      9da49006-d888-4566-b8bb-dd42bf0585b7-image.png

      1 Reply Last reply Reply Quote 0
      • T
        tiran
        last edited by

        I created 2 seperate frontends and 2 backends for the matrix federation and for the matrix client.

        Then set the default backend for each frontend.

        Matrix Federation(Frontend) points to -> matrix federation (backend) server port 8448
        Matrix Client(Frontend) points to -> matrix client backend server port 80

        Under advanced settings in the frontend I put this stuff.
        Screenshot 2023-09-11 at 19.42.20.png

        Don't forget to add a firewall rule in the WAN section to allow the port 8448

        Screenshot 2023-09-11 at 19.53.28.png

        Hope that helps.

        S 1 Reply Last reply Reply Quote 0
        • O
          OCT0PUSCRIME @OCT0PUSCRIME
          last edited by

          @OCT0PUSCRIME said in Matrix Synapse behind HAProxy on pfSense:

          @tiran Did you ever figure this out? I have got the URL: https://MYDOMAIN/.well-known/matrix/server resolving to the correct json file, but the federation tester is still giving me a timeout for some reason.

          Following up: my issue was really dumb. If you feel you have everything set up correctly, but you are getting timeouts on the federation tester, make sure your GeoIP blocking isn't too restrictive with pfBlockerNG, if you have that set up.

          1 Reply Last reply Reply Quote 0
          • S
            sub2010 @tiran
            last edited by

            @tiran
            Damn i didnt get it. I need the matrix well know configuration.

            frontend https
              acl matrix-well-known-client-path path /.well-known/matrix/client
              acl matrix-well-known-server-path path /.well-known/matrix/server
              use_backend matrix-well-known-client if matrix-well-known-client-path
              use_backend matrix-well-known-server if matrix-well-known-server-path
             
            backend matrix-well-known-client
              http-after-response set-header Access-Control-Allow-Origin "*"
              http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
              http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
              http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.example.com"},"m.identity_server":{"base_url":"https://identity.example.com"}}'
            
            backend matrix-well-known-server
              http-after-response set-header Access-Control-Allow-Origin "*"
              http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
              http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
              http-request return status 200 content-type application/json string '{"m.server":"matrix.example.com:443"}'
            
            
            1. I created a frontend
              e0387618-bce9-4a82-86be-53d45e9dc8ab-image.png
              845d6894-d564-418a-b9e4-9b27f1c2a4c4-image.png
              0abdceab-d046-4299-aad4-dba31de71fb4-image.png
              acl matrix-well-known-client-path path /.well-known/matrix/client
              acl matrix-well-known-server-path path /.well-known/matrix/server
              use_backend matrix-well-known-client if matrix-well-known-client-path
              use_backend matrix-well-known-server if matrix-well-known-server-path
            
            1. I created two backends
              d8d34d9a-398c-406a-b668-0e704bf7f584-image.png
              db7bd034-82f3-4797-9ea5-5c08f367cb9d-image.png
            http-after-response set-header Access-Control-Allow-Origin "*"
            http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
            http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
            http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.ibinary.de"},"m.identity_server":{"base_url":"https://matrix.org"}}'
            
            

            The Error Code is
            460aaed7-1044-4a82-aa1a-c49e9d3281ed-image.png

            Errors found while starting haproxy
            [NOTICE] (59904) : haproxy version is 2.7.6-4dadaaa
            [NOTICE] (59904) : path to executable is /usr/local/sbin/haproxy
            [WARNING] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:71] : a 'http-request' rule placed after a 'use_backend' rule will still be processed before.
            [ALERT] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:72] : error detected while parsing switching rule : no such ACL : 'matrix-well-known-client'.
            [ALERT] (59904) : config : parsing [/var/etc/haproxy_test/haproxy.cfg:73] : error detected while parsing switching rule : no such ACL : 'matrix-well-known-server'.
            [ALERT] (59904) : config : Error(s) found in configuration file : /var/etc/haproxy_test/haproxy.cfg
            [ALERT] (59904) : config : Fatal errors found in configuration.
            
            O T 2 Replies Last reply Reply Quote 0
            • O
              OCT0PUSCRIME @sub2010
              last edited by OCT0PUSCRIME

              @sub2010

              Take away the advanced passthrough stuff. Create one frontend called WELL-KNOWN and 2 backends. WELL-KNOWN-MATRIX-CLIENT and WELL-KNOWN-MATRIX-SERVER.

              In the front end create 2 Access Control Lists.

              NAME EXPRESSION VALUE
              SERVER path starts with /.well-known/matrix/server
              CLIENT path starts with /.well-known/matrix/client

              and 2 Actions

              ACTION CONDITIONAL ACL NAME
              Use Backend WELL-KNOWN-MATRIX-SERVER SERVER
              Use Backend WELL-KNOWN-MATRIX-CLIENT CLIENT

              For your backends do not put any configuration except in the advanced passthrough:

              WELL-KNOWN-MATRIX-SERVER
              http-after-response set-header Access-Control-Allow-Origin "*"
              http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
              http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
              http-request return status 200 content-type application/json string '{"m.server":"[YOURDOMAIN]:[YOURPORT]"}'

              WELL-KNOWN-MATRIX-CLIENT
              http-after-response set-header Access-Control-Allow-Origin "*"
              http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
              http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
              http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://[YOURDOMAIN]"}}'

              Note: the json response will vary depending on how you have your matrix server configured. I am using delegation on mine so my port is 443, but yours might be 8448 depending how you are serving.

              Note 2: This is a completely separate frontend than my matrix frontend. It only handles the well-known requests. I have a separate frontend for my actual matrix server and you might also have a third frontend for your federation.

              Note 3: I also notice on your matrix frontend you do not have use forward for checked. Check that. That takes away the need to have the X-Forwarded Proto in your advanced config. I don't think it's necessary on the well-known frontend, but I tend to just leave it checked on most of mine.

              1 Reply Last reply Reply Quote 1
              • T
                tiran @sub2010
                last edited by

                @sub2010

                I serve my well-know from the matrix server. I just published the whole domain on port 443.

                Anyway you error suggest that there is no 'matrix-well-known-server' and 'matrix-well-known-client'.
                ACL rule and indeed you call your rules 'matrix-well-known-client-path' and 'matrix-well-known-server-path' and under actions you use the names without the -path suffix.

                Also the actions are redundant and is where the error is coming from.

                Use either actions and Access Control List where you set the matrix.ibi.. stuff.
                Or remove the actions and use advanced pass thru option.

                I would suggest remove the advanced pass thru on the frontend entirely.

                Just add 2 more ACLs matching the actions you already added
                Screenshot 2023-09-12 at 08.06.32.png

                Don't forget to add -path at the end of you actions Condition acl names.

                S 1 Reply Last reply Reply Quote 1
                • S
                  sub2010 @tiran
                  last edited by

                  @tiran
                  @OCT0PUSCRIME

                  I'm sorry I didn't respond earlier. I had to replace my Synology Reverse Proxy in the background. I implemented your settings as follows, but unfortunately, without success. I must be doing something wrong; could you please check my configuration?
                  In Matrix Federation, I'm getting an error.

                  Frontend:
                  Element-Matrix Frontend
                  fa60bf47-74ae-44a8-a3e4-d032070f5756-image.png

                  Shared Frontend
                  Matrix-Synapse

                  41ab6159-21b6-47a2-8f6c-b5e63a277677-image.png
                  f8aa692b-68e0-414b-9417-0d2a6eec352c-image.png
                  fd2334e1-555b-41f0-a98f-4b182f6242a6-image.png
                  f63d361f-cb90-4413-b838-5afbbc9f381b-image.png

                  Well-Kown Config

                  99fa0ce1-c953-4343-8073-82b5c6e5e99d-image.png
                  f7797ba5-ad81-4dec-bbc4-64014fd53c50-image.png

                  Backend

                  43e713e7-1912-4a6c-b964-28270c80b433-image.png

                  Well-Kown Config

                  3a0b4acf-a005-4b91-b87c-a97169149f4a-image.png
                  ce2bae79-a680-485e-88db-bc8f78c57876-image.png
                  610c6be7-3342-481d-8e62-7cfb8dc3e048-image.png
                  059492d2-c9e8-45cf-be7e-75f278f26a7a-image.png

                  1 Reply Last reply Reply Quote 0
                  • T
                    tiran
                    last edited by

                    Hi,
                    Looks fine to me. Except the frondend IPs

                    I don't know you setup with pfsense but mine is used as main router connected to a WAN interface.
                    Thats where the frontend has to listen.
                    Yours displaying a private 192.168... address.

                    You should be able to navigate to your site. matrix.[yourdomain].de/.well-known/matrix/server
                    And get the json response.

                    Thats only happening if you make your HAProxy listening on you WAN IP.
                    Also your DNS should point to you WAN IP.

                    These are my frontends

                    Screenshot 2023-09-19 at 11.01.16.png

                    One for Federation.
                    One for the web stuff. Which includes the /.well-known stuff and all clients like element and cinny I let the server handle that for me.

                    This are my backends
                    Screenshot 2023-09-19 at 11.25.07.png Screenshot 2023-09-19 at 11.27.57.png

                    I don't use the advanced configuration.

                    I also added two allow rules to the WAN interface on the firewall for port 443 and 8448 as I mentioned in an earlier post.

                    Not sure whats the problem is withs yours. I could be the wan configuration or the backend configuration.

                    Try debugging it with the hosts file. And add your domain to it with the pfsense local IP/matrix server IP and see if it works locally before testing it over the internet.

                    /etc/hosts

                    192.168.1.1 matrix.yourdomain.com
                    

                    Then your computer will use your local ip for your domain.

                    BTW I use this to manage my server.
                    https://github.com/spantaleev/matrix-docker-ansible-deploy

                    Since matrix has a lot of moving parts its nice to have a consistent installation.

                    Hope you get it figured out.

                    1 Reply Last reply Reply Quote 0
                    • S
                      sub2010
                      last edited by sub2010

                      Damn it, i didnt get it run. I dont know where my error is.

                      I dont wont to give up. Can you further support me?

                      Lets start from beginning. My synapse config shows this configuration:

                      server_name: "domain.de"
                      public_baseurl: "https://matrix.domain.de"
                      serve_server_wellknown: true
                      

                      DNS Settings
                      Host A Entry
                      domain.de -> IPv4 Adresse
                      Host A Entry
                      matrix.domain.de -> IPv4 Adresse

                      Front End Matrix
                      Listen addresse (Shared Frontend)
                      CARP WAN 1 Port 443 SSL Offloading
                      CARP WAN 2 Port 443SSL Offloading
                      Access Control lists
                      matrix Host Matches matrix.domain.de
                      Actions
                      use backend matrix matrix-synapse
                      Use "forwardfor" option.

                      Backend
                      Server list
                      192.168.8.8 8448

                      Front End Federation
                      Listen addresse
                      CARP WAN 1 Port 443 SSL Offloading
                      CARP WAN 1 Port 8448
                      Access Control lists
                      e2a1b312-70e4-4b45-80bb-662ebd8265ac-image.png

                      Actions
                      f11beb96-ec2a-4b3a-9db2-e14300b06186-image.png
                      Use "forwardfor" option.
                      Server Certficate

                      Backend matrix-well-known-client
                      Advanced Settings
                      http-after-response set-header Access-Control-Allow-Origin "*"
                      http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
                      http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
                      http-request return status 200 content-type application/json string '{"m.homeserver":{"base_url":"https://matrix.domain.de"}}'

                      Backend matrix-well-known-server
                      Advanced Settings
                      http-after-response set-header Access-Control-Allow-Origin "*"
                      http-after-response set-header Access-Control-Allow-Methods "GET, POST, PUT, DELETE, OPTIONS"
                      http-after-response set-header Access-Control-Allow-Headers "Origin, X-Requested-With, Content-Type, Accept, Authorization"
                      http-request return status 200 content-type application/json string '{"m.server":"matrix.domain.de:443"}'

                      Federation Tester failed
                      79bd8dba-6df0-40a2-b6cd-5eb7bd09334f-image.png

                      O 1 Reply Last reply Reply Quote 0
                      • O
                        OCT0PUSCRIME @sub2010
                        last edited by OCT0PUSCRIME

                        @sub2010

                        I mean, a lot of this is going to depend on your matrix config as well.

                        It looks mostly fine to me, but firstly:
                        serve_server_wellknown: true

                        Should not be set in your homeserver.yaml becuase you are serving the well-known from the reverse proxy. This setting makes it available to serve from your matrix host, which is not needed for you. This isn't your issue though.

                        I see your back end is forwarding to 8448. That requires TLS certs on your matrix server and your homeserver.yaml configured to use them. If that's the case, then I think your well-known/matrix/server should be set to port 8448 and not 443.

                        I would recommend configuring your matrix listener to listen on 8008 instead and your backend configured to go to 8008 without TLS. With this you dont need a frontend for 8448 and you can leave your well-known/matrix/server port at 443.

                        S 1 Reply Last reply Reply Quote 0
                        • S
                          sub2010 @OCT0PUSCRIME
                          last edited by sub2010

                          @OCT0PUSCRIME

                          First, thank you very much for your support.

                          Done

                          It looks mostly fine to me, but firstly:
                          serve_server_wellknown: true
                          

                          Done

                          I see your back end is forwarding to 8448. That requires TLS certs on your matrix server and your homeserver.yaml configured to use them. If that's the case, then I think your well-known/matrix/server should be set to port 8448 and not 443.
                          

                          Done

                          I would recommend configuring your matrix listener to listen on 8008 instead and your backend configured to go to 8008 without TLS. With this you dont need a frontend for 8448 and you can leave your well-known/matrix/server port at 443.
                          

                          The Federation Matrix Error Code

                          {
                              "WellKnownResult": {
                                  "m.server": "",
                                  "result": "Get \"https://domain.de/.well-known/matrix/server\": x509: certificate has expired or is not yet valid: current time 2023-10-05T08:24:35Z is after 2023-06-14T15:01:22Z",
                                  "CacheExpiresAt": 0
                              },
                              "DNSResult": {
                                  "SRVSkipped": false,
                                  "SRVCName": "",
                                  "SRVRecords": null,
                                  "SRVError": {
                                      "Message": "lookup _matrix._tcp.domain.de on 8.8.8.8:53: no such host"
                                  },
                                  "Hosts": {
                                      "domain.de": {
                                          "CName": "domain.de.",
                                          "Addrs": [
                                              "2a03:4000:61:5c1f::19:967",
                                              "87.122.55.127"
                                          ],
                                          "Error": null
                                      }
                                  },
                                  "Addrs": [
                                      "[2a03:4000:61:5c1f::19:967]:8448",
                                      "87.122.55.127:8448"
                                  ]
                              },
                              "ConnectionReports": {},
                              "ConnectionErrors": {
                                  "87.122.55.127:8448": {
                                      "Message": "Get \"https://87.122.55.127:8448/_matrix/key/v2/server\": EOF"
                                  },
                                  "[2a03:4000:61:5c1f::19:967]:8448": {
                                      "Message": "Get \"https://[2a03:4000:61:5c1f::19:967]:8448/_matrix/key/v2/server\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
                                  }
                              },
                              "Version": {
                                  "error": "Get \"matrix://domain.de/_matrix/federation/v1/version\": EOF"
                              },
                              "FederationOK": false
                          }
                          
                          O 1 Reply Last reply Reply Quote 0
                          • O
                            OCT0PUSCRIME @sub2010
                            last edited by OCT0PUSCRIME

                            @sub2010
                            The error is telling you exactly what's wrong. You have an expired cert that expired on June 14th. Check the cert on your haproxy front end and the one on your matrix host if you have one.

                            S 1 Reply Last reply Reply Quote 0
                            • S
                              sub2010 @OCT0PUSCRIME
                              last edited by

                              @OCT0PUSCRIME
                              I didnt get it. AGAIN ;-).

                              First i have a wildcard Certificate, this ist for matrix.domain.de
                              But my matrix Server runs on servername domain.de

                              How i can configure the cert for any the domain.de?

                              This is my frontend configuration
                              06a99932-fc2e-46b6-bdd9-80df827c9663-image.png

                              S O 2 Replies Last reply Reply Quote 0
                              • S
                                sub2010 @sub2010
                                last edited by sub2010

                                @sub2010

                                Is this SRV Entry correct?
                                d157f624-7679-40ab-ab63-fb6de22a23ae-image.png

                                Same Error as well.

                                {
                                    "WellKnownResult": {
                                        "m.server": "",
                                        "result": "Get \"https://domain.de/.well-known/matrix/server\": x509: certificate has expired or is not yet valid: current time 2023-10-05T15:37:02Z is after 2023-06-14T15:01:22Z",
                                        "CacheExpiresAt": 0
                                    },
                                    "DNSResult": {
                                        "SRVSkipped": false,
                                        "SRVCName": "_matrix._tcp.domain.de.",
                                        "SRVRecords": [
                                            {
                                                "Target": "matrix.domain.de.",
                                                "Port": 443,
                                                "Priority": 1,
                                                "Weight": 1
                                            }
                                        ],
                                        "SRVError": null,
                                        "Hosts": {
                                            "matrix.domain.de.": {
                                                "CName": "matrix.domain.de.",
                                                "Addrs": [
                                                    "87.122.55.127"
                                                ],
                                                "Error": null
                                            }
                                        },
                                        "Addrs": [
                                            "87.122.55.127:443"
                                        ]
                                    },
                                    "ConnectionReports": {},
                                    "ConnectionErrors": {
                                        "87.122.55.127:443": {
                                            "Message": "Non-200 response 503 from remote server"
                                        }
                                    },
                                    "Version": {
                                        "error": "Get \"matrix://domain.de/_matrix/federation/v1/version\": x509: certificate is valid for *.domain.de, not domain.de"
                                    },
                                    "FederationOK": false
                                }
                                

                                How can i change the certificate?

                                1 Reply Last reply Reply Quote 0
                                • O
                                  OCT0PUSCRIME @sub2010
                                  last edited by

                                  @sub2010
                                  I use the same config. domain.tld and matrix.domain.tld. I'm not sure about your srv record, I dont use one.

                                  For my certificate I use 1 certificate. In acme you can specify multiple domains for one certificate. Mine includes. *.domain.tld and domain.tld

                                  Get a cert like that, put it on your haproxy frontend and also put it on your matrix host and point your homeserver.yaml to it and restart matrix. The error is still saying your cert is expired, so I am assuming the cert you have on your matrix host that your homeserver.yaml is pointing to is expired.

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.