@mcury Thanks a lot, i am working with Gateway Groups and VIP addresses to check that everything is working as intented Warm Regards

correction - it worked for a while... (until it was cached?)

@zewl23 If your Mail server uses protocols other than HTTP/HTTPS (such as IMAP, SMTP, POP3, etc.) that also require SSL/TLS certificates, combining everything behind a single public IP while managing traffic through pfSense + HAProxy can be a bit more involved. These protocols also need to be handled securely using SSL/TLS certificates. Here's how you can approach this: List the Protocols Mail Uses: HTTP/HTTPS (Webmail, Admin Interface): Typically served over port 80 (HTTP) and 443 (HTTPS). SMTP (Mail Transfer): Typically served over port 25 (unencrypted), port 465 (SMTPS), and port 587 (Submission). IMAP (Mail Retrieval): Typically served over port 143 (unencrypted) and 993 (IMAPS). POP3 (Mail Retrieval): Typically served over port 110 (unencrypted) and 995 (POP3S). All of these protocols need to work with SSL/TLS, and they may use the same Let's Encrypt certificates or other SSL certificates. Challenges of SSL Termination for Non-HTTP Protocols: While HAProxy can handle TCP and HTTP(S) traffic, it doesn’t natively support protocols like IMAP, SMTP, POP3 with their specific TLS mechanisms. However, you can set up SSL passthrough or use a combination of HAProxy and stunnel to handle those protocols securely behind pfSense. Approach: There are two main ways to handle SSL termination or passthrough for multiple services using different protocols: Use SSL Termination for HTTP/HTTPS with HAProxy and SSL Passthrough for IMAP/SMTP/POP3. Combine HAProxy with stunnel to manage TLS termination for all protocols. Option 1: HAProxy (SSL Termination for HTTP/HTTPS) + SSL Passthrough for Other Protocols For this approach: HTTP/HTTPS traffic will be handled by HAProxy, where SSL termination will happen. IMAP/SMTP/POP3 will be configured for SSL passthrough, allowing the traffic to flow encrypted directly to the mail server, which will handle the decryption. Step-by-Step Setup Install and Configure HAProxy for HTTP/HTTPS Traffic: For HTTP/HTTPS, follow the same steps as mentioned earlier to perform SSL termination for your webmail: Terminate SSL for HTTPS traffic at pfSense using HAProxy. Forward decrypted traffic to the Mail server over HTTP. 2. Configure SSL Passthrough for Other Protocols: To handle protocols like IMAP, SMTP, and POP3 (SSL Passthrough): SMTP/IMAP/POP3 SSL Ports should be configured as TCP mode in HAProxy, and the traffic will be forwarded as-is to the backend server. Here’s how to do it: Configure a TCP Frontend for IMAP/POP3/SMTP: Go to Services > HAProxy > Frontend and click +Add. Name: Set it to something descriptive like smtp_frontend or imap_frontend. Bind Address: Use your WAN IP and the respective ports for IMAP, SMTP, and POP3: SMTP: WAN_IP:465 (SMTPS) IMAP: WAN_IP:993 (IMAPS) POP3: WAN_IP:995 (POP3S) SSL Offloading: Disable SSL offloading (since we're using passthrough). Mode: Set to TCP. Backend Server: Configure backends to forward traffic to your Mail server. Configure Backend Servers for Each Protocol: Go to Services > HAProxy > Backend and click +Add. Name: Set it to smtp_backend, imap_backend, or pop3_backend based on the protocol. Mode: Set this to TCP (SSL passthrough). Server List: Add your Mail server with the respective port for each service (e.g., 192.168.1.10:465 for SMTP, 192.168.1.10:993 for IMAPS). Repeat this for all the services (IMAP, SMTP, POP3). This approach forwards the SSL/TLS encrypted traffic for non-HTTP protocols directly to your Mail server, which then handles the encryption and decryption. Example Configuration: Frontend for SMTP (port 465): Bind Address: WAN_IP:465 Mode: TCP Backend: smtp_backend Backend for SMTP: Server: 192.168.1.10:465 (Mail server on internal network) Frontend for IMAPS (port 993): Bind Address: WAN_IP:993 Mode: TCP Backend: imap_backend Backend for IMAPS: Server: 192.168.1.10:993 Option 2: Combine HAProxy with stunnel (SSL Termination for All Protocols) If you want to terminate SSL/TLS for all protocols (HTTP, IMAP, SMTP, POP3) at pfSense and filter the decrypted traffic, you can use stunnel along with HAProxy. stunnel is a proxy designed to add SSL encryption to existing clients and servers without changes to the application code. You can use stunnel to decrypt IMAP/POP3/SMTP traffic and forward it to HAProxy or directly to your backend. Steps: Install stunnel on pfSense: Install the stunnel package from System > Package Manager > Available Packages. Configure stunnel for Mail Protocols: Set up stunnel to listen on the respective ports (465, 993, 995) and decrypt traffic using your Let's Encrypt certificate. stunnel will forward the decrypted traffic to your iRedMail server over plain IMAP/POP3/SMTP. HAProxy for HTTP/HTTPS: Use HAProxy to handle SSL termination for HTTP/HTTPS as explained earlier. Firewall Rules: Configure firewall rules to allow traffic on the mail ports (25, 465, 993, 995, etc.). Example Configuration Summary (Using SSL Passthrough for Mail Protocols) Frontend (HTTPS, SMTP, IMAP, POP3): HTTPS (port 443): Bind Address: WAN_IP:443 SSL Offloading: Enabled (HAProxy manages SSL for HTTPS). Backend: Webmail backend on HTTP. IMAPS (port 993): Bind Address: WAN_IP:993 SSL Offloading: Disabled (SSL passthrough). Backend: iRedMail server on port 993. SMTP (port 465): Bind Address: WAN_IP:465 SSL Offloading: Disabled (SSL passthrough). Backend: iRedMail server on port 465. Backend Servers (Decrypted HTTP, Encrypted Mail): Backend1 (HTTP for Webmail): 192.168.1.10:80 Backend2 (IMAPS for Mail): 192.168.1.10:993 Backend3 (SMTPS for Mail): 192.168.1.10:465 Conclusion: To handle a setup where Mail uses multiple protocols over SSL/TLS: For HTTP/HTTPS (web interfaces), you can use SSL termination at HAProxy to decrypt and filter the traffic at pfSense. For SMTP, IMAP, and POP3, it’s better to use SSL passthrough to forward encrypted traffic directly to the Mail server, which will handle SSL/TLS for these protocols. Alternatively, you can use stunnel to decrypt these protocols if you want to inspect/filter the traffic. This combined approach gives you flexibility and control over traffic while maintaining security for all protocols. Let me know if you need further assistance with any specific configuration!

Set the the OpenVPN server to listen on localhost. Then setup port forwards on both WANs to localhost for the port the OpenVPN traffic is arriving on. Clients will be able to connect to either WAN and replies will go back correctly.

Additional information On my cluster where dpinger is failing, I added a floating rule attached to the WAN interface and set it to match. I then set the rule to apply to basically any traffic outbound from the WAN interface and I set it to log connections in the system log. Then I restarted dpinger. Sure enough, dpinger on the secondary is trying to use the CARP address as its source (which the secondary doesn't have right now). So that explains that. I performed the same test on my cluster where dpinger is working correctly. On that cluster, the rule recorded that the secondary used the interface IP address as its source, NOT the CARP VIP like my misbehaving cluster did. The oubtound NAT rules on both clusters are indeed THE SAME. As an experiment, I tried changing the NAT rules on the secondary that used localhost / loopback as its source so they natted to the WAN interface IP and not the CARP VIP. No change, secondary still trying to use the CARP VIP that it doesn't have. When I added the aforementioned NAT rule to the secondary that referenced This Firewall (self) as the source and the destination as any non-private IP address, then dpinger on the secondary starts using the WAN address instead of the WAN CARP VIP. However, this behavior doesn't occur on the primary when we manually failover to make the secondary the master. The only thing I can tell that's different between the configurations, is that in the cluster that is working, we are running Netgate appliances with pfsense+ 23.09.1. The cluster that is working incorrectly is running pfsense CE 2.7.2 on third party hardware. Possibly a bug in CE that's not present in pfsense+? Regardless of the cause, I can only conclude that dpinger is behaving differently in one of my clusters than it is another.

We just ended up adding additional IP addresses. It was easier to do that than to experiment on this after hours.

Yes you can setup config sync only. However somethings are automatically converted for use on the secondary during the sync. Mostly the CARP VIPs which you wouldn't be using but also notably the DHCP server config. If you use Kea that wouldn't be an issue in 24.03 and in 24.08 you can disable it.

@balmmva A WAN interface behaves basically like any other. So for HA you need three IPs in a subnet. However, CARP isbnot compatible with PPPoE. You will have to put a router in front of the WANs.

@edgarquadros If it's only on one VLAN, be sure everything is tagged through your switches like the rest.

I have actually figured out how to setup the automated unit tests to run, so my code changes have passed all the required tests and I'm just waiting for the fix to be accepted. Hopefully the patch will be rolled out soon.

@Gertjan Looks very similar. Thank you!

I have some new information about this problem. Unfortunately, I still haven't solved it. Today I discovered that by changing the NAT settings I am able to restore full internet speed. If I select the "WAN interface" option in "Translation Address", the full speed appears. However, if I select "CARP WAN VIP", the speed drops to a few Mbit/s. I recorded packets with wireshark from the WAN port. Both IP addresses (IP Address from WAN interface and CARP WAN VIP) come out with the same MAC Address. Maybe this is the problem? I tried to connect a switch + two routers to the cable from the ISP in which I set the same MAC Address but different IP addresses. After starting the routers, they interfered with each other and worked interchangeably, but neither cut the other's speed. Maybe if they came out of the same physical device, there would be a different symptom. Maybe it is somehow related to the ISP. But I can't confirm or deny it. Does anyone have any ideas what it could be?

@mikethiessen Thank you, I ended up doing just that and once the 8th spot was made, I deleted the other 2.

@Tadmin A CARP setup requires at least two IPs within a subnet for the two nodes and a shared CARP VIP. All three have to be static to get CARP work out of the box. A shared DHCP VIP is not supported. However, people posted workarounds here to get it up with DHCP as far as I remember. But this needs some scripting. Try the forum search.