multiple https with haproxy

PiBa · Oct 13, 2018, 1:35 PM

Ping/traceroute, that they fail to reach the intended target is okay(well its because there is some issue of course we are trying to diagnose..), but what messages are displayed does the ping complain about no route? Does the trace get 1 hop out.?

The pcap files can be read / analyzed by a program like Wireshark if you copy them to your local computer. They are in a 'binary' format that cat and 'edit file' wouldnt know how to display..

Not a single difference in the rules.debug before/after trouble?

Try rebooting the Arris modem next time instead of pfSense ?

Dns settings for your public domain and where they are hosted or pointing to really shouldn't affect pfSense in any way.. Unless someone is trying to ddos your domain..

coreyjohnson75 · Oct 13, 2018, 4:59 PM

@piba & @Mats

Thank @Mats I've been able to access my sites externally with SSL so other than the wildcard part I think I have that setup. If you think that makes a difference then I will definitely look into it. I'll post my configuration below if you don't mind glancing over it.

Well, I thought about going back to duckdns but at least at the time I couldn't get 2 sites to work.

Here is where I am right now, about 4 hours into a stable system where I went to GoDaddy (vs Cloudflare), bought a 2nd domain (.xyz versus .us) and have the DNS through them. I have 2 https and 1 http site all working and have not dropped outbound internet. I didn't change anything in my configuration other than the .xyz in my haproxy frontends. I'll need to get new SSL certs if this holds up as I am currently pushing through the previous .us one and of course getting an error but can reach my backends nonetheless (?).

Now that doesn't mean anything until I at least go 12-24 hours without dropping outbound internet, fingers crossed.

Now, IF this all holds up, I'll be where I wanted to be a week ago and trying to figure out how to HAproxy the rest of my mailserver ports through lol.

global
	maxconn			10
	stats socket /tmp/haproxy.socket level admin 
	uid			80
	gid			80
	nbproc			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	tune.ssl.default-dh-param	2048
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats refresh 10
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend Frontend_www
	bind			XX.XX.XXX.XX:80 name XX.XX.XXX.XX:80   
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			www_acl	var(txn.txnhost) -m beg -i www.wwolf.xyz
	http-request set-var(txn.txnhost) hdr(host)
	use_backend Test_Backend_ipvANY  if  www_acl 

frontend CF_443-merged
	bind			XX.XX.XXX.XX:443 name XX.XX.XXX.XX:443   ssl crt-list /var/etc/haproxy/CF_443.crt_list  
	mode			http
	log			global
	option			http-keep-alive
	timeout client		30000
	acl			zim_acl	var(txn.txnhost) -m beg -i webmail.wwolf.xyz
	acl			ha_acl2	var(txn.txnhost) -m beg -i ha.wwolf.xyz
	acl			Zimbra_acl	var(txn.txnhost) -m str -i webmail.wwolf.xyz
	http-request set-var(txn.txnhost) hdr(host)
	use_backend Zimbra_Backend_ipvANY  if  zim_acl 
	use_backend HA_Backend_ipvANY  if  ha_acl2 
	use_backend Zimbra_Backend_ipvANY  if  Zimbra_acl 

backend Test_Backend_ipvANY
	mode			http
	id			103
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk GET / 
	server			testpage 192.168.30.11:80 id 104 check inter 1000  

backend Zimbra_Backend_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk GET / 
	server			Zimbra_Backend_Server 192.168.30.5:443 id 101 ssl check inter 1000  verify none 

backend HA_Backend_ipvANY
	mode			http
	id			100
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	option			httpchk GET / 
	server			HA_Backend_Server 192.168.30.6:18122 id 101 check inter 1000

PiBa · Oct 13, 2018, 5:29 PM

If this change to GoDaddy 'fixes' the issue, then its likely still a problem waiting to happen. Does Cloudflare do 'healthchecks' or something.?. So they might be requesting the website main page every few seconds.? and filling up some 'state table' somewhere (arris modem?) which eventually overflows and causes trouble.?.

@coreyjohnson75 said in multiple https with haproxy:

how to HAproxy the rest of my mailserver ports

That will be limited to 'mode tcp' and without any 'smart' backend selection.. It can balance between multiple servers for the same domain if required.. It for example wont switch port 25 between 2 servers that only accept mail for specific different domains..

coreyjohnson75 · Oct 14, 2018, 6:47 PM

I wanted to let you guys know that the problem ended up being rules related on the primary VLAN that kept going down. I'm still confused on why it would all work for a while and then go out AND why it would work longer (never actually crashed) with GoDaddy but I reworked all my rules with the help of someone and moved everything back to Cloudflare and have been up for over 20 hours without ANY issue. I had the right rules but they were below some bad rules. I have since moved many Iot devices onto their own separate vlan to make everything prettier and more simple.

Now... if either of you could assist me with 1) best practices for running an email server behind pfSense and 2) not get my emails flagged as spam, that would be awesome.

Mats · Oct 15, 2018, 6:40 PM

Well, I can give you my best practice. IE it works for me (tm).

I have a windows server running Hmailserver behind a Pfsense box.
I don't use HAproxy for my Mailserver so I simply got NAT rules for Port 25 and 993 to my mailserver. I only allow encrypted IMAP to the mail server so therefore i only need one port besides port 25.
I also allow port 25 outbound from my mailserver to my ISP:s mailrelay for sending mail. My isp dosen't allow direct traffic from port so I have to send it through them.

There can be many reasons for your mail being marked as spam.
being on DHCP is one. Having a PTR record in DNS that doesn't match the hostname of your mailserver is another.

Therefore it's hard to answer that part but one way that might helt is to use a Mail relay service with a decent (or better) reputation. One example is amazons simple Email services (haven't tested it myself though)

coreyjohnson75 · Oct 16, 2018, 1:00 PM

@mats I'll definitely take a look at this. For now I am taking a break as my nerves need a rest, lol. Thank you both for all the assistance.

coreyjohnson75 · Oct 30, 2018, 6:01 PM

@mats So part of the problem was my rules but the final stick ended up being a bug in the upgrade to the most current release where my default gateway was no longer 'marked' default.... what a headache.

I did finally manage to get Zimbra OSE running in a docker alongside a NextCloud docker to work for files as well. Only then did I fully realize that since I chose the Open Source edition, I cannot fully integrate it into my android for calendars and such which is a major letdown. I was so impressed with the webmail features that I over looked the limitations of the free version.

I may look into iRedMail. I didn't like the webmail portions but in the end I doubt I will really be using it compared to the ability to have fully integrated calendar & contacts.