Multi WAN and incoming connections
-
Ok, I thought I had it all sorted. I have pfsense, 2x internet connections, multi-WAN enabled and seems to be functioning well most of the time.
I have several real and virtual servers setup, no 1:1 setup but all the rules appear to be working as expected.
There are minimal ports open to the internal network, but they are 2 web servers sharing an internal IP on a VM being served by apache virtual hosts.
I know it wasn't required but I did split the web servers via DNS to come in on each of the external IPs. ie, each domain name came in on one IP only.
Both domains work without any hitches what-so-ever under normal operations.
My connections are 100MBit and 20MBit and I can generally download 120MBit anytime.
I have the loadbalancer setup and it mostly seems to work.
Tonight we had an actual outage on one of the lines and unfortunately it was the incoming for the website I am currently working on. Fine for me locally, but the remote users couldn't access it.
What was interesting is that I couldn't get to either of them. Initially I didn't think much of that and I logged into my DNS and added the IP address of the server incoming connection that was up and added the website I was working on.
To my surprise, I could not get to either of the websites.
I struggled with it for some time before I gave up.
Low and behold, when the service comes back up, I am able to access both webservers externally again.
Internally there was never a problem, but externally, while one of the lines were down, I couldn't get to either and that was under normal conditions (each server with it's own public IP, even if one of them was down) and even with both servers having both IPs entered into the DNS.
From the top, my routing are the first three images. My gateway groups are the 4th image.
The remaining images are the firewall NAT and rules for each interface.
The most odd about it though, was that the firewall logs were indicating that a connection was being made and there was traffic being passed through. But I wasn't seeing it on the phone I was using that was disconnected from the wifi.
Apache logs from the webserver also confirm there were no active connections until both lines were up.
I know I probably haven't included enough yet, but if anyone could steer me in the right direction I'd appreciate it.
