Odd IP issue - firewall or HA related?

Rod-It

First of all, this is my second topic, apologies if this is in the wrong category.

A little info about the setup - in case it helps.

HPE DL360 G9 Server, 4 Broadcom Corporation NetXtreme BCM5719 NICs (2 in use)
ESXi 6.7u3a
Modem to NIC3 (number doesn't matter), External IP on Virtual PF box, second NIC to physical switch (TP-Link 24 port gig POE).

For the discussion, I am ruling out the switch and physical kit, I can replicate the issue within VMs only - it never leaves the network.

I dont know how well I can explain this, so bear with me.

PfSense is hosting 6 VLANS, but I am only interested in 2 of them for now.

I also run HAProxy as a reverse proxy - and I am not sure if this is what is causing my issues - let me explain and it might make more sense why I am blaming this.

VLAN1 - typical LAN traffic
VLAN2 - servers

This is the 3rd time now this has happened.

Exchange - let's say .2
NextCloud - call it .3

Exchange has been running since I setup Pf behind the reverse proxy, internal and external. Today I tried to add Nextcloud to the HA proxy, nada - I still got Exchange OWA - I checked logs and settings, after about 40 minutes I gave up and undid the backend and frontend - putting it back to how it was with exchange backend, exchange frontend and a shared frontend.

OWA still worked for external people, but internal, VLAN1 (and all other VLANs) can no longer access it.
Likewise, Nextcloud is also now broken on the internal LAN

Both boxes ping just fine and show HTTP/S available through a port scan, but none work, except if I am on the same VLAN as them.

I am guessing this is a firewall or proxy issue, running a firewall trace with success logged, I see the request and a pass, but the request never returns a webpage, unless I am doing so from the same subnet, this also does not affect external users, only other VLANs.

Here's the odd piece, if I change exchange from .2 to .4, it all starts working again - I am guessing that HA or the firewall is creating a block or lock on the IP - pings work, but HTTP does not, moving the IP resolves this, but I can't be moving IPs on servers each time this happens

I am not an expert on firewalls, networking or any form of Linux/BSD by any means, but I've got this far

All builds and packages are the latest, all drivers are the latest and the issue seems to only spawn when I try to add another site to HA

Any pointers on where to look, as it stands now, any of the 3 IPs that I've had to change, if I move a device back to them that needs HTTP/S, it still doesn't work, so something is preventing me using it.

Not sure if this is related - and I will log this separately if you think they are not, but I had similar issues with my APs, as soon as I put them on a separate VLAN for guests, I cannot connect to the HTTP/S portal to authenticate - it's as if the server is not listening.

EDIT:

This does not affect all HTTP/S services on that VLAN - only those that have been near HAProxy, so for example my EFA on the Server VLAN remains working.

I have all servers working again currently by giving them new IPs on the server VLAN.

Apologies too if this is confusing, I struggled how to word it so it should make sense.

Rod-It

I want to rule out HAProxy at this point, since externally it still works and internal traffic should not touch it.

I want to rule out the firewall side of things, because I see the traffic allowed, but I dont believe it actually does pass (at least not correctly), port scans will reveal the port open, the firewall shows a pass in traffic but the websites never load.

Someone advised me to check states, clear arp cache etc, all of which I have tried, but to no benefit

The issue seems to only affect services I try to proxy.

Does anyone have any idea where to start to try and troubleshoot?

Rod-It

Well I fixed most of my own issues.

Interesting that no one even bothered to ask or post anything

bmeeks

@Rod-It said in Odd IP issue - firewall or HA related?:

Well I fixed most of my own issues.

Interesting that no one even bothered to ask or post anything

Don't get too down. HAProxy is likely not a widely used package. I know I have no personal experience with it, thus would not have much to offer to this thread.

However, if you have managed to figure out the cause of your issue and solve it, then it would be kind of you to post the solution here for anyone else in the future who experiences the same issue to find it via a search and implement it to solve their issue.

Rod-It

@bmeeks

I understand why you suggested I post a solution, but I am not going to as this is likely specific to me and my setup, plus if no one understood my question or asked for clarification, the fix will likely only confuse things.

As for HA proxy not being widely used, it has been recommended multiple times on this forum alone to use this over Squid, hence why I use it. Plus there are options it has that squid does not support.

My main issue is not solved and that is the more annoying thing - I cannot use the IPs that seem to have blocked ports and I would ideally like help troubleshooting those, but alas, not a single reply to even ask for more details. I accept it was a convoluted, long and likely confusing post.

For what it's worth, I am a member of another forum, for which I am one of the most active members, so I understand forum etiquette, I have 25 years IT background, but not in networking/firewalls and I am no expert when it comes to Linux, BSD or their variants.

So I don't really post many questions, mostly replies and typically only when I have exhausted my own fault-finding and troubleshooting.

Thanks for the reply though.

bmeeks

Have you checked the HAProxy package setup documentation here? https://docs.netgate.com/pfsense/en/latest/packages/haproxy-package.html.

This is a link to the pfSense Redmine Bug Reporting site. This is a search of all open issues related to HAProxy: https://redmine.pfsense.org/projects/pfsense-packages/search?utf8=%E2%9C%93&q=haproxy&scope=&all_words=&all_words=1&titles_only=&issues=1&open_issues=1&attachments=0&options=1&commit=Submit.

Do any of these have symptoms that look similiar to yours?

Do you have a recent config.xml backup that you could restore? Say one that was taken when your Exchange configuration was working? It might be that in the backing out of the NextCloud/Exchange changes some little setting somewhere did not toggle.

Rod-It

@bmeeks

I will take a look at the links later - thanks.

As for a backup config.xml - this was a clean HA install, so the config should also be clean, however I have resolved the proxying issue, the ports being blocked I believe is more firewall related since I cannot connect to those IPs internally either, which do not go through the proxy, but will go to PfSense as a gateway.

I appreciate the help, but I think for the most part I have moved on from this for now, my only real issue is one I can bear with for now since I have used different IPs.