HA Proxy: Trying to get two different addresses to go to the same IP, but different ports
-
I have two DYNDNS hosts, lets call them www.1.com and www.2.com. Both points at the same WAN address. I use HA Proxy to be able to get to them both from inside my local network and from outside. So for this I have a common backend listening on port 80 on any network:
webpage-1 - host matches www.1.com
webpage-2 - host matches www.2.comIn the table I have:
webpage-1: use backend webpage-1
webpage-2: use backend webpage-2Finally, the back ends are
webpage-1: adress+port 192.168.1.53:80
webpage-2: adress+port 192.168.1.50:9876Those are two differet VM's. But now I would like to consolidate the VM's and set up both back ends to point to 192.168.1.53, only with port 80 and 9876 (with one of the webservers set up on that port). And I can't get that to work. When I shut down the VM 192.168.1.50 that was running the 9876 webserver, I can't get to that page on 192.168.5.3 even if the back end is changed. So it looks like that in some strange way, even with the back end changed and the service reloaded it still has the old address in it's tables. Is that possible, and what do I need to do to get it working?
Edit: I have tried to create a copy of the backend called webpage1-new and point to that, but it didn't change anything.
-
@Mastiff looks like you have a typo so let's clear that up first just to be 100% sure: you wrote "I can't get to that page on 192.168.5.3" but you meant 192.168.1.53 correct?
And, if you're on the internal LAN, are you able to fetch
http://192.168.1.53:9876? That should work even if you shut down HAProxy and pfSense completely. If THAT doesn't work, then you've got to sort that first before you have any hope of making pfSense + HAP pass a connection through. -
This post is deleted! -
@luckman212 Sorry, you're correct, of course. Typo. And yes, I can get to any port that I try on 1.53.
-
Ok, hmm. More questions:
- Do both
www.1.comandwww.2.comresolve to the EXTERNAL WAN IP of your pfSense? Even from the LAN? - Trying to understand... are you saying when both VMs are up, everything works as expected?
- Do both
-
@luckman212 Thanks for answering again!
-
Yes, it does.Sorry, I misread the question. Actually I think HA Proxy somehow captures the external addresses that fit the patterns set up in it and routes those directly to the IPs and ports that I have set up when it comes from inside the LAN. I doubt that the device on the LAN goes to DYNDNS first and then back to my network. The network has two levels, one inside my Windows server and one between the Windows server and the pfSense box handling WAN and this routing - the 192.168.1.x network is on the pfSense network, outside the Windows server, and 192.168.0.x is inside the Windows server. I can get to both 1.53:89 and 1.50:9876 from both the outher network and the inner network. -
Yes. And when when I try to route both to the same VM. But when I shut down the VM that now should noe be used, the one that was there stops working. Which is why I wonder if I have to reset routes or something, because a reboot didn't help.
-
-
Maybe I haven't had enough coffee yet, but somehow I'm getting more confused. What is
1.53:89… another typo?So your network looks something like this? Can Client B access both www.1.com and www.2.com successfully?
┌───────────────────────────────┐ │Client B (someone from outside)│ │ IP 100.90.80.70 │──────────┐ └───────────────────────────────┘ │ │ │ │ ├─────────────────────────────────────────┐ │ │ ▼ │ ┌───────────────────────────────┐ │ │ pfSense - WAN 9.8.7.6 │ │ ├───────────────────────────────┤ │ │ HAProxy │ │ │ listening on :80 │ │ ├───────────────────────────────┤ │ │ pfSense - LAN 192.168.0.1 │ │ └───────────────────────────────┘ │ ▲ │ │ │ │ │ ▼ ▼ ┌───────────────────────────────┐ ┌───────────────────────────────┐ │ Windows (Hyper-V) server │ │ Client A (Your PC) │ │ LAN 192.168.0.?? │ │ LAN 192.168.0.10 │ │ Internal 192.168.1.4 ?? │ └───────────────────────────────┘ └───────────────────────────────┘ trying to access www.1.com:80 ▲ (DNS resolves to 9.8.7.6 public IP…) │ NAT? │ ▼ ┌─────────────────────┐ │ webserver VM │ │ 192.168.1.50 │ │ running nginx etc │ │ 2 backends ? │ └─────┬─────────┬─────┘ │ *:80 │ │ *:9876 │ └─────────┘ -
@luckman212 Sorry. Not a typo, I just shortened them down and removed 192.168., I guess that was not smart.
A few comments to the map:
HAProxy is only listening on 80, not HTTPS 443.
Client A is normally on 192.168.0.x
Windows server is 192.168.1.4 WAN and 192.168.0.1 LAN
One webserver VM is 192.168.1.53, the other is 192.168.1.50, both are bridged to the network 192.168.1.x, they are not on 192.168.0.x.But except for that it seems quite right, yeah. Client B can access both servers, and so can Client A.
-
@Mastiff More confusion…
You said you were trying to consolidate to a single VM, no? But you're still talking about "One webserver VM is 192.168.1.53, the other is 192.168.1.50" ??
I adjusted the map, please tell me if it's "correct" now (I doubt it is because these WAN/LAN IPs you're talking about don't make sense to me…) If it's not please draw your own map, showing pfSense WAN/LAN IPs, Hyper-V interface IPs, and VM IPs.
Don't use
*.xanymore, and don't shorten anything. Use the real IPs please. Also don't say "WAN" and "LAN" when talking about internal NAT'ted networks e.g. Hyper-V. Just use internal/external or public/private. -
@luckman212 Thanks for taking the time, even if I'm obviously only making it harder for you... I sat down and thought this through, and I guess I have to show the full layout, not just the simplified layout that I thought would be enough for this to be solved. I apologize if that has thrown you off.
The VM with the webservers (actually two home automation programs with built-in webservers) has been behind two pfSense VMs NAT-ing one port each. This was unecessary complicated, something a forum member with a devil as his avatar pointed out in no uncertain terms when I got help on the forum to set it up around 8 years ago, but it has actually been running perfectly until now. Only now I wanted to simplify the setup and get rid of the virtual pfSense boxes I am having problems.
The main thing is that the address must be the same both for client A and client B, so the addresses aren't different when we're on the house wifi and on 5G outside the house.
This is what I have been running so far:
┌───────────────────────────────┐ │Client B (someone from outside)│ │ IP 100.90.80.70 │──────────┐ └───────────────────────────────┘ │ │ │ │ ├────────────────────────────────────────────────────────────────────────────┐ │ │ ▼ │ ┌───────────────────────────────┐ │ │ pfSense - WAN 9.8.7.6 │ │ ├───────────────────────────────┤ ▼ │ HAProxy │ ┌───────────────────────────────┐ │ listening on :80 │ │ Client A (Your PC) │ ├───────────────────────────────┤ │ LAN 192.168.0.10 │ │ pfSense - LAN 192.168.1.1 │ └───────────────────────────────┘ └───────────────────────────────┘ trying to access www.1.com:80 ▲ (DNS resolves to 9.8.7.6 public IP) │ │ ▼ ┌───────────────────────────────┐ │ Windows (Hyper-V) server │ Bridged network │ LAN 192.168.1.4 │ <──────────────────────────┐──────────────────────────┐ │───────────────────────────────│ ▼ ▼ │ Internal 192.168.0.1 │ ┌─────────────────────┐ ┌─────────────────────┐ └───────────────────────────────┘ │ Virtual pfSense 1│ │ Virtual pfSense 2│ │ 192.168.1.50 │ │ 192.168.1.53 │ │─────────────────────│ │─────────────────────│ │ 192.168.5.1 │ │ 192.168.6.1 │ └─────────────────────┘ └─────────────────────┘ ▲ ▲ │ │ NAT of port 80 NAT of port 9876 │ │ ▼ ▼ ┌─────────────────────────────────────────────┐ │ 192.168.5.100 │ 192.168.6.1 │ │ VM running Windows with 2 automation apps │ │ with built in webservers │ └─────────────────────────────────────────────┘ │ *:80 │ │*:9876 │ └─────────┘ └─────────┘What I want to achieve is this, so HAProxy to only one address, but two different ports, without the extra virtual pfSense instances:
┌───────────────────────────────┐ │Client B (someone from outside)│ │ IP 100.90.80.70 │──────────┐ └───────────────────────────────┘ │ │ │ │ ├────────────────────────────────────────────────────────────────────────────┐ │ │ ▼ │ ┌───────────────────────────────┐ │ │ pfSense - WAN 9.8.7.6 │ │ ├───────────────────────────────┤ ▼ │ HAProxy │ ┌───────────────────────────────┐ │ listening on :80 │ │ Client A (Your PC) │ ├───────────────────────────────┤ │ LAN 192.168.0.10 │ │ pfSense - LAN 192.168.1.1 │ └───────────────────────────────┘ └───────────────────────────────┘ trying to access www.1.com:80 ▲ (DNS resolves to 9.8.7.6 public IP) │ │ ▼ ┌───────────────────────────────┐ │ Windows (Hyper-V) server │ Bridged network │ LAN 192.168.1.4 │ <──────────────────────────────────────┐ │───────────────────────────────│ │ │ Internal 192.168.0.1 │ │ └───────────────────────────────┘ ▼ ┌─────────────────────────────────────────────┐ │ 192.168.1.100 │ │ VM running Windows with 2 automation apps │ │ with built in webservers │ └─────────────────────────────────────────────┘ │ *:80 │ │*:9876 │ └─────────┘I'm not used to creating these maps, I did it in Notepad++. There's probably better ways. But at least I got something to show for my hour of work.
So what happens when I set HAProxy to point directly to 192.168.1.100 it works, but when I shut down the two pfSense VM's it doesn't. And I can't understand why, in my head it shouldn't matter if the previous IP they were pointed to is there or not. Which is why I wonder if there is some routes stuck in the system.
-
Ok I agree about getting rid of the virtual pfSense instances, they are not necessary.
So,
192.168.1.100is the IP of your bridged Ethernet inteface of the Windows VM hosting the 2 "automation apps"? And that VM has a DNS and Gateway IP of192.168.1.1?Remaining notes/questions:
- what is
192.168.0.1on your Hyper-V server for? - I assume
192.168.5.1and192.168.6.1are only set like that because your virtual pfSense's were NAT'ting 192.168.1.x → the "inside" network. Since you've eliminated that, you need to make sure the webserver(s) of the automation apps are bound to either0.0.0.0or192.168.1.100 - HAProxy should be configured to:
- point
www.1.com→192.168.1.100:80 - point
www.2.com→192.168.1.100:9876
- point
- what is
-
@luckman212 Yes, 192.168.1.100 is the "virtual automation server", which is bridged to the ethernet port going to pfSense 192.168.1.1. That is also the DNS and Gateway IP.
It is a fully fledged Windows Server 2022 with DHCP, DNS and RAS, and it's running the VM's in VMWare Workstation. So it's not a pure hyper-V server. 192.168.0.1 is the private/internal network, so that's the network clients on the house wifi connects to. Hence client A 192.168.0.10.
The webservers on the automation apps are bound to all network cards on that VM, I assume that's 0.0.0.0.
And that's exactly how HAProxy is configured. And it works as long as the "detached" VM pfSense instances are running, but the second I shut them down, both stop working. That's the problem, which I was struggling with for hours yesterday before I started this thread.
-
I would appreciate if you could share ALL of the details - you are still leaving key things out:
- where did you mention anything about VMware workstation?
- what is providing DNS/DHCP/NAT for the
192.168.0.xnetwork? You don't show that anywhere, not on your "main" pfSense…
At this point, I can only assume that one or both of those virtual pfSense's that you had running in VMware were providing some sort of proxy or NAT between the
192.168.0and192.168.1subnets, which is why when you shut them down, your192.168.0.xclients start failing. I may be wrong there, but again, I'm working off of what appears to be an incomplete picture.The map you drew is a good start, but please go back to the drawing board and update it. Remove the incorrect information about Hyper-V, add the VMware, add all interfaces on your physical and virtual pfSense appliances, and then let's see where we are after that.
-
@luckman212 Update: I just checked and it's working on the phone from 5G cell network, so it's only from the internal network it's not working.
Edit: Simultaneous posting. The network maps I drew are the complete picture, as far as I can understand. The only difference is that it's a Windows Server 2022 running virutal machines on VMWare Workstation. There are no other connections for the two virtual pfSense instances. And I did say they were VM's in my first post, I didn't know that it mattered what kind of a host they were running on.
The Windows server is doing DNS/DHCP/NAT for that private/internal network, as I wrote in the previous post.
The two pfSenses could not possibly have been doing any proxy or NAT between those two subnets because they were only doing NAT to the single 192.168.1.100 virtual network card of the virtual home automation machine. They were not connected to any part of 192.168.0.1 and they were not connected to anything on the physical server. So this is what it looks like now after disconnecting the virtual pfSense instances.
Edit 2: Map edited:
┌───────────────────────────────┐ │Client B (someone from outside)│ │ IP 100.90.80.70 │──────────┐ └───────────────────────────────┘ │ │ │ │ ├ │ ▼ ┌───────────────────────────────┐ │ pfSense - WAN 9.8.7.6 │ ├───────────────────────────────┤ │ HAProxy │ │ listening on :80 │ ├───────────────────────────────┤ │ pfSense - LAN 192.168.1.1 │ └───────────────────────────────┘ ▲ │ │ ▼ ┌───────────────────────────────┐ │ Windows Server with VMWare │ Bridged network │ LAN 192.168.1.4 │ <──────────────────────────────────────┐ │───────────────────────────────│ │ │ Internal 192.168.0.1 │ │ └───────────────────────────────┘ ▼ │ ┌─────────────────────────────────────────────┐ │ │ 192.168.1.100 │ │ │ VM running Windows with 2 automation apps │ │ │ with built in webservers │ │ └─────────────────────────────────────────────┘ │ │ *:80 │ │ │*:9876 │ │ └─────────┘ ▼ ┌───────────────────────────────┐ │ Client A (Your PC) │ │ LAN 192.168.0.10 │ └───────────────────────────────┘ trying to access www.1.com:80 (DNS resolves via Windows Server 192.168.0.1 to pfSense box 192.168.1.1 to 9.8.7.6 public IP)``` Edit 3: There is nothing going on related to this on the Windows Server, the ports and addresses are not touched by anything there. I just checked DNS, DHCP and RAS, and the addresses are not mentioned anywhere there. My uninformed thought is still that there is something in HAProxy that "remembers" that it used to route those to the virtual pfSense instances and tries to do it still, even if they are shut down. -
I was editing while you were editing. But, is this more correct?
┌───────────────────────────────┐ │Client B (someone from outside)│ │ IP 100.90.80.70 │ └───────────────────────────────┘ ║ ║ ║ ▼ ┌───────────────────────────────┐ │ pfSense - WAN 9.8.7.6 │ ├───────────────────────────────┤ │ HAProxy │ │ listening on :80 │ ├───────────────────────────────┤ │ pfSense - LAN 192.168.1.1 │────────────────bridged ────────────────────┐ └───────────────────────────────┘ network │ ▲ │ │ │ │ │ ▼ │ ┌─────────────────────────────────────────┐ │ │ ├───────────────────┐ │ │ Windows Server + VMWare (workstation) │ LAN: 192.168.1.4 │──────bridged ─────┤ │ providing DHCP/DNS/NAT for Internal net ├───────────────────┘ network │ │ │ │ └─────┬────────────────────────────┬──────┘ │ │ Internal NIC: 192.168.0.1 │ │ └────────────────────────────┘ ┌────────────────────────────┐ ▲ │ webserver VM (Windows) │ │ │ 192.168.1.100 │ │ │ running 2 automation apps │ ▼ │ with built in webservers │ ┌───────────────────────────────┐ └────────┬─────────┬─────────┘ │ Client A (Your PC) │ │ *:80 │ │ LAN 192.168.0.10 │ │ *:9876 │ └───────────────────────────────┘ └─────────┘ trying to access www.1.com:80 (DNS resolves to 9.8.7.6 public IP…)If this is the correct topology, and if you say Client B is working fine, then we need to figure out what IP is being handed to client A when it tries to resolve
www.1.com. Is it9.8.7.6? This seems like an asymmetric route issue... guessing the webserver VM is sending the reply packets back to the wrong router (either VMware or pfSense which may not have originated the state) -
@luckman212 The map looks like it is correct, yeah. If you have any suggestions as to how I can check what's happening, I'm ready to download any applications that can be used to it. I tried a traceroute on my client 192.168.0.10, but it is just getting timed out after 192.168.0.1.
Edit: It has nothing to do with being on 192.168.0.10 inside the Windows sever. I connected a computer as 192.168.1.10, so inside the pfSense firewall but outside the Windows Server, and that didn't help.
-
Okay, that's good that it also fails from
192.168.1.x…On your webserver VM
192.168.1.100, install Wireshark and start a packet capture on the interface that hosts those automation apps. Usehttpas a filter.
Then, from a laptop on the
192.168.1.xnetwork, open a terminal window and runcurl -v www.1.comAfter that completes, stop & save the Wireshark capture. If it doesn't contain any confidential info, upload the result of the
curlcommand and the .pcap so we can take a look.If you can't upload it, at least try to inspect the source and destination IPs of the HTTP traffic and report back.
-
@luckman212 I think the curl was enough. I'll try with Wireshark if you don't think so, but I believe this should really explain it:
curl -v www.1.com * Host www.1.com:80 was resolved. * IPv6: (none) * IPv4: 192.168.1.53 * Trying 192.168.1.53:80... * connect to 192.168.1.53 port 80 from 0.0.0.0 port 62784 failed: Timed out * Failed to connect to www.1.com port 80 after 21072 ms: Could not connect to server * closing connection #0 curl: (28) Failed to connect to www.1.com port 80 after 21072 ms: Could not connect to serverSo if that redirection is made by HAProxy it seems that HAProxy is actually remembering the IP previous address, as I suspected, and directs the call to that instead of the current 192.168.1.100. It's not the callback it's the call.
Edit: To make sure I didn't mess up anything, I double checked that the backend was pointing to 192.168.1.100, and I tried to go to 192.168.1.100 with the browser, and that worked as it should. Same result with www.2.com, this one pointing to 192.168.1.50 and working when I tried 192.168.1.100:9876
Edit 2: OK, this is confusing. I disabled HAProxy and it's still pointing to 192.168.1.53. But now it doesn't work from the outside. I need to do a reboot of pfSense to see if maybe HAProxy won't be fully disabled until I do that, or if there is something else in pfSense that's redirecting this. But it will have to wait an hour, I have something work related so I can't break the connection to my company right now.
-
@luckman212 I give up my 60 year old brain, and I apologize for wasting your time! I finally found the problem, there is a DNS Override that I had no recollection of. So in the DNS server of pfSense the addresses www.1.com and www.2.com were set to 192.168.1.53 and 192.168.1.50. I finally caught on when after the reboot of pfSense the addresses were still pointing to those old addresses with HAProxy off. So I guess HAProxy has been taking care of anything that comes from public/WAN, while the DNS override has taken care of anything from private/LAN. But that doesn't work with any other ports than 80, so I removed the overrides. Now it seems like it's working exactly as it should, without the VM pfSense instances.
Again I apologize for wasting your time, and I will now crawl back in to a suitable hole in the ground...
-
You didn't waste my time, I am happy we got to the bottom of it.
There's this saying...
Privacy Policy · Cookie Policy