Multiple cPanel/WHM servers
Hello, I'm not sure if this is mostly about routing, NAT, or another topic.
We've got multiple cPanel/WHM servers going into a new datacenter, with a pfSense device at the front. We've had one server live for quite a while now and operating fine, but I'm trying to bring up 2 more servers and I'm having problems binding them to the correct IPs.
We have a single public IP, 22.214.171.124, which the pfSense device will respond on but otherwise shouldn't be used (other than as the public IP for the LAN devices, the cPanel servers on the WAN interface should use their own IPs). We have a /26 network at 126.96.36.199/26 which we want to assign to the cPanel servers. We have .130 pointing to the current live server, and if I log in and send a curl request to a service that responds with the public IP, it does respond with the .130 IP and not the public IP.
However, the new servers will only bind to that 184 public IP. On each of them I have the config for the em1 device configured just like I have it set up for the working server, except different IPs (188.8.131.52 and .145). I can't reach those servers on those IPs though, I have to log in to pfSense on the VPN and then access them using their LAN IPs. I can't reach them on the public IP either because pfSense responds on that one (but I don't need to reach them on the public IP - only their IPs on the /26 network). Eventually these servers will have multiple IPs on that /26 network assigned to them, because they will be hosting multiple SSL domains.
In pfSense, I have the outbound NAT configured so that the LAN interface will use NAT, but the WAN interface will not. I have it set to hybrid where it set up 2 additional rules for the WAN interface. But when I log in to either of those servers via their LAN IPs on the VPN, sending the same curl request shows that they are sending on the public 184 IP.
This is the ifcfg for one of the servers:
HWADDR=F8:BC:12:44:17:04 TYPE=Ethernet BOOTPROTO=static IPADDR=184.108.40.206 GATEWAY=220.127.116.11 NETMASK=255.255.255.192 DEFROUTE=yes PEERDNS=no DNS1=18.104.22.168 DNS2=22.214.171.124 PEERROUTES=no IPV4_FAILURE_FATAL=no IPV6INIT=no IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_PEERDNS=yes IPV6_PEERROUTES=yes IPV6_FAILURE_FATAL=no NAME=em1 UUID=9240a470-2075-4efa-87d5-7b5a7bf3e066 DEVICE=em1 ONBOOT=yes
If I run ifconfig then it looks fine (it even indicates a small amount of traffic on that interface), but it's not reachable on that IP.
em1: flags=4163<up,broadcast,running,multicast>mtu 1500 inet 126.96.36.199 netmask 255.255.255.192 broadcast 188.8.131.52 inet6 fe80::fabc:12ff:fe00:1704 prefixlen 64 scopeid 0x20 ether f8:bc:12:00:17:04 txqueuelen 1000 (Ethernet) RX packets 456 bytes 50829 (49.6 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 27 bytes 4093 (3.9 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 56 em2: flags=4163<up,broadcast,running,multicast>mtu 1500 inet 192.168.10.36 netmask 255.255.255.0 broadcast 192.168.10.255 inet6 fe80::fabc:12ff:fe00:1705 prefixlen 64 scopeid 0x20 ether f8:bc:12:00:17:05 txqueuelen 1000 (Ethernet) RX packets 765 bytes 102287 (99.8 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 423 bytes 234116 (228.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 58</up,broadcast,running,multicast></up,broadcast,running,multicast>
In the cPanel/WHM setup, there is a page for Show/Delete Current IP Addresses. The entry for em1 shows the .145 local IP, and the public IP is given as "Not Routable". It also shows em2 with the LAN IP and the 184 IP listed as the public IP. There's also a link on that page labeled "Validate", and if I click that then I see several entries in the system logs in pfSense for the firewall which shows requests from the .145 IP being blocked to several cPanel IP addresses, although it's showing those requests on the LAN interface instead of WAN or DMZ. e.g.:
Sep 13 16:09:22 LAN 184.108.40.206:49816 220.127.116.11:443 TCP:S
Other than that, I don't see much in the logs for that IP. If I type the IP into a browser, for example, I don't see anything in the firewall logs about that request.
I'm posting in this forum because I think that this might be an issue that I would fix in pfSense, based on the fact that the other live server has been working for a while and they're all plugged in to the same switch going to pfSense. Are there any suggestions or other places where I can look?
If I connect via the VPN and then SSH to one of the new servers (e.g. the .145 one), then once I'm logged in there I am able to ping all of the other assigned /26 addresses. So I can ping 18.104.22.168 (the gateway), 22.214.171.124 (live server), 126.96.36.199 (the other server that I can't reach from outside), all of those work. So the IP is getting bound to the correct server, but it's not routable. I tried to ping an unassigned IP in the /26 network and it was not reachable, so it's not that all of the IPs are responding by something like pfSense, only the assigned IPs are responding.