Multiple cPanel/WHM servers



  • Hello, I'm not sure if this is mostly about routing, NAT, or another topic.

    We've got multiple cPanel/WHM servers going into a new datacenter, with a pfSense device at the front.  We've had one server live for quite a while now and operating fine, but I'm trying to bring up 2 more servers and I'm having problems binding them to the correct IPs.

    We have a single public IP, 184.0.0.1, which the pfSense device will respond on but otherwise shouldn't be used (other than as the public IP for the LAN devices, the cPanel servers on the WAN interface should use their own IPs).  We have a /26 network at 68.0.0.128/26 which we want to assign to the cPanel servers.  We have .130 pointing to the current live server, and if I log in and send a curl request to a service that responds with the public IP, it does respond with the .130 IP and not the public IP.

    However, the new servers will only bind to that 184 public IP.  On each of them I have the config for the em1 device configured just like I have it set up for the working server, except different IPs (68.0.0.140 and .145).  I can't reach those servers on those IPs though, I have to log in to pfSense on the VPN and then access them using their LAN IPs.  I can't reach them on the public IP either because pfSense responds on that one (but I don't need to reach them on the public IP - only their IPs on the /26 network).  Eventually these servers will have multiple IPs on that /26 network assigned to them, because they will be hosting multiple SSL domains.

    In pfSense, I have the outbound NAT configured so that the LAN interface will use NAT, but the WAN interface will not.  I have it set to hybrid where it set up 2 additional rules for the WAN interface.  But when I log in to either of those servers via their LAN IPs on the VPN, sending the same curl request shows that they are sending on the public 184 IP.

    This is the ifcfg for one of the servers:

    HWADDR=F8:BC:12:44:17:04
    TYPE=Ethernet
    BOOTPROTO=static
    IPADDR=68.0.0.145
    GATEWAY=68.0.0.129
    NETMASK=255.255.255.192
    DEFROUTE=yes
    PEERDNS=no
    DNS1=68.105.28.16
    DNS2=68.105.29.16
    PEERROUTES=no
    IPV4_FAILURE_FATAL=no
    IPV6INIT=no
    IPV6_AUTOCONF=yes
    IPV6_DEFROUTE=yes
    IPV6_PEERDNS=yes
    IPV6_PEERROUTES=yes
    IPV6_FAILURE_FATAL=no
    NAME=em1
    UUID=9240a470-2075-4efa-87d5-7b5a7bf3e066
    DEVICE=em1
    ONBOOT=yes
    
    

    If I run ifconfig then it looks fine (it even indicates a small amount of traffic on that interface), but it's not reachable on that IP.

    em1: flags=4163<up,broadcast,running,multicast>mtu 1500
            inet 68.0.0.145  netmask 255.255.255.192  broadcast 68.0.0.191
            inet6 fe80::fabc:12ff:fe00:1704  prefixlen 64  scopeid 0x20
            ether f8:bc:12:00:17:04  txqueuelen 1000  (Ethernet)
            RX packets 456  bytes 50829 (49.6 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 27  bytes 4093 (3.9 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
            device interrupt 56
    
    em2: flags=4163<up,broadcast,running,multicast>mtu 1500
            inet 192.168.10.36  netmask 255.255.255.0  broadcast 192.168.10.255
            inet6 fe80::fabc:12ff:fe00:1705  prefixlen 64  scopeid 0x20
            ether f8:bc:12:00:17:05  txqueuelen 1000  (Ethernet)
            RX packets 765  bytes 102287 (99.8 KiB)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 423  bytes 234116 (228.6 KiB)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
            device interrupt 58</up,broadcast,running,multicast></up,broadcast,running,multicast> 
    

    In the cPanel/WHM setup, there is a page for Show/Delete Current IP Addresses.  The entry for em1 shows the .145 local IP, and the public IP is given as "Not Routable".  It also shows em2 with the LAN IP and the 184 IP listed as the public IP.  There's also a link on that page labeled "Validate", and if I click that then I see several entries in the system logs in pfSense for the firewall which shows requests from the .145 IP being blocked to several cPanel IP addresses, although it's showing those requests on the LAN interface instead of WAN or DMZ.  e.g.:

    Sep 13 16:09:22	LAN	  68.0.0.145:49816	  208.74.124.133:443	TCP:S
    

    Other than that, I don't see much in the logs for that IP.  If I type the IP into a browser, for example, I don't see anything in the firewall logs about that request.

    I'm posting in this forum because I think that this might be an issue that I would fix in pfSense, based on the fact that the other live server has been working for a while and they're all plugged in to the same switch going to pfSense.  Are there any suggestions or other places where I can look?

    Thanks



  • If I connect via the VPN and then SSH to one of the new servers (e.g. the .145 one), then once I'm logged in there I am able to ping all of the other assigned /26 addresses.  So I can ping 68.0.0.129 (the gateway), 68.0.0.130 (live server), 68.0.0.140 (the other server that I can't reach from outside), all of those work.  So the IP is getting bound to the correct server, but it's not routable.  I tried to ping an unassigned IP in the /26 network and it was not reachable, so it's not that all of the IPs are responding by something like pfSense, only the assigned IPs are responding.