pfSense Setup with 250+ CARP VIPs
-
Hello! I'm installing pfSense in an HA pair in a hosting environment. Each client receives a /24 private VLAN and at least one external static IP in the WAN subnet (public /24) of the router. Since each but IP requires a VIP and and each VLAN requires the gateway to be a VIP, I setup my VIPs with the first IP on an interface being CARP VHID 1 then following VIPs to be IP Aliases. This started great but once I got done (around 30 public VIPs at the moment) I noticed that my network speed tanked. When using just a CARP public VIP or a CARP with 1 or 2 Aliases, my network speed is fine (gigabit). If I put all 30 public VIPs on one CARP IP my speed drops to around 100 Mbps. What am I doing wrong? Thanks in advance.
-
Probably some nonsense your hosting provider is doing.
You might want to explain it all again but actually using IP addresses and maybe a diagram.
-
Let me try rephrasing. I AM a hosting provider trying to install pfSense firewalls I have 30 internal VLAN interfaces each with a CARP VIP. I then have a WAN with 30 CARP VIPs. I'm trying to cut down the number of CARP VIPs because there is a limit of 255 CARP VIPs. I saw that you can add IPs as an IP Alias on a CARP IP but when I moved my WAN CARP IPs to IP Aliases, my bandwidth for cut from 1 gigabit to 100Mbps. How many IP Aliases can I have on a CARP IP before I see this performance impact? Is there a better way to do this?
-
How many IP Aliases can I have on a CARP IP before I see this performance impact?
Pretty much as many as you want. Certainly more than 30. You would be subject to any of the same things anyone else is regarding subnet size, ARP, broadcasts, etc.
I would not me looking at pfSense for the reason for the delays you are seeing. I would be looking elsewhere in your infrastructure.
I still can't get a handle on your design. A diagram might help get it across to a simpleton like myself. As a hosting provider I would expect you already have one.
-
It's a WAN interface with 30 CARP IPs. If I convert 29 to IP Aliases on the interface of the 30th CARP IP my performance tanks. I know I could leave all 30 as CARP IPs but that's not scalable. I have no idea how to diagram this.
-
Well, there is absolutely no difference except that with 30 CARP IPs, each IP address has a different MAC address. With a CARP IP and 30 IP Aliases, all IP addresses will ARP for the CARP address they are riding on.
This is perfectly normal and legal, which puts us back to something silly being done by whatever that WAN is connected to.
-
It sounds to me like you have the wrong service for that you are trying to do.
You should have a /24 routed to you instead of a /24 on your WAN interface.
-
I agree that the /24 should be routed but my ISP wasn't interested. Since I can't get that I'm using CARP/IP Aliases. As far as I can tell, this should work but from some reason the IP Aliases just aren't. It works fine if I CARP all the IPs but that's not really sustainable.
-
I'd get an ISP that is willing to do things right. Just sayin'.
I'd pcap and see exactly what's happening. Maybe they have something silly like an inability to ARP for more than X IP addresses per MAC address or something.
It is almost certainly NOT the pfSense software.
