CARP, and multiple networks on a single interface.
-
Okay, here's the short version:
I have outbound NAT turned off, straight routing here.
I have one public routable network on the "LAN" interface, and another on the "WAN" interface. This works, and works well, however those blocks of addresses are small, 8 IP's each.
I've been assigned 2 Class C blocks. I went into my two pfSense boxes with the intention of assigning them .1 off of each of those blocks as a gateway address, however when I go to create the CARP address I get this error:
The following input errors were detected: * Sorry, we could not locate an interface with a matching subnet for 206.80.88.1/24\. Please add an ip in this subnet on a real interface.The problem is that there's no "real" interface to provide an IP address to. I'm already using a single address on a real interface. My only other choice would be to manually alias an address from the command prompt, right? I could force it in /etc/rc.conf until aliases are supported, but that feels like a kludge. The odd thing is, it let me set this up on Beta4, and with the advent of RC1 now pfSense chokes on this. Was this an intentional change?
UPDATE: Huh. It seems that RC1 is very "crashy". I've been messing with it all night on my two hacom boxes, and both of them are crashing like crazy, and corrupting the filesystem in the process. Looks like I'm going to be downgrading to Beta4 in the morning. :( Very very very unstable.
-
Any chance there is an usb-nic involced in your setup?
-
None. This is the triple-gigabit hacom model. The only thing "different" is that on both of my boxes I've added a single 10/100 NIC to get a fourth interface. One was an rl and the other dc. (I think….I'm not at the data center right now).
You've seen this error before? Is it really required that you have an IP in the same subnet on a physical interface for a CARP virtual IP to work? Doesn't seem right to me for some reason.
EDIT: Yup, you've seen it before:
http://forum.pfsense.org/index.php?topic=1374.0
That behavior is pretty accurate, although there's a step here I guess I should have tossed in. Originally while I was waiting for my new hacoms to arrive, I have a wrap and a soekris(another thread with carp issues where the switch was to blame exists here...), and when the hacoms arrived, I exported the config.xml, then running beta4. I installed beta4 onto the hacoms, then imported config.xml for each, then ran the RC1 update. After I saw that was good, I shut each one down in sequence and installed an additional pci nic so that I would have another routable interface for the office. That started my troubles. I've tried three different driver sets, rl, dc, and de.
When pfSense comes back up, it wigs out a bit wondering what's up with the interfaces and runs the interface assignment code again. No big deal, I assign them, and as a result the interface I had labeled "pfSync" winds up losing it's identity, which was expected. I go back to that interface, enable it, and that's when I get a kernel panic regardless of which driver interface I add to the system. I also get the error above regarding CARP, again that doesn't make a lot of sense.
I hate giving actual IP's, but I feel like you need to see what I have going on here.
Originally we were only assigned a /28 worth of addresses, which we burned through in no time at all. We expected to get a permanent allocation from ARIN, but that STILL hasn't happened. We had this setup:
206.80.68.16/28, which we subnetted to a pair of /29's:
WAN side
.16 network
.17 upstream gateway
.18 pfsense1
.19 pfsense 2
.20 WAN CARP virtual IP for pfsense boxes
.21 and .22 vacant
.23 broadcastLAN side
.24 network
.25 pfsense1
.26 pfsense2
.27 LAN CARP virtual IP for pfsense boxes
.28-.30 servers
.31 broadcastThat worked just fine. Upstream gives me a pair of temporary /24 allocations. I want those on the LAN side, so I went into CARP and tried to add, as you see above 206.80.88.1/24 as a CARP on pfsense1. That worked on Beta4 without complaint. RC1 refuses with the above error. It is notable however that even though Beta4 allowed it, it didn't actually work so far as I could tell. Traffic wasn't actually getting passed from interface to interface. What is proper protocol for this?
-
None. This is the triple-gigabit hacom model. The only thing "different" is that on both of my boxes I've added a single 10/100 NIC to get a fourth interface. One was an rl and the other dc. (I think….I'm not at the data center right now).
You've seen this error before? Is it really required that you have an IP in the same subnet on a physical interface for a CARP virtual IP to work? Doesn't seem right to me for some reason.
Yes, it's really required.
–Bill
-
You beat me to the post by a few seconds. See my post above. How would you approach my situation?
-
Side note - I tried to alias 206.80.88.2 and .3 to the LAN interfaces on each box. pfSense's code does not recognize aliases and still states that an IP on a real interface must exist. Proxy-ARP on each box won't do it either.
-
Well, I went back to the data center, and I tell ya, I'm starting to wonder about the power supply of these Hacom boxes. I had to play musical NIC's, but I did finally get a combination of cards that seem to be stable, a single rl and de. Everything else was causing kernel panics. I'm concerned that the quad 10/100 soekris cards I just ordered will be just as unstable. :(
After having spent $700 a pop on these systems, I get the feeling I'm going to have to go with a home grown solution, which really sucks as the form factor on these is good, but if the PCI slots are useless, then the entire system is. :\
-
Contact bao from hacom concerning your (possible) powersupply issues. I'm sure he's willing to help you or give you some advice.
-
Is there anything that can be done with the network layout above? It really blows not being able to have more than one CARP-able network per interface.
-
Afaik this is a limitation of how CARP works but somebody might proof me wrong.
-
Well, a way to test it would be to not use the web interface at all, and use the console to set up a carp VIP. Then go back and try to use aliases and set up a second one. Don't have a console handy to try it at the moment, but it would be useful to know.
-
As others have already told you this will not work.
-
Sorry, wasn't trying to push my luck, was simply trying to figure out where the limitation was, whether it was with CARP or with pfSense.
That said, my "crashy"-ness appears to be part of a known bug. 2 phone calls with Bao Ha came to this:
http://www.freebsd.org/cgi/query-pr.cgi?pr=i386/88610
They're going to try to beef up the power supply from the current 60 Watt. Going to be next week before I have further news on that front. Shame you can't use 5 or more interfaces on FreeBSD 6.0. No movement on that bug since November either.
-
Update - Bao tried a better power supply, but that doesn't appear to be the problem, there seems to be something wrong with the PCI bus, and is taking the matter up with the manufacturer in Hong Kong.
I hope he gets it resolved soon. These are pretty expensive door stops!
(I guess this became a hardware thread on me, didn't it?)
-
Another update. Hacom has pulled their boxes from their website. They've confirmed a serious issue with the PCI bus and are working to resolve the problem. They've since refunded me for my systems. Hope they get it resolved soon!
:o