My connection requires a unique MAC for each static IP. What do?
-
Edit: Long story short (don't read everything else unless you like tons of details)
It seems my cable modem connection requires that each IP in its pool of 5 static IPs be registered to a unique MAC ID. According to the pfsense docs, I should use a CARP VIP for this situation (man does that seem like a cludge), but it doesn't fix the problem for me. Any suggestions?
Network Setup:
3 Connections.
2 Cable (Coaxial) connections from the same ISP, using two identical cable modems, each set to bridge mode, with 5 static IPs on each connection (2 x 5 = 10 static IPs total). For each connection, the 5 static IPs are on a 255.255.255.0 subnet. However, both connections are on a different /24 subnet.
1 DSL connection from a different ISP, with 5 static IPs on a 255.255.255.248 subnet
All 3 connections are on different subnets, with different gateways.
In status, all three gateways show as online.
All 3 connections are using ONE of their 5 static IPs and that configuration works fine.
Problem:
Following the information from this thread: https://forum.pfsense.org/index.php?topic=102363.msg575440#msg575440
I was able to setup a Virtual IP (Proxy ARP) and a 1:1 NAT to an internal IP for my DSL connection (with firewall rules to allow data through for my internal box). That means pfsense is currently handling TWO of the five IPs for the DSL connection. One is for pfsense itself, and the other points to another box behind the router.
Following the exact same procedure for the two cable connections results in disaster. As long as pfsense is just managing one IP for each connection, everything works fantastic, but the second that I create another Virtual IP (Proxy ARP) for either of the cable connections, I start getting major packetloss on the interface, and a ping from outside the network goes from rock-solid to intermittent time-outs all over the place.
The weird thing is that the DSL connection works fine with the same configuration.
I tried a couple experiments to narrow down the problem. I disconnected everything from one cable modem and then tried connecting a single Windows box and assigning it a single IP: all good. Then I went into the properties for the adapter and tried assigning the same interface two of the static IPs. I started getting symptoms identical to when I was trying to have pfsense manage multiple IPs.
I also tried connecting two different physical devices to the modem, and assigning each device its own SINGLE static IP, and everything worked fine. So the modem definitely has no problem handling multiple IPs at once, just not both from the pfsense at the same time.
My best guess is that there is possibly some kind of MAC ID conflict, and the cable service doesn't like it when there is more than one IP registered to the same MAC address?
How can I fix this problem?
-
RTFM eh? I found a reference to Virtual IPs in the docs that seems to precisely address my situation, yet the proposed solution does not seem to work:
https://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses
Some upstream equipment requires each distinct IP address to have a unique MAC address. In such cases, the use of CARP VIP types may allow the additional addresses to function where they otherwise would not work with IP alias or Proxy ARP VIPs. This has been common to see in the past with AT&T Uverse equipment.
The MAC address of a VIP will change if the VIP entry is changed between a type that has a unique MAC address, such as CARP, to one that shares a MAC address with a parent interface, such as IP alias or Proxy ARP. Due to the MAC address change, other equipment on the segment may need to have its ARP cache cleared, it may need to be rebooted (cable modems especially), or there may be some other time period that must expire for the ARP cache to update. This may be as few as a couple minutes or up to four hours.
If a particular configuration does not work with IP alias or Proxy ARP type VIPs, try with a CARP VIP instead, or vice versa. Address or wait out the potential ARP concerns before declaring one particular type a failure, and always be on the lookout for IP conflicts.
This sounds fantastic… however I thought CARP was specifically designed for creating redundancy by sharing IPs. And that seems to be borne out in my experiment. I tried changing the Virtual IP type to CARP, but it won't let me create a CARP IP without putting a VHID password, which of course makes no sense in this use case. Anyway, I tried CARP and as soon as I created a CARP VIP on the interface, ALL IPs on that interface stopped responding to ping. Not just intermittent ping, NO CONNECTION AT ALL.
I also tried IP Alias and Other VIP for fun, and had no success.
In summary, I found these results:
IP Alias: Tried subnets of both /24 and /32 for the VIP. Rock-steady ping for the pfsense IP, but no response from the VIP. The 1:1NAT does not seem to work at all.
CARP: Lots of settings to fool with here, including VHID password, VHID group, and Base and Skew. I tried several different options, including /24 and /32 subnets, and changing the group/base/skew to random numbers, but in all cases using CARP just gets me NO PING on the pfsense IP AND the VIP
Proxy ARP: As detailed above, results in intermittent ping for both the pfsense IP and the VIP. With single host selected, there are no subnets to select, nor any other options.
Other: Again not really any options to change here for single host. Rock steady ping from pfsense IP, but no response from the VIP. 1:1NAT does not seem to work. -
With a cable modem it more than likely doesn't require a unique MAC per-IP, but you have to power cycle the modem often after changing, adding, or removing IPs. If it does require a unique MAC, then you want CARP IPs, and power cycle the modem after adding them.
-
@cmb:
With a cable modem it more than likely doesn't require a unique MAC per-IP, but you have to power cycle the modem often after changing, adding, or removing IPs. If it does require a unique MAC, then you want CARP IPs, and power cycle the modem after adding them.
I have tried restarting (power-cycling) both the modem and the pfsense box. It does not make a difference. Besides, I don't believe power-cycling is required in this case, seeing as how whenever I add the VIP I see an immediate change in pings.
But yeah… adding a CARP and power-cycling does not change my situation.
What settings, exactly, should I be using for CARP (VHID password, group, base, skew)?
Btw, using CARP really seems like a piss-poor way of dealing with this situation. It seems to me that a simple, straightforward, and less obtuse solution would be a VIP option that has a field for MAC ID spoofing.
-
Use the defaults for all the CARP settings.
There is no way in the underlying OS to add an IP alias on a diff MAC. CARP's a perfectly fine means of doing so, and is widely used as such.
Most cable modems require a power cycle before they'll pick up a MAC change on static IPs. Changes in reachability from inside your network have no dependency on your cable modem even existing much less picking up changes you're making.
-
I've seen this MAC:IP limitation on a number of occasions, cable modems, DSL modems, WISP connections. The big telco ISPs raison d'être is all about making life hard any anyone but the most basic common user. If switching ISP is an option, let them know you are displeased by voting with your wallet.
In the meantime, CARP works because each VHID translates into a unique MAC address. As cmb pointed out, use the default CARP settings, just make each alias a unique VHID.
So…
VHID =1 = MAC: 00:00:5e:00:01:01
VHID =127 = MAC: 00:00:5e:00:01:7f
etc...
You can create a unique MAC for up to 255 VHIDs. -
Can you afford to lose two public IP addresses?
Install ESX on the hardware and create your pfsense as a virtual. Create three virtual switches and tie a physical NIC to each. Connect the three ISP connections to the three NICs. Create a fourth virtual switch and tie it to another physical NIC for your inside network.
For the pfsense virtual machine, create 10 (vmware limit) virtual NICs: 4 on cable#1 virtual switch, 4 on cable#2 virtual switch, 1 on DSL virtual switch, and 1 on inside virtual switch.
This assume that pfsense can handle 9 outside interfaces and one inside.
NOTE, this would also give you the flexibility to create multiple pfsense instances - each tied to different ISPs or public IP addresses. That way, you won't lose any public IP addresses. Depending on what you are doing, this may suit your purposes better.
If you've never used vmware before, don't be put off by learning it. ESX is fairly simple to configure for simple setups like this.
-
1. There are very limited choices in the US for ISPs as it is. I'm in a third-world country doing this setup and we have even less choices here, so voting with my wallet is not really an option.
2. pfsense is already running on ESXi v6 actually. I have two virtual routers on the machine, using the same physical interface. They both have 1 static IP each and that works fine. It is only when I assign another static IP to pfsense that the two pfsense IPs crash and burn.
3. If I could get more statics from my ISP, then I could afford to burn two, but 5 IPs is the max for this ISP, and as I said before, I don't have any other choices.
4. I thought of making a virtual NIC for each static IP, but the problem then becomes that pfsense can not deal with more than one interface using the same gateway.
5. I've tried unplugging the modem completely, adding the CARP VIP, and then plugging the modem in. Still destroys the reliability of the two IPs, as if there was some kind of conflict.
I am at wits end trying to figure out what the problem is here. It doesn't seem like the CARP VIP is working as it should be.
-
With ESX, you probably missed the config at the ESX level to allow multiple MACs to the VM.
https://doc.pfsense.org/index.php/CARP_Configuration_Troubleshooting#Hypervisor_users_.28Especially_VMware_ESX.2FESXi.29 -
Can you tell us what it you want to do with three ISP connections? Are you just needing a whole bunch of public IP addresses?
-
Maybe get a tunnel with static IPs on it instead?
-
You could look around for an ipv4 tunnel broker in any country you choose and setup a tunnel with them. You stay on dynamic IP, your fixed addresses are routed to you.
-
Setup an AWS micro instance, run pfSense in it and setup an OpenVPN link from AWS with fixed IP to your dynamic IP.
-
