Mac Address Binding - Static IPs from ISP - ProxyArp/IP Alias VirtualIPs do not work
-
Hi everyone. I apologize if this is not the appropriate forum to post this, and if so please let me know and I'll move it. I am about at my wits end and I don't know what else to do.
Recently, I switched from Comcast to a local fiber based ISP. Luckily, they allowed me to purchase blocks of static IPs to use each month (block of 5 usable or /29) which was great! I set up pfsense like I have done literally 100s of times for my professional work including setting up virtual IPs using ProxyArp/IP Aliases (again, done this 100s of times with many MANY different ISPs). They weren't working, which was a bit confusing, but if I static IP'd them to another device say my laptop, it would work. Odd...so, after trying to change the primary IP of my firewall to the same IP, however, I realized that my ISP deploys mac-binding which means that my mac-address of my laptop's NIC was now stuck to that IP. No problem, I thought, I'll just call them and have them clear it up. Sure enough they cleared it, but Virtual IPs still don't work. After discussing the matter further with their level 3 tech who has been there for 20 years, he said that because of the mac-binding, it's just not going to be possible to do proxyarp/ip alias virtual IPs because it's a 1 to 1 mapping of IP to mac and they don't allow duplicate mac addresses to bind to multiple IPs.
I asked him how other customers do this, and he seemed to think everyone just assigns out IPs to the specific devices that need them which seems very dangerous IMO if their config is not tightly secured. Either way, that's not what I want as I want to use all of these behind my pfsense NAT.
I'm trying to figure out a way around this, but I'm at a loss. I thought about putting in additional NICs and assigning those additional IPs to them, but that won't work since you can only use the same gateway on one NIC in the same subnet. I don't want to get into silly setups like purchasing routers that sit in front of my pfsense with the aforementioned static IPs and then port forwarding everything to pfsense, but that's about the only way around this that I can think of. Also since my connection is 8 Gigabit symmetrical, purchasing the equipment needed to do this would get very spendy and again, it's such a silly and hokey setup, and I really don't want to go there.
Can any one think of ANY other way to do this? I don't suppose there's a way to assign a mac-address to a VIP? That might solve my problem, but I can't seem to find anything out there to support this on pfsense. Thoughts?
-
@byusinger84 I'm thinking this is not too far from what this thread is about? https://forum.netgate.com/topic/189451/multiple-wan-with-static-ips-dhcp-assigned-from-isp/8
And my idea was to virtualize pfsense in e.g. Proxmox which allows you to assign multiple interfaces to one physical port, each interface having it's individual MAC.
But now that I think about it, this may be similar to your alternative of adding NIC's, in that you do create separate WAN's = gateways... But you could possibly group them together couldn't you, as a completely balanced gateway group?
And your performance may or may not suffer if you virtualize. Never tested anything above 1G like that. I do have pfsense running under Proxmox and reach ~8/8G on a 10/10G connection with suricata active in legacy mode.
-
@Gblenn I was virtualized before in proxmox but the most I could get without nic pass through was about 3 gigabit on a Dell R430. Now I'm running bare metal on a Dell R340.
At any rate, yes that's what I was trying to emulate by adding additional physical nics, but obviously that won't work because each nic would need it's own ip address and share the same gateway, which isn't allowed. Of course that makes sense, but yeah I just don't know what else to try here. The tech at the isp just called me and told me he has tested with other vendors like Cisco in his lab with my config and it works just fine so idk what's different that Cisco is doing vs what I'm doing.
-
@byusinger84 Ok, so only 3 Gigabit with virtualized NIC's, that was a dissapointment of course... So then that path isn't really an option, unless there are ways to tweak it and get better performance.
A question... I don't really understand the "isn't allowed" part about having multiple NICs? What is it that isn't allowed, from what side, the ISP?
-
@Gblenn sorry no, pfsense literally will not let you do it.
-
@byusinger84 Hmm, you mean because they are on the same subnet which means they all have the same gateway at the ISP... got it, so that's why you were talking about Virtual IP's...
It's interesting though that I happen to have two IP's from my ISP, but in my case with very different subnets (and gateways).
So, if you were to connect it like this, you would run into trouble... But aren't there ways around that then?
I'm sure there are others much better equipped to answer this but the things I can think of are these:
- Disable gateway monitoring for all but one of the interfaces, or monitor some external IP's like e.g. 8.8.8.8, 1.1.1.1 etc.
- Adjust firewall rules on each interface to send traffic out the appropriate (gateway) WAN (under Advanced)
- I'm also thinking you would have to have manual outbound NAT rules for each WAN to make sure traffic exits the right way?
Could that work?
-
@Gblenn I don't think number 1 will help. Number 2 is interesting...maybe this is doable with some static routes? But sounds less than ideal/messy. 3 yes, this will work but only when the VIPs work or if #2 could work maybe I could force things out that way as well. I have done outbound NAT just fine using VIPs at other sites.
ISP guy called me back and he's stumped. He has tickets in with the network vendor because he doesn't know why it's not working because it works in his test lab.
-
@byusinger84 I agree #1 will not fix things but the point is you have to monitor different gateway IP's for each one...
I don't know how "messy" things will get, it's just one setting but of course it needs to be applied (edited) to each rule. So yes some manual work is required...
#3 is just one rule per WAN I suppose, so not much to do there really...So your ISP is saying that your original setup with VIP's works, but with a Cisco router? I don't know VIP's or Cisco so It's way out of my league, but hopefully you can get it to work as initially planned.
-
@byusinger84 Perhaps I'm overthinking things but you are not using load balancing or failover are you? In the other thread I referenced, @chpalmer posted a comment saying that it shouldn't be a problem unless you have failover/balancing implemented.
And sure enough, the thread owner has it working, just not at the speed he wants until HW is upgraded. -
@Gblenn I don't believe that is true, and in either case, I tried that. Did not work. The ISP literally doesn't see the IPs binding to the mac-addresses on the pfsense. He said that his Cisco ASA and his Palo Alto work just fine in the test lab. I tried to swap out my NICs in case I had something not playing nice. That didn't work either.
-
@byusinger84 But the fact that pfsense may not like have multiple WANs going to the same gateway shouldn't have anything to do with the ISP not seeing the individual MACs.
How do you connect the ports towards the ISP?
I just placed a managed switch in between but I suppose any dumb switch would do. Which in fact is what the other thread had... And in both cases all the IP's are DHCP, although mine never change and my ISP needed to register the MACs...