PFSense configuration for 48 public IP's over DHCP



  • Hello

    After a lot of experimenting I have decided to post and explain my situation here and ask for help.

    For our applications we need a lot of bandwidth, this is a hobby project so we do not have colocation in a datacenter. Our equipment is located in an old house. To be able to get enough bandwidth we have 3 subscriptions with the same cable ISP. This allows us to combine as speed of 3x 240 Mbit download and 3x 20 Mbit upload. The ISP is called Telenet.
    We have 3 seperate DOCSIS3 modems.

    So far we have always used a setup with 3 WAN's with 1 modem connected each. In PFSense we can balance the load and like that we are able to pull a total of 60 Mbit upload and 720 Mbit download. This config also gives us 3 IP's.
    The ISP actually allows us a total of 48 IP's (16 IP's for every modem) but the IP's are dynamic and handed out based on the mac. The IP's are not static and they are mostly not on the same range.

    The 16 IP's per modem we have tested by connecting 16 clients to 1 modem by using a switch. This works.
    So we know want the same results as the previous setup but we want to be able to use all 48 IP's. We don't really care for asigning them to PC's, the are just randomly used as where the 3 IP's originally.

    What we tried:
    1. First of all we tried using vlans. For this I have used a TP-Link SG2424 switch. I placed the switch between the modem and tried using 16 vlans. This results in all kinds of strange behaviour. When testing with just 1 vlan it works, but as soon as we change the mac address the gateway stops working and there's no way to get it back to working except for resetting Pfsense to factory settings. The interface always gets a valid IP, but there is no traffic out, I cannot ping or anything else. When I add 2 vlans as interfaces and I assign them each another spoofed mac address the router turns very slow and it still doesn't work.

    We have spent so much time o this we then considered using PCI-e splitters to be able to add up to 48 physical WAN's. This would be very expensive and seems like a total waste.

    Another option would be to connect USB3 ethernet adapters, this might work but again is not cheap for 48 ports and we will not know if it actually works stable on that scale before we actually tested it.

    Does anyone have any thought about how we can do this? I find it strange that this is so difficult. The option for adding virtual NIC's to Pfsense is very hard to find any info on. Odd, we already worked with a computer that ran 16 VM's on 1 modem and all of them got there own IP over the same NIC and worked perfectly fine.

    Thanks in advance for your advice.


  • Netgate

    It's hard because that's a cockamamie way to get multiple IP addresses.

    They can't/won't just route subnets to you?



  • Nope, I know it may be stupid but there is no other way to get the IP's.



  • 3x Telenet modems towards your pfsense will not help you achieve 3x240mbit. generally all modems in the same street get the same gateway (unless they assign you static ones or you are very lucky).
    pfSense does not support multi-wan with identical gateway. (the only way to work around it, is a mess by putting cheap dlinks between 2 of your WANS and double NAT – you lose the option of wan->VLANS).

    also, telenet does route you as many ip's as you want, if you get the correct subscription. see: http://business.telenet.be/nl/internet/corporate-fibernet

    so basically, what you are trying todo will not work in any way with pfSense.



  • @heper:

    3x Telenet modems towards your pfsense will not help you achieve 3x240mbit. generally all modems in the same street get the same gateway (unless they assign you static ones or you are very lucky).
    pfSense does not support multi-wan with identical gateway. (the only way to work around it, is a mess by putting cheap dlinks between 2 of your WANS and double NAT – you lose the option of wan->VLANS).

    also, telenet does route you as many ip's as you want, if you get the correct subscription. see: http://business.telenet.be/nl/internet/corporate-fibernet

    so basically, what you are trying todo will not work in any way with pfSense.

    Actually that's not correct. With our config with multiple servers we get the full speed 3x 240 Mbit. Modems do not get the same gateway. If you are talking about a default gateway, we get a random gateway on every modem. When we connect 16 clients on a modem, randomly they may have the same default gateway but that does not happen very often. They do not assign static ones but a mac address keeps the IP for a very long time. We have seen leases with the same IP for over 2 months.
    About the speed, Telenet only places a few housenumbers on every hub. We are the only one on our hub which gives us full speed. Telenet guarantees the full speed for up to 4 modems per household (this may be usefull for some situations with student appartments in big cities for example).

    Pfsense obviously does support it as once again, we have used it for almost 6 months with 3 modems. There has been a situation after a reset from Telenet where all 3 modems got an IP on the same subnet, which resulted in all 3 wans having the same gateway. This worked flawlesly. If I were to put 48 NIC cards in this PC I would just add 48 WAN's and load balance them. This would definately work.

    Telenet does not route as many IP's as you want. The biggest subscription in terms of bandwidth is the one we have 3 times. Corporate fibernet is a very expensive solution that is not available everywhere because it uses another layer of the network with a higher prioirty. Furthermore most Corporate Fibernet subscriptions are backed up with a VDSL line. They also come with 8 static IP's. This is the only pro they have. They are twice as expensive and they only offer half the bandwidth. That being said this is not an option as we are not considering paying what we pay now x6. On top of that it is not possible to have 6 subscriptions in the same household.

    What we do will work, we know ways as with the USB adapters that will just work, only they are a solution that isn't logical at all.

    As mentioned in my OP, the situation of the WAN is fixed. I have no questions about how Telenet technically works as I know everything from bottom to top about that. What we described in OP is what we already have working right now. No need for explanations saying that it is impossible.
    I am asking how to deal with it on PFsense side.



  • @NaHoW:

    … with the USB adapters that will just work...

    Do not use USB ethernet adapters with pfSense/FreeBSD. Just don't.
    Save that time for troubleshooting and forum bandwidth.

    @NaHoW:

    …to add up to 48 physical WAN's...

    Since 16 of them are connected through the same modem they will have the same gateway. This renders it useless as additional WANs.

    @NaHoW:

    … tried using 16 vlans ... as soon as we change the mac address ...

    Yeah, that's not supported.
    You can only use one MAC address on each physical interface even though the GUI let's you change it for each VLAN.
    The last configured MAC will be used for all virtual interfaces, unfortunately. Been there, done that.



  • @jahonix:

    @NaHoW:

    … with the USB adapters that will just work...

    Do not use USB ethernet adapters with pfSense/FreeBSD. Just don't.
    Save that time for troubleshooting and forum bandwidth.

    @NaHoW:

    …to add up to 48 physical WAN's...

    Since 16 of them are connected through the same modem they will have the same gateway. This renders it useless as additional WANs.

    @NaHoW:

    … tried using 16 vlans ... as soon as we change the mac address ...

    Yeah, that's not supported.
    You can only use one MAC address on each physical interface even though the GUI let's you change it for each VLAN.
    The last configured MAC will be used for all virtual interfaces, unfortunately. Been there, done that.

    I agree with the USB. On paper it will work but I've seen a lot of complaints about strange issues.

    To clear up the situation once more. The same modem does not mean the same gateway. This modem has no gateway assigned. It will randomly hand out IP addresses to up to 16 different mac's. These 16 IP's can be on completely different subnets or have completely different gateways.

    On the vlans; Definately not. The switch between the modem and the router reported 2 different mac adresses spoofed, one for each vlan.



  • @NaHoW:

    [These 16 IP's can be on completely different subnets or have completely different gateways.[/quote]
    can be is not will be.
    What if they are on the same gateway one day?

    As for the spoofed MACs: Great it's working for you!
    But wait, what's your problem then?



  • @jahonix:

    @NaHoW:

    [These 16 IP's can be on completely different subnets or have completely different gateways.[/quote]
    can be is not will be.
    What if they are on the same gateway one day?

    As for the spoofed MACs: Great it's working for you!
    But wait, what's your problem then?

    No it is not guaranteed. As mentioned in my earlier post sometimes they are and it doesn't matter, I still have 3 different IP's on the same subnet and get the full speed for my servers, so that's not a concern.

    The problem is that Pfsense pulls an IP and then gets bugged as I mentioned in the OP.
    I'm looking for help with my problem, not comments about how wrong I am by using 3 docsis modems.



  • Well, good luck then, if none of the replies seems to fit your expectations.



  • This will not work perfectly, you will have problems, and it will require a tremendous amount of manual intervention to get "working".

    Because all of your addressing and gateway assignments are out of your control, expect it to break and also expect randomly bizarre networking issues to occur…alot.  Managing 48 dynamically assigned addresses to ensure none have the same gateway is a fool's errand.

    It is more prudent to get 3 modems with 16 static IPs all going to different gateways.

    I know you mentioned this is a hobby thing, but if you're at that point where you have a requirement to use all 48 unique IP addresses and the combined, load-balanced bandwidth, you need to make some investments for it to work.  Or, you can modify the pfSense code base to better accommodate your very unique situation.



  • @tim.mcmanus:

    This will not work perfectly, you will have problems, and it will require a tremendous amount of manual intervention to get "working".

    Because all of your addressing and gateway assignments are out of your control, expect it to break and also expect randomly bizarre networking issues to occur…alot.  Managing 48 dynamically assigned addresses to ensure none have the same gateway is a fool's errand.

    It is more prudent to get 3 modems with 16 static IPs all going to different gateways.

    I know you mentioned this is a hobby thing, but if you're at that point where you have a requirement to use all 48 unique IP addresses and the combined, load-balanced bandwidth, you need to make some investments for it to work.  Or, you can modify the pfSense code base to better accommodate your very unique situation.

    It indeed feels like very hard to set this up. Once again for the gateways, this already occurs and it does not result in any problems at all. If 2 of the IP's have the same gateway then PFsense just keeps functioning as usual.

    It's not really an investment thing, there simply is no possibility to get 48 static IP's with the modems with Telenet.



  • Pfsense obviously does support it as once again, we have used it for almost 6 months with 3 modems. There has been a situation after a reset from Telenet where all 3 modems got an IP on the same subnet, which resulted in all 3 wans having the same gateway. This worked flawlesly. If I were to put 48 NIC cards in this PC I would just add 48 WAN's and load balance them. This would definately work.

    Ok here it my tip on this, if you are sure this will working, then please have a look on LANNER´s website
    they are offering device such as the FW-889x series with 64 LAN GB Ports, as per 8 LAN Ports on each module.

    It's not really an investment thing, there simply is no possibility to get 48 static IP's with the modems with Telenet.

    And there is no other ISP you can invite or call? Today getting static IP Adresses is not the real point I think.

    We have spent so much time o this we then considered using PCI-e splitters to be able to add up to 48 physical WAN's. This would be very expensive and seems like a total waste.

    For sure but trying out so much and don´t reach the goal is also is also a pain in the ass.

    For this I have used a TP-Link SG2424 switch.

    I hope not in the front of the pfSense.

    hat we do will work, we know ways as with the USB adapters that will just work, only they are a solution that isn't logical at all.

    I really hope this is only a hobby thing of yours!

    • get static IP addresses
    • get enough IPs from more then one ISP
    • get your hands on a LANNER FW-889x device (ask them what is running proper pfSense)
      Don´t use the FW-8895 the BIOS is not right and capable to install pfSense on it
    • Take enough modems and stitch them in the LANNER device.


  • @BlueKobold:

    Pfsense obviously does support it as once again, we have used it for almost 6 months with 3 modems. There has been a situation after a reset from Telenet where all 3 modems got an IP on the same subnet, which resulted in all 3 wans having the same gateway. This worked flawlesly. If I were to put 48 NIC cards in this PC I would just add 48 WAN's and load balance them. This would definately work.

    Ok here it my tip on this, if you are sure this will working, then please have a look on LANNER´s website
    they are offering device such as the FW-889x series with 64 LAN GB Ports, as per 8 LAN Ports on each module.

    It's not really an investment thing, there simply is no possibility to get 48 static IP's with the modems with Telenet.

    And there is no other ISP you can invite or call? Today getting static IP Adresses is not the real point I think.

    We have spent so much time o this we then considered using PCI-e splitters to be able to add up to 48 physical WAN's. This would be very expensive and seems like a total waste.

    For sure but trying out so much and don´t reach the goal is also is also a pain in the ass.

    For this I have used a TP-Link SG2424 switch.

    I hope not in the front of the pfSense.

    hat we do will work, we know ways as with the USB adapters that will just work, only they are a solution that isn't logical at all.

    I really hope this is only a hobby thing of yours!

    • get static IP addresses
    • get enough IPs from more then one ISP
    • get your hands on a LANNER FW-889x device (ask them what is running proper pfSense)
      Don´t use the FW-8895 the BIOS is not right and capable to install pfSense on it
    • Take enough modems and stitch them in the LANNER device.

    Wow, that is indeed it! Any idea where I can buy this?



  • Wow, that is indeed it! Any idea where I can buy this?

    LANNER Inc.



  • You mention VLANs, are you using Virtual IPs? (VIP)?

    This is essentially same as my setup. I have one DOCSIS 3.0 WAN1, dynamic IP and one (Verizon FiOS) Fiber>>Ethernet WAN2 with 5 static IP. (so 2 WANs and 6 IP addresses, no different than 30 WANs 700 addresses)

    I have pfsense "router on a stick" setup here: https://www.google.com/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRxqFQoTCPbX_bKbqscCFQIgPgodBPwDdg&url=https%3A%2F%2Fwww.highlnk.com%2F2014%2F06%2Fconfiguring-vlans-on-pfsense%2F&ei=GLvOVfbrMYLA-AGE-I-wBw&bvm=bv.99804247,d.cWw&psig=AFQjCNEy8TWbrKc3ohc7soAwNRJVWyTW6Q&ust=1439698072038640
    (sorry for long hyperlink)

    I have WAN1 and WAN2 going into a switch. I have pfSense also going into same switch via 2x 1GbE uplinks in a LACP LAGG. On my WAN1 with dynamic IP I use a VLAN and forward the connection to WAN1 on firewall. On WAN2 (Verizon FIOS 5x static IPs) I use Virtual IPs (VIP) and forward each over a VLAN same as I do WAN1.

    I had trouble with STP and MAC addresses on my switch but I am new to L3 so it was probably my inexperience.

    I'm on 2.2.4-RELEASE (amd64). Go to Firewall>> Virtual IP… Create VIP.

    Hope that helps.

    EDIT: To be clear, you would have your 3x modems going into same L2/L3 switch (make sure your switch fully supports L2, some do not), VLANs + VIPs would then carry your 48 WAN IP addresses to pfSense.



  • SOLUTION

    Thanks everyone for replying. A lot of information given in the replies was incorrect, other information did help me.
    I am posting this for other that may run into a similar situation.

    This is possible with Pfsense.
    Some notes for this case:

    • Loadbalancing multiple modems will combine all of the speed together. But, this is only usefull when you run multiple machines. When you run speedtest on 1 machine, the machine will only archieve the speed of 1 modem. In our situation there are 60 PC's and 3 modems with 240 Mbit. With all 60 PC's we now have 720 Mbit.
    • Having the same gateway multiple times on different NIC's does not matter. With the 48 we indeed have 5 IP's on the same gateway. There is no loss in speed or packets. All works as expected.
    • Everything is stable.

    Possible solutions:

    • The Lanner device is probably the cleanest solution. I it is very hard to find them and they are very expensive.
    • Our solution is using 3 external PCIe cases from Startech. In these we place 4 Intel quad wan cards. Meaning we have 16 NIC on one PCIe lane in the actual router. This works very good. It does take some extra spaces and every external PCIe cases is limitted to bandwidth of 2 Gbit. That is not a problem for us as 1 modem can only draw 240 Mbit.
    • The VLAN may also be a solution. I experimented with this but I have never used VLAN before so I failed. In the beginning it worked but the VLAN showed very strange behaviour. On top of that  the modem was getting stuck by it which makes me suspect there may be a problem with it somewhere there.

    The very best solution would be if PFsense gets support for virtual DHCP addresses. In this case all it would require is a virtual network interface with another mac. This is definately possible. After all, if you have a machine with 16 VM's on it, the virtualisation software also does that.

    Hope this helps the few others that run in this problem too.



  • @NaHoW:

    • Loadbalancing multiple modems will combine all of the speed together.

    Nope, that is not correct.
    All you can use is 3x 240 Mbps separately, which is the opposite of "combined" as you wrote above.



  • @jahonix:

    @NaHoW:

    • Loadbalancing multiple modems will combine all of the speed together.

    Nope, that is not correct.
    All you can use is 3x 240 Mbps separately, which is the opposite of "combined" as you wrote above.

    Please make sure you read my situation. It is working as we speak. The network is using a total download speed of 500 Mbps. As I clearly explained this work in our situation because we have 60 PC's in this network. 1 PC will only get the maximum speed of 1 uplink. Next time please quote the following 2 lines as well.



  • You said: "combine all of the speed together" which is wrong, was wrong and will be wrong forever.
    And it's not the setup you are actually describing.

    But I don't care anymore since you know everything better than the persons trying to help with your questions. I'm outta here.



  • Combine still means together. And yes I do use all 3 modems combined or together for my network. And together they archieve a speed of 720 Mbps. It is not about knowing better. It is about giving correct info. All info you provided so far is wrong. Let me tell you even more about the combined speed, with some adjustments and help from ee freelancer I found out how I can "combine" the speed of all 3 modems together. Allowing me to download a torrent file at a speed of 720 Mbps with just 1 pc involved. Meaning I can combine the speed. It was correct yesterday, it is correct right now and it will still work tomorrow.

    In the end BlueKobold helped me out. This is what helped me find the solutions. You should stick in the "it is not possible topics".

    My solution is possible and I share it with others that may have a similar case it future. Dont come here saying all I say is wrong when I have it working perfectly in real life.



  • Torrenting a file me as you can have multiple sessions opened up downloading from any possible gateways available. This is very different from combining all gateways to manage a single download session.

    Without binding your circuits together, you cannot technically share all of the available bandwidth in one session, it requires multiple sessions.

    In your case pfsense is load balancing and distributing that load, however you have it architected. It's not combining the bandwidth. A single session is still limited to the maximum throughout from the gateway it is going out.



  • @tim.mcmanus:

    Torrenting a file me as you can have multiple sessions opened up downloading from any possible gateways available. This is very different from combining all gateways to manage a single download session.

    Without binding your circuits together, you cannot technically share all of the available bandwidth in one session, it requires multiple sessions.

    In your case pfsense is load balancing and distributing that load, however you have it architected. It's not combining the bandwidth. A single session is still limited to the maximum throughout from the gateway it is going out.

    Exactly.
    I still dont agree on the combined thing. This discussion is useless. I clearly explained in what background is use it. In my case it is the combined speed that matters. I also said that this is not the case for single threaded downloads.



  • The discussion is important. Many people incorrectly assume that you can combine the bandwidth of multiple WAN connections with pfSense or some other device. At least once a month this conversation is had, and many times people need clarification when they observe torrenting speeds exceeding the bandwidth of a single circuit.

    While the conversation may be redundant, it's definitely not useless. IMHO.



  • @tim.mcmanus:

    The discussion is important. Many people incorrectly assume that you can combine the bandwidth of multiple WAN connections with pfSense or some other device. At least once a month this conversation is had, and many times people need clarification when they observe torrenting speeds exceeding the bandwidth of a single circuit.

    While the conversation may be redundant, it's definitely not useless. IMHO.

    In a certain way you are actually combining bandwidth if you have multiple connections, and that was clearly what I meant. I know it's not possible to archieve it on a single connection and I don't claim that either. As said we always have multiple connections.



  • In a certain way you are actually combining bandwidth if you have multiple connections,

    MLPPP (MPLS) can do this and yes also pfSense is able  to do so, but the certain point is, that your ISP
    must also offer you this ability as a service!!!

    I know it's not possible to archieve it on a single connection and I don't claim that either.
    As said we always have multiple connections.

    Load Balancing or fail over set up would be the other abilities that makes it happen to use any connections
    together.