IP's are not assigned (via ISP's DHCP) to multi-WAN interfaces.

ASTRApogianos · Feb 1, 2019, 12:45 PM

Hello everyone.
I have pfSense 2.4.4 installed as Hyper-V VM (Gen1) under Windows Server 2012 R2 Datacenter.
I have 1 dedicated NIC as WAN split into 4 virtual switches directly connected with the ISP and another one for my LAN.
On the WAN side, the provider gives me 4 static (leased via DHCP) IPv4 addresses through the same cable.

The issue is that whenever I restart the pfSense VM, 2 out of 4 (sometimes 3 out of 4) interfaces are not able to obtain the IP from the ISP. The workaround I'm following is to release and renew the interfaces several times until an IP is assigned. The interfaces have static MAC addresses.
On the other hand, with any other router (LinkSYS, TP-Link, ZyXel) I don't have issues like that as the IP is assigned in an instant and without having to release it from the settings.

My experience with Linux and its commands is very limited and I was wondering if any of you encountered anything like this and provide with any possible solutions.

If you need any logs, please guide me on how to get them.
Thank you

ASTRApogianos · Feb 5, 2019, 9:55 AM

It seems that no-one ever came with an issue like that.
One thing that came up after reading similar issues (with 0.0.0.0) was the hardware itself.
The NIC that I'm using is the Intel PRO/1000 MT Desktop Adapter.

Do you think that if I will change it to a Server Adapter will do any difference?

johnpoz · Feb 5, 2019, 10:55 AM

So your trying to give pfsense interfaces 4 IPs in the same range? Yeah I don't think that is a supported scenario.

Sure isn't via setting them static. So why should it work via dhcp?

ASTRApogianos · Feb 5, 2019, 11:29 AM

I'm not sure I understand what you mean...
In pfSense I have the following interfaces. hn0(wan1), hn1(lan), hn2(opt1-wan2), hn3(opt2-wan3) and hn4(opt3-wan4).
The interfaces hn0, hn2, hn3 and hn4 are virtual adapters created from the same physical nic.
The IP's that the ISP is providing are static leased (via DHCP) from the same subnet (/22).

I'm not using any Load Balancing or anything. The reason I want them is that I have enabled 1on1 on them for specific services I'm running on the connected servers behind those interfaces.
Moreover, once I release and renew, they all work just fine with no issues.
The issue is that on the initial boot of pfSense, some of them can't get an IP from ISP.

johnpoz · Feb 5, 2019, 11:41 AM

@astrapogianos said in IP's are not assigned (via ISP's DHCP) to multi-WAN interfaces.:

is providing are static leased (via DHCP) from the same subnet (/22).

Yeah that is not a supported scenario from my understanding. You can not put multiple interfaces in the same network.. If you want to use multiple IPs in the same network you would use a VIP setup.

Try creating a static IP for another interface where the IP you set is in the same network as another network and it will scream at you.

So I don't see why assigning the IP via dhcp would work either.

Hmmm
https://www.netgate.com/docs/pfsense/book/firewall/methods-of-using-additional-public-ip-addresses.html

Additional IP Addresses via DHCP

Some ISPs require additional IP addresses to be obtained via DHCP. This is not a good means of obtaining multiple public IP addresses, and must be avoided in any serious network. A business-class connection should not require this. pfSense is one of the few firewalls which can be used in any capacity with additional IP addresses from DHCP. This offers limited flexibility in what the firewall can do with these addresses, leaving only two feasible options.

Pseudo multi-WAN

The only option for having the firewall pull these DHCP addresses as leases is a pseudo multi-WAN deployment. Install one network interface per public IP address, and configure each for DHCP. Plug all the interfaces into a switch between the firewall and the modem or router. Since the firewall will have multiple interfaces sharing a single broadcast domain, enable Suppress ARP messages on System > Advanced, Networking tab to eliminate ARP warnings in the system log, which are normal in this type of deployment.

The only use of multiple public IP addresses assigned in this fashion is for port forwarding. Port forwards can be used on each WAN interface that uses an IP address assigned to that interface by the ISP DHCP server. Outbound NAT to the OPT WANs will not work because of the limitation that each WAN must have a unique gateway IP address to properly direct traffic out of that WAN. This is discussed further in Multiple WAN Connections.

ASTRApogianos · Feb 5, 2019, 4:09 PM

Thank you for the info.
I have the exact same setup with the Pseudo multi-WAN except the following.
I don't have any modems or routers set but I do have a small switch as described.
Moreover, I didn't had the "Suppress ARP messages" enabled but I do now as it was the only thing missing from my setup.

What I don't understand is how this is preventing the interfaces obtaining IP's from the ISP.

johnpoz · Feb 5, 2019, 5:37 PM

It shouldn't sniff, you should should see the dhcp either the discover or the request go out, and then you should get a response from your ISP... You need to validate that pfsense is asking or not asking before you we figure out where the problem is.

I would suggest you setup so you can sniff this traffic on your switch to see exactly what is happening.

edit: When i get a chance I will duplicate such a setup on my VM pfsense.

edit2: Ok I fired this up on a pfsense Vm, added some more interfaces set to dhcp - setup some reservations for their macs, and boom they getting IPs no issues.

If I had to guess its problem with your ISP handing you your lease - simple enough to validate with a sniff.

Whats your dhcp log say... I see it asking for and getting the IPs

Here is the log from one of them for example

Feb 5 11:23:35 	dhclient 	17615 	DHCPDISCOVER on em3 to 255.255.255.255 port 67 interval 2
Feb 5 11:23:35 	dhclient 	17615 	DHCPOFFER from 192.168.2.253
Feb 5 11:23:35 	dhclient 		ARPSEND
Feb 5 11:23:37 	dhclient 		ARPCHECK
Feb 5 11:23:37 	dhclient 	17615 	DHCPREQUEST on em3 to 255.255.255.255 port 67
Feb 5 11:23:37 	dhclient 	17615 	DHCPACK from 192.168.2.253
Feb 5 11:23:37 	dhclient 		BOUND
Feb 5 11:23:37 	dhclient 		Starting add_new_address()
Feb 5 11:23:37 	dhclient 		ifconfig em3 inet 192.168.2.102 netmask 255.255.255.0 broadcast 192.168.2.255
Feb 5 11:23:37 	dhclient 		New IP Address (em3): 192.168.2.102
Feb 5 11:23:37 	dhclient 		New Subnet Mask (em3): 255.255.255.0
Feb 5 11:23:38 	dhclient 		New Broadcast Address (em3): 192.168.2.255
Feb 5 11:23:38 	dhclient 		New Routers (em3): 192.168.2.253
Feb 5 11:23:38 	dhclient 		Adding new routes to interface: em3
Feb 5 11:23:38 	dhclient 		Creating resolv.conf
Feb 5 11:23:38 	dhclient 	17615 	bound to 192.168.2.102 -- renewal in 86400 seconds.
Feb 5 11:23:39 	dhclient 		PREINIT

ASTRApogianos · Feb 5, 2019, 5:51 PM

This was my thinking from the beginning but unfortunately it's up to what they will say until otherwise can be proven, and this can be done only with some logs.
Just to give you an idea, the ISP advertises those IP's as static and not as leased. Go figure...

Can you please provide a guide on how to capture and view such logs?

johnpoz · Feb 5, 2019, 7:22 PM

Your dhcp log is right there just click status, system logs, dhcp

As to sniffing - your connected to a switch right? is it smart - then just do a span port and sniff with some pc or laptop.. Kind f hard to sniff from pfsense on bootup, etc. So you would need to do it with something connected to the switch.. if the switch your using is dumb - you can get a cheap smart switch that can do span ports for like 40$

Just to give you an idea, the ISP advertises those IP's as static and not as leased.

Huh? Sounds like miscommunication on terminology is all.. Its a LEASE.. be it you call reservation or static means nothing... If the client asks dhcp for an IP its going to be a lease.

ASTRApogianos · Feb 11, 2019, 5:30 PM

Sorry for the delay but I couldn't test the connection.
Today though we had a power outage and after restarting the server, I was able to test it and get the following logs.

Feb 11 19:14:40 dhclient FAIL
Feb 11 19:14:40 dhclient 55379 No working leases in persistent database - sleeping.
Feb 11 19:14:40 dhclient 55379 No DHCPOFFERS received.
Feb 11 19:14:36 dhclient 55379 DHCPDISCOVER on hn4 to 255.255.255.255 port 67 interval 3
Feb 11 19:14:23 dhclient 55379 DHCPDISCOVER on hn4 to 255.255.255.255 port 67 interval 13
Feb 11 19:14:23 dhclient PREINIT
Feb 11 19:14:23 dhclient Deleting old routes
Feb 11 19:14:23 dhclient EXPIRE
Feb 11 19:14:23 dhclient PREINIT

Any thoughts?

johnpoz · Feb 11, 2019, 5:55 PM

@astrapogianos said in IP's are not assigned (via ISP's DHCP) to multi-WAN interfaces.:

Feb 11 19:14:40 dhclient 55379 No DHCPOFFERS received.
Feb 11 19:14:36 dhclient 55379 DHCPDISCOVER on hn4 to 255.255.255.255 port 67 interval 3
Feb 11 19:14:23 dhclient 55379 DHCPDISCOVER on hn4 to 255.255.255.255 port 67 interval 13

Well looks like to me you sent out discover, waited and didnt get a response so it failed.. Contact your ISP with that info.

ASTRApogianos · Feb 12, 2019, 11:33 AM

I have sent them a mail with the logs I got.

Meanwhile, I searched for "No working leases in persistent database - sleeping" on Google just to see what it will show me and amazingly I found the same issue with a Linux (Ubuntu) user having the exact same issue.
The solution is drafted below.

Well thanks to everyone, but problem was actually solved by editing dhconfig.conf file: i added a string
send vendor-class-indentifier "MSFT 5.0";

Do you think that the above could provide some positive results on my issue?

johnpoz · Feb 12, 2019, 1:13 PM

depends on the dhcpd that is serving your leases..