DHCP SERVER - REBIND fail with multiple access point with same ssid

johnpoz · Sep 28, 2016, 10:24 AM

So can we please see a sniff on pfsense for this request for rebind. So you actually mean rebind not renewal. So typical renewal will be a directed packet to the original dhcp server this is t1. Once you go into t2, since you did not get a response from dhcp server on your renewal request and are trying to rebind you would be sending broadcast.

Also the mac of the client is the same, nothing funky going on where the mac is from the AP.. So your AP are they working in a cluster or standalone? I do not have much experience with those low end AP from cisco. We always use a controller with cisco and in that case the controller normally runs in dhcp proxy mode, etc. But I do not believe those wap351 can work with a controller - and not sure exactly the details of what goes on when they are setup in a cluster, etc.

So I am curious why a client would ever get to t2 and be trying to rebind vs just doing a renewal in the dhcp process? Are you saying renewals and rebinds do not work? How exactly is this presenting itself as an issue? Would not the clients that can not renew or rebind just switch to discover mode once the lease has expired and then get a new IP? So are you users complaining of intermittent connectivity? When their IP release? What is the lease time your running?

What would be fantastic would be a watching the process from a client on AP1 as it goes through a renew, then move that client to ap2 and watch what happens as it tries to renew and then rebind and finally gives up and does discover, etc.

villasoftware · Sep 28, 2016, 11:36 AM

So can we please see a sniff on pfsense for this request for rebind. So you actually mean rebind not renewal. So typical renewal will be a directed packet to the original dhcp server this is t1. Once you go into t2, since you did not get a response from dhcp server on your renewal request and are trying to rebind you would be sending broadcast.

Yess, i will reply with the pcap later as i reach it on my other pc.

Also the mac of the client is the same, nothing funky going on where the mac is from the AP.. So your AP are they working in a cluster or standalone? I do not have much experience with those low end AP from cisco. We always use a controller with cisco and in that case the controller normally runs in dhcp proxy mode, etc. But I do not believe those wap351 can work with a controller - and not sure exactly the details of what goes on when they are setup in a cluster, etc.
The mac still the same, the wap351 cisco are in cluster one of this act as cluster controller, when the device manage the 802.11r (roaming capable) everithings work fine.

So I am curious why a client would ever get to t2 and be trying to rebind vs just doing a renewal in the dhcp process? Are you saying renewals and rebinds do not work? How exactly is this presenting itself as an issue? Would not the clients that can not renew or rebind just switch to discover mode once the lease has expired and then get a new IP? So are you users complaining of intermittent connectivity? When their IP release? What is the lease time your running?
The situation is when the device (windows) moving far from AP1 to near AP2 he change his connection asking for DHCP_REQUEST not DHCP_DISCOVER on this situation PFSENSE didn't answare.

What would be fantastic would be a watching the process from a client on AP1 as it goes through a renew, then move that client to ap2 and watch what happens as it tries to renew and then rebind and finally gives up and does discover, etc.
I've the packet capture from pfsense where i can see DHCP_REQUEST many time and PFSENSE didn't replay.

So on this situation the t1 and t2 are not considered from the device because he roam from and existing SSID to the SAME SSID as reconnection so he REBING asking for DHCP_REQUEST.

If i use a ZYWALL as dhcp everithings is very fine.

I've a pcap where i've used first the zywall and the the pfsense as dhcp server, the capture is on the lan port where the wap are connected, if you need a different capture just explain me better i can capture also the traffic from wap.

Thank You
Best regards

villasoftware · Sep 28, 2016, 11:51 AM

Hello here joined the packet capture from device where:

10.0.0.1 ZYWALL USG20 used as dhcp server
10.0.0.2 PFSENSE

84:a6:c8:ee:75:91 The device connecting to wifi.

On this file both situation where first the zywall as dhcp and then the pfsense as dhcp.

ThankYou
Best regards.

packetcapture.zip

johnpoz · Sep 28, 2016, 1:41 PM

So you have both dhcp servers running at the same time??

Where did you do this sniff? I am seeing multiple networks here 10.0, 10.14, 192.168, 10.17, 10.0.2, 10.0.3

What masks are you running? Also see something with a 169.254 - so not getting an IP from dhcp or just something bogus using that as loopback? I have to run for work - but will look bit deeper at this packet capture. But I see NACK to one request that pfsense saw.. For a 192.168 address. Your not running multiple dhcp server along with multiple layer 3 networks on the same layer 2 are you??

villasoftware · Sep 28, 2016, 1:47 PM

OK,
i have one dhcp server active at time.

The capture is from pfsense, standard installation lan,wan.

the 192.168.x.x came öfrom other network before joined, the last try with 192.x.x.x is after many ignored request with the right class.

Seeml like pfsense don't like rebind so early like a protection of REBIND attack.

Thank You
Best regards

johnpoz · Sep 28, 2016, 5:38 PM

so your capture was just from the lan interface.. Why is it seeing layer 2 traffic for multiple networks? Ah your mask on your 10? /8? Why??

And dude I see both Dhcp on at the same time - you can see here you got offer from 2 different dhcp servers both 10.0.0.1 and 10.0.0.2, see the transaction ID is the same and the offers are less than 1 second apart (.64 seconds).

For starters using /8 is a bad IDEA.. That sort of mask is good for a route, or maybe a firewall rule. But as mask on a specific host - do you really think your going to have 16 million hosts on the same network?

Lets have only pfsense dhcp running.. And lets see where your client asks for lease and doesn't get a response or gets a nak? Is it sending discover or request?

villasoftware · Sep 28, 2016, 5:51 PM

Well the class 10.x.x.x came from one of the so much tries just for test purpose….

The two simultaneous i believe is a transition when i've enabled the dhcp on pfsense and disconnect the zywall, this is an hotspot of one hotel so isn't easy to have a single request on trace.

I will enable the dhcp on pfsense and disconnect the zywall, and execute the same test connecting and roaming, i will capture again from the lan side of pfsense..... will follow soon the packet capture.

Thank You

villasoftware · Sep 28, 2016, 6:31 PM

OK well the test described before is joined, the client is alwais 84:a6:c8:ee:75:91 and pfsense is 10.0.0.2.

Due to server limitation in upload size i've filtered the pcap with the mac address of the client…

Thank You
Best regards

retry.zip

johnpoz · Sep 28, 2016, 7:19 PM

Ok Im a bit confused… Seeing a bunch of discovers and requests all asking for 10.0.1.4, then I see a request for 10.17.73.107 - but where is the offer for that IP. Where did the client get this IP?? I see acks for inform packets coming from the client at 10.17.73.107 that all get answered with acks..

But I don't see any offers at all. So why did the client switch from asking for 10.0.1.4 to having an IP of 10.17.73.107 since from this sniff that IP was never offered to him. But clearly he got it from somewhere, since he sends requests and discovers requesting that IP.

So here he sends out a discover with a requested IP in it, and then a request again with the IP in it. Where did it get at that IP?? But every time he sends a inform from that IP it gets a ack back.

villasoftware · Sep 28, 2016, 7:39 PM

He get that ip from previous situation (zywall offer 10.0.1.x, pfsense offer 10.10.x.x),

but as you can see there are discover and request where pfsense never answare.

I've changed the offer range to understand where is the trouble.

The question is why so many request and discover where pfsense didn't replay ..

Thank You

PS: Keep note with zywall usg20 as dhcp server no any trouble.

johnpoz · Sep 29, 2016, 10:36 AM

I don't think he will reply if asking for IP that is not in his range, is 10.0.1.4 in his dhcp pool

Again use of such a large network /8 is NOT a Good idea!!!

Do you have the ignore checkbox checked
Denied clients will be ignored rather than rejected.

If you check that even when a nak should be sent it won't be.

You shouldn't send a nak if IP addressed for is on your local network and just not in your pool. So your network is /8 or 10.anything - if client is asking for say 10.0.1.4 and your pool is only say 10.1.0.1 to 10.2.255.254 then he shouldn't send a nak because maybe there is a failover dhcp server that will give him an IP and if I NAK than you got problems, etc.

Again I am going to stress use of /8 for a local network is BAD BAD idea.. There is no possible reason to use such a large network..

villasoftware · Sep 29, 2016, 7:59 PM

Hello,
i will change this subnet soon,

Do you have the ignore checkbox checked , Denied clients will be ignored rather than rejected.
No i don't have so i'm expecting a NAK but no answare, this is the point.

I will try to clear my notebook before retest so i believe will not be present any strange proposition of other IP,

Anyway if you check you can see also a request with IP 0.0.0.0 with no answare.

If you shutdown the laptop, wait 10 minutes and power on and connect the procedure: DISCOVER,OFFER,REQUEST,ACK is correct, the trouble is when the notebook roam to another AP on the same network with the same SSID there the notebook will not get an answare at the REQUEST no ACK no NAK at all as you see.

There is any way to have a detailed log from ISC-DHCP to better inspect this issue ?

Thank You

PS: I will change the /8 ….. :)

johnpoz · Sep 29, 2016, 11:28 PM

Again what part did you not understand that the client will dhcp server will not send an nak if the IP asked for is on the local network ie your /8 but not in the pool.. your not going to get a nak if that is the case.

I can look again but I don't recall seeing a discover or a request that did not have a requested IP in it.. like I showed you it went from asking for 10.0.1 and then 10.173 I never saw a request did not have a requested IP in it.

villasoftware · Oct 11, 2016, 7:18 PM

Hello to everybody,
i've inspected more about dhcp fails with pfsense.

Now my network is composed with multiple Cisco WAP351, these AP have integrated a SWITCH.

On the real world whenwindows connect for the first time ask for a dhcp to obtain the ip address, in this case the switch is clean, no client mac address on the table so the client will reach the router and the beside AP without problem.

The trouble born when the client switch from one AP to another, in this case windows as for a DHCP_REQUEST with the previous IP address.

Now here are the difference between PFSENSE and ZYWALL.

PFSENSE sequence is:

DHCP_REQUEST ….
DHCP_ACK

ZYWALL sequence is:

DHCP_REQUEST ...
ARP (Who has 192.168.x.x tell to router ....
ARP answare with the mac ...
DHC_ACK

The ZYWALL use the ARP protocol to doscover duplicate address, the pfsense NOT!
Also the ZYWALL answare with NAK if the client ask for a ip out of subnet, PFSENSE NOT!

The most important things is the ARP, sending the ARP request the ZYWALL let the Cysco switch update the mac address table and let the client be reachable from the ROUTER.

PFSENSE is a Great project, hope they will solve this issue one day, waiting for this i wrote my own dhcp server thath use arp as complementary protocol to check for duplicate and kick the switch.

Thank You to everybody has replay on this task.

Best regards.

johnpoz · Oct 11, 2016, 9:09 PM

What???

out of the box pfsense dhcp (dnsmasq) will try and ping for an IP before it sends an offer.. See my attached, if pfsense did not have that IP in its arp table then it would have to arp for it first.

So here you see I did a release and renew of my client with 192.168.3.100, right after the pfsense dhcp sees the offer he sends out a ping, he did not have to arp because that was in his arp table. If you want I can clear the pfsense arp table and show him arping for it as we..

[2.3.2-RELEASE][root@pfSense.local.lan]/root: arp -a | grep 192.168.3.100
? (192.168.3.100) at 00:0c:29:07:3c:fd on em3 expires in 1127 seconds [ethernet]
[2.3.2-RELEASE][root@pfSense.local.lan]/root: arp -d 192.168.3.100
192.168.3.100 (192.168.3.100) deleted
[2.3.2-RELEASE][root@pfSense.local.lan]/root: arp -a | grep 192.168.3.100
[2.3.2-RELEASE][root@pfSense.local.lan]/root:

See 2nd attachment.

Pfsense is not going to send a NAK for an IP that range that it has set on its interface you had a /8 - which again BAD idea!!! why should it send a NAK? What the IP requested is in the IP network that his dhcp server is running in.

So I moved my vm over to another vlan my lan 192.168.9/24 it got an IP 192.168.9.234, I then moved it back into the dmz vlan and had it try and renew - as you can see it asking for its IP 192.168.9.234 and pfsense sending it NAK… Sorry but that is not IP is not good here..

It then sends out a discover, not asking for any specific IP. Pfsense want to give it 192.168.3.100 which you can see it pinging for.... (only reason its ping and not arp is because that IP is already in its arp table from before.

So at a complete loss what you think is going on. But pfsense does send NAK when the IP that is asked for is not in the IP range of the interface pfsense is running its dhcp server on. But since your mask was /8 no it wouldn't send a NAK for a 10.x.x.x address. Your wanting to send a NAK because the IP is not in the pool of the dhcp server? Well maybe your running a 2nd dhcp server as failover with another part of the range of the network that pfsense dhcp server is on. So sending a nak would be bad idea in that scenario.

From attached you can see pfsense dhcp (dnsmasq) is operating exactly how one would expect a dhcp server to act. It checks if an some other client is out there with the IP before it sends an offer, and if an IP requested is from a different network it sends nak.

villasoftware · Oct 12, 2016, 9:41 AM

Well, the process you describe is correct.

just i've see thath if there is an asked IP out of the "interface" range the DHCP didn't reply with NAK.

The most blocking things on my case is the PING instead or ARP.

I've 3 access point WAP351 Cisco, where when the device move from one to another the integrated switch didn't update his table also because one of these is between two other switch.

Testing the difference between the ZYWALL that let the WIFI work perfectly and PFSENSE (just the dhcp server) i'v seen that the ARP from DHCP_REQUEST to the DHCP_ACK kick all switch to update the path.

So could be a solution on these case use ARP instead of PING because this let the switch update the routing and let the client run.

I've well tested this using the OpenDHCP and adding the ARP packet between the REQUEST and ACK, with this everythings is fine, without the client stuck.
Zywall 20 do exactly this: DHCP_REQUEST - ARP - ANSWARE_OF_ARP - ACK.

Thank You again.
Have a nice day

johnpoz · Oct 12, 2016, 10:33 AM

And again arp would all depend on if the dhcp server has the IP that is requested in its arp table. The standard is to ping..

I am at a loss to what you think there needs to be an arp or ping for in the first place. It is NOT a requirement to get an IP from a dhcp server.

Clients can quite often also test before they request and IP they get in an offer for dupes, etc. Normall process through discover, offer, request and final ack is all broadcast traffic - switches do not need any mac in their arp table to pass on this info.

If your having to rely on dupe IP detection as part of your dhcp process you have other issues that is for sure. Why/How would there be duplicates? Are users setting static IPs on their machines?

If you have switches not passing broadcast traffic then you have another sort of problem..

villasoftware · Oct 12, 2016, 12:14 PM

Actually my problem is PFSENSE !

Replacing this with ZYWALL everythings work fine.

So in your opinion ZYXEL (ZYWALL) is WRONG ? (arp instead of ping)

The Cisco WAP351 access point thath work well with ARP is WRONG ?

Shoud everybody using Cisco WAP351 AP consider of don't use PFSENSE ?

Thank You in any case for your time.

Best regards.

johnpoz · Oct 12, 2016, 12:33 PM

I didn't say it was wrong, I said and showed you that the dhcpd will send an arp if the IP is not in the cache. But the standard dhcpd option for ip dupe detection is just ping.

If a device has in its arp cache the mac of an IP, why would it arp for that IP? I don't know what you exactly your issue is.. But there is nothing that the dhcpd in pfsense that is a problem.

And going to state this again.. IP dupe detection is not a requirement or needed for proper dhcp operation.. Is your pool exhausted or asking for an existing IP that has no lease assigned also comes into play. Where in the RFC do you see that the dhcp server has to send out an arp..

What I see in rfc 2131 is this statement

In some environments it will be necessary to reassign network addresses due to exhaustion of available addresses. In such environments, the allocation mechanism will reuse addresses whose lease has expired. The server should use whatever information is available in the configuration information repository to choose an address to reuse. For example, the server may choose the least recently assigned address. As a consistency check, the allocating server SHOULD probe the reused address before allocating the address, e.g., with an ICMP echo request, and the client SHOULD probe the newly received address, e.g., with ARP.

Says the server should PING.. And the client should ARP.. Nothing saying this is some mandatory requirement of the protocol..

What I can tell you is if your switching setup or wifi is not getting an IP because there is not an arp for the IP.. Something else is going on.. Who exactly is answering this arp if the address has not been assigned yet.. What exactly does that have to do with a switch passing a broadcast packet??

villasoftware · Oct 12, 2016, 6:35 PM

Well i try to explain better my trouble.

My installation is:

IBM-SERVER with ESXi where PFSENSE is main firewall for an HOTEL.
4 managed SWITCH ZYXEL
3 Access point Cisco WAP351

Pfsense is a great FIREWALL well working serving VOIP trunk with about 30 extension, 4 vlan handling fire alarm, local network, guest access, Credit card.

Everithings work perfectly except this stupid Cisco Wifi WAP351.

The Wap 351 network is :

SWITCH - WAP1 -WAP2

WAP3

Now if i use ZYWALL as dhcp server leasing the address of pfsense for dns and gatewai is all perfect, if i take out the zywall ans use the internal dhcp of pfsense the wifi suffer this trouble:

suppose to start from cold start, all table clean

The device connect to the AP1, this receive the IP and pfsense ARP for it.

At this time all device work great

The device move from AP1 to AP2, this device ask for DHCP_REQUEST with the previous IP, PFSENSE ACK with this ip.

At this time The device is insulated from anythings !!

I've made many many test and discussion with Cisco support without any solution.
Finally i've discovered thath the ZYWALL between DHCP_REQUEST and ACK send an ARP.

I've made some test usign the OpenDHCP and adding a code to send the ARP (who has tell) between REQUEST and ACK even the device is already known.

This let these stupid WAP351 (Cisco) work very well.

Now i can be really with you sayng these device are buggy because need kick the table with ARP message.

Now we can say ZYWALL USG is an accepted standard in the industry and this damn WAP351 run well with this.

Now the rfc as you say describe PING instead of ARP, on my side i don't have many solution, i can replace the WAP351 or use a different DHCP server.

One possible solution consider ZYWALL procedure is add a checkbox in the PFSENSE that let this always send an ARP message for these buggy device.

As i sayd i very like this PFSENSE as firewall is really well working except for this issue, yess isn't a fault of PFSENSE but finally i can't use this .

Today Cisco deliver a lot of access point with multiple ssid and vlan enought cheap about 250$ each and the intergrated switch for some installation is a good solution.

So by the way is someone need these access point can't use PFSENSE until he can't send a arp message.

Again ZYWALL send the ARP message anytime he receive the DHCP_REQUEST, is out of standard maybe yess but work.

Really thank you for your time, again a great job, i very like the new interface, looks very professional and as i said the voip is really well working with it, consider this hotel handle about 20000 call /year.

Best regards.