DHCP SERVER - REBIND fail with multiple access point with same ssid

johnpoz · Sep 29, 2016, 11:28 PM

Again what part did you not understand that the client will dhcp server will not send an nak if the IP asked for is on the local network ie your /8 but not in the pool.. your not going to get a nak if that is the case.

I can look again but I don't recall seeing a discover or a request that did not have a requested IP in it.. like I showed you it went from asking for 10.0.1 and then 10.173 I never saw a request did not have a requested IP in it.

villasoftware · Oct 11, 2016, 7:18 PM

Hello to everybody,
i've inspected more about dhcp fails with pfsense.

Now my network is composed with multiple Cisco WAP351, these AP have integrated a SWITCH.

On the real world whenwindows connect for the first time ask for a dhcp to obtain the ip address, in this case the switch is clean, no client mac address on the table so the client will reach the router and the beside AP without problem.

The trouble born when the client switch from one AP to another, in this case windows as for a DHCP_REQUEST with the previous IP address.

Now here are the difference between PFSENSE and ZYWALL.

PFSENSE sequence is:

DHCP_REQUEST ….
DHCP_ACK

ZYWALL sequence is:

DHCP_REQUEST ...
ARP (Who has 192.168.x.x tell to router ....
ARP answare with the mac ...
DHC_ACK

The ZYWALL use the ARP protocol to doscover duplicate address, the pfsense NOT!
Also the ZYWALL answare with NAK if the client ask for a ip out of subnet, PFSENSE NOT!

The most important things is the ARP, sending the ARP request the ZYWALL let the Cysco switch update the mac address table and let the client be reachable from the ROUTER.

PFSENSE is a Great project, hope they will solve this issue one day, waiting for this i wrote my own dhcp server thath use arp as complementary protocol to check for duplicate and kick the switch.

Thank You to everybody has replay on this task.

Best regards.

johnpoz · Oct 11, 2016, 9:09 PM

What???

out of the box pfsense dhcp (dnsmasq) will try and ping for an IP before it sends an offer.. See my attached, if pfsense did not have that IP in its arp table then it would have to arp for it first.

So here you see I did a release and renew of my client with 192.168.3.100, right after the pfsense dhcp sees the offer he sends out a ping, he did not have to arp because that was in his arp table. If you want I can clear the pfsense arp table and show him arping for it as we..

[2.3.2-RELEASE][root@pfSense.local.lan]/root: arp -a | grep 192.168.3.100
? (192.168.3.100) at 00:0c:29:07:3c:fd on em3 expires in 1127 seconds [ethernet]
[2.3.2-RELEASE][root@pfSense.local.lan]/root: arp -d 192.168.3.100
192.168.3.100 (192.168.3.100) deleted
[2.3.2-RELEASE][root@pfSense.local.lan]/root: arp -a | grep 192.168.3.100
[2.3.2-RELEASE][root@pfSense.local.lan]/root:

See 2nd attachment.

Pfsense is not going to send a NAK for an IP that range that it has set on its interface you had a /8 - which again BAD idea!!! why should it send a NAK? What the IP requested is in the IP network that his dhcp server is running in.

So I moved my vm over to another vlan my lan 192.168.9/24 it got an IP 192.168.9.234, I then moved it back into the dmz vlan and had it try and renew - as you can see it asking for its IP 192.168.9.234 and pfsense sending it NAK… Sorry but that is not IP is not good here..

It then sends out a discover, not asking for any specific IP. Pfsense want to give it 192.168.3.100 which you can see it pinging for.... (only reason its ping and not arp is because that IP is already in its arp table from before.

So at a complete loss what you think is going on. But pfsense does send NAK when the IP that is asked for is not in the IP range of the interface pfsense is running its dhcp server on. But since your mask was /8 no it wouldn't send a NAK for a 10.x.x.x address. Your wanting to send a NAK because the IP is not in the pool of the dhcp server? Well maybe your running a 2nd dhcp server as failover with another part of the range of the network that pfsense dhcp server is on. So sending a nak would be bad idea in that scenario.

From attached you can see pfsense dhcp (dnsmasq) is operating exactly how one would expect a dhcp server to act. It checks if an some other client is out there with the IP before it sends an offer, and if an IP requested is from a different network it sends nak.

villasoftware · Oct 12, 2016, 9:41 AM

Well, the process you describe is correct.

just i've see thath if there is an asked IP out of the "interface" range the DHCP didn't reply with NAK.

The most blocking things on my case is the PING instead or ARP.

I've 3 access point WAP351 Cisco, where when the device move from one to another the integrated switch didn't update his table also because one of these is between two other switch.

Testing the difference between the ZYWALL that let the WIFI work perfectly and PFSENSE (just the dhcp server) i'v seen that the ARP from DHCP_REQUEST to the DHCP_ACK kick all switch to update the path.

So could be a solution on these case use ARP instead of PING because this let the switch update the routing and let the client run.

I've well tested this using the OpenDHCP and adding the ARP packet between the REQUEST and ACK, with this everythings is fine, without the client stuck.
Zywall 20 do exactly this: DHCP_REQUEST - ARP - ANSWARE_OF_ARP - ACK.

Thank You again.
Have a nice day

johnpoz · Oct 12, 2016, 10:33 AM

And again arp would all depend on if the dhcp server has the IP that is requested in its arp table. The standard is to ping..

I am at a loss to what you think there needs to be an arp or ping for in the first place. It is NOT a requirement to get an IP from a dhcp server.

Clients can quite often also test before they request and IP they get in an offer for dupes, etc. Normall process through discover, offer, request and final ack is all broadcast traffic - switches do not need any mac in their arp table to pass on this info.

If your having to rely on dupe IP detection as part of your dhcp process you have other issues that is for sure. Why/How would there be duplicates? Are users setting static IPs on their machines?

If you have switches not passing broadcast traffic then you have another sort of problem..

villasoftware · Oct 12, 2016, 12:14 PM

Actually my problem is PFSENSE !

Replacing this with ZYWALL everythings work fine.

So in your opinion ZYXEL (ZYWALL) is WRONG ? (arp instead of ping)

The Cisco WAP351 access point thath work well with ARP is WRONG ?

Shoud everybody using Cisco WAP351 AP consider of don't use PFSENSE ?

Thank You in any case for your time.

Best regards.

johnpoz · Oct 12, 2016, 12:33 PM

I didn't say it was wrong, I said and showed you that the dhcpd will send an arp if the IP is not in the cache. But the standard dhcpd option for ip dupe detection is just ping.

If a device has in its arp cache the mac of an IP, why would it arp for that IP? I don't know what you exactly your issue is.. But there is nothing that the dhcpd in pfsense that is a problem.

And going to state this again.. IP dupe detection is not a requirement or needed for proper dhcp operation.. Is your pool exhausted or asking for an existing IP that has no lease assigned also comes into play. Where in the RFC do you see that the dhcp server has to send out an arp..

What I see in rfc 2131 is this statement

In some environments it will be necessary to reassign network addresses due to exhaustion of available addresses. In such environments, the allocation mechanism will reuse addresses whose lease has expired. The server should use whatever information is available in the configuration information repository to choose an address to reuse. For example, the server may choose the least recently assigned address. As a consistency check, the allocating server SHOULD probe the reused address before allocating the address, e.g., with an ICMP echo request, and the client SHOULD probe the newly received address, e.g., with ARP.

Says the server should PING.. And the client should ARP.. Nothing saying this is some mandatory requirement of the protocol..

What I can tell you is if your switching setup or wifi is not getting an IP because there is not an arp for the IP.. Something else is going on.. Who exactly is answering this arp if the address has not been assigned yet.. What exactly does that have to do with a switch passing a broadcast packet??

villasoftware · Oct 12, 2016, 6:35 PM

Well i try to explain better my trouble.

My installation is:

IBM-SERVER with ESXi where PFSENSE is main firewall for an HOTEL.
4 managed SWITCH ZYXEL
3 Access point Cisco WAP351

Pfsense is a great FIREWALL well working serving VOIP trunk with about 30 extension, 4 vlan handling fire alarm, local network, guest access, Credit card.

Everithings work perfectly except this stupid Cisco Wifi WAP351.

The Wap 351 network is :

SWITCH - WAP1 -WAP2

WAP3

Now if i use ZYWALL as dhcp server leasing the address of pfsense for dns and gatewai is all perfect, if i take out the zywall ans use the internal dhcp of pfsense the wifi suffer this trouble:

suppose to start from cold start, all table clean

The device connect to the AP1, this receive the IP and pfsense ARP for it.

At this time all device work great

The device move from AP1 to AP2, this device ask for DHCP_REQUEST with the previous IP, PFSENSE ACK with this ip.

At this time The device is insulated from anythings !!

I've made many many test and discussion with Cisco support without any solution.
Finally i've discovered thath the ZYWALL between DHCP_REQUEST and ACK send an ARP.

I've made some test usign the OpenDHCP and adding a code to send the ARP (who has tell) between REQUEST and ACK even the device is already known.

This let these stupid WAP351 (Cisco) work very well.

Now i can be really with you sayng these device are buggy because need kick the table with ARP message.

Now we can say ZYWALL USG is an accepted standard in the industry and this damn WAP351 run well with this.

Now the rfc as you say describe PING instead of ARP, on my side i don't have many solution, i can replace the WAP351 or use a different DHCP server.

One possible solution consider ZYWALL procedure is add a checkbox in the PFSENSE that let this always send an ARP message for these buggy device.

As i sayd i very like this PFSENSE as firewall is really well working except for this issue, yess isn't a fault of PFSENSE but finally i can't use this .

Today Cisco deliver a lot of access point with multiple ssid and vlan enought cheap about 250$ each and the intergrated switch for some installation is a good solution.

So by the way is someone need these access point can't use PFSENSE until he can't send a arp message.

Again ZYWALL send the ARP message anytime he receive the DHCP_REQUEST, is out of standard maybe yess but work.

Really thank you for your time, again a great job, i very like the new interface, looks very professional and as i said the voip is really well working with it, consider this hotel handle about 20000 call /year.

Best regards.

johnpoz · Oct 13, 2016, 8:50 AM

To your point #2, why would client moving from AP1 to AP2 ask for new IP? Why would he send request? Are you saying he had just happen to roam to AP at the time of lease renewal?

" At this time The device is insulated from anythings !!"

What does that mean.. You mean the client got his renewal but can not talk to anything?

Why would dhcpd have to send an arp?? That doesn't even get answered for for the offer to be sent and get through to your client asking? Makes ZERO sense!!!

So your AP2 is daisy chained off the first AP1 using what interface? Lan 2 I assume are you powering the AP2 via poe? Or you using one of the switch ports? Your not using the captive portal in in the AP are you? Your networks are all the same are you doing vlan tagging? If you daisychain can you config the ports for the uplink to the other AP2 to be trunked?

Why can you not just connect AP2 direct to the switch? So do you have this same problem when you move from AP1 to AP3, or AP3 to AP1 ?? Or only when client moves to AP2?? If client stays on AP1 he can not renew his lease? What is your lease time?

As to your ZYWALL sending arp.. He sends arp because he does not have the IP cached in his arp cache? What is the length of arp cache in the zywall? Pfsense has a fairly long arp cache.. Like 20 minutes..

villasoftware · Oct 13, 2016, 9:00 AM

#2, i have 3 AP, when the client phosically move far from AP1 and reach the AP2 on wireshark i can see it connect to AP2 and ask for DHCP_REQUEST (rebind).

THE WAP351 is a Cisco accesspoint thath have 5 LAN interface integrated switch, so the 3 WAP are on serial, the LAN of these switch are defined trunking and i vahe 3 ssid with 3 vlan.

Insulated meant when teh client move from AP1 to AP2 at this time i can ping only the AP2, no way to ping any other things on the network, if the router send a ARP request between REQUEST and ACk everytings work fine, the client can move from ap1 to ap2 and back withouth problem.

The WAP351 (Access point) are clear, no captive portal.

If i connect AP2 and AP1 to one switch i have the same problem, already play a lot with cable and switches. The switches are ZYXEl managed switch.

The zywall send ALWAYS arp any time he get the DHCP_REQUEST !!

When i try the network i move from AP1 to AP2 between a minute so is not a case of zywall arp cache.

Now is possible to reduce the ARP cache time on the PFSENSE ?

Thank You
Best regards.

villasoftware · Oct 13, 2016, 9:45 AM

#2, when the client move fro AP1 to AP2 he didn't release the ip, he just connect to the new AP and ask for a DHCP_REQUEST with the last IP got. (WINDOWS 7)

With OSX and IOS (MAC and tablet) he do somethings more, after he got the address he send an "grazious ARP" informing of his new address, and in this way he work well also with pfsense because he kick the switch by hisself !

Thank You.

johnpoz · Oct 13, 2016, 10:00 AM

"THE WAP351 is a Cisco accesspoint thath have 5 LAN interface integrated switch,"

Not really sure that is the case.. It has lan1 is the PD port and lan2 is the PSE port and then switch ports lan3-5..

So here is the thing.. What does the dhcp renewal have to do with it?? And your saying he gets that? Here I will move my phone from one AP to another AP and looking for dhcp requests on that vlan..

Ok so see attached pic… So I took my phone connected to AP in the kitchen.. I moved back and forth a few times, I finally did a capture where I got all the stuff I wanted to show in it. So you can see clearly my wifi controller showing that my phone moved between AP a few times.

So in that first part of the sniff you can see where I auth to the AP, I then send out a discover to get an IP.. I didn't see pfsense send out any pings or arps here - My guess to this would be that I have a dhcp reservation for my phone. So there shouldn't never be a reason to check for dupe IP and should always have that in his arp cache, etc. I then started a ping on the phone to 8.8.8.8.. I then moved into the back bedroom so that phone would switch over. YOu can see where AP 192.168.2.4 (guestroom) sends auth to my radius server.. Client gets accepted, and then he sends dhcp request, and gets back his ack right away.. Notice ping is still going through this whole process. The client never missed a ping moving from 1 AP to the other AP..

If I had to guess your problem its more to where your AP and switches think the mac is.. If all your saying you can talk to is the AP your connected too... Then your switch is prob got wrong arp cached entry to where to send the traffic, etc. Maybe that arp is clearly your switch cache???

Would love to be there and be able to troubleshoot this myself. BTW my 2 AP that I roamed between are on 2 different switches as well.

But I really don't see how dhcp really has anything to do with your problem of loosing connectivity when you roam from 1 AP to another AP..

villasoftware · Oct 13, 2016, 10:14 AM

Please try with Windows 7 not ipnohe

johnpoz · Oct 13, 2016, 10:27 AM

What would it matter what device it is?? But sure I can fire up my wife's laptop… Ok so see it connected to AP in guestroom... Then moved it to front of the house and roamed to kitchen AP just like my phone did. What would it matter what the client is??

villasoftware · Oct 13, 2016, 11:08 AM

For sure my trouble is the Cisco WAP351, he has a arp cache and use it for deliver the packet.

Now with Iphone i can see he send a GRACIOUS ARP autonomouse that make the deifference, with this the switch of the wap351 undestand where he should send or block the packet.

I've teste with iphone and macbook pro is fine (always) with windows 7 enterprise and pro NOT. This never send a Graciuous arp message autonomous.

The true is ZYWALL every time send this ARP (who as tell) thath in fact let the WAP351 work well, like sayd i've tested the OpenDHCP instead of PFSENSE DHCP server and got the same issue, after i've add on the source code (and recompile) the ARP request these AP work well also with windows 7.

Today i still using pfsense as main firewall and this zywall just as dhcp server delivering the address of pfsense as router and dns, on this situation everithings is fine, BUT i need 3 wifi network so i should use more than one zywall of buy somethings like zywall 200, but in this case pfsense will be replaced by this….

Please try to capture the traffic durring dhcp with wireshark, filter with "bootp and arp" and you will see the difference between IOS and Windows7.

I believe the only solution will be arrange a new virtual machine with another dhcp server instead of use thath one of pfsense, is a stupid solution but works!, hope one day i can use just pfsense with these ap because many times when you need to install these ap the internal switch is a good idea for cable reduction, guess an hotel with all room inline just interconnect all ap toghether...

Best regards.