[SOLVED] 2 SSID with different subnets - DHCP NAK

pszafer

Hi,
We have 2 offices with same SSID of wlan network, both with radius with dynamic VLAN assigned by Windows NPS.
On both sites there are pfsense routers with DHCP server on each.

Site 1 - 10.11.0.0/24 - Unifi - VLAN15
Site 2 - 10.16.0.0/24 - Cisco - VLAN15 (vlan shouldn't matter as those devices don't see each other).

In one office there is Unifi hardware, in second Cisco AP's.
On one laptop with Linux system which is using Gnome NetworkManager to connect beginning this year there is a problem.
There is problem only when moving.

Going to Site2 it is doing:

DHCP Discover
DHCP Request 10.16.0.x

Going from Site2 to Site1 it is doing:

DHCP Request with IP 10.16.0.x and it gets DHCP NAK.
it never tries to do DHCP Discover.

Forgetting network, connecting to Site1 it is doing:

DHCP Discover
DHCP Request 10.11.0.x

When going from Site1 to Site2 - didn't check yet what it is doing with tcpdump but it successfully change IP address from 10.11.0.x to 10.16.0.x.

Any idea if this is something with NetworkManager or something wrong with pfSense configuration.
I don't think there where any changes in configuration that I'm aware of in pfsense routers or laptop itself.

JKnott

@pszafer said in 2 SSID with different subnets - DHCP NAK:

DHCP Request with IP 10.16.0.x and it gets DHCP NAK.
it never tries to do DHCP Discover.

That sounds like a DHCP problem on that laptop. When it gets the NAK, it should try to discover. Do any other devices exhibit the same behaviour?

pszafer

No, but all other devices travelling across places are Windows.
I don't understand why it can move from Site1 to Site2, but not from Site2 to Site1...

Now I'm thinking if it is like it, it might be Unifi is blocking some DHCP requests after it sees DHCP NAK.

JKnott

@pszafer

Well, it's time to start some packet captures. PfSense has Packet Capture built in and you can run Wireshark on the laptop. However, if that Unifi is blocking something, it should also affect Windows.

Perhaps you can set up a test lab with a couple of APs or routers to see if you can replicate the issue.

johnpoz

@pszafer said in 2 SSID with different subnets - DHCP NAK:

and it gets DHCP NAK.

You know this how? Log, sniff on the dhcpd? Did you validate that the client actually gets the NAK?

As JKnott stated already when the client sees the nak, it should then send a discover.. Possible it does see the nak, and does send the discover but the dhcpd is not seeing the discover. Or the client is never getting the nak?

But even if the client never gets the nak, after its request is not answered - after a few attempts, it should revert to sending discover..

More info for sure is needed. A sniff on the dhcpd and the client, at the same time while this happens would be very insightful to where the problem is.

pszafer

You know this how? Log, sniff on the dhcpd? Did you validate that the client actually gets the NAK?

I know it from tcpdump. I couldn't validate it on client itself, but Unifi controller confirms in webui that client got DHCP NAK.

As JKnott stated already when the client sees the nak, it should then send a discover.. Possible it does see the nak, and does send the discover but the dhcpd is not seeing the discover. Or the client is never getting the nak?

But even if the client never gets the nak, after its request is not answered - after a few attempts, it should revert to sending discover..

More info for sure is needed. A sniff on the dhcpd and the client, at the same time while this happens would be very insightful to where the problem is.

Tcpdump on router side sees only DHCP request all over again. No DHCP Discover at all from this client as long as I don't delete file

/var/lib/NetworkManager/internal-*-wlan0.lease

and restart NetworkManager service.
Tomorrow morning I will try to sniff traffic from client side and I will get back to you.

Thank you for your help!

pszafer

NetworkManager doesn't try to do DHCP Discover after NAK.

It looks like this:

Frame 30: 333 bytes on wire (2664 bits), 333 bytes captured (2664 bits) on interface wlp4s0, id 0
Ethernet II, Src: MAC, Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Internet Protocol Version 4, Src: 0.0.0.0, Dst: 255.255.255.255
User Datagram Protocol, Src Port: 68, Dst Port: 67
Dynamic Host Configuration Protocol (Request)
    Message type: Boot Request (1)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x65cd814a
    Seconds elapsed: 0
    Bootp flags: 0x0000 (Unicast)
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: IntelCor_MAC (MAC)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (Request)
    Option: (61) Client identifier
    Option: (55) Parameter Request List
    Option: (57) Maximum DHCP Message Size
    Option: (50) Requested IP Address (10.16.0.41)
    Option: (12) Host Name
    Option: (255) End

and response:

Dynamic Host Configuration Protocol (NAK)
    Message type: Boot Reply (2)
    Hardware type: Ethernet (0x01)
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x65cd814a
    Seconds elapsed: 0
    Bootp flags: 0x8000, Broadcast flag (Broadcast)
    Client IP address: 0.0.0.0
    Your (client) IP address: 0.0.0.0
    Next server IP address: 0.0.0.0
    Relay agent IP address: 0.0.0.0
    Client MAC address: IntelCor_MAC (MAC)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: DHCP
    Option: (53) DHCP Message Type (NAK)
    Option: (54) DHCP Server Identifier (10.11.0.1)
    Option: (56) Message
    Option: (255) End
    Padding: 0000000000000000000000000000000000

and no DHCP traffic later at all. Some ICMPv6 traffic and MDNS.
Seems for me to some issue with NetworkManager and it's dhcp client, what do you think?

Found an issue with NetworkManager - https://gitlab.freedesktop.org/NetworkManager/NetworkManager/issues/325

johnpoz

You validated that client is getting the NAK? Seeing that the dhcpd sent it is only half the battle.. When a client doesn't get a nak.. It could assume that the IP is still good and will continue to try and use it.

but Unifi controller confirms in webui that client got DHCP NAK

How is that?? I have unifi controller - and have never seen any info it at all about dhcp anywhere

So that link looks like there is a bug on your linux networkmanager - looks like it is fixed from post a day ago with specific commit.
"This should be fixed by !387."

pszafer

I see it. It will be available any minute now. It is already in main repo https://www.archlinux.org/packages/extra/x86_64/networkmanager/ and I'm waiting when it will be available on all mirrors.

@johnpoz about Unifi
I have DHCP Snooping enabled and if you get DHCP NAK, then it shows up in Topology map.Instead of green line it is red, with info about DHCP NAK.

Thank you again for your help!

johnpoz

Ah - yeah I don't have that on... See no point to it in my setup. But will check it out thanks!

You have unifi switches? I don't so that setting makes no sense in my setup.

edit: But your saying these would show red?

And if you highlight them or something shows that there was a nak.. From my quick look into dhcp snooping on unifi, other than lots of people complaining about bugs with it.. You need to have unifi switches it seems, nothing done on the AP.

pszafer

@johnpoz I had to wait to see red line and get back to you.
Anyway I see red line and info that there is DHCP Timeout. Check dhcp settings on your router. On pfsense I see DHCP NAK.
I don't have Unifi switches.

johnpoz

Where do you have this snooping enabled exactly? My understanding does nothing if you don't have their switches, same with the dhcp guarding.

pszafer

@johnpoz

Settings -> Site -> Services section -> DHCP Snooping

On option which requires Unifi Gateway there is info USG REQUIRED.

johnpoz

Ok no switches, but a USG is required - don't have that.. Do you, thought you were running pfsense?

pszafer

@johnpoz this feature doesn't need USG. I have only 3 Unifi AP-AC-Pro, nothing else from Unifi.

johnpoz

I don't see that anywhere in the new ui.. But I find it on the old ui..

It is enabled on mine, I think its default... But I find it unlikely the AP would do anything with that setting. I have never seen such an issue - All of my ssids are different IPs.. I have devices that move between these networks without issue, I have moved laptop from wired to wireless - again different vlans, never an issue..

So lets start again... On the client sniffing do you see the offer? Do you see the nak? Do you see it send a discover after nak?

On the wire before it goes to the AP do you see the nak, do you see an offer, etc..

What is your dhcp server? Pfsense, the controller? Something else?

If you think unifi is doing something with the dhcp traffic that you don't want it to - then you should get with them about the problem. If pfsense is your dhcpd, and you see it send the nak, there is nothing else pfsense could do if the client doesn't see that, or doesn't then do a discover, etc.

pszafer

I think there is misunderstanding. I edited subject to SOLVED, but I've should post info, that fix included in NetworkManager 1.22.4 solves this problem, it doesn't exists anymore.

I just wanted to reply to you what I see in Unifi web ui.

johnpoz

Ok so was that client bug then.. Great..

Well like I said I can not find that on the "beta" ui, but I do see it on the classic view.. I do not recall ever enabling that, nor would I have reason to.. And some threads I found people complaining that it is on by default, etc..

Glad you got it sorted in the end..