VoiP LAN device stuck on DHCP renewal

gryest

Yes, I know and I can't capture same situation with DD-WRT.
Actually this capture done by pfSense "packet capture" function and opened by wireshark. Port mirroring I was using with DD-WRT setup.

"BTW, if you filter on DHCP, you won't get those pings cluttering things up." - not sure what you mean. I ran capture full detail on purpose do not miss anything.

JKnott

If you're trying to resolve a problem with a protocol, it generally helps to capture only that protocol. Packet Capture will allow you to do that.

gryest

@JKnott:

If you're trying to resolve a problem with a protocol, it generally helps to capture only that protocol. Packet Capture will allow you to do that.

OK, understood your point, thanks. Here is same moment with DD-WRT captured with switch mirrored port and Wireshark but VoiP controller remains online after following sequence:

–----------------------------------------------------------------------------------------------------------------------------------------
No. Time Source Destination Protocol Length Info
584 740.807538000 0.0.0.0 255.255.255.255 DHCP 302 DHCP Discover - Transaction ID 0xbe67eb

Frame 584: 302 bytes on wire (2416 bits), 302 bytes captured (2416 bits) on interface 0
Ethernet II, Src: VoiP_aa_00_bb_22 (00:25:f6:00:bb:22), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Internet Protocol Version 4, Src: 0.0.0.0 (0.0.0.0), Dst: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67)
Bootstrap Protocol (Discover)

No. Time Source Destination Protocol Length Info
585 740.837757000 D-LinkIn_e1:f2:9a Broadcast ARP 60 Who has 192.168.70.102? Tell 192.168.70.1

Frame 585: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: D-LinkIn_e1:f2:9a (c0:a0:bb:e1:f2:9a), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request)

No. Time Source Destination Protocol Length Info
586 741.835987000 D-LinkIn_e1:f2:9a Broadcast ARP 60 Who has 192.168.70.102? Tell 192.168.70.1

Frame 586: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: D-LinkIn_e1:f2:9a (c0:a0:bb:e1:f2:9a), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request)

No. Time Source Destination Protocol Length Info
587 742.891015000 D-LinkIn_e1:f2:9a Broadcast ARP 60 Who has 192.168.70.102? Tell 192.168.70.1

Frame 587: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: D-LinkIn_e1:f2:9a (c0:a0:bb:e1:f2:9a), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request)

No. Time Source Destination Protocol Length Info
588 743.424168000 169.254.255.1 255.255.255.255 DHCP 342 DHCP Offer - Transaction ID 0xbe67eb

Frame 588: 342 bytes on wire (2736 bits), 342 bytes captured (2736 bits) on interface 0
Ethernet II, Src: D-LinkIn_e1:f2:9a (c0:a0:bb:e1:f2:9a), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Internet Protocol Version 4, Src: 169.254.255.1 (169.254.255.1), Dst: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: 67 (67), Dst Port: 68 (68)
Bootstrap Protocol (Offer)

No. Time Source Destination Protocol Length Info
589 743.424341000 VoiP_aa_00_bb_22 Broadcast ARP 60 Gratuitous ARP for 192.168.70.102 (Request)

Frame 589: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: VoiP_aa_00_bb_22 (00:25:f6:00:bb:22), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request/gratuitous ARP)

No. Time Source Destination Protocol Length Info
590 743.424353000 VoiP_aa_00_bb_22 Broadcast ARP 60 Gratuitous ARP for 192.168.70.102 (Reply)

Frame 590: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: VoiP_aa_00_bb_22 (00:25:f6:00:bb:22), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (reply/gratuitous ARP)

No. Time Source Destination Protocol Length Info
591 743.424575000 192.168.70.1 192.168.70.102 ICMP 62 Echo (ping) request id=0x24d2, seq=0/0, ttl=64 (no response found!)

Frame 591: 62 bytes on wire (496 bits), 62 bytes captured (496 bits) on interface 0
Ethernet II, Src: D-LinkIn_e1:f2:9a (c0:a0:bb:e1:f2:9a), Dst: VoiP_aa_00_bb_22 (00:25:f6:00:bb:22)
Internet Protocol Version 4, Src: 192.168.70.1 (192.168.70.1), Dst: 192.168.70.102 (192.168.70.102)
Internet Control Message Protocol

No. Time Source Destination Protocol Length Info
592 743.424768000 0.0.0.0 255.255.255.255 DHCP 314 DHCP Request - Transaction ID 0xbe67eb

Frame 592: 314 bytes on wire (2512 bits), 314 bytes captured (2512 bits) on interface 0
Ethernet II, Src: VoiP_aa_00_bb_22 (00:25:f6:00:bb:22), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Internet Protocol Version 4, Src: 0.0.0.0 (0.0.0.0), Dst: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: 68 (68), Dst Port: 67 (67)
Bootstrap Protocol (Request)

No. Time Source Destination Protocol Length Info
593 743.479401000 169.254.255.1 255.255.255.255 DHCP 342 DHCP ACK - Transaction ID 0xbe67eb

Frame 593: 342 bytes on wire (2736 bits), 342 bytes captured (2736 bits) on interface 0
Ethernet II, Src: D-LinkIn_e1:f2:9a (c0:a0:bb:e1:f2:9a), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Internet Protocol Version 4, Src: 169.254.255.1 (169.254.255.1), Dst: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: 67 (67), Dst Port: 68 (68)
Bootstrap Protocol (ACK)

No. Time Source Destination Protocol Length Info
594 743.662369000 VoiP_aa_00_bb_22 Broadcast ARP 60 Who has 192.168.70.1? Tell 192.168.70.102

Frame 594: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: VoiP_aa_00_bb_22 (00:25:f6:00:bb:22), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
Address Resolution Protocol (request)

No. Time Source Destination Protocol Length Info
595 743.662570000 D-LinkIn_e1:f2:9a VoiP_aa_00_bb_22 ARP 60 192.168.70.1 is at c0:a0:bb:e1:f2:9a

Frame 595: 60 bytes on wire (480 bits), 60 bytes captured (480 bits) on interface 0
Ethernet II, Src: D-LinkIn_e1:f2:9a (c0:a0:bb:e1:f2:9a), Dst: VoiP_aa_00_bb_22 (00:25:f6:00:bb:22)
Address Resolution Protocol (reply)

JKnott

OK, understood your point, thanks. Here is same moment with DD-WRT captured with switch mirrored port and Wireshark but VoiP controller remains online after following sequence:

You're doing it again. Please filter on DHCP, so that you don't get all that clutter. A DHCP sequence should be easy to follow. If you filter on it, you'll see the discover, offer, request and ack all in order, without anything else. With all that ARP & ICMP stuff in there, I now have to search for only the DHCP stuff. If you're trying to solve a DHCP problem, you should be focused only on DHCP. You'd only be interested in the ARP traffic, if you're trying to find out why offered addresses are being rejected. In this instance, the ARPs are used for duplicate address detection. which could cause an offered address to be rejected. IN that capture, I see 12 packets, of which only 4 are relevant.

Also, what's this about with the link local address? Does that D-Link have the address 192.168.70.1? Or 169.254.255.1?

No. Time Source Destination Protocol Length Info
593 743.479401000 169.254.255.1 255.255.255.255 DHCP 342 DHCP ACK - Transaction ID 0xbe67eb

The dhcp offer and ack, from the D-Link have a source address of 169.254.255.1, but other traffic uses 192.168.70.1. In fact, there shouldn't be any DHCP traffic with link local addresses, as they're only supposed to be used when no DHCP server is available.

Also, I hope you're capturing at the same point. That is a mirror port of the managed switch. If you're not doing that you're adding variables that may affect the results.

Incidentally, a while ago, I bought a cheap managed switch just for this purpose. I have port 2 mirrored to port 1 and I can then insert the switch between 2 devices and plug a computer running Wireshark into port 1 to monitor the traffic.

BTW, when I use Wireshark, I use both capture and display filters as needed to get the desired results. I much prefer Wireshark to Packet Capture, as it's much more flexible and can be viewed in real time. However, I have used Packet Capture many times and exported to Wireshark for analysis, as it displays more detail.

Also, it would be helpful to attach the actual capture file, as it shows more detail than what you show.

gryest

@JKnott:

OK, understood your point, thanks. Here is same moment with DD-WRT captured with switch mirrored port and Wireshark but VoiP controller remains online after following sequence:

You're doing it again. Please filter on DHCP, so that you don't get all that clutter. A DHCP sequence should be easy to follow. If you filter on it, you'll see the discover, offer, request and ack all in order, without anything else. With all that ARP & ICMP stuff in there, I now have to search for only the DHCP stuff.
Also, what's this about with the link local address? Does that D-Link have the address 192.168.70.1? Or 169.254.255.1?

No. Time Source Destination Protocol Length Info
593 743.479401000 169.254.255.1 255.255.255.255 DHCP 342 DHCP ACK - Transaction ID 0xbe67eb

The dhcp offer and ack, from the D-Link have a source address of 169.254.255.1, but other traffic uses 192.168.70.1. In fact, there shouldn't be any DHCP traffic with link local addresses, as they're only supposed to be used when no DHCP server is available.

Also, it would be helpful to attach the actual capture file, as it shows more detail than what you show.

I posted that fragment on purpose and 169.254.255.1 is mystery for me too. Router here is 192.168.70.1. Most of DHCP traffic with DD-WRT and pfSense is like: DHCP Request ->DHCP ACK between lease expiration. I was trying to find clue when renewal is not successful or have any hiccup which are very rare in DD-WRT case.

Looks like I need to repeat capture with pfSense and Wireshark+mirrored port. The good news are it's very repetitive failure.
Thanks.

gryest

Here is attached files:
1. filtered DHCP capture with pfSense and Wireshark+mirrored port. Lease time set to 3600 sec in pfSense
2. pfSense DHCP syslog at the same time

Packets 3, 5, 8, 9 - VoiP soft restart by dealing restart code from attached phone. Looks like legitimate DHCP renewal.
Somewhere after packet #592 VoiP controller going offline and indicator changing from GREEN to RED and disconnected from SIP. In this mode I can restart it again by dealing restart code from attached phone.

[pfSen+VoiP fail capture txt.txt](/public/imported_attachments/1/pfSen+VoiP fail capture txt.txt)
[pfSense DHCP log xls.xls](/public/imported_attachments/1/pfSense DHCP log xls.xls)

JKnott

Perhaps I'm missing, but those look entirely normal, other than somewhat short interval for a 3600 sec. lease time. I see the initial discover, offer, request & ack, followed by request & ack for each renewal. What is strange, however, is the ack is going to the broadcast address and not the assigned IP. Incidentally, I had asked you to attach the actual capture file, not a text interpretation of it. The reason I did was, as I mentioned, the capture file contains more detail than the text. For example, with the capture file, I could examine what the client sent to the server in the request. Did it actually request the offered address? Or somehow requested 255.255.255.255? I can't tell from the text file. Regardless, that ack to the broadcast address is wrong.

gryest

Sorry, I am bit hesitant to attach complete capture file and post online. There some MDNS and other garbage from Linux PC I don't know yet how to completely stop when wiresharking from Linux side. So here skimmed DHCP only version as you prefer as I understood. All UDP, ARP, ICMP, SIP filtered out.
I am really curious because of 2 reasons: Should I continue with my VoiP provider and can I full rely on pfSense for future projects and network expansion.
Thanks.

[pfSen+VoiP DHCP 3600s lease01.pcapng](/public/imported_attachments/1/pfSen+VoiP DHCP 3600s lease01.pcapng)

JKnott

There some MDNS and other garbage from Linux PC I don't know yet how to completely stop when wiresharking from Linux side.

This is where creative use of capture filters comes in. You could create an filter that allows DHCP and also filters on the client and server MAC addresses. For example, it might be something like port 67 or port 68 and (ether host <mac1>or ether host <mac2>)</mac2></mac1>, with <mac1>and <mac2>being replaced with the actual MAC addresses of the client and server. This should result in only the desired captures.

I can't speak about the VoIP provider, but pfSense is an excellent firewall that does much more than much of the competition.</mac2></mac1>

gryest

Understood, the key word is "creative". I need bit more practice in filters.
Here is complete file in case.

[pfSen+VoiP DHCP 3600s lease02.pcapng](/public/imported_attachments/1/pfSen+VoiP DHCP 3600s lease02.pcapng)

JKnott

Option 50, in the request, shows the address 192.168.1.107 is being requested. The ack shows the correct client address, but for some reason is being sent to the broadcast address. I have no idea why it would do that. On my own system, the ack is sent to the correct client address. The problem with sending to the broadcast address is the client has no idea the ack is for it. When the broadcast IP address is used, the broadcast MAC address is also used. A proper ack is sent to the assigned IP address, using the client's MAC address.

So, the question now becomes, why is the ack going to the broadcast address?

gryest

I have no idea. I become suspicious about broadcast address from the beginning after I've got mentioned failures and started capturing traffic.

gryest

I got another look on DHCP sequence and compared to other devices. In short, VoiP controller receiving what was requested in DHCP Discover poll. VoiP set Broadcast flag 0x8000 and pf sense replied to broadcast address accordingly. Other PC and devices do not set Broadcast flag and pfSense replying to machine IP address as mentioned by JKnott.
The other thing, Wireshark shows End option missing in DHCP Discover which is normally (255) End.
But most important, for some reason VoiP do not reply on received DHCP Offer basically not following protocol. Correct protocol handling happening during soft/hard restart only which make me pretty much sure it's VoiP device software problem.
Thanks for tips.