Cisco Voip TFTP connection diagnostics

plrpilot · Dec 27, 2019, 5:24 AM

I have the following network:

pfSense is my DHCP server.

If I assign pfSense as the TFTP Server and define it as option 66/150, my phones (example 10.10.3.11 in this case) will connect and download the current firmware (uploaded through the TFTP module).

However, if I assign my freePBX server as the TFTP Server in option 66/150, my phones do not appear to connect and configure themselves.

Checking the phone's web interface shows that it is getting correct DHCP information (good IP address, and proper TFTP Server address).

However, the logs on the phone show a timeout connecting to the TFTP Server.

Doing a manual connection attempt from my VPN machine (via pfSense) shows that the TFTP Server on the FreePBX is active. If I enable verbose logging on the TFTP Server, I can see attempts I make from my vpn machine working perfectly, but it shows no attempt by the phones. I'm able to ping the phones from the FreePBX terminal. Doing a packet capture doesn't appear to capture any attempt by the phones to make a connection -- just the initial DHCP assignment. Also, a NetScan from FreePBX doesn't show the phones on the network, although they do successfully ping from that machine's terminal.

I've banged my head on this for a week and can't figure out what I might be doing wrong or how to possibly diagnose it. Any ideas would be greatly appreciated. My gut tells me this is a firewall issue, even though they are on the same LAN and network rules shouldn't be affecting them. I'm both a pfSense and freePBX newbie, so I'm probably missing a simple step but I'm braindead on what to try next.

Ideas?

chpalmer · Dec 27, 2019, 5:43 AM

The pfsense box would have no bearing on this issue. LAN traffic does not get pushed to the router unless its out of LAN subnet traffic.

Id look at the firewall on your FreePBX box. Or a misconfigured switch.

plrpilot · Dec 28, 2019, 1:21 AM

@chpalmer

Yeah, my initial thought was the same, but here's where I'm confused.

If it were the switch, then my pings from the FreePBX box should not be successful. The switch is a 48 port ubiquiti gen 2 switch (US-48-500). If it were blocking TFTP requests, then the phone should not have been able to connect to the pfSense TFTP Server.
It's possible that it could have been the firewall from FreePBX, but it doesn't make sense that the firewall rules would disable TFTP, since it's how it manages firmware and settings updates to the phones (at least the ones that don't use an https interface). But to eliminate that, I temporarily shut that firewall off and rebooted the phone. Nothing changed, so I'd say it's not the FreePBX firewall.
I've enabled promiscuous mode and done a packet capture, but it's not capturing the any traffic from the phone outside of the DHCP challenge/response, which is not necessarily a surprise, since it's a managed switch and shouldn't necessarily route the FreePBX traffic to the firewall for local traffic on a different switch port.
If it were the phone, then it shouldn't have worked for the pfSense TFTP server.

I could bridge the port for the phone and do a packet capture there, I guess. I'm just lost on ideas I could use to figure out what's going on and why the phones are non-responsive.

chpalmer · Dec 28, 2019, 3:04 AM

Rule 1. Traffic within a subnet never.... never reaches the router. Push the "I believe" button and move to the next step.

Logical does not change. Our understanding of it does.

Now move to your thought. Do not trust the switch. Put a dumb switch in its place and try that. Test test test prove prove..

when in doubt go back to rule one.

It is always possible that a firewall will get in the way. But only if within the path. In subnet LAN traffic does not pass the router! Look at everything else in the way. Smart switch? Rule it out. Firewall on one of the clients or servers? Rule it out. Totally.

Are you sending traffic via DNS? that is a different situation. Id ask why but you might have your reasons.

If it were blocking TFTP requests, then the phone should not have been able to connect to the pfSense TFTP Server.

ehhhh.. could be but not totally true.. How well do you know its config? Engineers sometimes get in the way. Ask any Ford Mechanic..

but it doesn't make sense that the firewall rules would disable TFTP,

You need to go back to firewall school. Again.. Engineers sometimes get in the way. Ask any Ford Mechanic..

A Ford engineer will step over 20 hookers to screw a mechanic!

And for the arm chair IT guy.. DNS traffic is not in subnet traffic.. But roundabout LAN to WAN back to LAN traffic and not included! Get off your game and get a job! ;)

plrpilot · Dec 28, 2019, 3:12 AM

@chpalmer

Thanks for the reassurance. I can promise you that I've spent hours on this trying to make assumptions. I would agree with your assessments. And short of swapping out the switch, I've exhausted my troubleshooting options.

I did post a question in the ubiquity forums.

I didn't exactly spell out my current dilemma -- I'm working remotely and won't have physical access to swap out the switches for another week. Fortunately, the phone isn't a critical one, and I have vpn access to be able to manage all of the components I need to change (except that danged switch). If I had thought this out in advance, I'd have plugged one of the phones into a spare port on my netgate firewall (xg-7100) and configured the port for the phone. It would have required a POE injector, but that's a minor issue. That's why I'm posting and searching for a possible solution in the meantime.

Thanks for your response!

chpalmer · Dec 28, 2019, 5:19 AM

LOL.. Ive got a counterpart trying to access a system in California from Idaho right now.. Whats in the path past the modem?? A managed switch. He cant see squat past the switch.

Says he left it as "dumb" as he could get it and works on site..

Good luck!

plrpilot · Jan 2, 2020, 7:17 PM

Update on this for anyone having similar issues. I ran a tcpdump on the FreePBX server and and realized the phone was trying to contact it. The firewall wasn't blocking it, but the TFTP Server wasn't responding due to an improperly configured option in the setup file.

Although I could connect locally in my testing, I didn't try to download a file. Once I realized I couldn't download a known file, I was able to go into the configuration and fix the error.

The phone is correctly connecting to the server and I have a new set of issues to deal with, but unrelated to network on the local network.

Takeaway -- utilize tcpdump (if available) to diagnose the local interface traffic. You can then take that pcap file and analyze it in Wireshark to quickly identify issues.

Happy New Year!