Eventual TFTP failure - "couldn't forward tftp packet: Permission denied"
-
Hello everyone, I posted this question a few months ago in NAT but never received any replies and I'd love to figure out a troubleshooting method as the issue has cropped up again.
I've got a Netgate SG-3100 device that has been giving me an issue. We've got a handful of SIP phones behind the Netgate, which use a PBX in a datacenter for calling and TFTP config. SIP calls all work great, and TFTP works for a while after reboot of the firewall, but eventually time will come to add a new handset or alter an existing handset's config. After X amount of time, the TFTP helper seems to stop functioning, and I start seeing the RRQ and WRQ lines followed by "couldn't forward tftp packet: Permission denied" in the Netgate logs. Sure enough, the traffic isn't making it to the TFTP server at the PBX. If I restart the Netgate, all the phones start doing their normal TFTP pulls successfully and everything is fine again for a while - not sure how long it takes to break. Regardless of the TFTP helper functioning, the phones all continue to make and receive calls without issue, it's only the TFTP link that breaks down.
I've searched for this error message without success. It doesn't seem to be related to the PBX or TFTP server as I can still access TFTP configs from another location behind another firewall. Netgate's config is pretty basic, I have the TFTP server specified so the phones pull the address when doing DHCP, I have a few DHCP static maps for the phones, and I have the TFTP helper enabled for LAN, and Manual Outbound NAT. ISP is a gigabit fiber link with static IPs, also configured in the Netgate. I have configured the OpenVPN server on the Netgate to access assets behind the firewall. Not sure what to do from here, but it's definitely a hassle to reboot the firewall each time I need to add/alter a SIP handset. Any ideas on what my issue might be?
-
@TreyD if after a reboot your TFTP sessionsl are ok, It seems that your TFTP client is being blacklisted.
The logs don't register a deny for your client? Look at the "x" and see if it is being blocked by some service like Snort. If so, look at the serice alert and the rule that triggered the violation. Analyse it and see if the suppression of that rule isn't a security risk and disable it.
-
@jorlando Thank you for the reply! So I took a look at the logs on every end I know to look. The client, a Polycom phone, logs the failures something like:
|Prov|Download of master configuration file failed
The server, the PBX itself, doesn't see the request come in, and this reinforces the idea that the request is indeed being blocked by the pfsense. The pfsense sees this in System Logs/System/General for each request:
Dec 9 15:36:11 tftp-proxy 48855 couldn't forward tftp packet: Permission denied
Dec 9 15:36:11 tftp-proxy 48855 <phone-internal-ip>:21238 -> 127.0.0.1:6969/<pfsense-wan-ip>:53532 -> <pbx-external-ip>:69 "RRQ <phone-mac-addr>.cfg"The structure of the tftp-proxy logged request looks identical if everything is functional, just the "couldn't forward" message isn't there. Snort isn't installed, and the pfsense WAN IP matches to an allow all rule straight to the PBX. The blacklist idea is an intriguing one, but if that is what is happening it seems to be blacklisting every internal IP behind pfsense, so the entire LAN network and not just a single LAN client. Also, pfsense seems to be the device doing the blacklisting, but I'm not sure to what end, and with no record of any other service showing up in the pfsense logs, it appears to just be the service running tftp-proxy. No firewall blocks seem to happen during this period on pfsense, there's just passes from wan-ip:randomport > pbx-ip:69 UDP.
-
@TreyD take a look at this site.
It seems that VoIP phones need some extra tweaks:https://support.onsip.com/hc/en-us/articles/204029430-PFSense-Firewall-Settings-for-VoIP?mobile_site=true
-
That SIP advice is pretty much obsolete at this point. Almost all VoIP services and PBXs will accept random source ports.
Phones behind pfSense connecting to something external should work fine with the default settings.Anyway this is not a SIP/VoIP problem.
What happens kill the xinetd service and then re-save the Sys > Adv > Firewall&NAT page to restart the tftp-propxy? Does it start working again?I would have bet on Snort or Suricata blocking the service with that error too.
Steve
-
@stephenw10 Thanks for the response - here's the log output from those actions:
Dec 9 18:44:56 tftp-proxy 79829 pf connection lookup failed (no rdr?)
Dec 9 18:42:17 xinetd 79803 Started working: 1 available service
Dec 9 18:42:17 xinetd 79803 xinetd Version 2.3.15 started with libwrap loadavg options compiled in.
Dec 9 18:42:16 check_reload_status Reloading filter
Dec 9 18:42:16 check_reload_status Syncing firewall
Dec 9 18:41:40 xinetd 36487 Exiting...After this I reset one of the handsets behind the firewall, and it looks like the result is the same. I can see each of the Polycom read requests and write requests in the pfsense log, and each one is still followed with:
tftp-proxy <pid> couldn't forward tftp packet: Permission denied
I actually just noticed something that may be a clue - I was going to say "this last time the system seemed to work without issue for over two months because I just set up a new phone in this office a week ago," and I went to check uptime to make sure my math was correct. Turns out the system has only been up for about two days, which means power failure to the office, which means pfsense probably came up before the fiber network out to the internet. Is it possible that something upstream coming online after pfsense could be causing it to get "stuck" in this behavior? Like the gateways would have been offline, so definitely DNS too...kinda reaching at straws. I can always reboot the system after hours, but it's not a great situation to have to explain to users in that office that I can't get their phone back online for the day.
-
If you run a packet capture on the WAN side can you see the tftp-proxy sending any packets?
I had assumed that error was the proxy unable write out the packets to the WAN (blocked outbound by Snort for example) but it could be simply logging a denied message received back from something upstream.
Steve
-
@stephenw10 Good call - I'll start here next time pfsense is in this state and see if those requests are making it out of the firewall. I ended up rebooting pfsense last night and everything came back up fully functional. I have no doubt this situation will happen again, and I usually have a chunk of time to mess around with it.
If this indeed only happens during a power outage or loss of provider, I can at least know to reboot the system when the automated alerts tell me things have come back up - that alone is a big win. I can't confirm as I didn't think to check the uptime in the past, but it seems as good a theory as any right now.
I'll keep an eye on this post for any new comments, and I truly appreciate everyone's time and assistance in helping me resolve this issue.