Issue with TFTP IP Phone Provisioning for phones behind PFSense
I have run into a rather strange problem where a client's XG7100 appliance seems to be preventing their Polycom phones from being provisioned via TFTP.
First some basic info about what we have set up:
We are hosting their FreePBX sever at an external location (datacenter). All of their phones are sitting behind a Netgate XG7100 appliance (running 2.4.4-RELEASE-p1). Inbound and outbound calling seem to work just fine; however, after a few hours of trial and error, I have been unable to get these phones to automatically provision successfully.
As a quick test I brought one of the phones to my office (we run a Sonicwall here). Within seconds the phone provisioned itself. As a 2nd test, I set up an old Netgear firewall at the client's office. I assigned it a separate Public IP address from the clients ISP block, set up a separate LAN, and connected another test phone. Again, it provisioned almost instantly. This seems to rule out the ISP being the issue, and seems to point to the PFSense. I assume its a configuration error on my end but I cannot figure it out. Here is what I have tried:
- Verified the client's public IP is in the trusted network list on the FreePBX firewall.
- Created both an inbound and outbound "allow any" rule for traffic coming from or going to the FreePBX public IP (I know this isn't necessarily the safest thing to do, but for testing purposes I felt it necessary)
- Enabled the TFTP Proxy helper in PFSense. First I tried enabling it on just the LAN, then just the WAN, then both the WAN and LAN simultaneously.
- Created an inbound and outbound allow rule for traffic going to or coming from the FreePBX on port 69.
Another thing I noticed: watching the TFTP logs on FreePBX indicates TFTP traffic coming from the phone is timing out (i.e. the PBX attempts to send TFTP data but the phone never responds). I checked the firewall log in PFSense and I do not see anything that indicates TFTP traffic is being blocked. In fact, I see the firewall is passing this traffic in the logs.
At this point I assume its something to do with NAT but I just cannot figure it out. This is the last issue I need to resolve before we move the PBX into production so I am hoping someone can point me in the right direction.
Thank you for your time!
I would run a packet capture filtered by the PBX IP on both WAN and LAN and see what's actually happening.
You can also check the state table for states with the PBX IP. You should see the tftp traffic redirected to localhost on LAN and then another state going out on WAN.
I expect you will see that since the PBX shows traffic arriving. More interesting it whether whatever the PBX replies with ever appears back on the pfSense WAN and whether that is passed.
Thanks for your help! I went ahead and re-enabled TFTP Proxy on the associated VLAN. Immediately after enabling the process I see this in the log:
"172.16.XX.XXX:32632 -> 127.0.0.1:6969/96.81.XX.XX:63309 -> 162.255.XX.XX:69 "WRQ 64167fxxxxxx-app.log""
I assume this is good news since it shows the TFTP helper is functioning.
I also went ahead and ran a packet capture as you suggested. The first capture I ran on the WAN and filtered for the FreePBX public IP. I also filtered for port 69. Every single packet captured was similar to the one below (bad UDP checksum). I am hoping this has pointed us in the right direction. Could it be possible the firewall is somehow corrupting the TFTP packets? Here is a sample from the capture:
13:38:11.741303 IP (tos 0x0, ttl 64, id 58595, offset 0, flags [none], proto UDP (17), length 71, bad cksum 0 (->414c)!)
96.81.xx.xx.57546 > 162.255.xx.xx.69: [bad udp cksum 0x54bb -> 0x466b!] 43 WRQ "64167fxxxxxx-boot.log" octet blksize 4096
That can just be an artifact in the packet capture but you should try disabling
Disable hardware checksum offloadin System > Adv > Networking. I have seen that break tftp previously.
Do you see packets coming back from the PBX too? Since you were seeing logs in the PBX I assumed it would be responding.
Thanks for the info! I will give disabling hardware checksum offload a try. I checked our firewall and the "Disable hardware checksum offload" is currently unchecked. Just to confirm you would like me to try checking this option?
I do indeed see packets coming back from the hosted PBX, so I am confident this is not a firewall/rule issue.
Yes, check that. You will probably have to bring down the interface and back up to apply it. Or reboot.
I went ahead and checked the "disable hardware checksum offload" option, then rebooted the firewall. I was still unable to provision the phone after rebooting (phone never pulls the TFTP files) so I went ahead and reverted back to the original setting, then rebooted again. I think I may end up setting up a test FreePBX and PFsense box at my office to try and replicate the issue. At this point the client's phones are functioning; the only issue being I have to manually provision them (which really isnt a big deal since they only have a few phones).