Avaya IP Phone VPN Issue *Seems resolved
I originally was going to put this in general, but after typing most of it, I tried something else (posted below, and as a result, decided to put it in hardware.
I'm still posting it even though I seem to have figured it out to see if anyone has any thoughts as to why my problem was happening, and to hopefully help someone in the future.
I have an Avaya 5610SW IP phone that is running the most current VPN Firmware and connects back to my office which has an ASA5510 for a firewall.
At home, I am connecting through a pfSense firewall that is virtualized on VMWare ESXi 5.1, running on a Poweredge R710. It has a WAN, LAN, DMZ, and a fourth segment for testing. Here, my IP Phone connects back to the office without issue. Has worked great since day one.
At my mothers house, I have a pfSense firewall running on an IBM NetVista SFF PC, with the NanoBSD version installed and running off a CF card that's connected via SATA. When I'm there, the phone will not work. Period. The VPN Connects, goes through the motions, then sits at 'Discovering x.x.x.x' (where x.x.x.x is the IP address of our IP Office). I have gone over the configs the best I can and everythign is the same, with the obvious exceptions being LAN Addresses and NAT entries. I haven't looked into it further as I'm not near her house and can't just go back and forth to troubleshoot. Right now I have two internet connections at home, so I decided to try and duplicate it.
Initially, I had a PIX520 on the second connection. It behaved in the exact same fashion: the phone would go through its motions, VPN would connect, but then it would sit at 'Discovering'.
I then grabbed an IBM ThinCentre that was sitting around and threw in a dual port Intel NIC. Loaded a default build of pfSense on it, and get the same thing.
After typing most of this up, I decided to try something else: Changing NICs. My firewall at my mothers has one xl0 NIC (WAN) and one fxp0 (LAN). I started the test firewall at home with a dual port Intel 82559 based NIC (fxp), using both ports on the NIC. I then dug an Intel FW82546 based Dual port NIC out and tried that… SUCCESS! Swapped back to the original NIC - FAIL. I then dug out several other NICs - BCM7503, BCM5701, 3Com 905CX, ALL of which worked fine, both as pairs (the ones I had two of) and along with the built in Intel 82573E NIC. As a final test, I put the original card back in, but only using one port..... and it WORKED! Very confused now, I swapped the ports.... and it failed. It turns out, for whatever reason, with the LAN using the FXP NIC (as it is at my mothers), the VPN does not work right. If it's on the WAN, it works fine. The PIX, incidentally, is using the exact same multi-port 82559 based NIC as I was initially using in the test pfSense box here.
Here's the real kicker that had me going nuts: Even when I established a site to site VPN from my mothers pfSense to the ASA at work and had full bi-directional communication for EVERYTHING ELSE (and the phones VPN functionality disabled), the phone STILL would not work. Other VPNs such as the Cisco IPSec VPN work fine as well. Just the Avaya phone was failing.