@cmb:
I was helping someone on IRC last week with an Avaya phone with 2.2. Sounds like a bit different of a circumstance, but the phone was sending malformed traffic. It apparently worked with 2.1.5. It appeared strongswan was doing something differently than racoon which triggered a bug in the phone's IPsec client. He had no means of getting to the phone's management interface so we were stuck.
You have a spare phone or two you could contribute to the cause? If you can ship me one, I'll experiment and see what works and what changed in behavior between racoon and strongswan there. PM me if you (or anyone) is willing to give us one and I'll get you an address. I'm in the US, FYI, in case shipment destination and associated cost influences your decision.
That…. might've been me actually. Encapsulation and rekey was working on 2.1.5, but unfortunately at this point both have to be disabled for it to work properly with 2.2/strongswan.
I have an open support ticket and have done some back and forth with jimp on this(as always, he's extremely helpful). I mentioned a very similar sounding situation but with Avaya + Cisco ASA... tl;dr: cisco expects a nat-d payload type 20, and avaya only does nat-d payload type 15. Same solution - disable encapsulation. It appears to have happened when cisco changed some of their more forgiving backend. Link: http://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/116294-problem-nat-00.html
I really feel that this is a problem with Avaya's implementation, and that raccoon was simply more forgiving – still kinda a bummer.
I'd like to try and get a phone to you. It's not my decision but I might be able to make it happen. They're ~$230 on amazon. I think you would have to set up an IP Office PBX to really get to the root of this issue; with encapsulation enabled the tunnel connects, but the phone just will not see/register with the PBX. iirc, a developer account with avaya can get the 'server edition' that you could run on a vm at no cost.
Thank you for your interest in this, Chris. I really appreciate it.
sidenote: read the blog post last night. Instantly bought a ticket for the hype train! Choo-choo!