It's been while since I tried it but I think you had to deploy it as a profile to OSX to use anything but the default options there.
However since that hangout was made I also think OSX may have stepped up the encryption levels it uses by default... so maybe a bit of both in play here. I know at the time we chose those settings as the only thing that would work with everything.
Try setting it to the values in the hangout to make sure it connects and it is a encryption settings issue. If so look at deploying via a profile.
Steve