I've got the Open-VM-Tools-8.8.1 package installed on the pfSense vm I'm running under ESXi 5.1 (installed via the System menu in the pfSense webGUI > Packages). From what I remember, the regular Open-VM-Tools package gave me some errors, but the 8.8.1 version seems to be fine. I'm running the i386 2.0.1-Release. My vSphere client reports the tools as installed and 3rd party/Independent. You may consider upgrading to 5.1, as it fixes a problem where the pfSense vm (or any other vm for that matter) wouldn't auto-start after the ESXi host was booted. Not sure if auto-start is important to you or not.
There's another release something like 5.0.1 update 1b that also fixes it. Off the top of my head build number is 8xxxxx