The admin group should have access, but apparently does not.
It looks like any group you give ssh access does not get it setup properly, but if you add it manually to a specific user then they do get it.
I opened a ticket for this:
http://redmine.pfsense.org/issues/614
I also found two other user manager issues while testing that.
http://redmine.pfsense.org/issues/612
http://redmine.pfsense.org/issues/613