I currently have multiple servers running and I want to keep local traffic separated from the "outside" and away from server to server along with separation of the admin/ILO network.

I'm not sure how good of an idea it would be to keep all the traffic going through a single switch and split everything into VLAN's OR...
separate everything onto its own network and switch, this was each network is separated physically even if PFsense does fail of keeping an intruder out

Each one poses in one way another some challenges...
The first scenario is complexity and having to "port" and setup VLAN's and TAG's on each server and PC
Downfall - single point of failure

The second is all static IP's and NAT.
Downfall - initial start up requires extra hardware to be purchased.

Currently I have a Woven LB4 acting as my main switch and everything is sorta fanning out from there including my media server and the WiFi.
The server back bone is intertwined across a Infiniband network and the iLO, KVM and Management Network right now are disconnected and don't want to connect them unless i know for sure they are secured.

I can pick up a couple of switches and the only headache would be the cabling to run and terminate ( I would make my own custom cables of course )

decisions decisions decisions...