If you need help setting that up - just ask..
But if your goal is isolation - which I assume it is because your asking how to only allow 1 IP, and block others. Then you really need to create two different L2 networks (vlans or completely different physical networks - 2 interfaces on pfsense with 2 different dumb switches).
Another option would be to just put them all on the same L2 (same L3 as well), but make it a private vlan... And then you can let X talk to Y, and A talk to D, but block Z from talking to A, etc. Via setting on your switch that support private vlans. But you need a switch that supports that.
Simple solution to keeping A from talking to B, is put them on different actually isolated networks. And then filtering whatever traffic you want to allow/block on pfsense.