tl;dr check if file format is .iso not .iso.gz
I'll reply to this thread even when it's old as it was one of the first which came up when googling "Operating system not found".
I spent quite awhile searching for an answer and then I realised that the pfSense ISO file, which I downloaded from official site, is in iso.gz format. I didn't notice this first as I'm working with small laptop screen and the file format was always cut from the screen.
You have to extract the .iso.gz file so you have only .iso file.
I installed pfSense CE 2.4.4 with Guest OS: Other and Guest OS Version: FreeBSD 11 (64-bit) on ESXi 6.7 U2, though I used VM compatibility ESXi 6.7.