By default the ALIX board is 38400 for the BIOS settings, so you have to use that speed to set them. I typically do the first connect at 38400, go into the BIOS, set it for 9600, CHS mode, then save and adjust back to 9600/8/n/1.
Neither one of those outputs looks good though.
Have you been over the embedded-related choices on this page?
http://doc.pfsense.org/index.php/Boot_Troubleshooting