PFsense TLS not working with multi WAN

[Eskir]

Hey guys, I've recently run into a problem with TLS when I'm elevating both my WAN interfaces into the same tier, this results in this:

Error:        	GnuTLS error -110: The TLS connection was non-properly terminated.
Status:      	 Server did not properly shut down TLS connection
Error:        	Transfer connection interrupted: ECONNABORTED - Connection aborted
Error:        	Connection timed out after 20 seconds of inactivity
Error:        	Failed to retrieve directory listing

That's from FileZilla.
I searched around and I read a lot about NAT and what not, but I'm not using anything special. My setup is just LAN > PFSense > WAN1/WAN2

I tried enabling Sticky Connections, that didn't change anything. I read about the FTP client proxy as well, tried to set it up, didn't change anything but in any case: I also read that the client proxy doesn't tamper with SFTP/FTPS connections anyway.

What should I do? Where can I start looking? It's very frustrating as it works sometimes, I get my directory listing etc but then when I do something it fails to connect again 99% of the time.