Very poor OpenVPN bandwith

illumina7

Hello,
I am using a self build pfsense in our office.
Our hardware is:

I5-3330
8GB DDR3 Ram
Quad Intel Server NIC, I think its with Intel i350 chipset
2 hdds zfs mirror
2 WAN connections, 500/50 and 100/40 Mbit (down/upload speed), fixed IPv4 WAN IPs on both connections
What I setup for OpenVPN:
Hardware crypto acceleration enabled and rebooted
Data Ciphers: AES-256-GCM, AES-128-GCM, CHACHA20-
POLY1305
Digest: SHA3-256
D-H Params: Disabled, ECDH Only
OpenVPN via TCP, with UDP it was too bad to use
Server parameters: fast-io;sndbuf 0;rcvbuf 0;tun-mtu 1400;mssfix 1360;
Client parameters: fast-io;sndbuf 393216;rcvbuf 393216;tun-mtu 1400;
I was working around for about two weeks to found settings which work somehow good, but it is far from the optimum.
With iperf on VPN the speeds are around maximum of 20 MBit, average around 8-10 Mbit/s. SMB filecopy over VPN works with full bandwith around 4 MB/s, but after ~30s on copy the bandwith dips to 0-300kB/s and gets high again.
RDP session to our terminalservers are incredible slow and stuttering all the time. Ping spikes up when bandwith dips to about 1000ms, otherwise it is around 20-30ms through the VPN.
Already tested OpenVPN via UDP, there is the bandwith much worse, around 1-4 Mbit/s.
Same effect is on our site2site IPSec VPN to branch office, which has an pfsense with almost same hardware (same CPU, 8GB Ram, zfs hdd mirror, quad intel i350 nic).

CPU load is about 1-10%, nothing special.

Why is the OpenSSL connection so incredible slow?
Someone has a hint for me?

Edit: I've tried almost every single tip I found on the internet, but I can't get it work as it should. Even with hardware aes acceleration disabled, the hardware should be possible to max out the available bandwith.
With different speed and down/upload tests, the bandwith is totally okay and we get maximum on both WAN connections. I've also called the providers more then 5 times to check the WAN connections for errors, but there was never found anything.