GRE tunnel results in dead slow upload

[Operations]

I am using Pfsense 2.4.5 (have tried VM on ESX7 and Pfsense barebone on Supermicro setup) with a Fiber 1000/1000 connection. Intel NICs and hardware specs are more than enough.
Which works fine, when i do several speedtests and 1GB file down i am getting around 900/900.

Since i like to use several external IPs i am using a GRE tunnel with an IPv4 /29 subnet. So i setup a GRE connection, static route, second gateway, virtual IPs. Combined with sNAT and dNAT rules (I selected Hybrid NAT).

My problem is that after setting up the GRE tunnel my DL/UL speed drops a lot. I am getting 600/150 at best.... Also when i do a speedtest straight from pfsense. I am often getting less than 10 Mbit upload. Download is always between 400/600. Biggest problem is the upload speed.

Ofcourse i get that using a GRE tunnel comes with a performance penalty, but this is too much for sure. I think there are some settings which i need to change but i dont know which ones.

Any ideas?