Im not really sure exactly what was wrong...
I've started from scratch, and came to the same or new issues. Doing troubleshooting, i found that the broadcast address was way off, which I did not really understood. I then found that the VLAN interface was created as /32 CIDR, which it defaults to, so its highly important to remember to change this.
Changed it to /24 CIDR, and then it started working.