DHCP relay Hyper-V issue

[Syndrose]

I will try and keep this short.
I have a Windows Server 2012 R2 server and running PfSense in Hyper-V.
PFSense is version 2.4.0 (will try and upgrade soon but I don't think it will help)
The VM has 3 gigs of memory and no limit on CPU Usage.
I have found other threads recommending to set MBUFbuffer (not sure if i spelled that right) to 1 million, which didn't help.
I have 3 Vlans
Vlan 1 - default, DHCP servers live on this network, no issues
VLAN 100 and 150  - Guest Wifi and VOIP network, uses DHCP relay in PfSense to get addresses from servers on VLAN 1 sometimes devices don't get dhcp leases. get the following in the DHCP  system log.
dhcrelay send_packet: No buffer space available
This happens frequently, especially with the higher traffic VOIP VLAN.
The Server has a 4 port nic "HPE Ethernet 1Gb 4-Port 331i Adapter"
Latest driver and firmware from HP
2 are dedicated and teamed for other services offered by Window Server 2012
1 is dedicated for the WAN connection
1 i dedicated for the LAN and VLAN connections.
Switch appears to be configured correctly as it assigns VLANs and sometime devices get addresses.
I have  "Virtual Machine Queue" enabled on both the Hyper-V interface in the VM as well as the driver for the adapter.
No IPSEC or anything fancy, just standard IPv4 networking. We don't use IPv6 at all.
Bandwidth management is NOT enabled in Hyper-V.
In the Hyper-V Switch Manager the option to "Allow management operating system to share this network adapter" is NOT enabled, so it should be dedicated to the VM(one would think).

Please let me know if you have any question/suggestions, help is much appreciated.