Multiple IP single interface w/traffic shaping

[Max Power]

Hopefully someone can help me and this can be done.

I have 1 WAN connection coming into building.  It is a small 5/2M connection but that is all that is available in our area.  We were provided 2 public IP addresses, but they have 2 different gateway IP's. I need to have a public IP on my Asterisk server and 1 public IP for my main internet (through pfsense).

The 5/2 limitation is limited by port speed of the physical connection.

I need to somehow implement traffic shaping to not disrupt VOIP calls.

Is there any way this can be done?

My current setup is this:

Wan Connection - ESXi virtual switch - wan of pfsense
                                                      - eth0 on asterisk

I hope this makes sense.

Thanks!