Multiple wan causing packet loss

[Cheetohz]

I am preparing to migrate to a new modem, new speed tier, etc with the same ISP. I have dynamic addressing for both.

Previously, I just had a very simple setup. Modem via ethernet to em1(WAN) on pfsense.

What i've done now is set up multiple vlan on my switch

vlan 48 - untagged port 48, tagged port 46, port 47 not member
vlan 47 - untagged port 47, tagged port 46, port 48 not member
vlan 46 - untagged port 46, port 47&48 not member

on pfsense I have em0(LAN) and em1(WAN)
I have created 2 vlans on pfsense, vlan48(wan1) and vlan47(wan2) both of which have em1(WAN) as the parent interface
vlan priority is set to 0 for all, as I don't see it being needed here.

I have enabled all 3 interfaces.
em1(WAN) has a unique mac (00deadbeef00) ipv4 and ipv6 set to none and hostname has been set as host5.domain.com
em1(wan1) has a unique mac (11deadbeef11) ipv4 is set to dhcp and ipv6 is set to none. Hostname is also set to host5.domain.com
em1(wan2) has a unique mac (22deadbeef22) ipv4 is set to dhcp and ipv6 is set to none. Hostname is also set to host5.domain.com

I have created a gateway group (WanGroup1)
I have set wan1_dhcp and wan2_dhcp both to Tier 1 and triggered on packet loss or latency. wan_dhcp is set to never

Now currently the modem on the other end of wan1 is unprovisioned, it is not giving a valid dhcp lease.
The modem on wan2 is provisioned and giving a public IP.

When ethernet is plugged into wan2, i get a great connection. 0% packet loss, amazing ping times.
When I plug ethernet into wan1, I start to see unacceptable amounts of packet loss, sometimes 15%, sometimes 60% it jumps around. I am still able to browse the web, I am able to ping external hosts, but with a lot of loss.

I've tested and even swapped multiple known good cables, thinking it was a cable issue, but it is not. Whenever both links are plugged in, I see this behavior. Is it something in my settings? What should I be looking at here? I am noticing cpu utilization has risen significantly while preforming these tests.

Good news in, I've been able to get some progress done on some much needed cable management while diagnosing this.