Multiple wan causing packet loss

  • I am preparing to migrate to a new modem, new speed tier, etc with the same ISP. I have dynamic addressing for both.

    Previously, I just had a very simple setup. Modem via ethernet to em1(WAN) on pfsense.

    What i've done now is set up multiple vlan on my switch

    vlan 48 - untagged port 48, tagged port 46, port 47 not member
    vlan 47 - untagged port 47, tagged port 46, port 48 not member
    vlan 46 - untagged port 46, port 47&48 not member

    on pfsense I have em0(LAN) and em1(WAN)
    I have created 2 vlans on pfsense, vlan48(wan1) and vlan47(wan2) both of which have em1(WAN) as the parent interface
    vlan priority is set to 0 for all, as I don't see it being needed here.

    I have enabled all 3 interfaces.
    em1(WAN) has a unique mac (00deadbeef00) ipv4 and ipv6 set to none and hostname has been set as
    em1(wan1) has a unique mac (11deadbeef11) ipv4 is set to dhcp and ipv6 is set to none. Hostname is also set to
    em1(wan2) has a unique mac (22deadbeef22) ipv4 is set to dhcp and ipv6 is set to none. Hostname is also set to

    I have created a gateway group (WanGroup1)
    I have set wan1_dhcp and wan2_dhcp both to Tier 1 and triggered on packet loss or latency. wan_dhcp is set to never

    Now currently the modem on the other end of wan1 is unprovisioned, it is not giving a valid dhcp lease.
    The modem on wan2 is provisioned and giving a public IP.

    When ethernet is plugged into wan2, i get a great connection. 0% packet loss, amazing ping times.
    When I plug ethernet into wan1, I start to see unacceptable amounts of packet loss, sometimes 15%, sometimes 60% it jumps around. I am still able to browse the web, I am able to ping external hosts, but with a lot of loss.

    I've tested and even swapped multiple known good cables, thinking it was a cable issue, but it is not. Whenever both links are plugged in, I see this behavior. Is it something in my settings? What should I be looking at here? I am noticing cpu utilization has risen significantly while preforming these tests.

    Good news in, I've been able to get some progress done on some much needed cable management while diagnosing this.