CARP on WAN with redundant uplinks
In previous setups I run the ISP connections directly into the PFSense boxes. This time I needed to be able to plugin another set of FWs so I added some switches WAN side.
I realized the C and D connections are physically separate which caused some un-expected behavior with failover.
What is the proper way to setup and maintain full redundancy WAN side without creating loops? I could stack the switches or trunk.
In that configuration you are relying on the ISP to forward the CARP advertisements. That might not be the best idea.
Can the link to the ISP be LACP?
The ISP wasn't forwarding CARP advertisements which is what was causing the issues with failover since the networks were split. Trunking the edge switches took care of that but now I believe we're relying on STP which I'm not sure is a good idea. The ISP should be able to accommodate whatever config we need.
This is my "perfect world" diagram…
If that ISP link is a loop with RSTP it should be reasonable as long as you ensure the ISP is the root bridge and everything is sane there. You won't get the aggregation while both links are up however. And some providers charge you for two circuits.
The LAGG connections are working great! I believe there is some room to improve the responsiveness of CARP failovers. I'm running a 3750 stack do you know if portfast should be on the port channel or members of the channel?
Nothing different should apply. That is all dependent on your STP configuration but it would generally be safe to have portfast enabled I would think.