Thanks for your answer! I've forwarded your answer to the network team of my provider and they told me, that this isn't supported. Well… Ok, it's no issue of pfSense. Great! :) #Closed

Make sure to set a rule on each internal vlan interface within pfsense to pass said traffic to your given gateway for each vlan. Firewall rule, on vlan20 create rule that sends traffic from * to * to use gateway WAN1 Then similar rule on vlan30

When I recently set something up similar, I could only ever get one WAn connection operational, reason is the MAC address used by the vlan's on pfsense are the parent interface MAC, are your three ISP connections all from the same ISP? You can try setting a bogus MAC on the vlan interfaces to see if that gets it working….it may work for you, I tired it but it didn't work for me, I had to use a virtual machine, vswitch/vnics to do my vlan'ing. Also make sure your ports connecting to the ISP devices are not tagged.

This is an internal routing question, not really a PFS query. You have to set your routing within the internal DLink switch - assuming it's manageable. You will also need to get the phone company to set a similar route back to your network on their switch to ensure traffic can be returned. Otherwise, if you only need to set a route between two hosts across both networks, you can assign these statically on each of the hosts in question. If you have more than two or three hosts requiring the routing info, then it would be better to set this as switch level, otherwise managing the whole thing could get messy.

Lan Side consists of a 3750x with L3 routing on. If this is a Layer3 switch you can or must be ensure where and who should route your entire network traffic, the pfSense or the Layer3 Switch, also the entire VLANs. Often and mostly are some points against or for the one or other method! This might be tending and pending on other needs or benefits you was not telling around in the opening post. Doing the routing only with pfSense might be a single point of failure if the pfSense is failing Doing it without pfSense some things could be not working really smooth and liquid. But let the pfSense routing only the WAN - LAN and WAN - DMZ traffic would be bringing you up to own two routing points and if then the pfSens is failing the rest of the LAN is working well through the Layer3 switches.

I found another post related to interfaces and saw a PHP reference, so I did some digging and found where the mismatch reallocation is triggered, so I will be looking into the /etc/rc.bootup file and see if I can disable an interface if it is missing and is a ue type.

@Phishfry: That is how I set up pfSense MiFi's - Modem on WAN, Local Ethernet on LAN and ATH0 on OPT1. Depending on your carrier you may want to consider OpenDNS. You may want to enable "Disable Gateway Monitoring" if you are on a data  limited plan. It constantly pings the gateway eating up data. It is under the >System>Routing tab on the menu.. Will keep that in mind, thank you.

That's what we did - we have two LAN switches so we created a small VLAN (3 ports) on each switch and put CABLE on one switch and DSL on the other. We will be scheduling a maintenance window in a couple weeks to go on site and test the various failover scenarios. For now CARP is running great and the secondary pfSense has their WAN ports disabled until we can test fully.

Right, got this figured out and it's working brilliantly. For anyone else interested; Bridging a PPPoE WAN to an OPT interface did not work for us. Although iftop did show attempted connections, no traffic passed. This may well be down to how our ISP handles traffic - I get the feeling they only allow a single MAC per subnet(!). So, to get it working, we used three pfSense boxes. One to deal with PPPoE authentication and routing, the other two to as the HA cluster. Connect the PPPoE line to the WAN interface and configure it as normal. The ISP provides the first usable address in the subnet as a /29. Reconfigure the LAN interface to something suitably small (/29 will do) and stick it in it's own VLAN - we left room at the beginning of the subnet for the cluster. Add a static route from the LAN interface to the rest of the network, via the IP that the cluster's CARP VIP will use in that VLAN (so you can manage it from the main network). Assign another interface as OPT1 and configure it with the next available WAN IP as the gateway, and the next IP after that for the interface itself. Set the MTU to 1492. Connect something to OPT1 and set it to an available WAN IP using the pfSense interface OPT1 address as the gateway. Test it - you should be connected using a public IP. Disable all packet filtering/NAT on pfSense, and turn on all NIC offloading if your NICs support it. Turning on fast forwarding and device polling is also a good idea, if your NICs support it. Reboot pfSense and test again, it should still work. Disconnect your device from OPT1, and connect a switch to it instead. Connect both your pfSense cluster box WAN interfaces to the switch, and configure each with a WAN IP using the upstream pfSense box as the gateway. If you have a /29 subnet for your WAN, this will leave two WAN IPs to use for CARP VIPs. Follow the normal HA instructions from here. One thing that's obvious is that you still have a single point of failure in the PPPoE authentication pfSense box, but that would also be true of using just one WAN connection. We are using two pfSense boxes for the PPPoE stuff, to two different WAN connections, and an EFM connection. They all have /29 subnets, and all go through a stacked pair of switches. The connections from the pfSense PPPoE boxes use LACP so a single switch failure won't kill a connection, same for the WAN input to the main boxes. Each WAN is in it's own VLAN. EFM is the only connection that only goes to one switch, but there isn't much we can do about that. If a switch dies, we can just re-patch it to the other. The above is not the most efficient use of IP addresses, but it solves the problem we were looking to solve, with an ISP who have chosen to implement a frustrating setup. EDIT: On the PPPoE box, you'll also need to; add an inbound rule to allow traffic destined for the OPT interface add an outbound rule to allow traffic not destined for LAN out of the OPT interface

What do your outbound nats look like? If they are set to manual you may need to add one for the ONLINE_DHCP

Thanks chris4916. Thanks for your help and advice on this setup. I am completely a noob at this, so I am glad you were able to explain and help me with this problem.

but could someone tell me at least if it's possible to create virtual Nic with actual Pfsense Only at the WAN Port or interface this will be needed and be able to realize. Go the way @jahonix was showing you up and then be happy with VLANs.

Do you mean this doesn't work when forcing gateway in FW rule ?

up …  ;)