Virtual pfsense - vlans - loadbalancing - gateway group

[fusionp]

Hi all,

I've been testing pfsense for some time now, and I'm about to change my current routing solution over to pfsense. I'm a WISP provider with between 700 and 1000 customers, I currently have 12 internet circuits that I load balance over, my current solution doesn't offer the "packet loss & latency" monitoring which pfsense does!

My current test setup is a virtual pfsense server, it has two physical cards, and the ESXi virtual Switch assigned to the VM has VLAN 4095 tagging, which carries all vlan's across the vswitch, i.e. trunking. I then have this connecting to a tplink managed switch, which then vlan's each internet circuit to it's respective ISP's modem/router, I set the public IP's directly on my pfsense interfaces and the gateways for each circuit are across the circuit link at the circuit providers equipment. I've created the vlan's within pfsense, I've created port 1 on my tplink switch which connects to the vSwitch as a trunk, I've created a gatewate group for balancing/failover. Communication from lan side to internet is working when I use "8.8.8.8" as my alternate monitor IP in the routing gateway. However when I set the alternate monitor IP to my circuits gateway, which is a 30ms hop to my ISP, and which responds to ICMP packets….when I do this I cannot connect to the internet...until I revert the alternate gateway IP back to "8.8.8.8".
I've also added a firewall rule to use the gateway group. Still not working.
Please see all the screenshots attached, what am I missing here, I don't believe it has anything to do with the vlan's but thought I'd add the details/screencaptures, why can I not use my circuits gateway IP address? Could it be because the latency/ping time to each circuits gateway is 30ms? Could it be too low? Any help greatly appreciated!

It's been a struggle to get my test setup working up to this point, but I cannot use public IP's like NTP and Google DNS as my alternate gateway monitors.

What are my other options here? Should I create separate groups for failover and load balancing?

Another question, this virtual pfsense box will push 350mb internet at peak times over the 12 circuits, as far as memory and cpu it should be fine, but does anyone think that the one physical card will cope with 12 vlans and the total data amount?

EDIT: I've just realised that whatever I set as my alt monitor IP address is the only thing I can then ping from pfsense....I cannot ping anything else...
So I guess next question is, do I need some sort of default route configured for each interface...And haven't created any additional filter rules, haven't added any additional packages, I have tested by creating a floating rule that passes all traffic to all interfaces....effectively disabling the firewall.

[fusionp]

Well, I've managed to answer and fix this myself, I had to set the gateway group on the firewall rule, I also had to change the ESXi vswitch configuration and instead of using vlans on pfsense I created seperate vlans on the port group and added a vnic for each vlan…working like a dream. Just thought I'd write back for anyone who may come to face the same/similar issues.