@johnpoz:

Pfsense is meant to be where your draytek are.. What your doing if natting is going to be terrible and you would have to port forward to allow traffic.  Did you disable natting on your pfsense?

Dear.
  Natting on Pfsense is active beacause I can ping is ok from Lan: 10.0.2.249 to Wan network of pfsens 2: 10.0.5.x. But it can't ping to wan adapter pfsense2 is 10.0.5.249 . I think the problem relate Pfsense 2.
  Wan network of pfsense1 can't ping wan adpater of fpsense1. To do this I think don't need nat or route beacause it is like together subnet mask.
  I think that steps I need to do is:
1. Ping Ok. From wan network of pfsense1 to wan adapter if pfsense1
2. Ping Ok. From wan network of pfsense 1 to Lan of pfsense 1
3. ping Ok. From wan network of pfsense 2 to Lan of pfsense 1

Thanks for your help.
P/s: I don't check System/Advanced/Firewall & NAT Disable all packet filtering.

Well, you will still have WAN, you just won't assign igb1 to any pfSense interfaces so untagged traffic on that interface will be dropped. If you look at the interface names you will be using they will be igb1_vlan80 and igb1_vlan90.

Found a similar issue here:
https://github.com/opnsense/apinger/issues/3

The problem I'm seeing is exactly the same as this post - " reetp commented on Aug 10 "

I also found more wan failover / recovery issues on the day I posted this.  I didn't save the links though.  No solutions there anyway.  Maybe old/known problem.

Also described here.  Odd that DHCP on the WAN causes problems, but static wan ip doesn't. 
http://serverfault.com/questions/611664/pfsense-dropping-gateway-interface-randomly

Yes, packet capture led to a not-yet-powered-off AP that for some reason still was cabled to the network and had the same IP as the gateway. From time to time the IP collision was won by that AP and the redirects began. All hail Wireshark, but i had to wait until today to get time to do some real packet capture…
Thanks for the help.

I forgot to mention, but of course this is enabled too:

Not sure what your trying to route?

So on your lan interface rules set the gateway you want these clients to use.  On your opt2 (lan2) rules set specific gateway to use.

You going to want to have a rule above these rules that call specific gateway to allow traffic between your lan and opt2 network - if you want to allow this traffic.

Also your going to want to make sure your outbound nat is setup correctly.. So I assume these pfsense wan address are made up.. since one is rfc1918 space and the other is public space at 11.11.11, etc.  So are both public or is one actually 10.10.10 and you made up the other one that is public to hide it?

Oh thanks. That's exactly what I did: I used different monitoring IPs for both gateways.

Can't comment. My ISP provides "Internet services needed by TV or anything connected to it" through IPTV's gateway.. Browse internet through TV settop box or whatever..

If you don't have a switch that supports vlans then get one.  Create a new vlan for the wireless and set your unifi AP to the vlan you set on the firewall and switch.  Create a gateway group for the 2 ATT links and set the gateway for that vlan to the group for ATT.

You will need to ad a rule or maybe 2 to allow traffic from the wifi vlan to get to the other respective networks before the primary rule with the gateway group

Thanx Mark for your reply, the problem was in the gateway menu, i should have put a value , other than "0" in "data payload" field.
But now I've got another question: why cant i acces web gui from wan using the second ip (static) ? It works only with the first wan ip (pppoe) !

huh???

Makes no sense…  So you want to replace your current router(s) with pfsense?  These routers you have at

172.28.64.2 and another one at 192.168.1.1, I assume you have different ISPs??  Why not just use pfsense with dual wan and use 1 network behind and then let pfsense load balance or failover or use policy routing to determine what host uses what wan connection based up whatever criteria you want to use, etc.

i've never watched anything pfSense related on youtube, so i don't know

Can anyone tell me how to port this feature in FreeBSD 10.3? For me as soon as I use route-to in the PF ruleset, it breaks the pfil  ordering.

For me, the input chain is ipfw –> pfil and output chain is pfil --> ipfw.

When I do not use the route-to rule, everything works fine, however as soon as I use the route-to feature in PF, it breaks the order.

you sure seem to have a lot of ports for no real reason ;)  And like to use them up via lagg that seem to just be there to use up ports not for any sort of real load balancing or failover need, etc.

What routing are you doing that you need downstream layer 3?  I doubt your pfsense box can route/firewall at 10ge - but what what sort of traffic would be going between segments that would need/use 10ge?

Why can you not just use your pfsense box as your router/firewall between all your segments and just use a switch be it the juniper or the other in layer 2 mode?  If you want line speed between say clients and your servers that are on different segments at 10ge then sure your going to need something that can do that as downstream router.

I love the 10ge and am a bit jealous to be sure.. But can you even leverage it?  What sort of speeds can you get out of your storage?

@DanC:

For switches - I'm using Ubiquiti Unifi 48 port switches.  It seems they do not have bandwidth limiting available.  My connection is a symmetrical gig.  I'd like to limit each tenant to 100 mbps (10 total in the building) and deny P2P.  Are you saying that pfSense is not capable of doing that (or at least not efficiently) across 10 VLANs

No, not at all. For that scenario I would use limiters on VLAN interfaces. That will be fine.

I'll talk with my boss and my ISP about increasing our subnet size.  I really only see this working if I'm supplying at least /30's to each tenant.  As you said - do it right the first time.  Is there anything stopping me from breaking up a subnet into mixed sizes, or is that just poor form?

For 10 customers you need at least a /26 to give them each a /30.

No, making different size subnets is fine. /31s are your friend here. You might want to leave

The way I'm planning on setting up the firewall - does that expose anything for me?  Is there a better configuration for that?  I need to make sure there are no security vulnerabilities as the LAN on that pfsense has the building's access control on it.  I also don't want to expose access to pfsense itself.

On the customer interfaces, pass anything on the firewall they need access to like DNS, then block any any any to This firewall and any management or private LANs, then pass all traffic.