@v3gard: Nope, not me (I think).I don't think this is an ISP issue as my ISP has standardized on IGMPv2. The problem is that I couldn't get IGMPv3 (which was installed on pfSense) to work seamless with IGMPv2. My main problem was that the active streams never disconnected. Oh, I thought you were that user, since you also switched to MikroTik. But based on your tcpdumps you are on a different ISP. I'm currently on Swisscom. @v3gard: I assume you can solve the problem if you manage to compile a version of the igmp tools where you force IGMPv2 - but I haven't tried it myself. I'm not sure if this is going to help. I configured FreeBSD now that it should use only v2 as default. Additionally I set the force_igmp_version in sysctl to "2". I have read somewhere, that igmpproxy uses v2 as default for downstreams. For upstreams it uses the version configured in the kernel igmp-module of the FreeBSD. This didn't really help for now. But based on your description, my problem might be a different one. At least the symptoms are different. As soon as I configure an upstream IGMP proxy to my ISP, my ISP router is restarting itself. This happens over and over again, as long as the upstream is active. Looking at the tcpdump didn't help at all. I just see that there are two IGMPv2 and IGMPv3 packets, both seem to be okay. After that the ISP router is gone (about 10 seconds after). Does anyone have any idea how I can solve this problem?

That makes a lot of (pf) sense like you phrase it. As a newbee, it was not clear to me that "Default" gateway behaves  this way and "Gateway Group" behaves that way, in the end of the day to me they seemed both gateways, the one called * and the other one called GTWYGRP. Anyways I hope others can benefit from this insight!

The ISP should be routing the /29 to your static IP.  You can do whatever you want with it.  You can assign it to another interface and pass all or some traffic.  You can assign VIPs.  Really depends on what you want to do.

I expect you can change to manual Outbound NAT and specify for each private-side IP subnet what public IP it shoud NAT to. You cannot use gateway groups and policy routing because all your public IPs actually go to a single gateway IP with your ISP.

So I finally wrote a basic IPv4 ACL tutorial for the Cisco SG300 series (I've been busy with work). It's really basic, it's a little guide for blocking traffic to 2 other VLANs and allowing all other traffic. With a little thinking someone could adapt it for more specific needs. http://kb.the-pds.net/?p=66 When I get some more time I'll post the tutorial on here.

After a lot of digging And Help The issue is resolved The problem was Router converted computer Yet there is something strange to me Why the latency is High on 65 now Usually between 8 and 13

Crisis averted. But no I'm cursed with Motorola NVG510's, so I only have IP Passthrough mode.  I've been through all the posts from previous folks with these same DSL modems and for whatever reason I am not afflicted with the same crap they were.  DHCP giving out the incorrect subnet, or the wrong gateway…  Maybe I can draw a decent picture for you to better understand.  Now I do understand that IP Passthrough by design should hand off the public IP to a specific internal MAC address, this doesnt happen in my setup for whatever reason.  I can, however, successfully browse the interwebs in all it's glory. I will admit though, prior to tonight I had the same subnet used for the DSL modems DHCP range and I did internally... 192.168.15.x I tried rebooting pf in an attempt to resolve the WAN2 being down from last night (which it did come back up) the problem was though, that for some reason even though the console was showing all interfaces up I couldnt even ping the LAN side.  i solved that by changing the subnets from 15 to 10 as I saw massive scrolling texts on the pf console complaining about the wan and lan sides having the same subnets. Now WAN is still down and I have a ticket open with ATT schedules for a tech to resolve it tomorrow... but as I sit right now I have WAN2-3 up and operational. [image: network_zpsukxno6b9.png] [image: Capture_zpsi76j2bjh.png]

I have seen problems when selecting a DHCP WAN that does not yet have an IP address. It happens for setting up various things, because the code is a bit dumb - the WAN interface does not yet have a real IPv4 address, and so the code complains that you cannot use it for IPv4 purposes. I think you need to get all the WANs to be up (at least have received their DHCP IPv4 addresses) in order to save the configuration in the webGUI. I have found this annoying in the past when trying to setup failover when the failover WAN is actually something that is not plugged in (and is only plugged in when needed).

Your setup as described sounds good. As long as the gateway group has all 3 WANs on tier 1, and all 3 WANs are up, then they should share the load. If there are only a couple of flows/states of stuff downloading, then you can get an imbalance because actually each individual flow/state can only be dedicated to 1 WAN. And are your rules matching all the traffic and feeding into that gateway group? (I expect so, since you are seeing balanced traffic on WAN2 and WAN3.)

I will give an OPT C that could work if you have access to both routers. If you run a routing protocol like RIPv2 on both sides both the routers will exchange their routing tables and communication should happen on both sides. To run RIP in PfSense you need to install a package like routed. You will have to run it on the opposite end too. Once that is done like Phil.Davis has pointed out with the firewall rules you should be good to go. I prefer this way because as you make changes to your network you can advertise those routes by just adding them under the RIP menu under services and the other side will know about them. Static routes are nice but you can make mistakes as your network grows. Also like Phil pointed out NAT hides the true originator of a packet so it makes it harder to block specific clients. P.S. Static routes add no over head to your router, while dynamic routing protocols add some overhead as they are constantly sending network topology updates (At least RIP does). Although I doubt you will see a CPU hit but just thought I through that out there.

@doktornotor: Really hope this thing gets rewritten or replaced ASAP. We're looking at rewriting it. I haven't found an alternative that's anywhere near complete as a drop-in replacement, though if anyone knows of something, suggestions would be appreciated. @doktornotor: Hmmm, not really for "want a fast switch over". First you need to figure out what's a genuine broken connection and what's just apinger being overzealous and doing more harm than good. (I.e., see the logs and try to find some patterns about what's still "normal" regarding delay/packet loss and when the WAN should really be flipped.) Needs local fiddling till you get some reasonable compromise; if things are flipping too often, I'd imagine this just disrupts connectivity in general, rending the failover to backup WAN more or less useless. Yes, this. You don't want it to be really sensitive (where "really sensitive" is relative to your connection's quality), as flapping back and forth across WANs unnecessarily will cause issues.

If it need to use broadcasts it needs to be on the same subnet.  If there's maybe, a DHCP option or something you can give to the guests, then maybe that would work.  No idea what that might be and would be a question for the playstation community. Ask them about playing playstation using the mobile app across routers/subnets.  It's either possible/supported or it's not.

@edmund: @doktornotor: What links? What kind of load balancing are you doing over a single uplink wire? I'd guess that it's a DSL connection?  If the individual account links are rate limited at the ISP then maybe this is possible? Perhaps it's one of those unexpected situations where something odd just works although I have my doubts as to whether this is really possible - calvinz, are you certain that both links were being established in the previous version AND that they both were transferring data? Edmund, yes it is a DSL (FTTH) connection. The rate is limited by the accounts subscribed with the ISP hence its independent. And yes i'm certain that both links are established properly and unique. I had execute test in parallel, speedtest of both were at the subscribed speed when test were done at the same time. Both connections had its own unique external IP address as well which is routable from outside.

It's in the asterisk CLI that I saw that message.  I've got the problem resolved though.  On the asterisk server, I had only the main IP listed.  I had to add to the network devices additional VLAN.  So by adding the VLANs to the operating system, I got everything to work and didn't really need to do any bridging of the VLANs with pfSense.

Can see 5.4.2.18 as latest. Overall, this looks like quality stuff, e.g. 5.0.5.10 release notes: Fixed the problem that the device in management VLAN100 is not able to communicate with devices in other VLANs or other devices on the Internet after running over about 6 to 8 hours. LULz?