@heper: is there a good reason your are making it more complicated by firewalling the "path" through the pfsense? I am transitioning multi-site company, something like this, star + redundant path topology: [image: Full_net.png] First reason, once I've done it goes to the local supporter. Which might firewall, block or screw it up by mistake. So I assume, it is already screwed :) And network must resist to such things from VPS A and B prospective Another reason, all this schema assume that traffic won't pass through the branch offices. It doesn't make sense anyway - kind of switching terminals in Frankfurt via Chicago @heper: so basically if "link-A' goes down, you don't want server2 to be able to reach server1 through the pfsense ? what do you mean by this? where ping won't go to the interface on the same machine, but going back through the seemingly direct path. Let's assume following fragment, where I set metrics on Company C, VPS A and VPS B. And trying to ping some interface (not shown) on VPS B, where central DNS server reside. : [image: Partial_net.png] Let's assume, tun0, 10.2.0.1 is this non P2P interface on VPS B tun1, is P2P interface on VPS,  [VPS B to VPS A P2P segment, path for ping red arrow 2] If I ping from PC on Company C, ip 10.2.0.1. Ping (request part) goes via tun1 on VPS B (Red arrow 2). And it is last point where it is seen. I assume, #3 is ping suppressed by loop prevention code If I disable Company A (which doesn't have metric set anywhere yet). Leaving net with VPS A, VPS B, and Company C, ping goes to 10.2.0.1. Moreover, on VPS B, ping 10.2.0.1 -I tun1 will drive ping away, as it assume path via Company A as more direct one. Obviously, ping never reach same machine interface, even it is less than 1ms away Yes, metric is not set up everywhere. Even if Company A maintainer won't screw up, it can be the case for non reachable part. VPS A to VPS B, metric 50 Company A => VPS 1, metric 10 (very fast line) Company A => VPS 2, metric 10 (very fast line) So, Company A will attract traffic to go via it's fast lines just to firewall it. P.S. What is the editor network used here  to quick draw diagrams, local or online? Coming from development field, I don't have any tool set worked out for it yet.

It's still listed in Diag>Routes with the same gateway IP? If it even lets you run the "route add" command without an error, there isn't a matching route there. Where there is, it'll just spit out "route: writing to routing socket: File exists". Something elsewhere in your configuration is deleting the route. Maybe OpenVPN, or a dynamic routing protocol. Or maybe a virtual IP configured that overlaps with that static route. You'll definitely want to find and fix the root cause, adding a route every 5 minutes is a bad hack, and needing to do so indicates you have a config problem somewhere.

Just to close this thread out. I tried the the failover goup stuff again and ran into the issues with services on the pfSense box (Squid and unbound) continuing to use  the default route no matter what I did.  I see that there are many threads raising the same issue. While trying to understand/resolve that I came across the "Enable default gateway switching" option.  Which I had previously been unaware. I now have this option enabled, two defined gateways (WAN and one of the microwave links), no gateway groups and no policy routing.  This achieves what I wanted to accomplish very simply. Thanks Mike

Your ICMP rule on LAN is IPv6, not IPv4.

I resolved it.  It turns out there was an issue with tagging on one of the ports on my switch.  It works now. Thanks for the reply.

Create a firewall rule –- In the advanced options, there is an option to choose a Gateway.

I don't think this is a firewall issues, it is a routing issue. When I run a trace route from an OpenVPN client trying to reach one of the branch offices this is what I get: C:>tracert 192.168.23.1 Tracing route to 192.168.23.1 over a maximum of 30 hops 1    15 ms    14 ms    12 ms  172.16.1.1   2    11 ms    10 ms    10 ms  domainname.com [206.201.5.180]   3    *    saddleback-rs-mv4 [64.58.128.9]  reports: Destination net unreachable Trace complete. So the HQ box is sending the packets to the default route instead of sending them through the proper IPSec to the branch office. Looks like it's not aware of the IPSec it's running on the same box. Thanks.

we were offered a grant to get this specific system. Nice gig for them.  I'd like some work like that.

@dtikev: Okay, now here are my current issues and questions: Is the Cisco even needed any longer? We initially tried to remove it from the equation (hired a consultant) but never got it working. It's been a while but I think the /30 handoff was a problem as we are using multiple IPs of the /24s to actually get traffic into our servers. If the Cisco can be taken out of the picture, how can that be done while still allowing all of the above? How do I make the server LAN go out AT&T and the user LAN go out TW? It's not working now… everything seems choose AT&T Once we're routing out correctly (above) how can I make a select group on the user LAN actually use AT&T? There's more I'd like to get figured out, but these core issues are on my current goals sheet. If it matters, the netgate hardware I'm running pfSense on has 6 cat5 gig interfaces and with the current setup I'm using 4 of them. I was hoping to add in our copper T1/last resort down the line but if all 6 ports are needed to get things working as above, the last resort failover option can be handled another way. I think theoretically you shouldn't need the Cisco anymore.  If your servers are all private IPs and you are NATing them, maybe you can create the Virtual IPs in pfSense and NAT them out the /30 connections that would come directly into the pfSense.  Just curious, why didn't you pay for pfSense support hours instead of hiring a consultant?  They should be able to tell you what is possible and help you with the config, and I doubt it would be more than paying an outside consultant. As others stated above, I think you need policy routing, either on the Cisco (if you keep it) or the pfSense (if you don't).  If you keep the Cisco, why do you have two links from the Cisco to pfSense?  Get rid of one, do the NAT on the Cisco, and set up policy routing on the Cisco to route the appropriate subnet or IP out the correct interface.  If you don't keep the Cisco, you probably want to set up gateway groups in the pfSense.  You can either load balance or create two gateway groups, each one sends traffic out an ISP and fails over to the other. I don't think you should need more interfaces on the pfSense unless some of your servers actually have public IP addresses and you're getting rid of the Cisco, then you need a physical interface on the /24 to connect the server to.  You could still use VLANs though to avoid extra interfaces.  If you have a T1 you will obviously have to connect it to a router first though. Hope this helps, let us know how it turns out.

cmb, I have a 2 port LAGG trunk with the internal network vlans. I have a single port trunk with the WAN network vlans. I have a GIF IPv6 tunnel. Ip my examples the VIP (192.168.100.161    link#25    UHS    0    16384    lo0) is removed, but the route statement like "192.168.100.0/24    link#25    U    681    1500    em1_vlan5" is not removed. The test is to have devices connected to different links using the same IP. Setup an alias IP to access the system on one of the links and make sure you can access it. Now delete the alias IP and then create it for the other link. Now try to access that device. Obviously there has to be something unique between the two device to ensure you know to which you are connected. Thanks, Rhongomiant

Just wanted to update this thread, there was nothing wrong with my pfSense config, this was an ISP port blocking issue. Thanks to all that took the time to review the thread.

The router I am using is an Asus AC56U which is a dual processor unit. Pretty much what I did was delete my OPT interfaces and rules. Recreate the inferfaces and rules giving the OPT an address in the same range as the old standard 192.168 and wallah; came right up. Thank you so much for your help with this. I clicked the thank you button on your last post.

"There's also another LAN2, 172.21.1.X/24, which is connected to LAN1 through a different gateway than pfSense." While I completely understand this scenario, normally you would NEVER see it, because normally you wouldn't ever set it up that way.  You would create a transit connection to this downstream router from pfsense so you never have asymmetric routing and you also control access from this downtream network to your lan network.  With a downstream router pfsense has no control over traffic initiated from that network in to your lan network. Or you don't even use that downstream router and just connect that segment direct to pfsense. While I agree with you that icmp redirects are part of the suite - I don't think they are intended for such a design flaw from the get go ;) I see that redirects can be useful - but I would never use them in this case, such a case should never be setup. Another option is if you want host on lan to use that downstream router to get to some downstream network, then just create a route on it telling it so, this is not the best solution - but also another way to solve asymmetric routing problems.

Not now. https://redmine.pfsense.org/issues/4231

I'm not sure I follow? I'm using the IPs I've been assigned by OVH, I can't just assign it an IP address that I don't rent. Again for clarification the IPs on the pfSense VM are: WAN: 178.xx.xx.104/32 (Public IP for LAN) WAN Alias 1: 178.xx.xx.195/32 (Public IP for OPT1) WAN Alias 2: 178.xx.xx.136/32 (Public IP for OPT2) LAN: 192.168.1.254/24 OPT1: 192.168.2.254/24 OPT2: 192.168.3.254/24 Main server IP: 188.xx.xx.172/32 Upstream WAN Gateway: 188.xx.xx.254/32 Bare in mind this configuration isn't the problem I am having, the upstream WAN gateway works fine as the pfSense machine can sucessfully route packets to and from the internet, it's a problem with NAT/Firewall rules I would of thought (port forwards not working for OPT1 and OPT2).