IIRC that's a limitation of pf. It can't use anything other than round-robin or round-robin+sticky when specifying multiple addresses in that way. To use hashing it would have to use a network in that context, which doesn't make sense for gateways. If you want to see something like that, you'll have to advocate to pf directly (OpenBSD) or perhaps FreeBSD since the pf in FreeBSD has diverged from that of OpenBSD.

Thank you  ;)

@cmb: That's the expected behavior. You need source NAT to access the system with backup status from a VPN via the system with master status. Hi CMB, Thank you for the clarification, I can understand why that might be the case, a bit unfortunate as the Source NAT feels like a bit of a hack but I'll try it out and continue with that :) Thank you! Edit: Just tested it and it works like a dream, anything to get rid of crappy static routes. Fantastic, thank you again!

Yes, I have traffic that matches Steam (UDP Destination Port 27000:27030 in this case) that goes out the default gateway. I can see these states under the Ficus interface in the Diagnostic > States viewer with matching destination ports. I would like to see outputs of pfctl -vvsr and pfctl -vvss when the firewall is in this mode. That will show exactly which rule is passing the traffic in question. It would be especially helpful if you could clear all states, generate the traffic in question, then take these samples. I realize it might be kind of large. There's probably a simple explanation for what you're seeing. Just don't know what it is yet.

Anyone?

why do you think the dns server for your opt1 network would be the lan interface of pfsense? Normally as kurianofborg stated you would just setup your dns on pfsense to also listen on this opt1 interface. BTW what mask did you put on your pfsense opt network.. I would hope you made it something realistic like a /24 and not a /8 because its 10.x.x.x

I really would not suggest you use your switch in layer 3 mode doing routing until you fully understand routing..  Once you do you will most likely see there is no point for the switch to be doing it, and you loose all the nice features of pfsense doing the routing/firewall between your segments. If you do decide to use your switch for routing, that keep in mind pfsense will need to be connected to it via a transit network or your going to run into asymmetrical routing issues.

I wanted to end this with the solution(s) to my problems. As said in this string, I didn't have the right interface assignment done.  But it should have worked after that… but it didn't. I screwed around with possible settings on the SG300 because I don't know the network world as well as others, but it turns out, that wasn't the issue. I had one firewall mistake.  I needed "ALL" instead of just TCP. Eventually I manually assigned an IP on the GuestWireless SSID and it was able to talk to the internet, but I still couldn't get an IP. I used the Capture network traffic on the PFsense to verify the DHCP request was going through, but no answer was coming back. I rebuilt everything including the vlan and interfaces, but that wasn't the issue. It turns out that I had to hit STOP on DHCP and then Start on DHCP and everything started working. The moral of the story (I think) is if you mess with interfaces, you need to stop and start DHCP service. Rich

@User40405: Ok so now I have managed to get whole Server PC to use WAN2 and rest of network to use WAN1. Now the question is how to get Plex Server on Server PC to use WAN1 but everything else on Server PC to use WAN2? Ypu misunderstand the way this works. This is not outgoing communication but incoming. In order to ensure that this service (Plex) is used only used on WAN2, you have to configure your external (public) DNS so that access is done only from WAN2. There is nothing else to be done  8)

When it comes to set-up OpenVPN with multi-WAN, one option is to configure OpenVPN server to listen on localhost (127.0.0.1) and then configure forwarding rules so that requests reaching each gateway on port configured on OpenVPN server is redirected to 127.0.0.1 This allows to have one unique OpenVPN server configuration available from multiple gateways. Difference between HTTPS and OpenVPN, when it comes to access pfSense GUI is that authentication in order to establish tunnel can be much stronger (and therefore more secure) than simple "login / password" requested by HTTP(S) web interface. Keep in mind that you are exposing your FW to internet is you authorise (GUI) admin access from internet  :o

Well, no response to my problem, so I did the right thing to do, and search in other threads to find a possible solution. I made some progress, but now I'm facing a new problem. Sooo, I learned that using the L3 switch as a router in this case is called a downtream router. Also, leaving the routing job to the L3 switch means that there's no need to load vlans and interfaces in Pfsense. what is needed is a different vlan between the switch and Pfsense (a transit network). I defined my transit network as 100.0.1.0/16 in vlan 100. Made the vlan 100 in my L3 switch, interface IP adress of 100.0.1.10/16 with one port tagged. Deleted all vlans in Pfsense, and created vlan 100 with interface IP adress 100.0.1.20/16. In routing, made a gateway pointing to the switch interface (100.0.1.10) and marked as default. Defined static routes so Pfsense can find the networks behind the switch. The networks fall under 10.0.0.0/24, so I made only one route with this adress and the gateway pointing the switch I made in the previous step. In the switch, defined ip route as 0.0.0.0 0.0.0.0 100.0.1.20. Defined the rules to pass any in LAN, and in the interface of the vlan. Now, from a host, I can ping pfsense, no problem there. The thing is, there's a loop now between Pfsense and the swith. From a host, if a traceroute to 8.8.8.8 it keeps jumping from the switch and pfsense. If I ping from Pfsense to 8.8.8.8, it says time to live exceeded error. I tried to change the gateway of the static route to WAN, but then the host can't ping anymore Pfsense, nor have internet access. But if I log into the CLI of the L3 switch, the switch can ping Pfsense AND 8.8.8.8. Any ideas or help guys? I'm going bald pulling my hair with this…

Even if he puts route on router A to get to the network behind router B… This PC on the 192.168.160 network is on the transit network - That is BAD design, and as cmb mentions you would have to use host routing on that PC or any devices on that transit or your going to have a bad day with asymmetrical routing when that box talks wants to talk to stuff behind B or B wants to talk to stuff on the transit, unless router B is also natting? To be honest why do you not just use 1 router.. This is much easier concept and easier setup for someone that does not self admitting knows little about routing and networking.. Normally routers are connected via transit networks, no devices are place on transit network other than "routers" these routers have routes to the networks behind the routers via the transit network(s)  You normally do not nat inside a rfc1918 network. What are you actually using for these routers?  Pfsense?  Is that router A some wifi router off the shelf device? Attached is a typical setup with downstream router via transit network. So edge router would have routes to the networks behind downstream router pointing to the downstream routers IP in the transit.  Downstream router just really needs its default route pointing to the nat routers transit IP.  The nat router needs to know to nat ALL the networks behind it to whatever its public IP is, etc. [image: typicalsetuptransit.jpg] [image: typicalsetuptransit.jpg_thumb]

Okay! I got this working finally! I am still playing around with the Protocol timing Settings, as sometimes my WAN does not renew properly. Please see new post: https://forum.pfsense.org/index.php?topic=114389.0

if your not worried about any rules, then just copy the default lan rule to your other interfaces which is any any. Now pretty much pfsense is just router between your local networks.  And nat/firewall to wan.. You do not need to set bidirectional rules since pfsense is stateful.  You just need to allow the traffic on the interface it first hits pfsense on, the return traffic will be allowed since there is a state already.  Rules are evaluated top down, first rule to match wins. if you have questions on only allowing specific sort of traffic or blocking something specific just ask.

You nailed it Chris.  I had it bound to only my internal networks and localhost under Network Interfaces, and only WAN/WAN2 under Outgoing Network Interfaces. Thank you.

I've been playing with this as well since i have connections with 6/1, 10/1, 30/2 and 100/8 speeds, all together in the same pfSense box. I'm starting to think the only way to achieve this is to create 1 VM for each wan link, with it's own pfsense and within them make the up/down limiting with the last box just doing all the current traffic. I really don't like that solution.