why do you think the dns server for your opt1 network would be the lan interface of pfsense? Normally as kurianofborg stated you would just setup your dns on pfsense to also listen on this opt1 interface.

BTW what mask did you put on your pfsense opt network.. I would hope you made it something realistic like a /24 and not a /8 because its 10.x.x.x

I really would not suggest you use your switch in layer 3 mode doing routing until you fully understand routing..  Once you do you will most likely see there is no point for the switch to be doing it, and you loose all the nice features of pfsense doing the routing/firewall between your segments.

If you do decide to use your switch for routing, that keep in mind pfsense will need to be connected to it via a transit network or your going to run into asymmetrical routing issues.

I wanted to end this with the solution(s) to my problems.

As said in this string, I didn't have the right interface assignment done.  But it should have worked after that… but it didn't.
I screwed around with possible settings on the SG300 because I don't know the network world as well as others, but it turns out, that wasn't the issue.
I had one firewall mistake.  I needed "ALL" instead of just TCP.
Eventually I manually assigned an IP on the GuestWireless SSID and it was able to talk to the internet, but I still couldn't get an IP.
I used the Capture network traffic on the PFsense to verify the DHCP request was going through, but no answer was coming back.
I rebuilt everything including the vlan and interfaces, but that wasn't the issue.
It turns out that I had to hit STOP on DHCP and then Start on DHCP and everything started working.

The moral of the story (I think) is if you mess with interfaces, you need to stop and start DHCP service.

Rich

@User40405:

Ok so now I have managed to get whole Server PC to use WAN2 and rest of network to use WAN1. Now the question is how to get Plex Server on Server PC to use WAN1 but everything else on Server PC to use WAN2?

Ypu misunderstand the way this works.
This is not outgoing communication but incoming.
In order to ensure that this service (Plex) is used only used on WAN2, you have to configure your external (public) DNS so that access is done only from WAN2.
There is nothing else to be done  8)

When it comes to set-up OpenVPN with multi-WAN, one option is to configure OpenVPN server to listen on localhost (127.0.0.1) and then configure forwarding rules so that requests reaching each gateway on port configured on OpenVPN server is redirected to 127.0.0.1

This allows to have one unique OpenVPN server configuration available from multiple gateways.

Difference between HTTPS and OpenVPN, when it comes to access pfSense GUI is that authentication in order to establish tunnel can be much stronger (and therefore more secure) than simple "login / password" requested by HTTP(S) web interface.

Keep in mind that you are exposing your FW to internet is you authorise (GUI) admin access from internet  :o

Well, no response to my problem, so I did the right thing to do, and search in other threads to find a possible solution.
I made some progress, but now I'm facing a new problem.

Sooo, I learned that using the L3 switch as a router in this case is called a downtream router.
Also, leaving the routing job to the L3 switch means that there's no need to load vlans and interfaces in Pfsense. what is needed is a different vlan between the switch and Pfsense (a transit network).

I defined my transit network as 100.0.1.0/16 in vlan 100. Made the vlan 100 in my L3 switch, interface IP adress of 100.0.1.10/16 with one port tagged. Deleted all vlans in Pfsense, and created vlan 100 with interface IP adress 100.0.1.20/16. In routing, made a gateway pointing to the switch interface (100.0.1.10) and marked as default. Defined static routes so Pfsense can find the networks behind the switch. The networks fall under 10.0.0.0/24, so I made only one route with this adress and the gateway pointing the switch I made in the previous step. In the switch, defined ip route as 0.0.0.0 0.0.0.0 100.0.1.20. Defined the rules to pass any in LAN, and in the interface of the vlan.

Now, from a host, I can ping pfsense, no problem there.
The thing is, there's a loop now between Pfsense and the swith.
From a host, if a traceroute to 8.8.8.8 it keeps jumping from the switch and pfsense.
If I ping from Pfsense to 8.8.8.8, it says time to live exceeded error.

I tried to change the gateway of the static route to WAN, but then the host can't ping anymore Pfsense, nor have internet access. But if I log into the CLI of the L3 switch, the switch can ping Pfsense AND 8.8.8.8.

Any ideas or help guys? I'm going bald pulling my hair with this…

Even if he puts route on router A to get to the network behind router B… This PC on the 192.168.160 network is on the transit network - That is BAD design, and as cmb mentions you would have to use host routing on that PC or any devices on that transit or your going to have a bad day with asymmetrical routing when that box talks wants to talk to stuff behind B or B wants to talk to stuff on the transit, unless router B is also natting?

To be honest why do you not just use 1 router.. This is much easier concept and easier setup for someone that does not self admitting knows little about routing and networking..

Normally routers are connected via transit networks, no devices are place on transit network other than "routers" these routers have routes to the networks behind the routers via the transit network(s)  You normally do not nat inside a rfc1918 network.

What are you actually using for these routers?  Pfsense?  Is that router A some wifi router off the shelf device?

Attached is a typical setup with downstream router via transit network.

So edge router would have routes to the networks behind downstream router pointing to the downstream routers IP in the transit.  Downstream router just really needs its default route pointing to the nat routers transit IP.  The nat router needs to know to nat ALL the networks behind it to whatever its public IP is, etc.

typicalsetuptransit.jpg
typicalsetuptransit.jpg_thumb

Okay! I got this working finally!

I am still playing around with the Protocol timing Settings, as sometimes my WAN does not renew properly.

Please see new post: https://forum.pfsense.org/index.php?topic=114389.0

if your not worried about any rules, then just copy the default lan rule to your other interfaces which is any any.

Now pretty much pfsense is just router between your local networks.  And nat/firewall to wan..

You do not need to set bidirectional rules since pfsense is stateful.  You just need to allow the traffic on the interface it first hits pfsense on, the return traffic will be allowed since there is a state already.  Rules are evaluated top down, first rule to match wins.

if you have questions on only allowing specific sort of traffic or blocking something specific just ask.

You nailed it Chris.  I had it bound to only my internal networks and localhost under Network Interfaces, and only WAN/WAN2 under Outgoing Network Interfaces.

Thank you.

I've been playing with this as well since i have connections with 6/1, 10/1, 30/2 and 100/8 speeds, all together in the same pfSense box. I'm starting to think the only way to achieve this is to create 1 VM for each wan link, with it's own pfsense and within them make the up/down limiting with the last box just doing all the current traffic. I really don't like that solution.

So I tried luckman212's suggestion and it worked! :)

I think something like this should be commented on within Sticky Connection topic.

Cheers all  ;D

You probably have an incorrect configuration for the monitor IP addresses and DNS servers for each gateway.

Set each gateway monitor IP address to a DNS server on System > Routing and set the DNS server to the same gateway under System > General Setup

If you have some things mixed up, such as all the DNS servers pointing to your WAN and also using them as monitor IP addresses on the other WAN gateways, that can cause exactly the behavior you describe.

See there's your problem - using logic and sense while evaluating how i got here:-)

I had no idea what pfsense could and couldn't do before i started this.  Never used it until a week ago, only marginally aware of it. I had the perception that it is a platform of components that one can use.  You see pfsense as a firewall - i see it is a comprehensive 'security platform' of things i can use as i see fit. Both are true.

I don't need the pfsense firewall / NAT(because i can't turn off the one i have.. and don't want to double NAT or double firewall)

As such all the outbound connections i want to block with my off the shelf pfsense box are done at the squidguard / suricata level.

This can be done in one of two locations - between the cable modem and my router or between my router and the rest of the network.  Either way i need to bridge as I want it transparent, i don't want to mess around with wpad.

pfsense gives me easy to use turnkey system to do this, i don't want to install linux - i have no no interest in maintaining a linux machine -  i bought a box with pfsense installed that does the job i need it to in an easy way is great - not sure why you are so horrified about what pfsense modules / features and packages I do or don't choose to use

I wasn't worried about the performance - I was just checking to see if the bridging might be causing the drop in throughput - turns out comcast mucked up my connection - pfsense in transparent mode has no impact in my scenario (home use)

I have a turnkey tool that does what I need it to and easy to get working and maintain - bloody brilliant in my book.  If you want to install a linux distro and install packages on that  etc etc more power to you i won't judge.  But that's not for me.  Having just learnt about security onion - maybe that's more suited to my need, thats where i will play and experiment next.

So consider this just a journey of discovery for me - i now understand what pfsense is.  I have made no call on what I will finally do.  And if and when UBNT let me turn off the firewall / NAT on my USG device i will do that and likely revert the pfsense to non-bridged mode and use it as my NAT and firewall at that point.

Hey, looking to do the exact same thing. Just one question, will the server still be on the local network even though it uses a different wan network? Basically will I be able to use Plex server that is hosted on the server that is using a different wan connection? Thanks.