The small sfp switch came in. Works great! /thread

A misconfigured outbound NAT could cause the same effect.

@viragomann: Change your pfSense WAN interface to 192.168.1.2/30 for the subnet of GW1 and add as virtual IP 192.168.1.6/30 for GW2. Now you can add separate filter rules for each WAN address and tag the packets coming in GW2 to direct responses back. That won't work to address reply-to though. Must be either a separate physical interface, or a tagged VLAN would work as well. No other option for proper reply-to functionality.

" that use the pfSense ip on the transfer VLAN as a GW." Huh??  How is that going to work - the gateway of any specific network/vlan would be an IP on that network/vlan. Do you mean they have a svi on the cisco switch in that specific vlan, and the gateway off the layer 3 switch is using the IP of pfsense in the transit network.  That is correct, but that is not how your statement reads. What are you rules on pfsense, and routes?  Are you pushing something out a specific gateway? What does this hypervisor trunk?  So you have vms in multiple vlans on there? So lets say you have 172.7.100 as vlan, and 172.17.110, and 172.17.120, 172.17.130..  These all point to say 172.17.x.1 as their gateway which all resides on the switch, except for the network that is hanging off pfsense, this 172.17.x.1 sits on pfsense.  And then you have a transit of say 192.168.0.0/30 So lets see your firewall rules for your home vlan and your transit network and your routes on pfsense.  So attached is how I would see your network, basically your esxi host is just switch with vlans hanging off of it that you have a trunk connecting that to your layer 3 cisco switch.  Where the vlans on that switch are all pointing to the svi on the cisco layer 3 for their respective vlans. Is this correct? [image: transitsetupvlans.png] [image: transitsetupvlans.png_thumb]

That's exactly what you want to do.

@johnpoz: "10.1.0.0/16" Really – why???  For what possible reason could you have to use a /16 on a lan segment??  That is a summary route type of mask, not a something you would put on actual network. Well AFAIK there is no performance impact or any other negative unless you actually put 65k devices (read: many devices) on such a subnet.  We have way more than 254 devices, so a class C subnet is not going to work.  I guess /20 would have been better, but it makes the IP address ranges harder to read and I wanted the new IT person to be able to quickly understand the network without having to figure out netmasks. Not an excuse, but since we only have about 700 or so devices I do not see an issue? @johnpoz: "LAN2 GW?" Why would lan2 have a gateway??  If it has a gateway its not a lan interface but a wan interface..  Do you mean you created a gateway in pfsense?  Or you actually put gateway on lan2 interface? Perhaps we have different reference points, hence our terminology does not align.  LAN2 GW referred to the point of view of a device attached to the LAN subnet.  It would see the LAN2 interface (IP) on the pfSense as its gateway for its subnet.  The LAN2 interface in pfSense does not have an upstream gateway as it is a LAN interface as you correctly mentioned. @johnpoz: If you do in fact have downstream router then pfsense should be connected to such a router with a transit network, or you run into asynchronous routing issue when devices from lan between pfsense and your downstream and devices in the downstream network talk to each other. I think this might be the issue.  Unfortunately I do not (yet) know the internals of what is behind LAN2 (Sonicwall firewall in my case, in your picture - the 192.168.1/24 network is unknown to me), I just know that LAN2 of my FW is attached to a Sonicwall firewall that has behind it some network 10.5.1.0/24. I fixed this by adding a static route - seems like the ping responses were a fluke and it did not work reliably as per your explanation.  A static route from 10.1.0.0/16 to 10.5.1.0/24 via LAN2 was the key. Thanks for your assistance.

Solving the connection problems would be the best solution, but this isn't possible at the moment as there are no alternatives for a more stable connection here. The side with the connection problems is the openvpn server side. I have now changed my openvpn configuration according to the "Multi-WAN OpenVPN" documentation. I should have found this document earlier. This setup allows the pfsense openvpn client boxes to connect to whatever connection is currently available and would not result in connection drop if the other connection comes back online again. This seems to work.

Where is your multiwan? Connect your comcast into 1 wan interface in pfsense, and your other wan into another pfsense interface.  Create whatever networks you want behind pfsense. https://doc.pfsense.org/index.php/Multi-WAN

I guess that 10.1.1.251 is not the default gateway for that network. I'm sure the pings are getting to the destination but are not getting back. If this is case, you need a static route on the default gateway of that network, routing 10.27.x.x through 10.1.1.251 Anyway, don't you have a VLAN capable switch? Can't you move one PC at a time by switching them to a different VLAN? Having several layer3 domains within the same layer2 domain is never a good idea

Sure thing! :) I've used R7000 in AP, but, recently i purchased/ordered UAP-AC-PRO. UAP-AC-PRO should support VLAN configuraiton.

Define your gateways (under system > routing > gateways tab), create a permit rule on your LAN interface that matches your private IP address 10.0.0.2, under this rule in the advanced option, select gateway, choose your gateway you want the rule to use. Make sure that this rule you created is listed ABOVE the permit any rule at the bottom of the interface rules.  (this is essentially a policy route matching 10.0.0.2 route out WAN2) There are other quirks you gotta watch out for as well such as making sure you have a NAT rule in place for allowing 10.0.0.2 to be natted out the 2nd internet connection.  This is really a basic configuration you're asking for, it gets complex once you start placing VPN's and DMZ interfaces in the mix, but not so bad once you start to understand WHY it is the way it is.

This is what a packet capture looks like 15:11:02.074448 IP 10.3.3.3.27036 > 255.255.255.255.27036: UDP, length 52 15:11:04.074080 IP 10.3.3.3.27036 > 255.255.255.255.27036: UDP, length 52 15:11:09.073963 IP 10.3.3.3.27036 > 255.255.255.255.27036: UDP, length 52

Ah, perfect. I managed to follow tutorials on how to get Local Port Forwarding to work in Putty on Windows and it worked a treat. Thanks for the tip

After updating my draytek 130 to firmware 3.7.9 the issue was solved and PPoE pass-through was possible.

Generally, yes. You would assign the /30 to an inside interface which would chew up 3 of the 4 addresses in the /30 leaving one for the device. You would make sure outbound NAT rules do not match the traffic so no NAT is performed out WAN. pfSense understands /31 netmasks so you could do two /31 interfaces and be able to use two of the addresses. The hosts involved would need to be good with /31 addressing too. You could, perhaps, make one /31 interface and route the other /31 to the host on that interface. That way you might be able to use 3 publics on that server (think http virtual hosts, etc). I've never tried it.

You don't have two separated VLANs anymore if you extend the netmask to /22 and serve all hosts from there. That's one big broadcast domain. If you want or need to separate segments then your setup is wrong. BTW, subnetting has nothing to do with your router (pfSense or Smoothwall). They both just serve what you configured.