It was a complete Noob moment Just to go over how I got there and what I did to make it right. I added 2 network adapters to the guest in esxi Configured and got working. Added a 3rd, configured and got working Added a 4th, configured and got working Added a 5th, configured and broke the system. What I didn't realize was that the adapters to the OS somehow changed. So adapter 1 was no longer vmx1 The noob moment was that I would have caught the issue had I looked at the mac address. My end config has 1 wan and 7 lan adpaters. 3 of the lan have virtual ip addresses attached. This is all interoffice so firewall rules are completely open. Setting up NAT outbound was fairly easy, need to think a little harder about the virtual ips. Going forward I need to eliminate adapters and use Vlans, but that's a story for another day

@Piyapong: PPTP WAN 2 is different. but when i'm switching the adsl modems (Provider 1 is now WAN1) the same problem occurs on WAN1 and not on WAN2

Don't add the gateway in the interface page. Having a gateway present there makes it assume that it's a WAN and to do NAT. Just add the gateways and static routes in System > Routing. You should be able to do internet failover between the two PFSense devices as well, simply by setting up a gateway group on each with its primary WAN as the Tier 1 and the address of the other PFSense as the Tier 2.

Hi, okay - found the issue… using /dev/cuaU0.0 and if I want to use this interface, I should also tick the "hidden" checkbox "enable interface" :-D -- closed ;-)

So how is the switch configured? What I can tell you for sure, is 10MBps seems really slow for just being a hardware hit in pfsenes. My pfsense is vm on old hp 40L hardware, me doing speed tests between network segments I see better than that. 10MB would be about what the limit is for a 100mbps connection.  You sure you don't have a 100mbps connection somewhere in the setup?  Your going to have 4 ports that could have this - maybe an uplink between switches for your other vlan/network? Here is my test setup.. see attached. Lan is em1 in my pfsense, goes through a vswitch that is tied to my sg300 and this port is access with my native untagged vlan.  Then I have a em2 in my pfsense vm that native untagged is my wlan network and then on top of that are a bunch of vlans.  So this connection to different physical esxi nic than the lan nic is trunked all the way to pfsense vm nic, ie it carries tags. Then I have my desktop (192.168.9.100) that is connect to same sg300 cisco switch to a port that is native untagged lan network.  And then I have a laptop (192.168.2.216) plugged in to another switch port that is in my wlan pvid untagged. If I do a simple file copy from my pc to the laptop and see over 19.. > robocopy c:\test \\192.168.2.216\test push.zip                          --------------------------------------------------------------------------   ROBOCOPY    ::    Robust File Copy for Windows                      --------------------------------------------------------------------------   Started : Saturday, August 13, 2016 6:17:05 AM                            Source : c:\test\                                                          Dest : \\192.168.2.216\test\                                            Files : push.zip                                                        Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30                          --------------------------------------------------------------------------                           1    c:\test\                                  100%        New File              3.6 g        push.zip                  --------------------------------------------------------------------------               Total    Copied  Skipped  Mismatch    FAILED    Extras        Dirs :        1        0        0        0        0        0      Files :        1        1        0        0        0        0      Bytes :  3.601 g  3.601 g        0        0        0        0      Times :  0:03:18  0:03:18                      0:00:00  0:00:00      Speed :            19446578 Bytes/sec.                                  Speed :            1112.742 MegaBytes/min.                              Ended : Saturday, August 13, 2016 6:20:24 AM                          That is with a really LARGE file.. Have you tested both directions?  What OSes are in play are you using smb, smb2, smb3?  There could be something just going on in your file copy method that is slowing you down.. What does an Iperf test show? what does a simple iperf test show [ ID] Interval          Transfer    Bandwidth [  4]  0.00-10.00  sec  180 MBytes  151 Mbits/sec                  sender [  4]  0.00-10.00  sec  179 MBytes  150 Mbits/sec                  receiver That is to same laptop from my pc..  If I put them on the same lan sure I see much higher.. [ ID] Interval          Transfer    Bandwidth [  4]  0.00-10.00  sec  1.09 GBytes  935 Mbits/sec                  sender [  4]  0.00-10.00  sec  1.09 GBytes  935 Mbits/sec                  receiver > robocopy c:\test \\192.168.9.239\test push.zip -------------------------------------------------------------------------------   ROBOCOPY    ::    Robust File Copy for Windows -------------------------------------------------------------------------------   Started : Saturday, August 13, 2016 6:35:46 AM   Source : c:\test\     Dest : \\192.168.9.239\test\     Files : push.zip   Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30 ------------------------------------------------------------------------------                           1    c:\test\ 100%        New File              3.6 g        push.zip ------------------------------------------------------------------------------               Total    Copied  Skipped  Mismatch    FAILED    Extras     Dirs :        1        0        0        0        0        0   Files :        1        1        0        0        0        0   Bytes :  3.601 g  3.601 g        0        0        0        0   Times :  0:00:34  0:00:34                      0:00:00  0:00:00   Speed :          112137010 Bytes/sec.   Speed :            6416.531 MegaBytes/min.   Ended : Saturday, August 13, 2016 6:36:21 AM So while yeah unless your pfsense hardware is capable of routing at your wire speed your not going to see the performance as switched network only..  I find it unlikely that with your hardware the performance hit would be as hard as your seeing.  Mine is on vm and see better than yours.  New esxi hardware is on my wish list and coming soon.  I just love running my pfsense on vm, but yeah its going to be a hit compared to hardware.  I might switch to hardware here soon though as saw some posts about psfense running on minnow board, etc. [image: testsetup.jpg_thumb] [image: testsetup.jpg]

be it your networks are tagged or untagged doesn't really matter.  I run multiple untagged and tagged (vlans) on pfsense works without any issues.  What switch are you using? "I already do this at the Layer2 router" There is no such thing as a layer 2 router, routing happens at layer 3. Yes the removal of downstream routing will simplify your network and allow for better control. You can have multiple network segments without he use of "tagging" if you want as long as you have physical interfaces in pfsense, and you setup your smart/managed switch appropriately or use different dumb switches for each network.

Squid can only grab HTTP transparently unless you jump through a bunch of hoops and install a custom CA on all clients to break SSL and intercept HTTPS (it's a bad idea – don't do it) If the user puts their proxy settings in the browser it can do both easily. Choosing to allow some clients to bypass or use a different VPN based on their source is easy, just policy route with a rule matching their source IP address and direct them to whatever gateway you want.

More pics [image: DMZ.png_thumb] [image: float.png] [image: DMZ.png] [image: float.png_thumb] [image: sita151.png] [image: sita151.png_thumb] [image: uznet27.png] [image: uznet27.png_thumb] [image: uznet213.png] [image: uznet213.png_thumb] [image: ng.png] [image: ng.png_thumb] [image: inosmi.png] [image: inosmi.png_thumb] [image: inosmi_ok_uznet9.png] [image: inosmi_ok_uznet9.png_thumb]

Your putting 3000 on the same broadcast domain.. That is a lot of broadcast traffic ;) /20 gives 4k users, which 1000 more than you say you would need.. To me /22 would be highest I would ever think of using for a segment with machines that would be broadcasting.  Window machines are chatty freaking things!!!  They like to send a lot of broadcast and multicast traffic out of the box..

Zinga. It's working. At some point, likely while trying to figure out how to make the ISP provided gateway a "dumb modem" or "pass-through" (according to what I've read), since it is unable to go into "true bridged mode" without losing its configuration for static ip's.. I managed to deviate from the original video in my OP. After the 1:1 NAT, I should have (and have now done) added the Firewall -> Rules, manually.  I did that in accordance with the video and, it works.  No Firewall -> NAT -> Port Forward, no Firewall -> NAT -> Outbound NAT, just Firewall -> Rules -> WAN. Ugh.  I'm sure there are some following giving the ole "SMH" and perhaps I will later down the line as well as I continue to learn, not just -what- to do, but why.  However, for now, I'm just happy things are working.  I feel comfortable I'll keep the business line and can now call tomorrow to cancel the residential. Derelict, I do greatly appreciate your assistance.  I hope I didn't frustrate you/matters too much.  I'll learn to walk one day, much less, get out of diapers.  And I promise to pay it forward once I know my knowledge is sound and am within my limits to assist properly.

Hello, could it be an netmask issue ? as the gateway is in the same subnet as clients ?

Well thanks for your respond Derelict but i think you miss understand me. I just didn't want pfsense to flush the states and causing disconnects when a dsl goes down Our problem has been resolved from the following setting: System/Advanced/Miscellaneous/Gateway Monitoring -> State Killing on Gateway Failure (Flush all states when a gateway goes down) Thanks a lot for your time and your help.

What is the source network you are connecting from? What are the WAN-side addresses of the upstream devices? That all looks like it should work if the port forwards in the upstream devices are correct.

"WRT-54GL" My gawd.. That hardware is what 10 years old.. How could you be still using G for your wifi??  You have to be faster just using your phone cell connection.  G would be like dialup.. Dude time to update… really!!! I can hear it now as people come over to the place to use your wifi Hey buddy whats your wifi password, oh shit G - no thanks I think I will just use my data plan, I want the info like this week ;)

Surely, this will be the best way. So you have both WANs on pfSense and the box can manage a failover. And also a new version of pfSense will be recommended.  :) My first version was 1.3, but only for play around.

what is the mask on this 10.90-91 network? (private network) and the lan network.. Its quite possible they overlap if your using say default /8 of a 10 network. Also what are the rules on your lan interface of pfsense if your dong any sort of policy routing out a specific gateway you have to have rules that allow the traffic to the other opt1 network before sending out a gateway, etc. Also what is the other vm - is it running any sort of software firewall, windows for example out of the box will block icmp from other than its local network.