The core issue appears to be a defect inside AT&T's cellular network. I have an MiFi which I pulled out to test, and I see the issue as well. I tested an iPhone hotspot on AT&T and it shows the same problem with both LTE and 4G. The smallest data payload acceptable is 20 bytes. I would report the defect to AT&T, but I don't know anyone inside. I'd also like to know if the issue exists in Verizon's network, but I don't have a Verizon phone to test with. If someone does, and would like to test, I'd appreciate it. No need to hook the device up to pfSense, you can test from your laptop. Just connect to the hotspot and try to ping. Example commands (for a Mac): ping -s 0 8.8.8.8   ping -s 16 8.8.8.8   ping -s 20 8.8.8.8   ping -s 56 8.8.8.8 Thanks.

Yes dude source nat it.. How does your client behind R2 know its wan is down.. So its always going to send traffic to its own gateway.. How exactly are you port forwarding via router R2 wan to something behind R1 anyway..  Some sort of failover dns on the internet?  How does client on the internet know to go to r2 wan if R1 is down or not down? Draw up your network please..

@jahonix: @marcvb: … i do not have to use lan rules ... As johnpoz already mentioned: it depends on what the hosts on your public subnet are supposed to do. A mail server surely would need outgoing rules if you want to send mail, some Windows servers would like to contact update servers quite frequently, … Thank you both i understand it now. My virtual demo servers and pfsense are working, installed a speedtest mini within iis and traffic shaping is also working. This is much better than our GTA firewall www.gta.com

Awesome. Thanks for the clarification, I was having the exact same problem, until I looked at the documents Forwarding mode is necessary for Multi-WAN configurations unless default gateway switching is enabled.

the problem would of been that when you create a new interface in pfsense, ie opt it does not create any default rules like it does on the default lan interface. So any traffic you would like devices on this network to be able to create wold have to be allowed for, example if you want them to be able to ping pfsense interface in that segment would have to be allowed, dns to pfsense or elsewhere allowed.  You could make it like the lan and put any any rule if you wanted, etc.

It was a complete Noob moment Just to go over how I got there and what I did to make it right. I added 2 network adapters to the guest in esxi Configured and got working. Added a 3rd, configured and got working Added a 4th, configured and got working Added a 5th, configured and broke the system. What I didn't realize was that the adapters to the OS somehow changed. So adapter 1 was no longer vmx1 The noob moment was that I would have caught the issue had I looked at the mac address. My end config has 1 wan and 7 lan adpaters. 3 of the lan have virtual ip addresses attached. This is all interoffice so firewall rules are completely open. Setting up NAT outbound was fairly easy, need to think a little harder about the virtual ips. Going forward I need to eliminate adapters and use Vlans, but that's a story for another day

@Piyapong: PPTP WAN 2 is different. but when i'm switching the adsl modems (Provider 1 is now WAN1) the same problem occurs on WAN1 and not on WAN2

Don't add the gateway in the interface page. Having a gateway present there makes it assume that it's a WAN and to do NAT. Just add the gateways and static routes in System > Routing. You should be able to do internet failover between the two PFSense devices as well, simply by setting up a gateway group on each with its primary WAN as the Tier 1 and the address of the other PFSense as the Tier 2.

Hi, okay - found the issue… using /dev/cuaU0.0 and if I want to use this interface, I should also tick the "hidden" checkbox "enable interface" :-D -- closed ;-)

So how is the switch configured? What I can tell you for sure, is 10MBps seems really slow for just being a hardware hit in pfsenes. My pfsense is vm on old hp 40L hardware, me doing speed tests between network segments I see better than that. 10MB would be about what the limit is for a 100mbps connection.  You sure you don't have a 100mbps connection somewhere in the setup?  Your going to have 4 ports that could have this - maybe an uplink between switches for your other vlan/network? Here is my test setup.. see attached. Lan is em1 in my pfsense, goes through a vswitch that is tied to my sg300 and this port is access with my native untagged vlan.  Then I have a em2 in my pfsense vm that native untagged is my wlan network and then on top of that are a bunch of vlans.  So this connection to different physical esxi nic than the lan nic is trunked all the way to pfsense vm nic, ie it carries tags. Then I have my desktop (192.168.9.100) that is connect to same sg300 cisco switch to a port that is native untagged lan network.  And then I have a laptop (192.168.2.216) plugged in to another switch port that is in my wlan pvid untagged. If I do a simple file copy from my pc to the laptop and see over 19.. > robocopy c:\test \\192.168.2.216\test push.zip                          --------------------------------------------------------------------------   ROBOCOPY    ::    Robust File Copy for Windows                      --------------------------------------------------------------------------   Started : Saturday, August 13, 2016 6:17:05 AM                            Source : c:\test\                                                          Dest : \\192.168.2.216\test\                                            Files : push.zip                                                        Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30                          --------------------------------------------------------------------------                           1    c:\test\                                  100%        New File              3.6 g        push.zip                  --------------------------------------------------------------------------               Total    Copied  Skipped  Mismatch    FAILED    Extras        Dirs :        1        0        0        0        0        0      Files :        1        1        0        0        0        0      Bytes :  3.601 g  3.601 g        0        0        0        0      Times :  0:03:18  0:03:18                      0:00:00  0:00:00      Speed :            19446578 Bytes/sec.                                  Speed :            1112.742 MegaBytes/min.                              Ended : Saturday, August 13, 2016 6:20:24 AM                          That is with a really LARGE file.. Have you tested both directions?  What OSes are in play are you using smb, smb2, smb3?  There could be something just going on in your file copy method that is slowing you down.. What does an Iperf test show? what does a simple iperf test show [ ID] Interval          Transfer    Bandwidth [  4]  0.00-10.00  sec  180 MBytes  151 Mbits/sec                  sender [  4]  0.00-10.00  sec  179 MBytes  150 Mbits/sec                  receiver That is to same laptop from my pc..  If I put them on the same lan sure I see much higher.. [ ID] Interval          Transfer    Bandwidth [  4]  0.00-10.00  sec  1.09 GBytes  935 Mbits/sec                  sender [  4]  0.00-10.00  sec  1.09 GBytes  935 Mbits/sec                  receiver > robocopy c:\test \\192.168.9.239\test push.zip -------------------------------------------------------------------------------   ROBOCOPY    ::    Robust File Copy for Windows -------------------------------------------------------------------------------   Started : Saturday, August 13, 2016 6:35:46 AM   Source : c:\test\     Dest : \\192.168.9.239\test\     Files : push.zip   Options : /DCOPY:DA /COPY:DAT /R:1000000 /W:30 ------------------------------------------------------------------------------                           1    c:\test\ 100%        New File              3.6 g        push.zip ------------------------------------------------------------------------------               Total    Copied  Skipped  Mismatch    FAILED    Extras     Dirs :        1        0        0        0        0        0   Files :        1        1        0        0        0        0   Bytes :  3.601 g  3.601 g        0        0        0        0   Times :  0:00:34  0:00:34                      0:00:00  0:00:00   Speed :          112137010 Bytes/sec.   Speed :            6416.531 MegaBytes/min.   Ended : Saturday, August 13, 2016 6:36:21 AM So while yeah unless your pfsense hardware is capable of routing at your wire speed your not going to see the performance as switched network only..  I find it unlikely that with your hardware the performance hit would be as hard as your seeing.  Mine is on vm and see better than yours.  New esxi hardware is on my wish list and coming soon.  I just love running my pfsense on vm, but yeah its going to be a hit compared to hardware.  I might switch to hardware here soon though as saw some posts about psfense running on minnow board, etc. [image: testsetup.jpg_thumb] [image: testsetup.jpg]

be it your networks are tagged or untagged doesn't really matter.  I run multiple untagged and tagged (vlans) on pfsense works without any issues.  What switch are you using? "I already do this at the Layer2 router" There is no such thing as a layer 2 router, routing happens at layer 3. Yes the removal of downstream routing will simplify your network and allow for better control. You can have multiple network segments without he use of "tagging" if you want as long as you have physical interfaces in pfsense, and you setup your smart/managed switch appropriately or use different dumb switches for each network.

Squid can only grab HTTP transparently unless you jump through a bunch of hoops and install a custom CA on all clients to break SSL and intercept HTTPS (it's a bad idea – don't do it) If the user puts their proxy settings in the browser it can do both easily. Choosing to allow some clients to bypass or use a different VPN based on their source is easy, just policy route with a rule matching their source IP address and direct them to whatever gateway you want.

More pics [image: DMZ.png_thumb] [image: float.png] [image: DMZ.png] [image: float.png_thumb] [image: sita151.png] [image: sita151.png_thumb] [image: uznet27.png] [image: uznet27.png_thumb] [image: uznet213.png] [image: uznet213.png_thumb] [image: ng.png] [image: ng.png_thumb] [image: inosmi.png] [image: inosmi.png_thumb] [image: inosmi_ok_uznet9.png] [image: inosmi_ok_uznet9.png_thumb]

Your putting 3000 on the same broadcast domain.. That is a lot of broadcast traffic ;) /20 gives 4k users, which 1000 more than you say you would need.. To me /22 would be highest I would ever think of using for a segment with machines that would be broadcasting.  Window machines are chatty freaking things!!!  They like to send a lot of broadcast and multicast traffic out of the box..

Zinga. It's working. At some point, likely while trying to figure out how to make the ISP provided gateway a "dumb modem" or "pass-through" (according to what I've read), since it is unable to go into "true bridged mode" without losing its configuration for static ip's.. I managed to deviate from the original video in my OP. After the 1:1 NAT, I should have (and have now done) added the Firewall -> Rules, manually.  I did that in accordance with the video and, it works.  No Firewall -> NAT -> Port Forward, no Firewall -> NAT -> Outbound NAT, just Firewall -> Rules -> WAN. Ugh.  I'm sure there are some following giving the ole "SMH" and perhaps I will later down the line as well as I continue to learn, not just -what- to do, but why.  However, for now, I'm just happy things are working.  I feel comfortable I'll keep the business line and can now call tomorrow to cancel the residential. Derelict, I do greatly appreciate your assistance.  I hope I didn't frustrate you/matters too much.  I'll learn to walk one day, much less, get out of diapers.  And I promise to pay it forward once I know my knowledge is sound and am within my limits to assist properly.

Hello, could it be an netmask issue ? as the gateway is in the same subnet as clients ?