Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [SOLVED] Large file transfers between interfaces dropping

    Scheduled Pinned Locked Moved L2/Switching/VLANs
    15 Posts 4 Posters 2.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      netnewb2
      last edited by netnewb2

      Foreword: I don't like networking so maybe I don't understand my own setup.

      I'm using a pfsense (2.4.4-p1) firewall to restrict VLAN access to some Proxmox VMs. My primary GW is a Unifi USG, connected to a Unifi Switch. It looks like this:

      Internet -> USG WAN port
      USG LAN1-> WAN port on pfsense router (pFs) -> LAN port to Unifi Switch (USW)
      USG LAN2-> USW -> Proxmox
      USW -> FreeNAS

      I've static entries in the USG for the subnets that pfsense manages. Pfsense has a static entry for the native LAN through WAN. USG manages only the native LAN (untagged). The VMs are using pfsense as a gateway. So, if I access a VM resource from PC, I basically have the following route:
      PC -> USW -> USG (I think) -> pFs WAN -> pFs LAN -> USW -> Proxmox (VMs);

      VMs are using a linux bridge so they can access resources on the same VLAN directly, but they're using pfsense for internet access and inter-vlan access.

      I've noticed my problem while trying to completely virtualize pfsense, but that's another story. My problem now, is that I was testing the network and I've noticed that large file transfers from FreeNAS, through pfsense VLAN, are failing. Example:

      PC -> USW -> USG -> pFs WAN -> pFs LAN -> pFs VLAN24 -> USW VLAN24 -> FreeNAS

      I've tried throubleshooting the problem by doing the following:

      • clear invalid DF bits (on/off)
      • Disable Firewall Scrub (on/off)
      • Static route filtering (on/off)
      • Disable hardware checksum offload (on/off)
      • Disable hardware TCP segmentation offload (checked)
      • Hardware Large Receive Offloading (checked)
      • created separate interface with an upstream gateway to the USG, to avoid WAN;
      • pfsense virtualized with 2 x NIC passthrough vs hardware pfsense SG-2220 - identical configs;

      I'm certain that it's a pfsense problem because:

      • created new VLAN on USG and FreeNAS to cut pfsense as the middle man and it worked (PC -> Switch -> USG -> FreeNAS);
      • Proxmox VMs on the same VLAN as FreeNAS work;
      • tried Windows and Linux for the file transfer;

      Does anyone have any suggestions?

      1 Reply Last reply Reply Quote 0
      • N
        netnewb2
        last edited by netnewb2

        I tried to reply to my post, but it's marked as spam.

        I now believe it's a FreeNAS thing. It seems it has something to do with assymmetric routing as outlined in this a FreeNAS forum post (I can't post the link cause I'm getting marked as spam). I still don't understand why I can't even access the web interface across VLANs.

        1 Reply Last reply Reply Quote 0
        • johnpozJ
          johnpoz LAYER 8 Global Moderator
          last edited by johnpoz

          @netnewb2 said in Large file transfers between interfaces dropping:

          Does anyone have any suggestions?

          Yeah draw up this mess...

          PC -> USW -> USG -> pFs WAN -> pFs LAN -> pFs VLAN24 -> USW VLAN24 -> FreeNAS

          How exactly are you using pfsense - you understand that out of the box if you setup a wan on pfsense its going to be natting!!! with stuff you put on its lan..

          if you just want to use pfsense as router/firewall between 2 of your local networks then you should not be using "wan" on pfsense but 2 lans - lan and opt, or you will want to make sure to turn of nat on pfsense, etc.

          And you have also introduced a downstream router so that mess up the whole thing as well. Even if you have disable nat on pfsense.

          And you have proxmox in there as well that has some issues - what version of proxmox?

          I don't like networking so maybe I don't understand my own setup.

          I love networking, do it for a living.. .Its my passion... And can tell you I don't understand WTF your trying to do with that mess ;)

          If you have usg.. and your happy with it - then use that to firewall between your local networks be they physical or virtual. Not sure why you feel throwing in a downstream router into your mix makes any sense - especially if you don't actually know and love and understanding networking.

          An intelligent man is sometimes forced to be drunk to spend time with his fools
          If you get confused: Listen to the Music Play
          Please don't Chat/PM me for help, unless mod related
          SG-4860 24.11 | Lab VMs 2.8, 24.11

          N 1 Reply Last reply Reply Quote 0
          • N
            netnewb2 @johnpoz
            last edited by

            @johnpoz

            Your post irked me cause it feels condescending. I hope it's not, since I'm just trying to learn. After all, even if I don't LIKE networking, it's a means to an end and I've learned just enough to get what I needed/wanted out of it.

            I've been using a Netgate SG-2220 for about 2 years and last month it started throwing some weird errors and it would lock up completely. Tried a few reinstalls to no avail, so I thought it was the Intel Atom crapping itself. I migrated pfsense to a VM and bought an USG for the simplicity and throughput (I have gigabit WAN).

            While pfsense was virtualized it was throwing the same errors so I realised it was due to one of the USB devices (UPS or Cellular Dongle) - even though I've been running it like this for over a year. Maybe some update messed things up.

            I don't have time to learn and migrate from pfsense to USG, I still need some features on the pfsense, such as routing all traffic from certain IPs through a VPN tunnel and pfsense no longer locks up, so I'm stuck with this Frankenstein setup.

            Overall, TF I'm trying to do, is just keep the homelab partially separated from the LAN so I don't bring down the house when I'm tinkering with networking and VMs.

            Proxmox is on 5.4.5.

            Your post does help though, cause I completely forgot about NAT from VLAN -> LAN. WAN to VLAN isn't natted.

            1 Reply Last reply Reply Quote 0
            • N
              netnewb2
              last edited by

              Problem solved by adding static routes to VLANs that weren't in the same network as FreeNAS. I assume it had something to do with asymmetric routing and with FreeNAS not setting gateways on VLAN.

              D 1 Reply Last reply Reply Quote 0
              • johnpozJ
                johnpoz LAYER 8 Global Moderator
                last edited by johnpoz

                If you felt it was condescending I apologize.. I tend to write from the hip and the heart and just let it flow..

                I am all for helping people learn what I love, etc.. But knowing this stuff that I do it can be frustrating watch someone butcher your passion ;)

                Its like a chef watching someone over salt a dish or, or putting ketchup on a $50 steak ;)

                I am more than willing to help you straighten out the mess it seems you have created from your description.. But lets start with a drawing of what you have - and what you want to accomplish and we can work out the best way to do that.. Throwing in another router/firewall when you already have one that your happy with and working is not always the best option.

                While I think you could replace that usg and be much happier.. If that is what you have to work with - lets work out if makes sense to even use pfsense at all.. I have a usg on my shelf - it works, and not a bad price point for what it can do.. But to be honest - its not the most friendly interface to work with.. And is way more difficult to do even the basic stuff that is simple and straight forward with pfsense. As soon as the pfsense box I got was off back ordered and delivered I could not get that usg off my network fast enough ;) Even though I love their AP - their usg is like having a chef being forced to use a plastic knife and toothpicks as their only tools.

                You can for sure use a downstream router and firewall in your network - but to be honest at your scale its more than likely just over complicating it all.

                If you could take the time to draw up what you currently have and describe what you want to accomplish from a filter standpoint - this can talk to that, but only on ports xyz, etc. We can work out the most efficient and simplest way to do that.

                An intelligent man is sometimes forced to be drunk to spend time with his fools
                If you get confused: Listen to the Music Play
                Please don't Chat/PM me for help, unless mod related
                SG-4860 24.11 | Lab VMs 2.8, 24.11

                N 1 Reply Last reply Reply Quote 0
                • N
                  netnewb2 @johnpoz
                  last edited by

                  @johnpoz

                  I appreciate your offer to help but I'm actually leaving for a longer business trip and I was making sure the network was stable. OP wasn't a problem for my workflow since I don't access FreeNAS across VLANs, it was something I noticed accidentally that I couldn't explain and I wrongly assumed it was a pfsense issue.

                  I wouldn't have bought the USG if not for the abovementioned lockups. When the SG-2220 "broke" I went for "it just works" and since I already have some Unifi equipment the USG was a logical alternative. Now, the setup isn't optimal but it's better from a wife perspective. I can homelab and break things while netflix and the internet still works for the rest of the house. I'm running about 15 VMs (some of which are internet facing), LACP, proxies, VPNs and I'm more comfortable doing these in pfsense since I've been using it for about 4 years.

                  1 Reply Last reply Reply Quote 0
                  • johnpozJ
                    johnpoz LAYER 8 Global Moderator
                    last edited by johnpoz

                    @netnewb2 said in [SOLVED] Large file transfers between interfaces dropping:

                    Unifi equipment the USG was a logical alternative.

                    You would think huh ;) I had gotten the usg for sim reasoning... I had bumped my connection from 100/10 to 500/50 and my pfsense on old N40L as vm could just not do that speed..

                    I knew I needed something quick that could push that speed and was "cheap".. Until I could figure out the direction I wanted to go. So I had gotten a usg3p for like $100.. And yeah I could get my 500/50 without too much issue.. As long as didn't turn off offload, like if wanted to play with their dpi stuff.. Then it was prob worse than my VM of pfsense..

                    After just a few minutes with it trying to setup just basic firewall rules, I could tell yeah not going to be using this.. So had to decide did I build a new VM host that could run pfsense and handle my speed - or did I go with actual hardware for pfsense.. Ended up with a sg4860.. And life was good again.. My usg sits there collecting dust - guess its a spare router/firewall... I can not even find a buyer for it for like $75 ;) You want to buy mine? I just turned it on the other day to update its firmware to current.. Which was fun since the current controller couldn't even adopt it with the firmware it had on it, etc.. I keep trying to come up with how I could actually use it.. And just can not come up with anything.. If they would enable it to just be say a monitor for dpi via simple bridge mode then that might be something I could do with - but yeah without some major playing in the cli, doesn't seem possible at all.

                    An intelligent man is sometimes forced to be drunk to spend time with his fools
                    If you get confused: Listen to the Music Play
                    Please don't Chat/PM me for help, unless mod related
                    SG-4860 24.11 | Lab VMs 2.8, 24.11

                    N 2 Replies Last reply Reply Quote 0
                    • N
                      netnewb2 @johnpoz
                      last edited by

                      @johnpoz said in [SOLVED] Large file transfers between interfaces dropping:

                      to play with their dpi stuff.. Then it was prob worse than my VM of pfsense..

                      Oh yeah, I didn't expect doing anything advanced with the USG since they have this "Warning: Enabling IDS/IPS will affect the device maximum throughput. USG: 85 Mbps, USG-Pro: 250 Mbps, USG-XG-8: 1 Gbps." . I've only blocked outgoing traffic for some IoT devices and forwarded ports to pfsense.

                      I'm getting a free sg-2220 and sg-2440 from work soon (they switched to fortigate) and was thinking what to do with them... maybe HA between the 2220s or using just the 2440.

                      1 Reply Last reply Reply Quote 0
                      • N
                        netnewb2 @johnpoz
                        last edited by netnewb2

                        @johnpoz I came back to say you were right. I've used 2 gateways and while it worked, it was also tiresome since I had to babysit NAT, multiple firewall rules and gateways. It's also annoying to downgrade to something as basic and featureless as the USG.

                        I've setup CARP with 2 virtualized pFs about a month ago and I'm pretty happy with the setup and now I'm trying to decide whether to keep pF virtualized or switch to 2 x netgate SG-2220 which I already own. Maybe you can give some advice.

                        So, I have a "compute" unit with Proxmox, a FreeNAS "storage" unit and an Intel NUC with proxmox which hosts the failover pfsense. "Compute" has a 4 x Intel network card (which are setup in one LAG) and 2 x Intel onboard ports - it's a Supermicro MB. FreeNAS also has 2 ports setup in a LAG.

                        Most of the traffic happens between the VMs on Proxmox and with FreeNAS. The pFsense VM is using the 4 port LAG. I've assumed that this setup should work faster than a hardware pFsense with only 1 LAN port, especially for traffic between the VMs (pFs is routing between VLANs).

                        1 Reply Last reply Reply Quote 0
                        • A
                          akuma1x
                          last edited by akuma1x

                          @netnewb2 You might not be able to CARP the SG-2220 boxes, since they have only 1 WAN and 1 LAN port. I might be wrong, however...

                          But, I thought there had to be a dedicated sync interface port between the 2 units for the boxes to keep updated with each other.

                          Jeff

                          N 1 Reply Last reply Reply Quote 0
                          • N
                            netnewb2 @akuma1x
                            last edited by netnewb2

                            @akuma1x it's recommended to have a dedicated sync interface and it can be done via VLAN as well. That's how I've set it up for now.

                            edit: strongly recommended

                            A 1 Reply Last reply Reply Quote 0
                            • A
                              akuma1x @netnewb2
                              last edited by

                              @netnewb2 said in [SOLVED] Large file transfers between interfaces dropping:

                              @akuma1x it's recommended to have a dedicated sync interface and it can be done via VLAN as well. That's how I've set it up for now.

                              That's what I was going to come back and add, that it might be able to sync over a VLAN. You beat me to it... Thanks!

                              Jeff

                              1 Reply Last reply Reply Quote 0
                              • D
                                dumdedumda @netnewb2
                                last edited by

                                @netnewb2 said in [SOLVED] Large file transfers between interfaces dropping:

                                Problem solved by adding static routes to VLANs that weren't in the same network as FreeNAS. I assume it had something to do with asymmetric routing and with FreeNAS not setting gateways on VLAN.

                                Currently dealing with a similar scenario myself-- where transferring large files inter-VLAN between FreeNAS and a client crashes the network.

                                Can you explain the process of "adding static routes to VLANs that weren't in the same network as FreeNAS"? Thanks!

                                N 1 Reply Last reply Reply Quote 0
                                • N
                                  netnewb2 @dumdedumda
                                  last edited by

                                  @dumdedumda said in [SOLVED] Large file transfers between interfaces dropping:

                                  @netnewb2 said in [SOLVED] Large file transfers between interfaces dropping:

                                  Problem solved by adding static routes to VLANs that weren't in the same network as FreeNAS. I assume it had something to do with asymmetric routing and with FreeNAS not setting gateways on VLAN.

                                  Currently dealing with a similar scenario myself-- where transferring large files inter-VLAN between FreeNAS and a client crashes the network.

                                  Can you explain the process of "adding static routes to VLANs that weren't in the same network as FreeNAS"? Thanks!

                                  Well, it’s been a while since the initial post and right now even I don’t understand what I was trying to do. Looks needlessly complicated.

                                  AFAIR, it has something to do with FreeNAS on multiple VLANs and asymmetric routing. Example:

                                  PC on 192.168.1.5
                                  FreeNAS on 192.168.1.100 and 192.168.100.100

                                  PC tries to access FreeNAS on 192.168.100.100. Works initially but after a while FreeNAS will try to respond via 192.168.1.100, as in, from the same VLAN as the PC.

                                  Or another issue when FReenas tries to answer back from 100.100 but doesn’t have a gateway set on that interface. The solution was to add a gateway on 192.168.100.0/24 but that wasn’t an option (in the freenas gui). So I had to set a static route from Freenas something like, 192.168.1.0/24 via 192.168.100.1 (router interface on that VLAN that can talk between VLANs).

                                  Tbh I can’t remember details and since then, I’ve moved on from freenas and pfsense

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.