[SOLVED] Large file transfers between interfaces dropping
-
Foreword: I don't like networking so maybe I don't understand my own setup.
I'm using a pfsense (2.4.4-p1) firewall to restrict VLAN access to some Proxmox VMs. My primary GW is a Unifi USG, connected to a Unifi Switch. It looks like this:
Internet -> USG WAN port
USG LAN1-> WAN port on pfsense router (pFs) -> LAN port to Unifi Switch (USW)
USG LAN2-> USW -> Proxmox
USW -> FreeNASI've static entries in the USG for the subnets that pfsense manages. Pfsense has a static entry for the native LAN through WAN. USG manages only the native LAN (untagged). The VMs are using pfsense as a gateway. So, if I access a VM resource from PC, I basically have the following route:
PC -> USW -> USG (I think) -> pFs WAN -> pFs LAN -> USW -> Proxmox (VMs);VMs are using a linux bridge so they can access resources on the same VLAN directly, but they're using pfsense for internet access and inter-vlan access.
I've noticed my problem while trying to completely virtualize pfsense, but that's another story. My problem now, is that I was testing the network and I've noticed that large file transfers from FreeNAS, through pfsense VLAN, are failing. Example:
PC -> USW -> USG -> pFs WAN -> pFs LAN -> pFs VLAN24 -> USW VLAN24 -> FreeNAS
I've tried throubleshooting the problem by doing the following:
- clear invalid DF bits (on/off)
- Disable Firewall Scrub (on/off)
- Static route filtering (on/off)
- Disable hardware checksum offload (on/off)
- Disable hardware TCP segmentation offload (checked)
- Hardware Large Receive Offloading (checked)
- created separate interface with an upstream gateway to the USG, to avoid WAN;
- pfsense virtualized with 2 x NIC passthrough vs hardware pfsense SG-2220 - identical configs;
I'm certain that it's a pfsense problem because:
- created new VLAN on USG and FreeNAS to cut pfsense as the middle man and it worked (PC -> Switch -> USG -> FreeNAS);
- Proxmox VMs on the same VLAN as FreeNAS work;
- tried Windows and Linux for the file transfer;
Does anyone have any suggestions?
-
I tried to reply to my post, but it's marked as spam.
I now believe it's a FreeNAS thing. It seems it has something to do with assymmetric routing as outlined in this a FreeNAS forum post (I can't post the link cause I'm getting marked as spam). I still don't understand why I can't even access the web interface across VLANs.
-
@netnewb2 said in Large file transfers between interfaces dropping:
Does anyone have any suggestions?
Yeah draw up this mess...
PC -> USW -> USG -> pFs WAN -> pFs LAN -> pFs VLAN24 -> USW VLAN24 -> FreeNAS
How exactly are you using pfsense - you understand that out of the box if you setup a wan on pfsense its going to be natting!!! with stuff you put on its lan..
if you just want to use pfsense as router/firewall between 2 of your local networks then you should not be using "wan" on pfsense but 2 lans - lan and opt, or you will want to make sure to turn of nat on pfsense, etc.
And you have also introduced a downstream router so that mess up the whole thing as well. Even if you have disable nat on pfsense.
And you have proxmox in there as well that has some issues - what version of proxmox?
I don't like networking so maybe I don't understand my own setup.
I love networking, do it for a living.. .Its my passion... And can tell you I don't understand WTF your trying to do with that mess ;)
If you have usg.. and your happy with it - then use that to firewall between your local networks be they physical or virtual. Not sure why you feel throwing in a downstream router into your mix makes any sense - especially if you don't actually know and love and understanding networking.
-
Your post irked me cause it feels condescending. I hope it's not, since I'm just trying to learn. After all, even if I don't LIKE networking, it's a means to an end and I've learned just enough to get what I needed/wanted out of it.
I've been using a Netgate SG-2220 for about 2 years and last month it started throwing some weird errors and it would lock up completely. Tried a few reinstalls to no avail, so I thought it was the Intel Atom crapping itself. I migrated pfsense to a VM and bought an USG for the simplicity and throughput (I have gigabit WAN).
While pfsense was virtualized it was throwing the same errors so I realised it was due to one of the USB devices (UPS or Cellular Dongle) - even though I've been running it like this for over a year. Maybe some update messed things up.
I don't have time to learn and migrate from pfsense to USG, I still need some features on the pfsense, such as routing all traffic from certain IPs through a VPN tunnel and pfsense no longer locks up, so I'm stuck with this Frankenstein setup.
Overall, TF I'm trying to do, is just keep the homelab partially separated from the LAN so I don't bring down the house when I'm tinkering with networking and VMs.
Proxmox is on 5.4.5.
Your post does help though, cause I completely forgot about NAT from VLAN -> LAN. WAN to VLAN isn't natted.
-
Problem solved by adding static routes to VLANs that weren't in the same network as FreeNAS. I assume it had something to do with asymmetric routing and with FreeNAS not setting gateways on VLAN.
-
If you felt it was condescending I apologize.. I tend to write from the hip and the heart and just let it flow..
I am all for helping people learn what I love, etc.. But knowing this stuff that I do it can be frustrating watch someone butcher your passion ;)
Its like a chef watching someone over salt a dish or, or putting ketchup on a $50 steak ;)
I am more than willing to help you straighten out the mess it seems you have created from your description.. But lets start with a drawing of what you have - and what you want to accomplish and we can work out the best way to do that.. Throwing in another router/firewall when you already have one that your happy with and working is not always the best option.
While I think you could replace that usg and be much happier.. If that is what you have to work with - lets work out if makes sense to even use pfsense at all.. I have a usg on my shelf - it works, and not a bad price point for what it can do.. But to be honest - its not the most friendly interface to work with.. And is way more difficult to do even the basic stuff that is simple and straight forward with pfsense. As soon as the pfsense box I got was off back ordered and delivered I could not get that usg off my network fast enough ;) Even though I love their AP - their usg is like having a chef being forced to use a plastic knife and toothpicks as their only tools.
You can for sure use a downstream router and firewall in your network - but to be honest at your scale its more than likely just over complicating it all.
If you could take the time to draw up what you currently have and describe what you want to accomplish from a filter standpoint - this can talk to that, but only on ports xyz, etc. We can work out the most efficient and simplest way to do that.
-
I appreciate your offer to help but I'm actually leaving for a longer business trip and I was making sure the network was stable. OP wasn't a problem for my workflow since I don't access FreeNAS across VLANs, it was something I noticed accidentally that I couldn't explain and I wrongly assumed it was a pfsense issue.
I wouldn't have bought the USG if not for the abovementioned lockups. When the SG-2220 "broke" I went for "it just works" and since I already have some Unifi equipment the USG was a logical alternative. Now, the setup isn't optimal but it's better from a wife perspective. I can homelab and break things while netflix and the internet still works for the rest of the house. I'm running about 15 VMs (some of which are internet facing), LACP, proxies, VPNs and I'm more comfortable doing these in pfsense since I've been using it for about 4 years.
-
@netnewb2 said in [SOLVED] Large file transfers between interfaces dropping:
Unifi equipment the USG was a logical alternative.
You would think huh ;) I had gotten the usg for sim reasoning... I had bumped my connection from 100/10 to 500/50 and my pfsense on old N40L as vm could just not do that speed..
I knew I needed something quick that could push that speed and was "cheap".. Until I could figure out the direction I wanted to go. So I had gotten a usg3p for like $100.. And yeah I could get my 500/50 without too much issue.. As long as didn't turn off offload, like if wanted to play with their dpi stuff.. Then it was prob worse than my VM of pfsense..
After just a few minutes with it trying to setup just basic firewall rules, I could tell yeah not going to be using this.. So had to decide did I build a new VM host that could run pfsense and handle my speed - or did I go with actual hardware for pfsense.. Ended up with a sg4860.. And life was good again.. My usg sits there collecting dust - guess its a spare router/firewall... I can not even find a buyer for it for like $75 ;) You want to buy mine? I just turned it on the other day to update its firmware to current.. Which was fun since the current controller couldn't even adopt it with the firmware it had on it, etc.. I keep trying to come up with how I could actually use it.. And just can not come up with anything.. If they would enable it to just be say a monitor for dpi via simple bridge mode then that might be something I could do with - but yeah without some major playing in the cli, doesn't seem possible at all.
-
@johnpoz said in [SOLVED] Large file transfers between interfaces dropping:
to play with their dpi stuff.. Then it was prob worse than my VM of pfsense..
Oh yeah, I didn't expect doing anything advanced with the USG since they have this "Warning: Enabling IDS/IPS will affect the device maximum throughput. USG: 85 Mbps, USG-Pro: 250 Mbps, USG-XG-8: 1 Gbps." . I've only blocked outgoing traffic for some IoT devices and forwarded ports to pfsense.
I'm getting a free sg-2220 and sg-2440 from work soon (they switched to fortigate) and was thinking what to do with them... maybe HA between the 2220s or using just the 2440.
-
@johnpoz I came back to say you were right. I've used 2 gateways and while it worked, it was also tiresome since I had to babysit NAT, multiple firewall rules and gateways. It's also annoying to downgrade to something as basic and featureless as the USG.
I've setup CARP with 2 virtualized pFs about a month ago and I'm pretty happy with the setup and now I'm trying to decide whether to keep pF virtualized or switch to 2 x netgate SG-2220 which I already own. Maybe you can give some advice.
So, I have a "compute" unit with Proxmox, a FreeNAS "storage" unit and an Intel NUC with proxmox which hosts the failover pfsense. "Compute" has a 4 x Intel network card (which are setup in one LAG) and 2 x Intel onboard ports - it's a Supermicro MB. FreeNAS also has 2 ports setup in a LAG.
Most of the traffic happens between the VMs on Proxmox and with FreeNAS. The pFsense VM is using the 4 port LAG. I've assumed that this setup should work faster than a hardware pFsense with only 1 LAN port, especially for traffic between the VMs (pFs is routing between VLANs).
-
@netnewb2 You might not be able to CARP the SG-2220 boxes, since they have only 1 WAN and 1 LAN port. I might be wrong, however...
But, I thought there had to be a dedicated sync interface port between the 2 units for the boxes to keep updated with each other.
Jeff
-
@akuma1x it's recommended to have a dedicated sync interface and it can be done via VLAN as well. That's how I've set it up for now.
edit: strongly recommended
-
@netnewb2 said in [SOLVED] Large file transfers between interfaces dropping:
@akuma1x it's recommended to have a dedicated sync interface and it can be done via VLAN as well. That's how I've set it up for now.
That's what I was going to come back and add, that it might be able to sync over a VLAN. You beat me to it... Thanks!
Jeff
-
@netnewb2 said in [SOLVED] Large file transfers between interfaces dropping:
Problem solved by adding static routes to VLANs that weren't in the same network as FreeNAS. I assume it had something to do with asymmetric routing and with FreeNAS not setting gateways on VLAN.
Currently dealing with a similar scenario myself-- where transferring large files inter-VLAN between FreeNAS and a client crashes the network.
Can you explain the process of "adding static routes to VLANs that weren't in the same network as FreeNAS"? Thanks!
-
@dumdedumda said in [SOLVED] Large file transfers between interfaces dropping:
@netnewb2 said in [SOLVED] Large file transfers between interfaces dropping:
Problem solved by adding static routes to VLANs that weren't in the same network as FreeNAS. I assume it had something to do with asymmetric routing and with FreeNAS not setting gateways on VLAN.
Currently dealing with a similar scenario myself-- where transferring large files inter-VLAN between FreeNAS and a client crashes the network.
Can you explain the process of "adding static routes to VLANs that weren't in the same network as FreeNAS"? Thanks!
Well, it’s been a while since the initial post and right now even I don’t understand what I was trying to do. Looks needlessly complicated.
AFAIR, it has something to do with FreeNAS on multiple VLANs and asymmetric routing. Example:
PC on 192.168.1.5
FreeNAS on 192.168.1.100 and 192.168.100.100PC tries to access FreeNAS on 192.168.100.100. Works initially but after a while FreeNAS will try to respond via 192.168.1.100, as in, from the same VLAN as the PC.
Or another issue when FReenas tries to answer back from 100.100 but doesn’t have a gateway set on that interface. The solution was to add a gateway on 192.168.100.0/24 but that wasn’t an option (in the freenas gui). So I had to set a static route from Freenas something like, 192.168.1.0/24 via 192.168.100.1 (router interface on that VLAN that can talk between VLANs).
Tbh I can’t remember details and since then, I’ve moved on from freenas and pfsense
