FTP passive port on demand opening
-
Not sure what your doing but sftp is not slow.. And sure and the F going to be faster than webdav.. If you want speed then use https for your transfer.. FTP should of died off 10 years ago, why its still in use is blows my mind.
-
With my test is slow... but the real problem of the SFTP is that it use a lot of cpu.
What happen when 10 users at the same time are downloading 150Gb each with 5 file transfers for each connection for a total of 50 sftp active connections?
With webdav I can reach the speed of 80MB/s , with ftp/s I reach 50/60 MB/s.. with Sftp the maximum is 24MB/s
-
If you have 10 users moving files - why would you not setup something like owncloud.. I just pulled a file from my shitty 15$ a year vps in Las Vegas, while I am in Chicago and hit 25MBps without issue for a 400MB file..
If you have users just downloading, why not just let them download via https?
This almost 2020, not 2000 - using ftp is not longer a valid protocol.. Why not just use active for your ftp server... Now you don't need to forward anything other than 21.. Since the ftp server will make the outbound connection to them..
edit: So I just pulled that file via https, and multiple streams so I could max out my pipe.. hit 60+MBps which is my full download pipe...
If you want users to download, that would be the way to go..
-
@johnpoz thank you , i will verify to works only with active connections.
Owncloud that use some like nginx/apache using (PHP?!?) can serve to download to 10 users a single file of 150gb?Now with ftp/webdav my user can download 150gb within 1 hour and 30 minutes (live gzip compression) (if there is only one download) with a 1gbps server virtualized.
-
I just grabbed 600MB file from my nextcloud running on one of my servers in CA.. Maxed out my download pipe.. Grabbed it in seconds..
So picked 3 files that it zipped up for me 1.5GB - again downloaded in seconds hit over 50MBps without any issues.. And that server is not any sort of rocketship...
-
you can try generating a big fake file of 150Gb and download it?
mkfile -n 150G sparseFile -
https://doc.owncloud.com/server/admin_manual/configuration/files/big_file_upload_configuration.html
This will expose the server to any kind of attack with php set to really high values..
-
Dude do what you want.. You don't think ftp is not a security freaking risk??
Thought you said they were downloading..
Here not going to do 150G, but here maxing out my pipe
There is little reason to be using such an antiquated protocol in this day an age..
-
please post the - htop -
screenshot when you download the filealso consider that I encrypt all the data on the server side with AES256 that need to be unencrypted when downloaded ... and 5gb are not 150gb as cpu load ;)
-
-
@openaspace said in FTP passive port on demand opening:
and 5gb are not 150gb as cpu load ;)
Huh.. Moving a file doesn't take more cpu just because the file is bigger.. The cpu load would just run longer is all..
The vps I was testing this off of does even 150GB of space for me to play with ;) And the host where I do have that kind of space is shared, and I don't have access to see what my process is doing to the overall load.
Keep in mind that is a shit lowend vps...
I would test what ftp does - but I don't even have it installed... I can move files to and from it with sftp without any issues. Zero use for ftp..
-
(decrypt 150gb on the fly instead of 5gb it's different)
Multiply this CPU usage for 5 contemporary connections transfer of each client, and multiply for 10 clients = 50. (it doesn't matter if you have a single core or eight cores, the server will became unstable).
And I'm not considering possible contemporary uploads of other users at the same time.. With others 20 connections for example..
.. With out transfer that are decrypted and input transfer that are encrypted.. (other CPU load)SFTP is very powerful and secure solution, but not applicable in heavy load environments.
Thank you :)
-
If you are running a current business and talking about "heavy load" but still use an antique protocol like FTP I'd say you're having other problems. No company I know, not even heavy hitters with big data pipes, are still using FTP these days. As @johnpoz said, it's 2020. There are other and modern ways to transfer files - securely! - without using antique techniques.
Also running 150GB files? What's the use case in that? I've seen companies dealing with 3D printing and architecture or constructions or media production using files this size, but even they won't touch FTP for transferring such numbers. And multiple clients that do up- and downloads of such overly big files at the same time? To somewhere through the internet into some cloud location? Seems strange.
(BTW: just have a look at how YouTube et al handle video upload. Ever seen a video in 4k/8k with 60fps? And they push that via HTTPS, too. Noone decrypts/encrypts the file on the fly because the services use TLS layer to encrypt. The file isn't encrypted - sent - decrypted. So I don't understand your logic with 150GB being more CPU bound then 5GB while transferring via modern protocols?)
-
I'm talking about the local storage encrypt, non about the transfer ;) all the files are encrypted on the disk. Files are decrypted when downloaded and are transfered over https.
I already use https from browser for upload and download and with webdav client the same using https.
-
Anyway, my question was for the on-demand opening of passive doors...
....and can be possible to interact with pfsense using API from a web page? .. i want to create a PASS firewall rule when a client compile a protected form where will write her static ip , in order to allow access only to him.
-
What indeed means, that it has nothing at all to do with running another protocol but is due to your local storage encryption thingy. So again, don't understand why you keep on mentioning that 150GB files would be harder/more CPU bound to transfer with any other possible (current, modern) transportation method. SFTP, HTTPS, etc. wouldn't need any more CPU cycles for 150G or 5G. If SFTP is slow I'd check with your target. Never seen that much difference on our customer hosting servers at all. More often it's a thing with the customers not updating their client software and using old/deprecated SSH/SSL ciphers/settings. I'd try updating clients etc. to ensure best protocols are used with hardware encryption in use etc. Had that discussions too many times like "But that tool worked for years!" - "Are there updates?" - updated tool multiple major versions "Wow it really is faster/is working now!" ;)
But again, to stay in the OP: No passive port opening automagically as passive ports are configured on the server side and every server can declare them as they want. pfSense has an active FTP helper for outbound connections that could help with that but it relies on the other side offering active FTP - not everyone does that (or uses FTP at all these days).
@openaspace said in FTP passive port on demand opening:
i want to create a PASS firewall rule when a client compile a protected form where will write her static ip , in order to allow access only to him.
Why? Passive FTP means you have inbound (if you host the server) or outbound connections with 21/tcp as well as xxxxx-yyyyy/tcp. So if it's your own FTP server where you upload your huge file stuff, why not allow your passive port range to your specific IP? Don't get the problem with opening passive ports here?
-
As I already answered in my first post, there is no passive helper or way for pfsense to auto open the passive ports. If you want to provide passive ftp as an option to your ftp server then you need to set your ftp server to use a specific range of ports for its passive connections. And then forward that range.
You can for sure lock down the source IPs on this forward/firewall rule.
-
Yes, I will see if is possible to interact with pfsense using api to do this
-
@JeGr it's well known that an sftp process use more cpu
-
Your more than welcome to tinker all you want - but would be akin to coming up with a faster way to read floppy disks.. Nobody uses them ;)
Sure there is a huge user base for something like this - the 3 people still using ftp will love you I am sure ;)
