Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for chunked / multipart HTTP uploads #1801

Open
alexitx opened this issue Nov 3, 2024 · 2 comments
Open

Support for chunked / multipart HTTP uploads #1801

alexitx opened this issue Nov 3, 2024 · 2 comments
Labels
suggestion Feature suggestion

Comments

@alexitx
Copy link

alexitx commented Nov 3, 2024

Is your feature request related to a problem? Please describe.

Many reverse proxies, tunnels and even some managed hosting services impose limits on the client request body size. For example, Cloudflare has upload body size limit of 100 MB on the free plan, which can only be increased up to 500 MB on the enterprise plan, excluding custom offers.

100 MB is very small file size by today's standards and often multimedia like videos and RAW photos rapidly exceed it. I personally use SFTPGo for easy access to and from private remote storage for backup and archival purposes.

Describe the solution you'd like

Note: not to be confused with the Multipart form submission or chunked Transfer-Encoding specifications.

Support for multipart HTTP file uploads where upload requests for large files are split into multiple smaller parts, similar to how S3 multipart upload functions. The maximum body size for individual requests can be a configurable limit, and the requests can be sent sequentially or in batches until the upload is complete.

This can be done using a specialized protocol for resumable multipart uploads like tus.io, a pure JavaScript client library like Resumable.js or custom implementation using the common Web Fetch API or XHR. Of course, this would require some work on the server side to accept the multiple parts and stitch them together.

Additionally, this has the benefit of offering reliable upload resumption in the case of disruptions. As far as I can see, SFTPGo doesn't currently offer HTTP upload resumption.

Describe alternatives you've considered

  • Painstakingly splitting large files into multipart archives in order to use HTTP upload
  • Disabling or reconfiguring tunnels or reverse proxies that offer caching and protection in order to remove the size limits, which might not always be possible

What are you using SFTPGo for?

Private user, home usecase (home backup/VPS)

Additional context

tus.io - open protocol for resumable HTTP file uploads
tusd - official reference implementation of the tus protocol in Go
Resumable.js - JavaScript library providing multiple simultaneous resumable uploads via the HTML5 File API
File Browser - (HTTP only) web file management interface, very similar to SFTPGo, with implementation of tus for multipart resumable uploads
Uppy - modular JavaScript file uploader using the tus protocol

@alexitx alexitx added the suggestion Feature suggestion label Nov 3, 2024
@adamzvolanek
Copy link

Any update or note about this being worked in the near future?

@turnerd10
Copy link

Would love to see this added as well, this product is excellent, but useless behind Cloudflare without it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
suggestion Feature suggestion
Projects
None yet
Development

No branches or pull requests

3 participants