Transfer is a small Nextcloud app created by Daniel Thwaites. It does one simple thing: you paste a URL, and the server downloads the file directly into your Nextcloud account using its own bandwidth. That means you don’t have to keep your laptop awake for a 40 GB download, or download a file locally just to upload it to Nextcloud again. I took over maintaining it a couple of months ago. This week I released version 1.0.
One of the features in 1.0 isn’t very glamorous. Users wanted downloads to start immediately instead of waiting for the next cron cycle. It was an old feature request from before I took over that never got implemented, and I can understand why. On the surface it’s a small request: skip the queue, start now. I ended up spending more time on it than on the progress bars, the settings page, and the Nextcloud 34 compatibility work put together.
The obvious version
The naive implementation is one endpoint. The browser posts the URL, the server downloads the file in that request, the request returns when the file is done. It works fine (with a small file).
But a download can be gigabytes, and it can take hours. A PHP web worker that is busy downloading is a worker that cannot serve anyone else, and there’s a finite pool of those. One user pasting a few links to large files can occupy the whole pool, and now the instance is slow for everyone. Nothing is broken, I just decided that a web request was a reasonable place to spend three hours.
This is the reason the app used a background job in the first place. A cron process does not occupy one of the web workers serving interactive requests. It’s the correct default, and it stays the default. The immediate option is opt-in, and it needs to be handled with some care.
The part that actually took the time
Once you accept that an immediate download holds a worker open, a second problem appears: what happens if the user closes the tab? They chose an immediate download because they’re waiting for it. If they close the tab or their laptop goes to sleep, there’s no reason to keep a web worker busy on their behalf anymore (and other users would probably like to have it back). So the download needs to be cancellable, and cancellation needs to work in three cases: the user clicks a cancel button, the user navigates away, and the user’s browser simply disappears.
I tried a few approaches first. An AbortController on the fetch, which only closes the browser’s end of the connection and leaves the server downloading happily. A sendBeacon on beforeunload, which fires (sometimes). A server-side trick to detect the dropped connection, which PHP’s execution model doesn’t really let you do mid-download. A separate worker process, which drags in exec() portability questions and has no precedent in the codebase. None of them survived all three cases.
What shipped is simpler than any of those. The browser asks the server to prepare a transfer, which stores the parameters and returns an id. Then it starts the transfer, which blocks. While that request blocks, the browser independently polls a progress endpoint every couple of seconds, and each poll doubles as a heartbeat. The download callback checks two things: has someone set a cancel flag, and has the heartbeat gone quiet for more than ten seconds. Either one aborts it.
prepare -> id
start -> (blocks, downloading)
progress -> progress + heartbeat, every 2s
no heartbeat for 10s -> abortThat covers all three cases. Clicking cancel sets the flag. Navigating away or losing the browser stops the polling, and ten seconds later the server notices and gives up. Either way no web worker stays tied up on a download nobody is waiting for.
The tab as a lease
At first I treated the open tab as a problem. The user might close it, and I’d have to handle that. What made the heartbeat click for me was reading it the other way around: an open tab is the user saying “I’m willing to dedicate a process to this”, and closing it says “I’m done, clean up”. The heartbeat turns that into a lease. As long as the browser keeps renewing it, the server keeps working. When the renewals stop, the server is allowed to stop too.
Someone will probably ask why I didn’t make it fully asynchronous instead. Partly because the worker-process route has the portability problems mentioned above. But mostly because then the open tab wouldn’t mean anything anymore, and here it does.
A large immediate download runs exactly as long as someone is watching it, and the cost is carried by the person who chose it. If you want to close your laptop, use the background queue, that’s what the app was always for. If you want the file now and don’t mind watching a progress bar, keep the tab open. That tab is your half of the contract.
Where it’s not available
Immediate transfers need a distributed cache (Redis or Memcached) because the progress and the heartbeat have to be visible across processes. If the instance doesn’t have one, the checkbox is hidden and the background queue keeps working. I didn’t want a feature that half worked on a plain setup and produced confusing failures.
What 1.0 means
The 0.7 releases were mostly about making Transfer maintainable again: simplifying the frontend, replacing legacy APIs, and cleaning up dependencies. Most of that was invisible to users. 1.0 is the first release where the visible surface grew: live progress, a settings page, and the immediate option above. After this the plan is boring on purpose. Bug fixes, keeping up with new Nextcloud releases, and the occasional feature.
Transfer is on the Nextcloud app store if you run your own instance. The source is on GitHub, it’s AGPL-3.0, and it supports Nextcloud 29 through 34.
