Transfer is a small Nextcloud app created by Daniel Thwaites. It does one simple thing: you paste a URL, and the server downloads the file directly into your Nextcloud account using its own bandwidth. That means you do not have to keep your laptop awake for a 40 GB download, or download a file locally just to upload it to Nextcloud again. I took over maintaining it a couple of months ago. This week I released version 1.0. One feature took far longer than the rest of the release combined.
The feature is not very glamorous. Users wanted downloads to start immediately instead of waiting for the next cron cycle. It was an old feature request from before I took over that never got implemented, and I can understand why. On the surface it is a small request: skip the queue, start now. I ended up spending more time on it than on the progress bars, the settings page, and the Nextcloud 34 compatibility work put together.
The obvious version and why it is wrong
The naive implementation is one endpoint. The browser posts the URL, the server downloads the file in that request, the request returns when the file is done. It works on the first try with a small file, but there’s a catch.
A download may not be a small file at all. It can be gigabytes, and it can take hours. A PHP web worker that is busy downloading is a worker that cannot serve anyone else. There is a finite pool of those workers. One user pasting a few links to large files can occupy the whole pool, and now the instance is slow for everyone, not because anything is broken but because I decided a web request was a reasonable place to spend three hours.
This is the reason the app used a background job in the first place. A background cron process does not occupy one of the web workers serving interactive requests. It is the correct default, and it is still the default. The immediate option is the exception, and exceptions are where you have to be careful.
The part that actually took the time
Once you accept that an immediate download holds a worker open, a second problem appears: what happens if the user closes the tab? The user chose an immediate download, which means they are actively waiting for it to finish. But what happens when that stops being true? If they close the tab or disappear entirely, there is no longer a reason to keep an interactive web worker occupied on their behalf. So the download needs to be cancellable, and cancellation needs to work in three cases: the user clicks a cancel button, the user navigates away, and the user’s browser simply disappears.
I tried multiple different approaches. An AbortController on the fetch, which only closes the browser’s end of the connection and leaves the server downloading happily. A sendBeacon on beforeunload, which fires sometimes. A server-side trick to detect the dropped connection, which PHP’s execution model does not really let you do mid-download. A separate worker process, which drags in exec() portability questions and has no precedent in the codebase.
Each one solved the demo and failed a real case.
The version that shipped is even simpler. The browser asks the server to prepare a transfer, which stores the parameters and returns an id. Then it starts the transfer, which blocks. While that request blocks, the browser independently polls a progress endpoint every couple of seconds, and each poll doubles as a heartbeat. The download callback checks two things: has someone set a cancel flag, and has the heartbeat gone quiet for more than ten seconds. Either one aborts it.
prepare -> id
start -> (blocks, downloading)
progress -> progress + heartbeat, every 2s
no heartbeat for 10s -> abortThis covers all three cases and ensures a web worker isn’t tied up for a long download that no one is actively waiting for. The heartbeat is the user’s part of the contract with the server: as long as it keeps arriving, the server knows someone is still actively waiting for the download to finish.
The tab as a lease
At first I thought of the open tab as an inconvenience. The user might close it, and I would have to handle that. The insight that solved the problem is to think of the tab in another way. The open tab is not an inconvenience. It is the user saying “I am willing to dedicate a process to this”. Closing it says “I am done, clean up”. The heartbeat effectively turns the open tab into a lease on the server-side work: it is how the browser keeps stating that the user is still interested, and the server is allowed to stop when it stops receiving that statement.
So the constraint that had been bugging me became the feature. A large immediate download runs exactly as long as someone is watching it, and the cost of it is visible to the person who chose to pay it. If you want to close your laptop, you use the background queue, which is what the app was always for. If you want to watch a progress bar and have the file now, you keep the tab open, and that tab is your half of the contract.
Scope, on purpose
Immediate transfers need a distributed cache (Redis or Memcached) because the progress and the heartbeat have to be visible across processes. If the instance does not have one, the checkbox is simply hidden and the background queue keeps working. I did not want a feature that half worked on a plain setup and produced confusing failures. It is present when it has the means to work correctly and is disabled when it does not.
What 1.0 actually means here
The 0.7 releases were mostly about making Transfer maintainable again: simplifying the frontend, replacing legacy APIs, and cleaning up dependencies. Most of that was invisible to users. 1.0 is the first release where the visible surface grew: live progress, a settings page, and the immediate option above. After this the plan is boring on purpose. Bug fixes, keeping up with new Nextcloud releases, and the occasional feature, each one weighed against the pool of web workers it might quietly consume.
Transfer is on the Nextcloud app store if you run your own instance. The source is on GitHub, it is AGPL-3.0, and it supports Nextcloud 29 through 34.
The capable version of a feature is easy to demo and expensive to own. The manageable version is the one where the constraint and the feature turn out to be the same thing. The tab has to stay open, not because I failed to make the feature fully asynchronous, but because keeping it open is the user’s half of the contract.
