Limitations
Persistent Storage, powered by Backblaze, is in Beta. Account owners and admins can join under Settings → Profile, or by ticking Join the beta program at signup. They can leave it the same way, under Settings → Profile. See Overview.
Persistent Storage is object storage with a filesystem layer on top. It looks like a folder, but it does not behave exactly like a local disk. Read this page before you put a workload on it.
Persistent Storage is object storage behind a filesystem layer. It has higher latency than the instance's local NVMe disk and only partial POSIX behavior. Keep hot data on local disk.
Slower than local disk
Every file operation that isn't already cached on the instance is a network request to object storage.
- Opening, listing and reading files has much higher latency than the instance's local NVMe disk.
- Many small files are the slowest case. Large files read or written whole work best.
- Recently used data is cached in the instance's memory, up to 1/8 of its RAM (at least 512 MB, at most 8 GB). Data outside the cache is fetched again.
Keep hot data on local disk. Copy what a job reads over and over to local disk first, run the job there, and copy the results back to /persistent/<name> at the end. Don't run databases, package caches, virtual environments or scratch space from Persistent Storage.
Not a full POSIX filesystem
| Behavior | On Persistent Storage |
|---|---|
| Hard links | Not supported. |
| File owner and permissions | Fixed. Every folder is 0777 and every file is 0666. chmod and chown don't stick. |
| Writes | Uploaded in the background after you write. A file isn't safe in storage until the upload finishes. Run sync /persistent/<name> before you stop or terminate. Plain sync doesn't wait for these uploads. |
One instance at a time
Attaching the same storage to more than one instance at the same time isn't supported and can corrupt your data. Every mount is read-write, and there is no file locking between instances. Attach each storage to one instance at a time: stop the instance that has it and detach it before you attach it to another.
If the same storage is mounted on two instances anyway:
- There is no file locking. Two instances can write the same file at the same time and one write is lost.
- Each instance caches folder listings and file details for about 60 seconds. One instance can act on old data and overwrite newer changes from the other.
To move storage between instances, see One instance at a time.
When storage can't be reached
Persistent Storage needs a working network connection from the instance to storage.
When the instance starts
- The instance always finishes starting. Storage that can't be mounted doesn't block it.
- Each storage gets about 30 seconds to mount, and then one test listing.
- If that fails, the folder
/persistent/<name>is removed if it is empty. A note named/persistent/<name>.MOUNT-FAILED.txtexplains why. - Don't write into
/persistent/<name>if the storage didn't mount. Anything written there is on the instance's local disk and is lost when you terminate.
To try again, stop the instance and start it.
While the instance is running
If storage stops answering while it is mounted:
- Reads of data that isn't cached, folder listings and uploads wait, then fail. Each request waits up to 15 seconds and reads are retried 3 times. Your program gets an I/O error (
EIO). - The folder stays mounted. It works again when storage can be reached.
- Data you wrote that hasn't finished uploading is held in the instance's memory. It is lost if the instance is stopped or terminated before the upload finishes.
What the website shows
- If the Massed Compute storage service can't be reached, the Storage page shows "Persistent Storage is temporarily unavailable. Try again in a few minutes." This notice doesn't cover Backblaze itself: if only Backblaze is down, the page still loads normally while the instance sees the errors above.
- If Persistent Storage is switched off, deploy refuses storage with "Persistent Storage is not available right now."
- Storage that failed to mount on an instance isn't flagged on the Storage page. Check for the
.MOUNT-FAILED.txtnote on the instance.
