Coding · ·

Persist files to the cloud with S3

The server your website is running on wipes all your files - other than those tracked by git - every time you git push or whenever I restart it. This means if you write files with eg Bun.write('file.txt', 'blah blah') the next time you git push and your changes go live, you'll have lost the file.txt you wrote with Bun.write() before.

This guide takes you through how to use Supabase (free) to add long-term persistence for files on your server, with cloud based file-storage (S3). In a later guide, you'll be able to use the same Supabase account with a PostgreSQL database. Both will persist data in the cloud, which can be accessed either from your online server or from programs running on your computer.

S3 counts as virtual storage - something you need to know about for exams.

Quick start

By the end of this 'Quick start' you'll be reading and writing files that are stored in the cloud as easily as reading and writing files stored locally.

go to https://supabase.com/ and create an account ('Continue with GitHub' is the easiest option)
if it asks you to 'Create a new organisation' just call it 'me' or whatever and select the free plan
create a new project with the default settings
- you will need to choose a 'Database password' even though we won't be using a database yet
  - I suggest you use a strong password chosen by your browser (you can change it later if you forget it)
it can sometimes take a minute or so until your project is created

Back in VSCode, in the root of your website's repo, create a new file simply called .env with the following contents:

S3_ENDPOINT=""
S3_ACCESS_KEY_ID=""
S3_SECRET_ACCESS_KEY=""
S3_BUCKET=""

From your Supabase project's dashboard:

go to 'Project settings' (bottom-left cog icon)
then 'Storage' (under 'Configuration')
then 'S3' (under 'Configuration' on the left)
copy the value from the 'Endpoint' field into the correct line in your .env
click 'New access key' below and give it whatever description you want
copy the 2 keys into the matching lines in your .env
click 'Done'
go to 'Storage' in the left-side menu
then 'New bucket', giving it whatever name you want and leaving it as private
whatever name you just used, put that into the matching line in your .env file too
save your .env file

Your .env file will now look something like this (but with your details not these random ones!):

S3_ENDPOINT="https://sadfjoweorjweojojsdj.storage.supabase.co/storage/v1/s3"
S3_ACCESS_KEY_ID="a3f7c1b2d49e58f0c7a1e9b43d56f2c8"
S3_SECRET_ACCESS_KEY="7f2c9b1de43a58f0b6d8a2e19cf47d3a1b59e0c4f872d65a0e3b1f9c2d47a8e5"
S3_BUCKET="stuff"

See if it works by making an s3-tests.js:

let path = 'test.txt'

// read
let fh = Bun.s3.file(path)
console.log(await fh.exists() ? await fh.text() : '[no file]')

// write
await Bun.s3.write(path, Date.now())

... and running it with bun s3-tests.js. You should see it log [no file] which is correct. Run it again and it should show the contents of the file you just created (a timestamp eg 1762520230138) instead of [no file]. The file test.txt is stored in the cloud with Supabase, instead of on your local file system. Once you can see it works, you can remove the s3-tests.js file.

Reading and writing files

You learnt in an earlier guide how to read and write local files with Bun:

// read local file
let fh = Bun.file('text.txt')
let text = await fh.exists() ? await fh.text() : '[no file]'
console.log(`The file has this in it: ${text}`)

// write local file
await Bun.write('output.txt', 'The quick brown fox')

Here's how you can read and write cloud files now you've set things up with Supabase and your .env file:

// read cloud file
let fh = Bun.s3.file('text.txt')
let text = await fh.exists() ? await fh.text() : '[no file]'
console.log(`The file has this in it: ${text}`)

// write cloud file
await Bun.s3.write('output.txt', 'The quick brown fox')

Did you spot the differences? Thankfully, it's almost identical!

instead of Bun.file() use Bun.s3.file()
instead of Bun.write() use Bun.s3.write()
alternatively:
- import { s3 } from 'bun' at the top of your file
- then just use s3.file() and s3.write().

When you run Bun, if it finds a .env file in your current folder it will read the values in as environment variables. When you use Bun.s3.file() it uses those environment variables to connect to your S3 'bucket' with Supabase.

Optional: Key-value store on top of S3

We're going to build a super simple client-side key-value store on top of Supabase's S3. Here's how we'll be able to use it from clients (such as from your website):

let highScore = await kv.get('highScore') // gets the current high score
await kv.set('highScore', highScore) // saves a new high score

Pretty neat right?! This comes with the warning that users could heavily abuse this, and it's probably not a good idea for that reason... but let's give it a go anyway!

Start by adding 2 route handlers to your server.js (inside Bun.serve({ routes: { ... here }})):

// within your 'routes' object
'/data/:key': {
  GET: async (req) => {
    // get a value for a key
    let key = req.params.key
    console.log('kv get', key)
    let fh = Bun.s3.file(key)
    return new Response(await fh.exists() ? await fh.text() : '')
  },
  POST: async (req) => {
    // set a value for a key
    let key = req.params.key
    let value = await req.text()
    console.log('kv set', key, value)
    await Bun.s3.write(key, value)
    return new Response('ok')
  }
}

That's it for the server.

Your client-side JavaScript needs this:

// very simple persistent key-value storage
let kv = {
  async get (key) {
    return await (await fetch(`/data/${key}`)).text()
  },
  async set (key, value) {
    await fetch(`/data/${key}`, { method: 'POST', body: value })
  } 
}

Which can then be used with eg:

let highScore = await kv.get('highScore') // gets the current high score
await kv.set('highScore', highScore) // saves a new high score

Here's an example HTML file that uses it, which you can plonk in your ./public folder called maybe high-scores.html:

<!doctype html>
<title>S3 KV persistence test</title>
<p>Testing persistence of eg high scores.</p>

<p>The current highest score is <b id="current"></b></p>

<form id="form">
  <input id="highScore">
  <button>Save high score</button>
</form>

<script>
// very simple persistent key-value storage
let kv = {
  async get (key) {
    return await (await fetch(`/data/${key}`)).text()
  },
  async set (key, value) {
    await fetch(`/data/${key}`, { method: 'POST', body: value })
  } 
}
  
getScore()

async function getScore () {
  current.innerText = await kv.get('highScore')
}

form.onsubmit = async (e) => {
  e.preventDefault()
  await kv.set('highScore', highScore.value)
  alert(`High score of ${highScore.value} saved`)
  getScore()
}
</script>