Here is what I learned building it.

Step 1: pick a model you're actually allowed to use

Most "remove background" tutorials use RMBG-1.4 or RMBG-2.0. They're very good, but their licences are non-commercial, and my site runs ads. ISNet via @imgly/background-removal is AGPL, which I didn't want in a closed-source site. That ruled out the usual suspects before I wrote any code.

What I tested next:

So the plan became: BiRefNet lite on WebGPU when the GPU can handle it, ormbg on the CPU everywhere else.

Step 2: run the model in a worker, not on the page

Inference blocks for seconds, so everything lives in a module Web Worker. Transformers.js does the heavy lifting:

const { pipeline } = await import('https://cdn.jsdelivr.net/npm/@huggingface/transformers@4.3.1/dist/transformers.min.js');

const segmenter = await pipeline('image-segmentation', 'onnx-community/ormbg-ONNX', {
  device: 'wasm',
  dtype: 'quantized',
  progress_callback: (e) => {
    if (e.status === 'progress_total') postMessage({ type: 'progress', loaded: e.loaded, total: e.total });
  },
});

The model files come from the Hugging Face Hub once and Transformers.js keeps them in Cache Storage, so the second visit starts without a download.

One gotcha cost me an hour: set onmessage before any top-level await import() in the worker. If the import comes first, the page's first message arrives while the module is still loading and is silently lost.

Step 3: don't feed a 12-megapixel photo to the model

Phone photos are huge, and segmentation models work at roughly 1024 px anyway. The page:

1. Draws a copy of the image scaled down to at most 1024 px on its longest side.
2. Sends only those pixels to the worker (as a transferable buffer, no copy).
3. Gets back a one-channel mask at that small size.
4. Scales the mask up to the original size and applies it as the alpha channel of the original pixels.

The result keeps the full resolution (a 2400×3000 photo comes back as a 2400×3000 PNG), while the model only ever sees a 1024 px image. The mask is soft at the edges after upscaling, which looks natural around hair.

Step 4: the WebGPU limit nobody mentions

My dev server has no real GPU, so the WebGPU path couldn't be tested properly there. The first time I ran it on a normal desktop GPU, the very first image failed with:

Too many storage buffers in shader. Current: 17, Max is 16

The BiRefNet ONNX export has a shader that binds 17 storage buffers. WebGPU exposes the limit as maxStorageBuffersPerShaderStage, and many GPUs cap it at 16. ONNX Runtime already asks for the adapter's maximum, so there was nothing to raise. On those cards this model simply can't run.

The fix is to ask the adapter before downloading 115 MB of model:

const BIREFNET_MIN_STORAGE_BUFFERS = 17;

async function choose(cpuOnly) {
  const cpu = { model: 'onnx-community/ormbg-ONNX', device: 'wasm', dtype: 'quantized' };
  if (cpuOnly) return cpu;
  const adapter = await navigator.gpu?.requestAdapter().catch(() => null);
  if (!adapter || adapter.limits.maxStorageBuffersPerShaderStage < BIREFNET_MIN_STORAGE_BUFFERS) return cpu;
  // fp16 halves the download (115 MB instead of 224 MB) but needs shader-f16.
  return {
    model: 'onnx-community/BiRefNet_lite-ONNX',
    device: 'webgpu',
    dtype: adapter.features.has('shader-f16') ? 'fp16' : 'fp32',
  };
}

Two more lessons from that bug:

• A failed WebGPU run can poison ONNX Runtime for the rest of the worker. After one GPU error, later runs in the same worker kept failing, even on the CPU backend. So the worker doesn't retry. It reports gpu: true, the page terminates it, starts a fresh CPU-only worker and sends the image again.
• Remember the failure. The page stores a cpu-only flag in localStorage, so the next visit goes straight to the model that works instead of failing once more.

To be honest about the outcome: on my GPU, and on many common cards, the limit is 16, so many visitors get ormbg on the CPU.

Step 5: let people fix what the model gets wrong

No model is perfect. ormbg sometimes keeps a soft shadow on a plain wall behind a person, and it does so confidently (mask values around 253 of 255), so no threshold can remove it without eating into hair as well.

Instead of fighting the model, I added an Erase / Restore brush: paint over the shadow to erase it, or paint back an earring the model dropped. The brush edits the mask, not the pixels, so restoring brings back the original image exactly. Two minutes of touch-up beats any amount of threshold tuning.

What I'd tell someone building the same thing

1. Check the licence first. The best-known models aren't usable on a site with ads.
2. Test on non-people images. Portrait models fail on pets and products.
3. Downscale for the model, upscale the mask. You get full-resolution output for the cost of a 1024 px inference.
4. Check adapter.limits before choosing a WebGPU model, and treat a GPU error as "start a new worker", not "retry".
5. Give users a brush. It turns "almost right" into "right".

You can try it here: free AI background remover. The model is cached after the first visit, and nothing is uploaded. If your GPU reports 17 or more storage buffers and BiRefNet runs for you, I'd love to know which card it is.