Seconding the last line hard. A backup you have never restored is a hypothesis, not a backup. Borrow or dig out an old handset once a year, restore onto it, and see what is actually there. Mine was missing an entire messaging app's history and had been for over a year, and the backup dashboard was cheerfully green the whole time.
Egon Rask
@egress_egon
Technical co-founder. Our hosting bill went from $70 to $2,300 in a month because of an image endpoint with no cache header, and I have been reading invoices line by line ever since.
21 credit Contributor
- From answers
- 0
- From questions
- 21
Ask whether you need to roll back at all. use cache and cacheComponents have been part of the stable 16 line since 16.0. The preview channel is where the newer navigation work lives.
If what broke your app was the i18n/auth interaction with caching, dropping to the preview build's predecessor on the stable line keeps your caching work and removes the moving target. If what you actually wanted was the new navigation behaviour specifically, then yes, you are choosing between that and a quiet week.
The 80% cut settles it, surely. If the model got five times cheaper then a $20 subscription buys five times as many messages, so the chat plan is now obviously the better deal per dollar.
Fair, I had welded the two together in my head. Checked, and the thing I actually care about is which surface my usage is billed from, which for me is the subscription and not the API. Adjusting my spreadsheet, and my confidence.
One thing nobody mentions until it hurts: log the inputs alongside the usage, not just the counts. When you find the expensive 3% of requests you want to be able to look at what they actually were. I keep a hash of the prompt and a truncated first 500 chars. Cost me about 40MB a month and saved a full day of guessing.
Do not turn anything off yet. First find out what it is - it takes about five minutes and guessing costs you more.
Cost Explorer, then set it up like this:
- date range: last 30 days
- granularity: daily
- group by: Usage Type (not Service - Service will tell you "EC2-Other" and you will learn nothing)
The daily view is the important part. You are looking for the day the line goes up, because that tells you what you changed. Usage Type tells you what you are being charged for in words specific enough to search.
On a small account, the top three causes by a mile:
- NAT Gateway data processing. Charged per GB through it plus an hourly rate. If your VPC tutorial put your compute in a private subnet with a NAT gateway, then every byte your hourly job pulls from S3 goes through it and gets billed. Same-region S3 traffic that should be free suddenly costs $0.045 a gigabyte. This is the single most common way a hobby account produces a four figure bill, and your description - private subnet, hourly job, S3 - matches it exactly.
- Cross-AZ data transfer. Chatty services in different availability zones, billed both directions.
- EBS snapshots or forgotten volumes left behind by an instance you terminated.
If it is number one, the fix is a VPC gateway endpoint for S3. It is free, it takes about two minutes in the console, and it routes S3 traffic inside your VPC instead of out through the NAT. Add one for DynamoDB too while you are there.
Be aware that interface endpoints for other services are not free - they have an hourly charge per AZ plus per GB. The gateway endpoints for S3 and DynamoDB are the free ones. Do not read "endpoints are free" and add fifteen of them.
Password manager, unique addresses, updates. Everything else is a hobby, and hobbies stop when you are busy.
No - the pitch is no egress charges. Operations are billed in two classes, and Class A is the expensive one at several dollars per million. It is genuinely cheap for normal usage, which is why nobody thinks about it until they put a list call in a hot path and do 47 million of them.
The mental model that fixes this permanently: object storage is a place to put bytes, not a place to ask questions. Any question - what exists, what changed, what belongs to this user - is a database query.
What that looks like concretely:
- one row per object in D1 (or Postgres, whatever you have): key, size, content type, owner, created_at, plus whatever you want to filter on.
- writes go through your code, so you insert the row in the same request that does the PUT.
- the gallery reads the DB. R2 is only ever touched to serve or store the actual bytes.
- a nightly reconciliation job lists the bucket once and fixes drift, because drift will happen.
That nightly job is one Class A operation per 1,000 objects, once a day. Your 47 million becomes something like 3,000 a month.
The pricing asymmetry is the thing to burn into memory: Class A (writes and lists) costs more than ten times what Class B (reads) costs. Any design where a read path triggers a Class A op is going to be expensive, and it will be expensive in proportion to your traffic, which is exactly the wrong direction.
Transform-on-upload is obviously right for my case and I think I avoided it because it means a queue and a worker. Which, fine, that is an afternoon and it is $263 a month.
Release is a deploy. Launch is a date plus a list plus a reason for the date. You have a deploy.
At £1,000 including a screen, a new laptop that handles two VMs comfortably is a stretch and a desktop is not. Compare what 32GB of RAM costs in each and the argument mostly settles itself. Whichever way you go, buy the RAM you need up front on the laptop, because you usually cannot add it later.
Take 1.6 per kilo, round it to 125, spread it across three meals, and close the calculator tabs. The precision you are chasing is smaller than your daily logging error.
Cache the query embeddings too, not just the document ones. In our logs about 38% of searches in any given week are a string we have embedded before - people re-run the same search, or five users in a team search the same project name. A hash-keyed cache with a 30 day TTL is twenty lines and shaves real money off the API line.
We added a second required reviewer, reasoning that more eyes would catch more. Throughput halved, both reviewers assumed the other was reading carefully, and the two worst bugs of the quarter went through with two approvals each. We reverted it after five weeks. Diffusion of responsibility is real and adding people to a review is the wrong lever.
Add to the list of things that quietly keep serverless databases awake:
- migration tooling that opens an advisory-lock connection at boot and holds it
- a background job runner polling a jobs table every few seconds
- an admin dashboard tab someone left open with a 10 second refresh
- log shippers or metrics exporters querying
pg_stat_* - your own local development environment connected to production because of a stale
.env
The last one has happened to two people I know and the tell is compute activity in a timezone where you have no users.
Default to off on lookup failure is the detail I would have got wrong and only discovered during an incident.
I have honestly never timed a rollback. Doing that this weekend.
At 8TB and growing 15% a month, move. Zero egress is not a marginal saving at your volume, it is the difference between a viable product and one where every new customer makes the maths worse.
Rough monthly egress at list price for 8,000GB:
- S3 straight to the internet: around $711
- S3 behind CloudFront: around $595 after the free tier
- R2: $0
You pay storage on R2 either way - 600GB is roughly $9 - plus operations, which for video streaming are Class B and negligible. So your $400 budget goes from impossible to mostly unspent.
The honest reasons not to move:
- You need S3 event notifications into an existing AWS pipeline. Your thumbnail Lambda is exactly this. R2 has event notifications too, but they land in a different queue system and you would rewrite that piece.
- Lifecycle rules into archival storage classes, if you have a lot of cold content.
- Any AWS-native feature applied on the object path.
Given you have exactly one Lambda hook, rewriting it is a day of work against $600 a month. That is not a close call.
One genuinely important detail: the zero-egress model assumes normal serving. Read the terms for your traffic type rather than assuming, and if you are ever unsure whether your usage fits, ask them before you migrate 600GB.
Concentrating also makes spot pricing viable, which is often less than half of on-demand for the same card. A queue that tolerates a machine disappearing mid-batch is not much extra code and it changes the maths more than any model choice.