Everything a prediction touches.
- 115 requests a second
- ~1M GPU-hours a month
- 95%+ of the bill, GPUs
The idea behind Replicate was to make AI models something you could build into ordinary software, like any other API, without needing to know the nitty-gritty. Just pick your model and run your prediction1.
A request reaches the API2, gets routed to a model running somewhere on a fleet of 1,500 GPUs3, gets handed to a process inside a Docker container running Cog4, and produces output that’s streamed back to the user5. Somewhere north of 10 million times a day.
Along the way it gets metered and billed, and also turns up in the revenue and margin reporting, which told us whether that request, and millions like it, had made us any money.
Ten million requests a day sounds like a lot, but it’s only about 115 a second, a modest load for a web service. What made Replicate big was how long each one ran. A lot of models took a long time to run – sometimes several minutes – so the real measure was GPU time: a fleet capable of around a million GPU-hours a month.
That made the economics strange. Most infrastructure bills are a blend: some compute, some databases, some storage, some bandwidth, nothing dominant. In Replicate’s early days, more than 95% of the bill was GPU time,7 and even once storage and bandwidth grew into real line items, GPUs still dominated. “Do we really need three instances of the web service, or can we get away with two?” stopped being a question worth asking. It was a rounding error, so you picked the bigger option and spent your attention where the money was: keeping the GPUs busy. The margin numbers drove how we ran the fleet.
The technical work of building the system was interesting, but the investigative work of figuring out which parts of our usage made us money was downright fascinating. Not many engineers get to follow a request all the way from the front door to the money, and maybe we’re all a little worse off for it.