Follow Cyber Kendra on Google News! | WhatsApp | Telegram

Add as a preferred source on Google

AWS's Trillion-Dollar Billing Bug Resists First Fix

AWS's billing bug showed trillion-dollar estimates; a rollback fix failed and mitigation is still underway. Real charges are unaffected.

AWS Billing Bug

Amazon Web Services customers spent Friday staring at bills that made no sense: hundreds of billions, low trillions, in a few reported cases, even quadrillions of dollars, for cloud usage that hadn't changed overnight. 

AWS confirmed within hours that the numbers were wrong, but more than twelve hours later, its first attempted repair had failed, leaving the Billing and Cost Management Console broken and finance teams stuck watching alerts that wouldn't clear.

The glitch surfaced late Thursday night and spread across X by Friday morning as screenshots piled up.

One widely shared post showed a projected monthly bill of $1.5 trillion. Another documented case put a single account's estimate at $502 billion, with a month-over-month change of more than 55 trillion percent. At more than one company, the first instinct wasn't to suspect a display bug — it was to assume a breach, and teams began rotating access keys before anyone confirmed the figures were fiction.

AWS traced the fault to Cost Explorer, the tool that forecasts spending using estimated pricing (projected rates applied to usage data) rather than the metered, validated figures behind real invoices. The company said the root cause was "an issue with unit pricing within the estimated billing computation subsystem" — in effect, a miscalculated rate that multiplied ordinary usage into astronomical projections.

The fix itself became the story: engineers tried two paths in parallel, rolling back the offending change while separately reverting to the last accurate estimates on file. By Friday morning, AWS conceded the rollback hadn't worked. Estimated billing remains paused while it recomputes accurate figures, a process it expects to take several more hours.

The deeper lesson lands on FinOps teams that pipe Cost Explorer straight into Slack or PagerDuty: a display bug can trigger the same scramble as a real breach. If your dashboard looks unhinged right now, don't touch anything or pay anything — AWS says no action is needed. 

Verify independently through Cost and Usage Reports, CloudTrail, and AWS Config instead. Only if a spike survives AWS's recalculation should you go looking for a genuine cause, like exposed credentials, cryptomining, or runaway autoscaling.

Post a Comment