Skip to content
Theme
OnDevice Tools

SoftwareCloud & Services

Google Cloud launches AI accelerator storage to cut idle compute costs

Google Cloud says slow storage can leave expensive AI accelerators idle and has launched two systems to keep compute busy. The products target enterprises running large AI training and inference workloads.

Illustration of a server rack showing AI accelerator storage and data throughput metrics

Key takeaways

  • Slow data input and output can leave expensive AI accelerators idle, according to the Google Cloud Blog.
  • Cloud Storage Rapid can reach 15 terabytes per second in throughput and handle 20 million queries every second with sub-millisecond latency.
  • Google Cloud Managed Lustre, built with DDN, provides up to 10 terabytes per second per instance for parallel workloads.
  • Thinking Machines Lab and Shopify report higher throughput and lower latency using the new caches.
In this article
  1. What Google Cloud announced
  2. Why storage matters for AI accelerators
  3. Two new storage options from Google Cloud
  4. What early customers report
  5. What it means for buyers

What Google Cloud announced

The Google Cloud Blog says slow storage can leave expensive AI accelerators idle, and that storage choice can put the rest of a compute investment at risk. In a post dated 16 November 2026, Google Cloud argues that enterprise AI has moved from simply buying accelerators to keeping them fully utilised. It is promoting two purpose-built storage systems under the banner of AI accelerator storage.

The post is vendor material, so its performance claims have not been independently verified. Google Cloud says data input and output is often the main challenge in AI infrastructure. When accelerators wait to read or write data, compute capacity goes unused. Standard enterprise storage, even systems built for fast analytics, can create bottlenecks that leave GPUs and TPUs sitting idle.

Why storage matters for AI accelerators

During model training, thousands of GPUs may try to save checkpoints at the same time. Traditional cloud storage is built for durability, so each save triggers many administrative database transactions. This can choke the network and freeze the cluster until the save finishes. Slow storage can also turn checkpoint recovery into a long delay, leaving expensive hardware idle.

Inference work requires systems to load and unload huge model stores quickly. Slow storage causes cold starts while GPUs wait for data. Models with context windows above 100,000 tokens can fill accelerator memory, forcing the hardware to recalculate previous inputs if no high-speed cache is available.

Two new storage options from Google Cloud

Illustration: cloud, Google logo, Google Cloud logo, storage and the labels AI Accelerators, Cloud Storage Rapid, Google Cloud

Google Cloud is offering Cloud Storage Rapid and Google Cloud Managed Lustre. Cloud Storage Rapid connects Colossus, the distributed file system behind Google Search, YouTube and Gemini, directly to accelerators. A single bucket can reach 15 terabytes per second in throughput and handle 20 million queries every second, with latency below one millisecond, according to the blog. Google Cloud says it spent two years co-designing this architecture with frontier labs and its own foundation model teams. It also claims that nine of the world's top AI labs run on its infrastructure.

Rapid Cache acts as a read accelerator alongside TPU and GPU clusters and can reduce cold-start delays from hours to minutes. Google Cloud Managed Lustre, built with DDN, provides up to 10 terabytes per second per instance for parallel file system workloads.

Its Dynamic Tier combines hot and cold data into one namespace. Cloud Storage Mirror copies datasets in the background so files are ready before training starts.

What early customers report

Thinking Machines Lab uses Cloud Storage Rapid Cache to keep read throughput above 1.8 terabytes per second and says it has reached an 89 per cent cache hit ratio. The company reports that rate-limiting surge errors have been removed across its multi-region setup.

Shopify has deployed more than 50 Rapid Caches worldwide and says it achieved over 2 terabytes per second of peak throughput with latency reduced by up to 90 per cent.

What it means for buyers

Organisations running large AI training or inference workloads should assess whether their current storage creates idle compute time. Google Cloud says its systems remove the need to move data into separate silos and that Rapid Cache requires no code changes.

The blog also cites a Gartner prediction that more than 150,000 AI agents will be deployed in coming years, which could increase pressure on storage systems.

Sources

  1. Beyond the compute bottleneck: Maximizing the ROI of AI accelerators with purpose-built storageGoogle Cloud Blog

Tags

Spotted a mistake? See our corrections policy. Read how News by OnDevice works in our editorial policy.