Copel cuts Reverse ETL complexity with BigQuery exports to AlloyDB

What Google Cloud announced

Google Cloud has announced the general availability of BigQuery exports to AlloyDB, according to the Google Cloud Blog. The feature lets data engineering teams move data from BigQuery, Google's analytical data warehouse, into AlloyDB for PostgreSQL, its transactional database, using a single declarative SQL statement.

The company positions this as a way to remove the middleware, staging buckets and orchestration that Reverse ETL, the process of syncing analytical results back into operational systems, has traditionally required.

Why Reverse ETL is usually hard

Analytical systems, known as OLAP, are good at scanning large volumes of data but are not built for the fast, row-level reads and writes that customer-facing applications need. Those workloads run on transactional databases, known as OLTP. Keeping the two in sync has historically meant building pipelines with many moving parts.

Google Cloud says this is especially true at enterprise scale, where teams often rely on heavy middleware, intermediate file storage and complex orchestration to move data between the two worlds.

Copel's legacy pipeline

Copel, a Brazilian electric power provider, uses BigQuery as the analytical home for telemetry from millions of smart electricity meters. To power what it calls its Systems of Action, including low-latency support dashboards, mobile apps and operational billing, that data must be continuously synced into AlloyDB.

Until recently, Copel ran this sync as a complex DAG, a directed acyclic graph of scheduled tasks, in Google's managed service for Apache Airflow. BigQuery enrichment queries wrote results into dozens of fragmented CSV files in a Cloud Storage bucket. Airflow then created temporary tables in AlloyDB to avoid locking production tables, sequential gCloud scripts loaded the CSV data into the cluster and its shards, and a final SQL transaction swapped the temporary table into production.

The team had to maintain the pipeline, manage file staging, handle IAM impersonation across several services and monitor the health of the whole chain.

The new architecture

With the export feature, Copel replaced that multi-step process with one SQL statement. BigQuery now talks directly to AlloyDB over parallelised connections, so there are no staging files in Cloud Storage, no gCloud scripts and no temporary tables to manage.

According to Google Cloud, the exported values map to AlloyDB's columns during the transfer, and nested structures remain fully queryable and indexable on the PostgreSQL side. Copel's Airflow DAG now contains a single BigQueryInsertJobOperator task that runs the export query, replacing dozens of lines of scripting and custom error handling.

Performance and business benefits

Copel validated the new setup with two benchmarks, comparing a smaller incremental export against a much larger full-table export. Google Cloud says the direct export sustains a throughput of roughly 4.2 million rows per minute on complex JSONB telemetry fields, a figure the company attributes to the combination of BigQuery's distributed processing engine and AlloyDB's storage layer.

Google Cloud lists several benefits from the migration. Skipping Cloud Storage removes staging storage costs and the related API operation fees. Data moves inside Google's secure internal network and is never written to readable CSV files, which improves security. Troubleshooting is simpler because errors are contained within one SQL statement, freeing engineers from monitoring intermediate storage and pipeline failures.

What it means for other teams

The feature is generally available now, so any team pushing BigQuery data into downstream transactional systems can try it. Google Cloud notes that similar built-in export functionality exists for Bigtable, Spanner and Pub/Sub.

For organisations still running file-based Reverse ETL pipelines, the Copel example suggests the maintenance and cost burden of staging-based architectures can be avoided. The claims about performance and benefits come from Google Cloud's own blog post and have not been independently verified.