Moving from Stitch to Zero-ETL
Stitch is a solution built on Singer, an open source ETL tool. I had been on Stitch for a while without any issues, but a couple things came up that had me exploring other ETL solutions.
-
Cost - As our volume of data scaled so did our Stitch costs. Stitch kept reminding us that we were going over our monthly alloted row count and that we'd need to pay about 3x our current rate to process our volume of data. With this significant cost jump I started looking at alternative solutions.
-
Latency - Stitch was currently running once per hour. At the time we were on the Standard Annual Plan which paywalled faster replication times. The business was asking for more up to date data and before jumping to upgrading our Stitch plan I wanted to explore other options.
Around this time AWS has just launched a new feature between Aurora PostgreSQL tables and Redshift called Zero-ETL. At a minimal extra cost (besides the increase compute on Redshift and Aurora's side) Redshift could ingest data at near real-time (~5 second lag time) a significant improvment over what Stitch could offer.
The hardest part about properly evaluating Zero-ETL as a viable replacement for Stitch was getting an accurate cost estimate. Since Zero-ETL isn't a separate service, but rather a feature that increases compute on the Producer/Consumer side of the pipeline I had.
I ended up creating a version of it on staging and then using this formula:
Production Cost = Staging Cost x (Production Data Size / Staging Data Size) x (Production Transaction Rate / Staging Transaction Rate)
and monitoring the increased compute costs on AWS. The estimated production costs ended up being significantly less than the estimated Stitch upgrade costs, and from there the decision was easy.
Moving from Airflow to Mage
Moving off Airflow wasn't really a data team decision. The company was consolidating its tech stack, and Airflow was one more tool every engineer had to learn and someone had to maintain.
To be clear, Airflow wasn't the problem. The DAG model is expressive, the ecosystem is huge, and almost any question you have has already been answered somewhere on Stack Overflow. It just carried more operational overhead than what we were actually doing with it justified.
Mage was the replacement.
The Rough Edges
The documentation is the biggest gap, and I feel it constantly early on.
When something breaks in Airflow, the path to an answer is short — years of Stack Overflow threads, blog posts, and GitHub issues cover nearly every edge case. When something breaks in Mage, the docs are often thin, the community forum has no relevant threads, and you end up reading source code or filing a GitHub issue and waiting.
Testing was harder than expected too. Mage's testing patterns aren't as established as Airflow's, and some of the CI/CD integration took more custom work than we'd planned for.
None of this was a dealbreaker. It's the growing pains of a younger tool, and the Mage team ships fast. But the first few pipelines will take longer than you expect, and it won't be because the work is hard — it's because the answers are harder to find.
Was It Worth It?
Yes, with caveats.
Mage is a genuinely good tool. The block model is the right abstraction for how pipelines actually get built, iteration is fast, and the operational overhead is lower than Airflow's. But its youth shows — the doc gaps and thinner community are real friction, and they add up during development and debugging. If Mage reaches the maturity Airflow has today, I think it has a real shot at becoming the new standard. The underlying model is better. It just needs time and ecosystem depth.
On the ingestion side there's less to caveat: the cost savings are real and so is the latency improvement.
If I were doing it again, I'd change the rollout, not the destination:
- Migrate one pipeline at a time, and keep Airflow running alongside Mage until each one has proven itself. Running both costs some extra attention, but it makes every cutover boring — which is what you want
- Pad your estimates for the first few pipelines while you learn where Mage's sharp edges are
- If you're on Postgres and Redshift, audit your schemas for incompatible types before you commit to Zero-ETL, not after
Once you're past the learning curve, the day-to-day really is better than what I left behind.