Cloud & Open-Source Data Engineering · Azure DP-900 Training · Recent Project
Two ETL projects, each built twice — once on Microsoft Azure, as the hands-on vehicle for DP-900 exam prep, and once entirely open-source and local, purely to see what the difference would actually be. It held up both times.
Background
The brief was DP-900 (Azure Data Fundamentals) exam prep, and the vehicle was Bee Haven — a beekeeping-data medallion pipeline (bronze → silver → gold) on Azure Data Lake Storage Gen2, processing hive sensor readings enriched with weather data via Azure Data Factory and Synapse Spark. As an open-source believer, I also built the same architecture on SeaweedFS — a local, Apache-2.0, S3-compatible object store — originally just to compare tooling, before it grew into a genuine parallel implementation worth documenting on its own.
After Bee Haven, I wanted a second, deliberately small project with real live data, to test something specific: would a genuinely simple automation be simple in Azure? A German electricity-price monitor (fetch a free API daily, clean, store, visualize) seemed like the right size to find out — and, again, got built as a pair: once on Azure, once as a local Docker Compose stack.
What follows is the honest record of both pairs — where each one actually finished, not where it was originally planned to.
The Work
Same architecture, same data, two platforms each — read any of them on their own.
ADF → Synapse Spark medallion pipeline, weather-enriched, full orchestrator run against fresh data. Scheduled trigger built correctly, never confirmed to fire.
→ Read the full buildThe same medallion architecture, rebuilt on a local, open-source, S3-compatible store — genuinely round-trips through the S3 API, browsable via WebDAV, its own real bugs found and fixed.
→ Read the full buildDeliberately "simple" side project — ADF + Azure Functions — that still took most of a day and ten distinct roadblocks to get running.
→ Read the full buildDocker Compose + Python + Plotly. Its own three real bugs found and fixed, and — unlike the Azure side — the dashboard actually shipped.
→ Read the full buildAcross All Four
The same handful of issues, twice, on two unrelated projects.
Bee Haven needed three separate managed identities each granted their own storage role. The electricity Function App hit the same lesson in a new context: the auto-granted role is almost never for the storage account you actually care about.
The same silent, non-firing ADF schedule trigger — correctly configured, confirmed "Started," never observed to actually fire — independently on two unrelated projects. A well-documented, long-standing community issue, not a config mistake.
Flex Consumption silently blocked on trial subscriptions. A user-assigned identity created by default when nothing asked for one. A resource provider that needs manual, one-time registration before anything works.
SeaweedFS lost a whole bucket once too (no volume mount on the first container). The local electricity build had three genuine logic bugs of its own. The difference was minutes to diagnose and fix, not days.
No cross-service managed-identity handshake, no black-box trigger, no separate compute tier to provision for a 17-row job — just a script talking directly to storage. Most Azure roadblocks above simply have no equivalent locally, because the layer that could fail isn't there.
Not a Clean Sweep
Neither open-source build finished exactly as ambitiously planned, and neither Azure build finished with a perfectly clean scorecard either. The electricity project's local build stayed flat: SeaweedFS sits in the Docker Compose file unused, storage went to the plain filesystem instead — no real S3-API parity attempt like Bee Haven's SW build has. The electricity project's Azure build, meanwhile, never got its dashboard: Power BI Desktop is Windows-only, this was all built on a Mac, and after two working core pipelines in one week, that felt like the right place to stop rather than force it. Bee Haven's Azure trigger has the same shape of gap — built correctly, never confirmed firing. This page states all of that plainly rather than smoothing it into a tidier story than what actually happened.
Status
Full Write-Ups
Same instinct — assignment turns into a working local build — showed up a third time, in a different domain entirely.
The full build, the identity saga, and the trigger bug that never got resolved.
→ ReadSame architecture, local S3-compatible store, its own bugs found and fixed cleanly.
→ ReadA deliberately simple project, and the honest gap between "simple" and "simple in Azure."
→ ReadThe version that actually shipped its visualization — plus three real logic bugs, fixed.
→ ReadA DP-900 podcast assignment that turned into a local AI production pipeline instead — same "just build it locally" instinct, applied to TTS and an LLM.
→ Read