Technical Deep Dive: Electricity Prices Azure Pipeline

All ten roadblocks, in order, with the actual symptom, root cause, and fix for each

1. Choosing the API — a non-issue, for contrast

Compared weather, air quality, public transit, and electricity prices on ease of access. aWATTar won cleanly — no API key, no registration, one clean endpoint. This is what "simple" actually looks like when a service is designed well.

2. ADF ingestion — genuinely smooth

REST linked service (Anonymous auth) → REST source dataset → JSON sink dataset → Copy activity. Built without real issues, reusing patterns already learned from Bee Haven (dynamic filenames via formatDateTime(utcnow(), ...), same linked storage account).

3. Deciding how to run the cleaning step

Initially defaulted toward Synapse, since that's what Bee Haven used. Real concern raised before starting: Synapse's Spark pool comes with a fixed 12-vCore trial quota regardless of how small the actual job is — a 17-row cleaning job would require spinning up the exact same pool, cold-start delay, and quota contention risk as a 2.7-million-row job. Correctly decided against Synapse for this reason before hitting the problem, and chose Azure Functions instead.

4. Flex Consumption not supported on trial subscription

Symptom: "Free trial subscription is not supported for Flex Consumption. Please select a different subscription."

Root cause: Microsoft's current Function App wizard defaults Linux Python apps toward the newer Flex Consumption hosting plan, deliberately excluded from free trial subscriptions. The portal's standard wizard, at time of building, didn't offer the older classic Consumption plan as a selectable option at all for Linux — only Flex, Premium, App Service, Container Apps, or Consumption (Windows, unusable for Python).

Fix: bypass the portal wizard entirely; create the Function App via VS Code's Azure Functions extension instead, which does support choosing the classic Consumption plan.

5. Missing host.json

Symptom: "No azure function project root could be found. This can be caused by a missing host.json file."

Fix: added host.json (standard v2 Python model config) and local.settings.json (expected by tooling even though not deployed).

6. Microsoft.Web resource provider not registered

Symptom: "The subscription is not registered to use namespace 'Microsoft.Web'."

Root cause: this subscription had never created an App Service/Function App resource before, so the underlying resource provider had never been activated — a one-time, per-subscription step unrelated to this specific project.

Fix: Portal → Subscriptions → Resource providers → search Microsoft.Web → Register. Took a couple of minutes to propagate.

7. First VS Code attempt still failed

Even after moving to VS Code, the first "Create Function App in Azure" attempt still defaulted toward a Flex-style App Service Plan (FLEX-AwattarApp-d8c6) and failed for the same underlying reason as #4, combined with #6 not yet having propagated. The second attempt, run a few minutes later, succeeded cleanly — likely a combination of the provider registration finishing propagation and simply retrying. Left an orphaned, empty resource group (awattarapp) behind — harmless, but a small cleanup item.

8. RBAC — same lesson as Bee Haven, in a new context

The deployment process auto-created a role assignment for the Function App's own internal housekeeping storage account (auto-created by every Function App) — easy to mistake for the actual data storage access needed. The real storage account (lake00bee) still needed its own, separate, manually-added role assignment for the Function's identity.

Fix: manually added Storage Blob Data Contributor for the Function's identity, explicitly scoped to lake00bee.

9. Portal's Test/Run button blocked by CORS

Symptom: "Running your function in portal requires the app to explicitly accept requests from https://portal.azure.com."

Root cause: the in-portal manual test button is itself a web client calling the function's admin API — like any cross-origin request, it needs the target to explicitly allow the portal's own origin. A one-time UI convenience setting, unrelated to the function's actual runtime security.

Fix: Function App → CORS → add https://portal.azure.com to Allowed Origins.

10. Wrong managed identity type in code

Symptom: DefaultAzureCredential failed with "App Service managed identity configuration not found in environment" despite the identity having correct storage permissions.

Root cause: VS Code's Function App creation flow, by default, creates a user-assigned managed identity (a separate named resource, e.g. awattarapp2-identities-a69038) rather than the simpler system-assigned identity. DefaultAzureCredential() called with no arguments only looks for a system-assigned identity by default.

# Fix: pass the user-assigned identity's Client ID explicitly
credential = DefaultAzureCredential(managed_identity_client_id="7820a2d9-...")

Success

Once all ten issues were resolved, the function ran correctly on the first genuine attempt: authenticated, read 4 bronze files spanning multiple days, correctly deduped and merged into a 32-row Silver table, archived and deleted the originals, completed in 943ms.

Reflection

None of the ten roadblocks were caused by the actual problem being solved, a mistake in understanding the concept, or the API being used — aWATTar was flawless throughout. Every single one was Azure-platform-specific tooling friction: subscription-tier restrictions not surfaced until attempted, a required config file not obvious from documentation, a one-time per-subscription registration step, confusing wizard defaults that don't match what a trial/personal-use developer actually needs, and a browser-security setting unrelated to the function's real security model.

Each individual issue, once hit, had a findable, documented, one-line fix. But there were ten of them, for a project whose actual logic is four bullet points long. Proportionally, given how much smaller this project's scope was than Bee Haven's, the friction-to-substance ratio was arguably worse — a 30-line script took longer to get running in Azure than it would have taken to write from scratch on a laptop with no cloud involved at all.

Closing update: the trigger bug, again

The scheduled trigger, correctly configured and confirmed "Started," did not visibly fire over several days of observation — resulting in partial-day Silver data (17/19/20 rows on three different days, versus the expected ~24) rather than missing days entirely. Confirmed correctly configured via the same verification steps used on Bee Haven. The dedup/merge logic itself was independently confirmed correct against the local Docker comparison build, which uses its own internal scheduler and reliably captures a full 24 rows/day — isolating the gap specifically to ADF's trigger reliability, not the Function App's cleaning logic.

Full story: project overview. Compare against the open-source parallel build.