Technical Deep Dive: Electricity Prices Local Pipeline

Docker Compose stack details, and the three real bugs found and fixed while getting the dashboard right

Stack

Docker Compose Stack
├── seaweedfs   – S3-compatible storage (present, unused by current pipeline)
├── etl         – Python service: fetch → process → merge → visualize
└── webdav      – Nginx server exposing the dashboard + charts

Data flow: aWATTar API → JSON saved to data/bronze/new/ → flattened & merged into data/silver/prices.parquet → Plotly charts + dashboard HTML written to webdav/ → served at localhost:8081.

Key classes/methods

ElectricityPricePipeline — fetch_prices() pulls today's prices from aWATTar; process_bronze_files() reads raw JSON, flattens, merges, triggers visualization; _flatten_prices() converts the API response into row dicts; _merge_to_silver() dedupes and merges into the master table; generate_visualizations() orchestrates chart + dashboard generation; run_once() / run_scheduled().

Bug 1 — zero-price records could be silently overwritten by stale data

Symptom: 0 prices weren't reliably reflected in the dashboard.

Root cause: _merge_to_silver() deduplicated on start_utc using pandas' default keep='first'. If an hour was already stored from an earlier fetch, that old row always won over a fresh fetch for the same hour — so a corrected price of 0 from a later fetch could never overwrite a stale non-zero value.

combined_df = combined_df.drop_duplicates(subset=['start_utc'], keep='last')

Also cleaned up _flatten_prices(), which filtered records using a mixed truthy/is not None check inside all([...]) — fragile and easy to misread:

if start_ts is None or end_ts is None or marketprice is None:
    continue

This makes it unambiguous that a price of 0 is valid data, not "missing."

Bug 2 — charts never appeared in the dashboard (blank iframes)

Symptom: the heatmap and time-series <iframe>s in the dashboard were empty.

Root cause: the dashboard HTML requests /visualizations/heatmap.html and /visualizations/timeseries.html, but generate_visualizations() was writing those files directly into viz_dir — the same directory Nginx serves as its web root. That put the files one level too high (reachable at /heatmap.html, not /visualizations/heatmap.html), so the iframe requests 404'd.

Fix: introduced a viz_subdir (/visualizations/visualizations inside the container, mapped to webdav/visualizations/ on the host) and pointed the heatmap/time-series writers at it, while dashboard.html and index.html stay at the viz_dir root — matching the URL paths the HTML already used. No docker-compose.yml changes were needed.

Bug 3 — emoji/symbols rendering as garbled text (mojibake)

Symptom: ⚡ displayed as âš¡ in the browser.

Root cause: the generated HTML had no <meta charset="UTF-8"> tag, and the file was written with open(path, 'w') — using the container's default locale encoding rather than an explicit one. The browser guessed the wrong encoding (typically Latin-1) when decoding the UTF-8 emoji bytes.

Fix: added <meta charset="UTF-8"> to the generated dashboard's <head>, and wrote the file explicitly as UTF-8: open(path, 'w', encoding='utf-8'). Optional hardening also recommended: charset utf-8; inside Nginx's server {} block, to force the correct Content-Type response header.

Verifying the fixes

# 1. Clear old/possibly-corrupted state
docker-compose down -v
rm -rf data webdav/*.html webdav/visualizations

# 2. Rebuild with the fixed function_app.py
docker-compose up -d --build

# 3. Run the pipeline once
docker-compose exec -e RUN_ONCE=true etl python function_app.py

# 4. Open the dashboard
open http://localhost:8081

Expected result, confirmed: stats card shows the correct min/current price (including any 0.00 values), both charts render inline, and all emoji/symbols display correctly.

Quick reference

Full story: project overview. Compare against the Azure build and its technical deep-dive.