Technical Deep Dive: Bee Haven SeaweedFS Pipeline

From a "just a tangent" Docker test to a genuine parallel medallion build, with both bugs fixed in full

Setup log

Project deliberately kept as its own sibling folder (seaweedfs-playground/), not nested inside the Azure project, with its own Python venv (sw, separate from bee).

docker run -d --name seaweedfs \
  -p 8333:8333 -p 9333:9333 -p 8080:8080 \
  chrislusf/seaweedfs server -s3
docker ps

First round-trip test — create bucket, upload, list, read back — worked on the first run:

✅ Created bucket: playground
✅ Uploaded hello.txt
📁 Bucket contents:
   hello.txt (21 bytes)
✅ Read back content: 'Hello from SeaweedFS!'

Prefix-based folder behavior confirmed next, since SeaweedFS has real filer/directory semantics under the hood rather than just flat-namespace prefix tricks:

s3.put_object(Bucket="playground", Key="bronze/incoming/test.csv", Body=b"a,b,c\n1,2,3")
s3.put_object(Bucket="playground", Key="silver/test.csv", Body=b"a,b,c\n1,2,3")
response = s3.list_objects_v2(Bucket="playground", Prefix="bronze/")
for obj in response.get("Contents", []):
    print(obj["Key"])
# → bronze/incoming/test.csv (only — silver/test.csv correctly excluded)

Real infra bug: no volume mount, lost data on container recreation

The original container had no -v flag, so SeaweedFS stored everything inside the container's own filesystem. The moment the container was recreated to add WebDAV support, the entire playground bucket — including hello.txt — was gone.

# Fix: add a named volume so data survives container recreation
docker run -d --name seaweedfs \
  -p 8333:8333 -p 9333:9333 -p 8080:8080 -p 8888:8888 \
  -v seaweedfs_data:/data \
  chrislusf/seaweedfs server -s3 -filer -webdav

Confirmed fixed: created a test file, ran docker stop seaweedfs && docker start seaweedfs, file was still there afterward via WebDAV at localhost:8888.

Architecture actually built

Two containers, mirroring the Azure design: beeblob (flat, mirrors the raw-landing account) and beelake (hierarchical). Final beelake structure:

beelake/
├── bronze/
│   ├── new/              # Raw CSVs
│   ├── archive/          # Archived originals with timestamps
│   └── weather/          # Raw weather JSON (backup)
├── silver/
│   ├── processing/       # STAGING - notebook output before final move
│   ├── flow/              # FINAL - cleaned flow data
│   ├── weather/           # FINAL - cleaned weather data
│   ├── humidity/          # FINAL
│   ├── temperature/       # FINAL
│   └── weight/            # FINAL
└── gold/                  # Future aggregations

Flow: local CSVs → beeblob → beelake/bronze/new → cleaned → staged in silver/processing/ → moved to per-type final folders on success, left in place for review on failure. Weather step reuses the already-staged sensor Parquet files to determine the real date range needed, fetches from Bright Sky in monthly chunks (~29 requests for the full 2.5-year range), applies the same cleaning as the Azure notebook (drop 100%-null forecast-only fields, forward-fill categoricals, interpolate numeric gaps), and stages/moves alongside the sensor data.

Bug 1 — flow departure/arrival sort order

# Before (bug):
df = df.sort_values(["timestamp_utc", "flow"])
# After (fix):
df = df.sort_values("timestamp_utc")

Sorting by ["timestamp_utc", "flow"] meant rows sharing a timestamp got ordered by their flow value, not their original file order — so "departure" ended up meaning "whichever reading has the lower number" instead of "whichever reading appeared first," silently changing the intended meaning (first occurrence = departure, second = arrival, based on original order). Fixed by sorting on timestamp_utc alone — pandas' stable sort preserves original relative order for ties.

Bug 2 — timezone fallback silently produced naive timestamps

# Before (bug) — fallback never actually localized to a timezone:
except Exception as e:
    df["timestamp_utc"] = pd.to_datetime(df["timestamp"])
    df = df.drop(columns=["timestamp"])

# After (fix):
except Exception as e:
    df["timestamp_utc"] = df["timestamp"].dt.tz_localize(
        "Europe/Berlin", ambiguous="NaT", nonexistent="shift_forward"
    ).dt.tz_convert("UTC")
    df = df.drop(columns=["timestamp"])

Symptom: humidity and weight ended up with plain, timezone-naive timestamp_utc columns, while flow and temperature correctly got timezone-aware ones — inconsistent across the four files. Only surfaced once the weather step tried to compute a combined min/max date range across all sensor files and hit Cannot compare tz-naive and tz-aware timestamps.

Root cause: the primary path uses ambiguous="infer", which needs several consecutive repeated timestamps to resolve a DST fall-back hour — works fine for high-frequency flow/temperature, fails for the twice-daily humidity/weight (not enough consecutive data to infer from), triggering the except branch. That fallback used plain pd.to_datetime() with no tz_localize() call at all — silently dropping timezone info entirely. Fixed with ambiguous="NaT" in the fallback, marking genuinely ambiguous hours as missing rather than guessing, while still producing a proper tz-aware UTC column either way.

Final verified result

• 4 sensor files cleaned, archived (2 versions, from testing before/after the timezone fix), and moved to final locations — all four now consistently timezone-aware
• Weather: 20,477 records across 29 monthly API chunks, cleaned, staged, and moved to silver/weather/weather.parquet
• Success-branch logic correctly moved everything only after all processing succeeded

Why this worked cleanly where the Azure version didn't

The real Azure pipeline hit a persistent AuthorizationPermissionMismatch even with correctly-assigned IAM roles verified directly in the portal — isolated to a specific integration seam between ADF's triggered execution context and ADLS Gen2 authentication, a widely-documented pain point, not a gap in understanding. Locally, there's no separate managed identity, no cross-service token handshake, no session-vs-trigger authentication context to diverge — just one script talking directly to storage. This local build isn't a replacement for the Azure work — it's proof the underlying architecture and logic are sound, demonstrated working end-to-end, even though the Azure deployment hit a platform-level limitation that stayed open.

Full story: project overview. Compare against the Azure build and its technical deep-dive.