How Duckietown.jl was built — what, why, and how

This document explains how this package was constructed: what each part is, why it exists, and how it was validated.

The provenance rule. Every factual claim in this document traces to one of five sources, and says which:

1. a gate document in docs/src/validation/FJ*_STATUS.md (the validation record),
2. a test in test/ (executable, re-runnable),
3. an artifact in artifacts/ (recorded evidence),
4. a script + recording in notebooks/ (the DORA case study), or
5. the file's own header comment (each source file states its scope and the reference code it ports — the per-file tables below quote those headers verbatim).

Anything not traceable to one of these is labelled a design decision, with its reason stated.

1 · What this is, and why a port instead of a wrapper

The package is a native Julia reimplementation of a Python Duckietown MDP — a small autonomous-driving decision problem (lane following, a crossing duck, a stop sign) — exposed through the POMDPs.jl interface so that the Julia planning ecosystem can be run against it.

Why a port rather than calling the Python simulator from Julia:

• Planners need branchable states. Tree search calls the transition many times from the same state with different actions. The Python simulator mutates a single global environment; the Julia transition is branch-pure — simulate_decision never mutates its input, so a planner may branch freely (stated and enforced in src/generative/transition.jl, validated in FJ8.0's generative-model contract, src/evaluation/benchmark.jl).
• Speed. The native backend exists to be "branchable, fast" (src/backends/native_julia.jl header); planner budget studies (artifacts/fj8/budget_study.md) are measured in generative calls, which a cross-language bridge would dominate with overhead.
• The Python code stays the ground truth. The port never replaces the reference — it is validated against it, live, at three levels (section 4).

The reference implementation is the sibling duckduck repository (supplementary material of the study this benchmark reproduces). Its trained checkpoints (policy.npy, policy.pt) are read-only frozen inputs: they are loaded, never rewritten (rule stated in src/solvers/adapters.jl header; enforcement: the FJ9.9c artifact ledger classifies them as provisioned frozen inputs, src/interfaces/reproducibility.jl).

2 · The method: validation gates

The port was built as a sequence of FJ gates. A gate is a bounded claim ("the ego dynamics match", "a full episode matches", "the trained policies match") with:

• a status document (docs/src/validation/FJ<N>_STATUS.md) recording what was measured, what deviated, and what was left undone;
• tests that re-run the measurement (test/test_fj<N>_*.jl);
• where applicable, frozen artifacts of the evidence (artifacts/).

A gate that fails is fixed or its failure is documented — it is never skipped silently. The gate sequence (from the README's build table, each entry linked to its document):

Standing rules the gates produced

Each of these exists because breaking it once cost real debugging time; each is enforced in code, not by convention:

• A changed world gets a new name, never an edit. default_config means "the Python source defaults", permanently. Corrected or extended worlds are new named scenarios. Enforced by the testset "named scenarios are new configs, not edits to the defaults" (test/test_configs.jl), which pins each scenario to exactly its documented deltas.
• The core must load with nothing but Julia. using Duckietown may not require Python, a solver, or a plotting library; solver and plotting code lives in package extensions (ext/, see section 6). Enforced by test/test_fj8_solver_independence.jl and the source-import audit in src/interfaces/reproducibility.jl, which bans solver tokens from src/.
• Artifacts are classed, and the class decides what a rebuild may do (ArtifactStatus in src/interfaces/reproducibility.jl): rebuilt artifacts are regenerated and compared; persisted-source artifacts are recorded evidence that can only be checked, never regenerated; provisioned frozen inputs (the trained checkpoints) are never written.
• Stale claims are banned by string. The documentation audit (src/interfaces/reproducibility.jl, run by test/test_fj99_closure.jl) scans the repository for claims that were once true and later falsified — for example, the claim that the car can never encounter the stop sign, which stopped being true when the scenario layer landed — so a fixed bug cannot survive as prose.
• Never judge a suite from truncated output. The suite writes a structured report (artifacts/fj9/test_report.json) built from the Test result tree; two independent summarizers (tools/suite_summary.sh, tools/suite_summary.py) must agree with it. This rule exists because the reporter's very first run disagreed with the summarizers — the reporter had a counting bug (finished testsets discard Pass objects and keep only n_passed; test/reporter.jl now reads counts at every level). The README records this incident.
• missing stays missing all the way to the renderer — never coerced to zero (FJ9 visualization contract, src/visualization/scene.jl).

3 · The source tree, file by file

Descriptions below are the files' own header lines (quoted or lightly compressed), so this table cannot drift from the code without the code's headers drifting too.

src/dynamics/ — the world and its physics (FJ3)

src/rng/ — exact randomness (FJ3.8)

Why this exists at all: episode-level parity (FJ6) is impossible unless both sides consume identical random streams; approximating the RNG would turn every parity mismatch into an unanswerable question.

src/model/ — what the agent sees (FJ2, FJ3.6)

src/reward/ — how behaviour is scored (FJ2)

src/generative/ — the transition (FJ3.7)

src/backends/ — the reference, live (FJ5)

src/solvers/ — the frozen baselines (FJ7)

Why native inference: running a learned policy must not require Python (the dependency rule of section 2). The weight exports live at artifacts/fj9/weights/ (produced by tools/export_actor_weights.jl).

src/evaluation/ — measurement without solver bias (FJ5–FJ8)

src/interfaces/ — the contracts

src/visualization/ — figures that cannot lie quietly (FJ9)

src/config/

ext/ — optional capability, isolated

DuckietownMCTSExt.jl, DuckietownMakieExt.jl, DuckietownPythonCallExt.jl — package extensions (Project.toml [extensions]) that activate only when the user loads MCTS, Makie, or PythonCall. Why: the dependency rule of section 2; the core package must never pull a solver, a plotting stack, or Python into a plain using.

4 · How "valid" is established: the three-layer parity architecture

1. Fixture parity (FJ2/FJ3). Pure functions are compared against JSON fixtures generated from the Python source (test/fixtures/fj2_parity.json, fj3_*.json, fj37_transition.json, fj38_rng.json).
2. Live one-step parity (FJ5). A running Python reference is driven to a state, the state is imported/exported through world_to_ref, both sides take the same action, successors are compared (src/evaluation/parity.jl, tools/parity/reference_server.py).
3. Free-running episode parity (FJ6). Both sides run whole episodes from the same seed and the trajectories are compared step by step (artifacts/fj6/rollout_*.csv, drift summarised in artifacts/fj6/drift_summary.json).

The suite runs all of it when the reference environment is present, and degrades honestly when it is not: test/reference_guard.jl skips the 26 reference-dependent test files by name and says so (README, section "Testing"). Measured counts (README): 201 test sets / 148 855 assertions with the reference, 17 / 167 without. The authoritative count is always artifacts/fj9/test_report.json.

5 · The scenario layer: making the task reachable without touching the defaults

default_config reproduces the Python defaults — including their inert switches: the duck crosses with probability 0.02 and no stop sign is injected, so the stop-and-duck machinery is effectively unreachable in casual use. Rather than edit the defaults (banned, section 2), named scenarios were added (scenario_config in src/config/yaml_loader.jl):

• :lane_following — the defaults, unchanged (test-pinned field by field).
• :stop_and_duck — the switches on (p_cross = 1.0, stop sign injected), plus spawn-sanity bounds; scoring untouched (test-pinned: same(sc.reward, d.reward)).
• :stop_and_duck_safe — the same world with the safety shaping switched on and the stop sign made reachable. Both changes are measured, not aesthetic:
  • With the source-default weights, the pedestrian term is zero whether the car brakes for a crossing duck or drives through it — and braking scores strictly worse (measured in notebooks/audit_reward.jl before the scenario existed; the comment block in yaml_loader.jl records the numbers). The weights used are the project's own SAC/TD3 values.
  • The source places the sign at rotate 180° "facing vehicles travelling east" — but the route travels west there, so traffic sees the back of the board and the detector rejects it on every frame; even turned around, the corner placement leaves the stop line geometrically unreachable (detection dies at d_stop = 0.395 m), and a mid-straight placement gives less than one decision of detection runway. All three facts were measured (notebooks/audit_sign_orientation.jl, notebooks/audit_stop_geometry.jl) before the final placement was chosen. :stop_and_duck keeps the source's own placement, warts included.
  • The testset in test/test_configs.jl pins the safe scenario to exactly these deltas — four reward weights and the sign pose — and nothing else.

6 · The DORA case study: what the finished model is for

With the model validated, an external solver (DORASolvers.jl, an online tabular-SSP solver) was brought to it. Every design step is a measured fact with its script in notebooks/:

Two Pluto notebooks make this runnable by anyone: notebooks/DORA_on_Duckietown.jl (the formulation, executed cell by cell, plus tile-by-tile replays) and notebooks/Playground.jl (pick a scenario × a driver: random, MCTS, DORA, or the four frozen baselines).

7 · How to verify any claim in this document

# the whole validation suite, complete log, real exit code
bash tools/run_full_suite.sh

# the authoritative result
cat artifacts/fj9/test_report.json

# any single gate
julia --project=. -e 'using Pkg; Pkg.test()'   # honest subset without the reference

# the case study, live
julia -e 'using Pluto; Pluto.run()'            # open notebooks/DORA_on_Duckietown.jl

Known limitations are not in this document by design — they are recorded where the audit checks them: artifacts/fj9/reproducibility_manifest.json and the README's Known limitations section.