Workflow: from render to full-data run

A port moves through a fixed sequence of gates. Each automated gate must pass before the next starts, with human reviews before the demo and before full-data execution. The generated repository's docs/IMPLEMENTATION_GUIDE.md is the authority for the ordered gates and their evidence; this page explains what each gate is for.

flowchart TD
    r[1. Human: render and supply sources] --> i[2. Agent: implement commands,<br/>tests/e2e.py, model.yaml, slurm/config.sh]
    i --> g1[3. Agent: local gate<br/>uv run pytest -rs]
    g1 --> g2[4. Agent: MEDS-DEV gate<br/>uv run pytest -m meds_dev -rs]
    g2 --> h1{{5. Pre-demo human review<br/>source fidelity, predicates, demo plan}}
    h1 --> g3[6. MIMIC-IV demo gate<br/>uv run pytest -m real_data -rs]
    g3 --> h2{{7. Demo human review<br/>target dataset, task bundle, resources}}
    h2 --> c[8. Commit and push<br/>clean tree, immutable SHA]
    c --> f[9. Full-data run via SLURM<br/>shared reviewed task bundle]
    f --> rep[10. Final report +<br/>scripts/github-sync.sh RUN_ID]

1. Render

uv tool install copier
copier copy --trust https://github.com/florian6973/MEDS_algorithm_template ../my-meds-algorithm

Copier asks for the algorithm name and slug, the profiles, whether MEDS-DEV commands should pass predicates, and the implementation source (paper and/or repository). The answers are recorded in .copier-answers.yml, so copier update can bring later template changes in.

The generated commands are deliberately stubs: STUB_PROFILES = set(PROFILES) in src/<slug>/<slug>.py makes the end-to-end tests skip with an explicit algorithm_stub reason until the real algorithm runs.

2. Implement

Using an agent for automated implementation

Open a coding agent, such as Claude, in the generated algorithm repository and give it the copyable algorithm-implementation prompt below. Replace every placeholder with the source paper or repository, generated repository path, target dataset and revision, and reviewed task-bundle location (or explicitly state that the bundle is not yet available). Supply relevant source configurations and checkpoint references as well. Select the repository's profiles when rendering; a single implementation can support several profiles through their separate algorithms/<profile>/model.yaml files.

The template supplies instructions, interfaces and executable checks. It does not launch an agent or run an unattended implementation service. The prompt works as ordinary agent input and includes the Ponytail implementation principles; installing a separate agent plugin is not required.

Stage	Agent's work	Human's responsibility
Handoff	Read AGENTS.md, SPEC.md, README.md and docs/IMPLEMENTATION_GUIDE.md; inspect the paper, source code and configurations; identify missing evidence.	Supply the intended source, profiles and dataset context; resolve scientific choices the source leaves ambiguous.
Implementation	Implement the algorithm and every declared profile's commands, descriptors, fixture adapter and primary-behavior tests. Reuse meds-algorithm-core for shared contract behavior. Record source fidelity and deviations in IMPLEMENTATION_REPORT.md.	Resolve missing inputs or consequential departures from the source when the agent raises them.
Automated verification	Run the local and MEDS-DEV gates, inspect failures, fix their causes and rerun the affected checks. Report exact commands, results, skips and remaining limitations.	Review the evidence, source fidelity, predicates and demo plan at the pre-demo gate.
Approved demo	After approval, run the MIMIC-IV demo gate and report execution, coverage and resource evidence.	Review the demo results and approve the full dataset, task bundle, access and resource plan separately.
Approved full run	Commit and push the completed implementation, run the approved SLURM job from its immutable commit, and prepare the final report and permitted structured run evidence.	Assess the scientific results and review the implementation and run evidence before merging.

Within the implementation and verification stages, the agent should continue the edit–test–inspect cycle without asking for approval for each routine change. It must surface missing source material, credentials or task bundles as blockers, and wait at the two human review gates before the corresponding data run. If work spans several sessions, the next agent reads the repository's implementation report and recorded evidence, identifies the last completed gate, and resumes there. Approval must be explicit; a passing test or an agent-written report does not grant it.

Tests must exercise the real algorithm behavior, including negative controls. The agent must not make a gate pass by weakening checks, substituting fixture-only behavior, or hiding an unfinished profile with a stub skip. Passing automated checks establishes the properties listed below; it does not independently establish fidelity to the paper or scientific validity. Keep the detailed instructions in the shared prompt and generated implementation guide rather than maintaining a separate prompt for each agent.

Typed implementation and audit trail

The generated algorithm owns its named class and frozen configuration (for example, RETAIN and RETAINConfig in the retain package). Profile ABCs require its methods; typed request objects carry artifact references and permitted inputs. The shared runner owns validation, measurement, staging, metadata and publication. Agents should implement cohesive algorithm components with explicit dependencies, without mutable global runtime state, unnecessary wrappers or copies of core utilities. Test config/request serialization and execution in a fresh process. See the shared architecture and UML for the implemented design.

Implementation deliverables

Implement every command in the union of declared profiles, tests/e2e.py, algorithms/<profile>/model.yaml, and slurm/config.sh. In tests/e2e.py, run_e2e runs the real profile on a small fixture to which it applies meds_algorithm_core.conformance.signal.add_split_canaries, and returns the artifacts, the evaluated index and splits, the exact external inputs, deterministic fixture seed and the canaries; fitted_codes reports every code the algorithm fitted a population statistic for; prepare_meds_dev_run configures the MEDS-DEV run. Complete tests/test_primary_behavior.py, which fails on purpose until it shows the algorithm's claimed behavior on a known signal, with a negative control. Record every source element as ported, adapted or omitted in IMPLEMENTATION_REPORT.md.

What each gate proves

Gate	Proves	Does not prove
Local (pytest -rs)	the contract holds (command registry, manifests and config_digest, publication and rollback, measurements, key coverage, prediction schema), the real profile runs end to end on a synthetic fixture with no evaluated-split label read across a fresh full-lifecycle rerun (no label at all for zero-shot) and no statistic fitted on tuning or held-out subjects, and the primary behavior appears	anything about real data
MEDS-DEV (-m meds_dev)	the package installs in MEDS-DEV's isolated environment and the algorithms/<profile>/model.yaml command strings run	scale or real-data behavior
Pre-demo review (human)	the port is faithful to the source, predicates are interpreted correctly, the demo plan is sound	—
MIMIC-IV demo (-m real_data)	the pipeline survives real data: serialization, scale transitions, resources, key coverage on MEDS-DEV's mortality/in_icu/first_24h with MEDS-DEV's MIMIC-IV predicates	algorithm quality: the demo validates execution only (its held-out split is tiny and may have no positives)
Demo review (human)	the evidence is sufficient; target-dataset, task and predicate compatibility; access and resources for the full run	—
Full data (SLURM)	the scientific result on the shared task bundle	—

A plain pytest run never clones MEDS-DEV or downloads data: the meds_dev and real_data tiers are opt-in markers. Set MEDS_DEV_DIR to reuse a MEDS-DEV checkout and MEDS_DEMO_DIR to reuse a built demo.

The full-data run

• The cluster runs an immutable commit from a clean checkout: slurm/submit.sh refuses to submit from a dirty tree.
• The tasks come from a shared, reviewed task bundle for the target dataset, built once upstream: clinical tasks defined and extracted with ACES, or randomly sampled code-by-horizon tasks (generated with meds-random-task-sampler and selected with meds-task-selection), containing each task's definition and its per-task index and labels. Zero-shot runs additionally need each task's ACES criteria. Every algorithm compared on a dataset uses the same bundle, and the bundle stays outside Git.
• Record the dataset name and revision, predicate provenance, bundle location, the commit of the tool that built the bundle, a digest of its task definitions, algorithm commit, configuration and job IDs.

Reporting results

scripts/github-sync.sh <run-id> pushes only the run's structured evidence (spec.yaml, result.yaml and the command manifests) on a runs/<algorithm>/<run-id> branch and opens a draft PR. Predictions, labels, data, checkpoints, raw logs and metrics are never pushed; metrics come from MEDS-DEV's evaluation step. See Running a algorithm for the exact list.

Copyable algorithm-implementation prompt

Use this after rendering a repository. Replace all placeholders before sending it to a coding agent. The full-data evaluation may target any suitable MEDS dataset (for example, MIMIC-IV or NWICU); the provided real_data test remains the MIMIC-IV demo smoke test.

The prompt adapts Ponytail's implementation principles to scientific algorithm development. It works with Claude and other coding agents without installing a plugin.

Based on the following source:

<PAPER_PATH_OR_URL_AND_OR_REPOSITORY_PATH_OR_URL>

implement the algorithm in the generated MEDS algorithm repository at:

<GENERATED_REPOSITORY_PATH>

For the full-data run, target this dataset in MEDS format (name, revision, and location):

<TARGET_DATASET_NAME_REVISION_AND_MEDS_PATH>

Use this dataset-specific shared task bundle if already available:

<FULL_DATA_WORK_DIR_OR_NOT_YET_AVAILABLE>

Read AGENTS.md, SPEC.md, README.md, and docs/IMPLEMENTATION_GUIDE.md in the generated repository
completely before writing code. Inspect the paper, supplementary material, source repository,
configuration files, checkpoints, preprocessing code, and evaluation code available from the supplied
source.

Apply the Ponytail principle of the smallest complete change: implement the complete source-defined
algorithm with the least additional infrastructure. Check meds-algorithm-core, existing algorithm-family code,
standard libraries and installed dependencies before creating helpers. Keep shared fixes in their
owning package; do not copy core utilities into the algorithm repository. Avoid speculative wrappers,
configuration options and compatibility layers. Trace affected commands, callers, fixtures and
documentation, and finish all required changes. Simplicity must preserve scientific fidelity, temporal
and label isolation, artifact validation, reproducibility, required evaluation evidence and human
review gates. Write focused tests for new behavior and failure cases. Report deviations and checks
that were not run.

Write minimalist, auditable, object-oriented algorithm code. Put source-defined hyperparameters and defaults
in the algorithm-owned frozen <AlgorithmClass>Config class in config.py; validate values and prove
serialization round trips. Keep the implementation in <algorithm_slug>.py and expose the named algorithm
and configuration classes from the package, so callers can import them meaningfully.
Implement the selected Algorithm profile methods with their typed request classes. Prefer small cohesive
components and explicit constructor dependencies; use inheritance for the profile contract and composition
for algorithm internals. Do not add unnecessary classes, wrappers, mutable module-level runtime state, hidden
singletons, import-time work, broad exception swallowing, or duplicated contract tables. Declarative
constants and the profile-to-implementation association are appropriate; fitted state belongs in artifacts.
Use core's runner for both CLI execution and tests. Algorithm methods write only to its staging destination
and return PayloadMetadata; do not reimplement publication, measurements or manifest/config/request writing.
Implement explicit payload compatibility checks. Prediction outputs contain keys and probabilities;
only the runner attaches evaluation truth after scoring. Demonstrate that saved configuration and requests
can be reloaded in a fresh process and executed without an earlier call on the same algorithm instance.

Preserve the selected algorithm profile and standardized command arguments, MEDS input boundary, artifact
manifest envelopes, MEDS-DEV model.yaml interface, and final MEDS prediction output. Intermediate payload
formats and execution mechanisms are algorithm-owned: choose whatever preprocessing, featurization, framework,
batching or non-batching strategy, external executable, storage format, and algorithm architecture most
faithfully reproduce the source.

When predicates are needed, reuse MEDS-DEV predicates for the selected dataset if available; otherwise,
use reviewed external predicates for that dataset and record their provenance. Decide within the algorithm
how those predicates become inputs. Validate dataset-specific codes, units, task definitions, and label mappings;
do not assume MIMIC mappings apply to another dataset. Keep such mappings in dataset-specific
configuration or algorithm-owned code. Do not introduce a generic predicate-derived feature representation
into the contract package. Keep the provided MIMIC-IV demo code, task, and MEDS-DEV predicates as the
canonical smoke test, regardless of the full-data target.

For a zero_shot_direct algorithm (answers queries directly, e.g. MOTOR, EveryQuery) or a zero_shot_materialized
algorithm (generates futures and evaluates the task on them; infer is task-independent and only predict
receives the task): the algorithm receives the original ACES task through
external_task_criteria_file (with the dataset predicates) and the upstream-built index through
external_index_dir. Never build, filter, or extend the cohort, and read the index only through
meds_algorithm_core.read_index. Reuse an existing aces-to-<algorithm> adapter for this algorithm if one exists;
otherwise write one on aces-relaxations, pinned to an exact version: reject tasks that are not future-only
or are degenerate, normalize, then apply only the approximations and assumptions the algorithm's query
language forces. Publish relaxations.yaml with every step, and report per task whether the answer is exact.
A materialized algorithm evaluates the task on generated futures instead and needs no relaxation beyond
sampling. Follow the shape for the selected profile in docs/IMPLEMENTATION_GUIDE.md.
Do not omit source behavior because it is inconvenient for a fixture; adapt the implementation or fixture
instead.

Record every material source element as ported, adapted, or omitted, with evidence and justification, in
IMPLEMENTATION_REPORT.md. Implement every selected typed algorithm method, finalize the implementation-specific
command strings in model.yaml, and implement tests/e2e.py. Keep STUB_PROFILES = set(PROFILES) until the actual selected
algorithm profile completes and produces standards-conformant predictions.

Then run:

    uv run pytest -rs
    uv run pytest -m meds_dev -rs

Complete the pre-demo sections of IMPLEMENTATION_REPORT.md and ask for human review of source fidelity,
predicate interpretation, and the proposed MIMIC-IV demo execution. Only after approval, run:

    uv run pytest -m real_data -rs

There must be zero algorithm_stub skips. Report exact pass/skip counts and account for every remaining skip.
Update IMPLEMENTATION_REPORT.md with the demo cohort, final-output coverage, predicate provenance,
environment, wall time, peak process-tree PSS/RSS, GPU measurements when available, failures, and fixes.
Ask for a second human review before launching the selected full dataset through the repository's SLURM
interface. Review target-dataset compatibility, access, and resources separately from the MIMIC-IV demo.
Use a shared, reviewed task bundle for the target dataset, built once upstream: clinical tasks defined in
ACES and extracted with the ACES package, or randomly sampled code-by-horizon tasks (generated with
https://github.com/florian6973/meds-random-task-sampler and selected with https://github.com/florian6973/meds-task-selection).
It must contain each task's definition and its generated index/labels; zero-shot algorithms also need each
task's ACES criteria. If a compatible bundle is not yet available, report that blocker before the
full-data run. Record the dataset name/revision, predicate provenance, bundle location, the commit of the
tool that built it, and a digest of its task definitions. Keep the bundle external to Git. Commit and push the implementation,
configuration, and report-so-far; run the cluster job from that exact clean commit. After the authorized
run, complete the report with full-data evidence, scientific deviations, and known limitations, then
synchronize structured results through the GitHub workflow. Treat
docs/IMPLEMENTATION_GUIDE.md as the authority for the ordered gates and their evidence.