Skip to content

Dashboard end-to-end operator workflow

This procedure configures project policy, selects an execution backend, discovers sources, composes a run, and follows durable evidence through Dash. The WALLABY local DALiuGE qualification checks the graph and publisher against direct loopback NM, DIM, and TM services; it is intentionally outside this Core/Dash workflow.

1. Prove system readiness

Open System and confirm PostgreSQL, queue, configured provider checks, and worker status match the intended environment. DALiuGE and Slurm tiles show whether a backend is configured or profile-managed; they do not prove network connectivity. Use Deployment target > Test profile for the selected profile and resolve critical diagnostics before creating external intent.

Equivalent CLI checks are:

beampipe doctor
beampipe status
beampipe worker list
beampipe doctor --profile PROFILE_NAME

2. Save project policy

Open Overview > Project policy, then choose an existing project or New. The editor covers:

  • identity and required adapters;
  • TAP timeout/retry policy, queries, and enrichments;
  • metadata mappings, transforms, flags, and signature fields;
  • graph source, manifest templates, and graph patches;
  • discovery, execution, and output-verification policy (in progress).

The YAML pane is canonical and synchronizes with the visual editor. Select Save version. Success creates and activates an immutable Core revision; contract errors remain visible and prevent a valid workflow from being assumed. Existing executions keep their pinned revision.

3. Configure the deployment target

Open Deployment target. For REST DIM, remember that TM and workers may need different DIM addresses. For example, a project-neutral local profile can use:

name                 local-rest
project              minimal_survey
default              yes
translator URL       http://translator.example.test
deploy host/port     manager.example.test / 8001
DIM host/port for TM manager.internal / 8001

When Core runs in Docker, enter names reachable from Core and TM containers, not addresses that work only in the browser. Save and select Test. Both the translator and manager paths must pass before execution.

For Slurm, configure the login node, account, absolute paths, resources, manager topology, modules, environment, and runtime inputs. The Test action checks SSH/Slurm connectivity and renders the effective request. Keys and passphrases remain external Core credential slots and never reach Dash.

4. Register and discover sources

Open Source registry and select Register:

  1. Choose the project.
  2. Enter one source identifier per line.
  3. Select Register sources.
  4. Select one or more same-project rows.
  5. Select Discover selected and confirm.

Discovery is asynchronous. Follow scheduler_tick and discover_batch in Jobs. Open a source to inspect readiness gates, metadata by group, current and last-executed discovery signatures, linked executions, provenance, enabled state, and stale-after policy.

A source can be composed manually when Admission is ready. When project execution automation is enabled, changed discovery also marks it pending for recurring admission.

5. Compose and start

Select same-project sources and choose Compose run, or open Runs > Compose run:

  1. Confirm project, archive label, and deployment-profile revision.
  2. Select sources and optional groups.
  3. Select Validate selection.
  4. Review source, group, and record counts.
  5. Resolve every blocker.
  6. For a full live run, keep Start immediately and Submit backend on.
  7. Keep Stage inputs on for a production staging graph; turn it off only for an explicitly no-download qualification graph. This does not disable that project's output-verification policy (in progress).
  8. Select Create + start.

Preparation calls Core's authoritative readiness endpoint. Changing a source, group, archive label, or profile invalidates the preview. Creation pins the active project revision and profile snapshot.

If creation returns an ambiguous network failure, Dash preserves the same idempotency key and freezes the request. Resume that exact intent; do not edit it into a different run or assume no ledger row was created.

6. Interpret the run explorer

  1. 01createdproject and profile revisions pinned
  2. 02runningleased job, events, backend identity
  3. 03reconciledbackend observations and output policy resolved
  4. 04terminaloutcome locked against regression
manifestsha256 + size source graphsha256 + size patched graphsha256 + size physical graphsha256 + size timelineevents + observations ledgersnapshot + run record
Tab Evidence
Overview Control, submission, scheduler, DALiuGE, output, terminal state; inputs and timing
Timeline Provenance events plus normalized and raw DIM/Slurm observations
Artifacts Manifest and graph kinds, hashes, sizes, and downloads
Manifest + graph Structured data exploration and EAGLE links
Ledger Compact snapshot plus run record, staging, backend merges, scheduler metadata, and timestamps

The bundled WALLABY no-download project is not an output-verification opt-out (the feature is in progress). It requires the two synthetic image and weights patterns, and its graph contains a mandatory native publisher. Core currently reconciles that handoff only for slurm_remote, by pulling the attempt-scoped inventory through authenticated SSH/SFTP after Slurm completes. A rest_remote run with required publication fails closed because Core has no trusted handoff retrieval path; DALiuGE finished alone must not become terminal success.

The direct local DALiuGE qualification proves the graph-side receipt but creates no Core execution or dashboard ledger evidence. Output not required is valid only for a different pinned project whose policy explicitly sets output_verification.required: false.

Terminal runs stop automatic detail polling. Use manual refresh when you need to re-read evidence after an external operator action.

7. Recover cautiously

For REST, expect TM translation, DIM deployment, and dim_poll. For Slurm, expect remote staging, sbatch, awaiting_scheduler, and batched reconciliation.

Use Retry only after reading terminal evidence and entering an operator rationale. Core derives the safe recovery phase from durable state. Cancel contacts the pinned backend first and records only a confirmed outcome. Never retry an uncertain submission while an external DIM session or Slurm job may exist; follow Recovery and cancellation.