Workplans ========= Workplans define the contract for requesting the execution of one or more :doc:`blueprints`. A user-configured workplan informs C-Star which ``Blueprint`` to execute, and in what order. Workplan Schema --------------- Workplans are defined in :class:`cstar.orchestration.models.Workplan`. .. rubric:: Workplan Attributes .. autosummary:: ~cstar.orchestration.models.Workplan.name ~cstar.orchestration.models.Workplan.description ~cstar.orchestration.models.Workplan.steps ~cstar.orchestration.models.Workplan.state ~cstar.orchestration.models.Workplan.compute_environment ~cstar.orchestration.models.Workplan.runtime_vars State ^^^^^ .. include:: snippets/in-development.rst A workplan may be configured in a *draft* or *validated* state using :attr:`~cstar.orchestration.models.Workplan.state` - a *draft* workplan can be freely edited and submitted for execution. - modifications to a *validated* workplan are restricted to ensure reproducibility. Compute Environment ^^^^^^^^^^^^^^^^^^^ .. include:: snippets/in-development.rst The desired compute environment characteristic are specified using :attr:`~cstar.orchestration.models.Workplan.compute_environment` Runtime Variables ^^^^^^^^^^^^^^^^^ .. include:: snippets/in-development.rst C-Star performs some simple templating for user convenience. The variables identified in :attr:`~cstar.orchestration.models.Workplan.runtime_vars` are meant to be provided at runtime, such that one workplan could be shared and used for a range of supported values without needing to modify the yaml. The user API for specifying runtime variables is still in development. Steps ^^^^^ The heart of a workplan is the collection of steps found in :attr:`~cstar.orchestration.models.Workplan.steps`. Steps have a 1-to-1 relationship with blueprints - each step must specify the path to a blueprint file. The step also specifies the application type to use for its execution. Step Schema ----------- See :class:`cstar.orchestration.models.Step` for complete details on configuring steps. .. rubric:: Step Attributes .. autosummary:: ~cstar.orchestration.models.Step.name ~cstar.orchestration.models.Step.application ~cstar.orchestration.models.Step.blueprint ~cstar.orchestration.models.Step.depends_on ~cstar.orchestration.models.Step.blueprint_overrides ~cstar.orchestration.models.Step.compute_overrides ~cstar.orchestration.models.Step.workflow_overrides .. _workplan_examples: Workplan Examples ----------------- .. tab-set:: .. tab-item:: Single-step The following example demonstrates the minimum possible workplan. It contains a single step to be executed. .. code:: yaml name: Simple Workplan description: Run a simulation state: draft steps: - name: job1 application: roms_marbl blueprint: /home/x-seilerman/wp_testing/2node_1wk_new_a.yaml .. tab-item:: Multi-step The following example demonstrates a workplan with multiple steps. Note that each step can reference different blueprints. Additionally, this example introduces a simple dependency with ``depends_on: job1``. A dependency must be specified if the steps of the workplan require specific ordering. Here, *job1* must complete successfully before *job2* will start. .. important:: A multi-step workplan without dependencies has no ordering guarantees. Jobs are scheduled immediately and executed as the system launcher permits. .. code:: yaml name: Multi-step Workplan Example description: Run multiple ROMS-MARBL simulations state: draft steps: - name: job1 application: roms_marbl blueprint: /home/x-seilerman/wp_testing/2node_1wk_new_a.yaml - name: job2 application: roms_marbl blueprint: /home/x-seilerman/wp_testing/2node_1wk_new_b.yaml depends_on: - job1 - name: job3 application: roms_marbl blueprint: /home/x-seilerman/wp_testing/2node_1wk_new_c.yaml .. tab-item:: Overriding Blueprints The following example demonstrates how to override configuration in a blueprint from the workplan. Overriding blueprints enables the same blueprint to be used with different inputs, data sources, etc. .. tip:: Blueprint overrides are supplied as a dictionary with :ref:`Blueprint schema` .. code:: yaml name: Workplan Overriding a Blueprint description: Run a blueprint ensemble varying parameters with overrides state: draft steps: - name: step 1 application: roms_marbl blueprint: blueprint.yaml blueprint_overrides: - name: Run blueprint with development UCLA-ROMS branch - code: roms: location: https://github.com/CWorthy-ocean/ucla-roms.git branch: develop - name: step 2 application: roms_marbl blueprint: blueprint.yaml blueprint_overrides: - name: Run blueprint with custom UCLA-ROMS fork - code: roms: location: https://github.com/github-user/ucla-roms.git branch: main Checking validity ----------------- Workplans can be checked for errors using the CLI and in code. .. tab-set:: .. tab-item:: Validating via CLI Use the ``check`` command from the ``cstar`` CLI. .. code-block:: console cstar workplan check my_workplan.yaml .. tab-item:: Programmatic Validation Use the ``deserialize`` method to validate a YAML file in Python. .. code-block:: python from cstar.orchestration.models import Workplan from cstar.orchestration.serialization import deserialize deserialize("my_workplan.yaml", Workplan) Execution --------- .. include:: snippets/review-config.rst .. attention:: An error will occur if the SLURM **account** and **queue** are not configured when running on a HPC. .. tab-set:: .. tab-item:: Run via CLI Use the ``run`` command from the ``cstar`` CLI to execute the workplan. .. code-block:: console cstar workplan run --run-id my_workplan.yaml .. tip:: Executing the command again with the same :term:`run ID` will attach to the previous execution and continue to monitor status until terminated. Specify a different :term:`run ID` to re-run the workplan from scratch. .. tab-item:: Programmatic Execution Use the ``dag_runner`` module to execute a Workplan in python. .. code-block:: python from pathlib import Path from cstar.orchestration.dag_runner import build_and_run_dag path = Path("/path/to/my/workplan.yaml") await run_workplan(path, run_id="my-unique-id") Checking Workplan Status ------------------------ .. tab-set:: .. tab-item:: CLI Status Check Use the ``status`` command from the ``cstar`` CLI to retrieve the current status of steps in a running workplan. .. code-block:: console cstar workplan status --run-id