Attention
This project is still in an early phase of development.
The python API is not yet stable, and some aspects of the schema for the blueprint and workplan will likely evolve. Therefore whilst you are welcome to try out using the package, we cannot yet guarantee backwards compatibility. We expect to reach a more stable version in 2026.
Release notes#
v0.6.0#
Breaking Changes#
Drops support for [C]Worthy ucla-roms versions <0.2: removes support for running roms with roms.in and the classic suite of .opt files. Replaces that functionality with support for ROMS namelists. This and future versions of C-Star will only run blueprints that point to namelist-compatible versions of ROMS. (#556)
Removes DatasetLinker concept, which existed to support inflexible .opt conventions (#556)
New features#
Supports [C]Worthy ucla-roms version 0.2 (runtime namelists) (#556)
Bug Fixes#
N/A
Improvements#
N/A
Miscellaneous#
N/A
v0.5.0#
Breaking Changes#
Deprecated/removed CSTAR_ORCH_REQD_ENV environment variable
Modified parameters to DagStatus.__init__
Removed workplan-monitor CLI command
Removed dag_runner.attach_to_run (duplicates process_plan functionality)
The flag constant is removed (#428)
Modified the interpretation of Status.is_running to exclude Unsubmitted (#461)
Rename env var group name constants to omit leading underscore (#462)
The roms executable is now executed in the work subdirectory instead of output (#465)
The output location in the runtime settings (roms.in) file is now _always_ modified to place outputs in the output directory (since we no longer run ROMS in the output directory) (#465)
ServiceConfiguration, JobConfig have been relocated to cstar.entrypoint.config from cstar.entrypoint.worker.worker (#503)
BlueprintRequest deprecated in favor of XRunnerRequest (#506)
BlueprintRequest has been deprecated and removed. (#510)
–stage CLI parameter for cstar blueprint run is no longer supported (#510)
deep_merge is a pure method and no longer changes the inputs (#498)
The -d short dry-run argument of cstar workplan run is removed to avoid a conflict. (#515)
The output-dir argument of cstar workplan run now uses –o and –output for consistency (previously -d and –output-dir). (#515)
ROMS-specific blueprint, runner, and transforms have been moved to align with the demo applications (now contained in cstar.applications.roms_marbl) (#519)
The ROMS-specific SimulationRunner has been replaced by the use of the generic BlueprintRunner and can no longer be invoked with the same entrypoint (#519)
Core contracts for application building (e.g. Transform, XRunner, RunnerRequest, etc) have been relocated `cstar.applications.core (#519)
Deprecated the Registrar (#519)
Deprecated unused ComputingPlatform model (#519)
The WorkplanTransformer no longer receives transforms to the __init__ method (#523)
Use of {{path: <step-name>}} in templates is no longer supported. Use {{work_dir: <step-name>}} instead (#522)
The RomsMarblBlueprint model has been updated. Old blueprints may be upgraded by using cstar blueprint migrate <blueprint-path> (#521)
JobFileSystemManager attributes have been modified for clarity and naming consistency (work_dir->`run_dir`, root->`root_dir`, output_dir->`working_dir`). (#521)
Sample blueprints are updated to the latest schema version. (#521)
the name of a task is now the step name (not step.safe_name). using cstar workplan status on old outputs will result in a key error (#532)
The _monitor loop_ has been removed after scheduling a run. cstar workplan log … should be used in it’s place (#546)
StateRepository class now comprises functions from cstar.orchestration.state (#546)
Env var constants missing correct type annotations are ignored. (#554)
All system contexts have been renamed (#558)
New features#
Enable user supplied variables for templating from cstar workplan run …
The orchestrator writes a sentinel file to disk to dynamically alter dependencies (#459)
Add trace log level to reduce log frequency during Workplan execution (#460)
Persist WorkplanRun record for every invocation of the orchestrator for run history and lookup (#461)
Add cstar workplan monitor –run-id <run-id> for reattaching to a running Workplan (#461)
Add nesting_info block to RomsMarblBlueprint (#465)
Create work-dir symlinks to predetermined paths for CDR and nesting info files (#465)
If no Makefile is provided in the compile_time additional code block, ROMS will copy and use the appropriate Makefile from the ROMS repo (#465)
Add automated release-note updater, utilizing PR template categories (#484)
Locally-launched steps now generate a .out log file consistent with slurm-launched steps (#493)
Added capability to add custom blueprints and orchestrate within a Workplan (#506)
Add a basic Plotter application as a demonstration of building a functional application (#499)
Add template-filling transform for user variables and paths (#500)
.ext files are now passed through extract_data_join at the same time as other files are passed through ncjoin (#495)
Add Directives to enable just-in-time blueprint manipulation by the runner. (#498)
Enable upgrading blueprint files using the cstar blueprint migrate command (#515)
Added support for dynamic discovery of applications residing in subpackages of cstar.application (#519)
Add cstar workplan log <run-id> <step> command to easily view logs from a workplan step. (#525)
The OverrideTransform is now applied automatically after all registered transforms and does not need to be registered for the application. (#523)
Enable callback chaining for CLI arguments (#534)
Execution of cstar blueprint run with any out-of-date blueprint automatically triggers a schema migration (#535)
Add –clobber flag to cstar blueprint migrate … to allow overwriting prior migrations (#535)
cstar workplan generate … enables the user to specify a glob pattern or file extension (#536)
Add feature flag CSTAR_FF_ORCH_TRX_TIMESPLIT_LONGNAME to enable using minimal length names for subtasks (to counter ROMS failures on long paths) (#537)
Add schema references automatically during blueprint serialization (#521)
Add CLI tool for generating updated schemas from blueprint models (see: CSTAR_FF_DEVELOPER_MODE=1 cstar blueprint schemas –help) (#521)
Enable automatic migration boundary discovery (#521)
Add support for deserializing files to a dictionary without immediate model creation (#542)
Added ExecutiveRunSummary containing useful metadata about assets and processes created during a run (#546)
Added try_deserialize to cstar.orchestration to enable loop-based deserialization without exceptions (#546)
Added user-facing display of ExecutiveRunSummary after run-scheduling (#546)
LocalLauncher now honors dependencies and can exit after scheduling tasks. (#548)
Setting CSTAR_FF_DEBUG_BUILD_MODE=1 results in ROMS build in debug mode (#553)
Inject per-system settings class into CStarEnvironment for customized behaviors. (#555)
Add environment support for Elja HPC (#545)
Add new nest_ic application for creating initial conditions of a child domain based on a restart file of the parent domain (#528)
Add new upscaler application for generating CDR forcing based on .uscl outputs from an inner domain (#528)
Add nest_from directive for modifying blueprints to receive IC and boundary files from previous steps (#528)
Security Fixes#
N/A
Bug Fixes#
Fix bug in reload_dag_status where the correct launcher may not be loaded
Fix comparison-to-self bug for orchestrator open_set
Fix mutable default attribute value in WorkplanRun
Remove repeated check identifying remote workplan URI
Fix bug causing intermittent FileNotFound errors
Fix workplan template with inconsistent blueprint path
Fix bug causing failures when invoking blueprints via cstar blueprint run
Fix bug handling user-relative paths in JobFileSystemManager (#430)
Fix regression causing blueprint output directory overrides to not be applied (#434)
Fix failure to include run-id in output path directory tree (#434)
Fix defect where steps defined before dependencies were not mapped correctly after time-splitting (#434)
Mitigate failures in the CachedRemoteRepositoryRetriever.refresh by automatically re-cloning if a refresh fails (#438)
Fix posix-path conversion bug when passing blueprint URLs to cstar blueprint run (#440)
Catch FileNotFoundError in cstar.cli.blueprint.check (#443)
Catch FileNotFoundError in cstar.cli.workplan.check (#443)
Update tests to ignore default factory where appropriate (#444)
Fix a JSON serialization bug with pydantic models using field aliases (#451)
Fix case-sensitivity bug when configuring split frequency via CSTAR_ORCH_TRX_FREQ env variable (#452)
Fix incorrect initial-conditions file paths created in RomsMarblTimeSplitter (#453)
Fix incorrect working directories resulting from LiveStep.from_step when a parent is specified (#453)
Use step.fsm.output_dir as the source of reset files instead of step.fsm.joined_output_dir (#455)
Fix failure to terminate health check thread (#456)
Fix issue when parsing SLURM status of the form cancelled by Username (#459)
Fix cstar workplan status … triggering a re-build of the entire workplan (#461)
Fix incorrect env var discovery if env_ appears repeatedly in variable name (#462)
Fix unhandled exception caused by attempting to execute typer app (#462)
post_run() no longer tries to join _ext files (#466)
Pin version of tutorial data repo (#475)
Bump typer version to avoid list-default bug (#476)
Fix uninitialized attributes error in SlurmBatch
Mitigate a dependency issue with prefect and fakeredis (pin fakeredis<2.35)
Fix possibly incorrectly renaming a marbl input file to the _ORIGINAL roms.in file (#491)
Mitigate possible ROMS output-name-length errors by using a relative output path in the roms.in (#491)
Fix incorrect status mapping of local runs, which caused dependencies to not be cancelled if upstream tasks failed. (#478)
Fix bug when merging dictionaries during application of blueprint overrides (#496)
Fix bug where SimulationRunner fails to display errors when parsing bad CLI arguments (#497)
Ensure typer CliRunner is used for tests of blueprint check command (#505)
Fix bug where automatic output directory overrides appeared dependent on time-splitting (#512)
Fix failure to handle SLURM timeout status (#508)
Generalize OverrideTransform for non-ROMS application types (#500)
Create work-dir earlier to avoid failure when making sym-links for CDR/nesting files. (#513)
File system’s .clear() method now correctly clears the work directory (#516)
Fixed a bug where a path that has been renamed logged the old path and the file appeared to have been deleted (#519)
Fixed a failure after joining partitioned files due to an aggressive string replacement (#519)
Fixed a bug where the Service signal handler failed to initiate shutdown (#519)
Fixed a bug caused by inconsistent __init__ argument ordering and removed the prior fix of adding a dupe attribute (#519)
Fix bug where CSTAR_CLOBBER_WORK_DIR would also delete important workplan artifacts (#527)
Fix bug in runner error handling (#526)
Fix a case-sensitivity bug in run-id autocompletion (#525)
Fix failing unit test causing new feature branches to fail (#530)
Remove python-specific named group syntax in checkpoint_frequency regex. (#531)
Enhance log-level callback to reset log levels to fix inconsistent logging levels when passing –log-level CLI parameter (#534)
Fix bug in ExecutionHandler where it attempts to open the log file before it exists. (#537)
Fix bug when null restart file is passed to adapter (#537)
Fix no module named psutil error after environment setup (#543)
Fix failure caused by cstar workplan status using SlurmHandle with LocalLauncher (#544)
Fix XDG directory caching bug resulting in writes to default directory (#547)
Fix failure to set CSTAR_RUNID and CSTAR_CLI_DRY_RUN when supplied in CLI (#547)
Fix _no-default-status_ bug in DagStatus (#546)
Fix _use-incorrect-working-dir-attribute_ bug in LiveStep and during automatic step overrides (#546)
Fix _step-working-dir-changes-after-deserialization_ bug (#546)
ROMS-MARBL app will now exit non-zero if ROMS crashes (#550)
Fix variable name with matching prefix causing erroneous registration (#554)
Fix overwriting existing metadata with secondary import (#554)
Avoid use of fixed port number causing tests to sporadically fail if port 12345 is taken (#557)
Fix cannot call asyncio.run on running event loop failure from CLI tests (#557)
Ensure LMOD_SYSTEM_DEFAULT_MODULES is set before manipulating modules (#545)
Fix bug in Workplan._check_steps that allows multiple steps to resolve to the same safe name. (#561)
Fix regression in test mocks after entities were renamed. (#564)
Fix missing raw status mapping to SLURM TIMEOUT (#563)
Fix incorrect test harness data formatting resulting in unknown statuses in tests (#563)
Fix failure to generate schemas for new applications with cstar blueprint schemas (#566)
C-Star log stream should now more fully capture ROMS log stream (#551)
Fix SlurmHandle has no attribute run_id bug (#571)
Fix unmapped OUT_OF_MEMORY status returned by SLURM resulting in run failure. (#569)
Fix index error when formatting validation error messages (#574)
Add missing import causing test failures (#574)
Use alternative method for identifying cluster on Elja compute nodes (#573)
Fix bug when merging dictionaries during application of blueprint overrides
Fix bug where SimulationRunner fails to display errors when parsing bad CLI arguments
Improvements#
Decouple launcher-specific precondition checks from the dag_runner
Use computed attributes in place of duplicating keys found in DagStatus.details
Re-use configure_environment where applicable
Refactor cli.workplan.run to make use of typer callbacks for input validation
Add –dry-run option to test workplan generation in cstar workplan run
Remove if is_feature_enabled(“CSTAR_FF_ORCH_TRX_OVERRIDE”) to always enable overrides. (#428)
Encapsulate creation of JobFileSystemManager in new LiveStep class (#434)
Perform input validation before executing workplan and blueprint in: (#443)
cstar.cli.blueprint.run
cstar.cli.workplan.run
Add additional tests for EnvItem functionality not tied to XDG home directories (#444)
Reduce module load times (#446)
Reduce package import time by lazy loading expensive modules (#447)
Extract nested/duplicate YAML representer functions for re-use (#451)
Add empty sections specified in release notes and _Code Review Checklist_ (#454)
Replace use of incorrect mp.Queue where queue.Queue is appropriate (#456)
Replace blocking time.sleep usage in health check thread (#456)
Remove default values provided in JobConfig class (#457)
Removed slow status retrieval loop for previously submitted jobs (#459)
Enable “path-less” CLI commands, such as cstar workplan status –run-id <id>. (#461)
Display detailed status information for tasks in a Workplan (#461)
Ensure env var display consistency in cstar env show output (#462)
The original runtime settings file and the one that is written and used by C-Star are now appended with _ORIGINAL and _PATCHED to reduce confusion. (#465)
Use computed attributes to in place of duplicating keys found in DagStatus.details (#467)
Improve error message when failing to load a remote YAML file. (#473)
Loosen Pydantic type validation for override dictionaries to mitigate pydantic warnings (#474)
Update unit tests that fail under specific user conditions on Anvil (#483)
Fix unexpected linting warnings about missing required parameters to ServiceConfiguration
Improve naming of module containing shared CLI callbacks
Add callbacks for automatic export of pertinent environment variables from CLI parameters
Reduce unit test duration
Improve help text for CLI subcommands and parameters (#487)
Mitigate a dependency issue with prefect and fakeredis (pin fakeredis<2.35) (#489)
Reduce usage of hardcoded strings in CLI and tests (#497)
Simplified env-var flag evaluation (#501)
Add –log-level and –clobber parameters to CLI methods for running blueprints and workplans. (#502)
Reduced amount of repetitive logs captured at the DEBUG level (#504)
Improved consistency in CLI outputs and error handling (#506)
Mitigated linter warnings from using deprecated collections (e.g. t.Mapping, t.Iterable) (#507)
Save the generated roms.in to the work directory, where roms is run. (#514)
Symlink the roms executable to the work directory, where roms is run. (#514)
Verbosity and dry-run CLI argument callbacks now ensure the appropriate environment variables are set. (#515)
Simplified application registration process via use of ApplicationDefinition (#519)
Enabled state tracking when running blueprints for improved progress visibility (#519)
Improved docstrings and type-hinting in affected code (#519)
Remove unnecessary use of Literal types (#525)
Replaced variable length tuple typehints with simpler Sequence (#523)
Removed unnecessary file creation during worker tests. (#533)
Removed unnecessary addition of version callback to all typer apps (#534)
Minor de-duplication of help text in CLI via adding constants (#535)
Refactored as needed to share migration code with manual migration triggered via cstar blueprint migrate … (#535)
Improved type-hints for InputDataset and RomsInputDataset (#538)
Mitigated linter warnings (#538)
Improved formatting, linting, and type hinting in serialization, tracking, and launcher modules. (#541)
Promote env show and workplan status to always-on & deprecate feature flags (#540)
Improved level of detail in cstar workplan status output (#532)
Reduce code duplication for producing minimally-padded indices (#539)
Improved exception logging during workplan run (#546)
Increased detail stored on ProcessHandle (#546)
log_path and script_path added to LiveStep and re-used where appropriate to encapsulate path building (#546)
Added _on-status-changed_ and _on-launched_ callbacks to orchestrator to reduce launcher coupling (#546)
Refactored StateRepository into class; subsumed sentinel path building (#546)
Improved discovery speed by caching var metadata (#554)
UX improved by printing ‘available step names’ hint for user when handling incorrect step name in input (#549)
workplan log CLI handler no longer reproduces internal path resolution (#549)
Extract lmod environment variable handling into standalone settings class (#555)
Mitigated incomplete typehints and used modern class member declarations in modified classes. (#555)
Moved fixed environment variable configuration into dotenv files, deprecated configure_environment (#555)
Remove unused parameter to load_lmod_modules (#545)
Improved type-hinting (#545)
Replaced looping over single module loads with one call loading multiple modules (#545)
Workplan run summary includes paths to Workplan YAML files for review. (#567)
Mitigate linter warnings in orchestration.py (#565)
Reduce warnings produced by linter (#568)
Reduce usage of hardcoded strings in CLI and tests
Miscellaneous#
Default to having prefect analytics collection disabled
—
For more details, please refer to the `commit history`_.
Rename constant and move from cstar.orchestration.utils to cstar.base.env (#432)
Update all references to the variable (#432)
Remove unnecessary TypeVar (#451)
Fix inconsistent test naming convention (#451)
Fix typos in tutorial (#464)
Fix unexpected linting warnings about missing required parameters to ServiceConfiguration (#481)
Ensure shutdown of prefect to release build agent after tests complete (#509)
Add blueprint schema reference to the documentation. (#529)
Updated string values in integer fields failing schema validation in plotting sample (#531)
Added shared functions for storing data in the typer context (#549)
Revert prior change resulting in less control of lmod module loading (#573)
Add tutorial to documentation for creating custom applications. (#560)
v0.4.0#
Breaking Changes#
Rename env var group name constants to omit leading underscore
Modified the interpretation of Status.is_running to exclude Unsubmitted
New features#
Enable status retrieval for multiple SLURM jobs in a single sacct call
The orchestrator writes a sentinel file to disk to dynamically alter dependencies
Add trace log level to reduce log frequency during Workplan execution
Persist WorkplanRun record for every invocation of the orchestrator for run history and lookup
Add cstar workplan monitor –run-id <run-id> for reattaching to a running Workplan
Security Fixes#
N/A
Bug Fixes#
Fix failure to override output directories for orchestrated blueprints
Fix defect where steps defined before dependencies were not mapped correctly after time-splitting
Fix unhandled exceptions in cstar [blueprint|workplan] check with invalid paths
Fix posix-path conversion bug when passing blueprint URLs to cstar blueprint run
Fix case-sensitivity bug when configuring split frequency via CSTAR_ORCH_TRX_FREQ env variable
Fix a JSON serialization bug with pydantic models using field aliases
Fix incorrect working directories resulting from LiveStep.from_step when a parent is specified
Fix failure to terminate health check thread
Fix issue when parsing SLURM status of the form CANCELLED by <username>
Fix incorrect env var discovery if env_ appears repeatedly in variable name
Fix unhandled exception caused by attempting to execute typer app
Fix cstar workplan status … triggering a re-build of the entire workplan
Improvements#
Improve error handling when CachedRemoteRepositoryRetriever.refresh fails
Validate inputs prior to execution with cstar [blueprint|workplan] run
Improve module load times
Extract nested/duplicate YAML representer functions for re-use
Replace use of incorrect mp.Queue where queue.Queue is appropriate
Replace blocking time.sleep usage in health check thread
Removed slow status retrieval loop for previously submitted jobs
Ensure env var display consistency in cstar env show output
Enable “path-less” CLI commands, such as cstar workplan status –run-id <id>.
Display detailed status information for tasks in a Workplan
Miscellaneous#
N/A
—
For more details, please refer to the `commit history`_.
v0.3.0#
This is a bug-fix release focused on stabilizing the orchestration of blueprints.
Breaking Changes#
Environment variables
CSTAR_HOMEandCSTAR_OUTDIRare deprecated in favor ofCSTAR_CONFIG_HOMEandCSTAR_STATE_HOMERelocated command conversion functions from cstar.orchestration.launch.slurm to cstar.orchestration.converter
Promote blueprint overrides to always-on, deprecate and remove feature flag
CSTAR_FF_ORCH_TRX_OVERRIDE
New features#
Add remote repository caching for re-use across simulations
XDG base directory standard is implemented
Security Fixes#
N/A
Bug Fixes#
Fix path concatenation bug resulting in invalid source data paths
Fix indentation creating an unreachable code block in git utils
Add conda installation of rsync to environment.yml (fixes bug related to a flag used in newer ucla-roms makefiles that is not available in MacOS’s default rsync installation)
Reimplement
CSTAR_CLOBBER_WORKING_DIRfor single-blueprint casesFix bug handling user-relative paths in
JobFileSystemManager
Improvements#
Avoid automatic failures by retrying failed git clone operations
Add informational logging when tasks fail and additional tasks are cancelled
Update default log format to include file name and line number
Add constants for the set of available feature flags
Implement
CSTAR_LOG_LEVELenv-varAutomatically retry partitioning without coarse dims if it fails when they are included
Enable orchestrated simulation execution using the local runner
Miscellaneous#
Additional details added in docstrings
Silence OMP warning originating from numba
Modify a blueprint in the tests directory for a refactored layout in the remote data repository
Move env-var handling methods to separate module to avoid circular imports
Remove
__init__-based import shortcutsAdjust error handling and presentation in the worker/service layer
—
For more details, please refer to the `commit history`_.
v0.1.0#
New features#
Add support for river forcing
Add support for forcing corrections (used in legacy ROMS configurations)
Breaking Changes#
Merge Case and Component classes into a single class, Simulation, simplifying internal/blueprint structure. Remove old Case and Component modules.
Rename caseroot to directory
Rename BaseModel to ExternalCodeBase
Remove start_time and end_time parameters from InputDataset.get(), these are now obtained from the corresponding attributes
Rename ROMSComponent.namelists to ROMSSimulation.runtime_code and ROMSComponent.additional_source_code to ROMSSimulation.compile_time_code
Internal Changes#
Update calls to roms-tools to reflect latest changes in API
Update internal/test blueprints to reflect new structure
New backend data retrieval system cstar.io
Add ROMSRuntimeSettings class with ability to parse and create roms .in files
Save partitioned ROMSInputDatasets in the same directory as their un-partitioned versions, rather than a subdirectory “PARTITIONED”
Documentation#
Run most examples in documentation locally (unless HPC-specific), so HPC access is not required
Add ‘How-to Guides’ section
Remove tutorial page on using roms-tools
Replace all URLs to other doc pages with relative links
Bugfixes#
Complete missing unit test coverage from Case and Component in new Simulation modules
Add expanduser to Path instances to allow tildes to represent root in paths
Correct issue where AdditionalCode.modified_files list indices did not correspond to AdditionalCode.files
v0.0.8-alpha#
New features#
Add Case.persist() and Case.restore() methods to allow continuation of work in a new session
Add ExecutionHandler class to track tasks run locally (LocalProcess subclass) or submitted to a job scheduler (SchedulerJob subclass)
Improved tracking of local InputDataset and AdditionalCode files to prevent repeat fetching
Add ability to read blueprints from URL
Remote yaml files are now accessed via requests rather than Pooch, negating need for hash checks
Bugfixes#
git and DataSource information are now read-only attributes throughout
v0.0.3-alpha (9th Dec 2024)#
New features#
Add support for SDSC Expanse HPC
Bugfixes#
Fix bug where in certain circumstances environment variables were checked for before being set, prompting user to install already installed externals
v0.0.1-alpha (6th Dec 2024)#
The first release of C-Star!
This release provides basic functionality, including the ability to create, import, and export a “blueprint” for reproducible ocean model simulations (a C-Star “Case”) using supported ocean models (ROMS, optionally with MARBL biogeochemistry) and run those Cases locally or on supported HPC systems (via Slurm and PBS). There is support for using existing model input data in netCDF format, or creating new input data via integration with the roms-tools library.
Note that the python API is not yet stable, and some aspects of the schema for the blueprint will likely evolve. Therefore whilst you are welcome to try out using the package, we cannot yet guarantee backwards compatibility. We expect to reach a more stable version in 2026.