API Reference

Pharaoh API Module

Lists all items that can be imported from pharaoh.api.

This module contains Pharaoh project-related API functions.

get_project(lookup_path: str | Path | None = None) → PharaohProject

Returns an instance of PharaohProject.

Parameters:

lookup_path – If None, the function tries to return the Pharaoh singleton instance that is already loaded in memory and raises and exception if there is none. If given a string or Path instance, searches the given path and all its parent folders for a pharaoh.yaml project file and returns a PharaohProject instance.

class PharaohProject(*args, **kwargs)

PROJECT_SETTINGS_YAML = 'pharaoh.yaml'

__init__(project_root: str | Path, overwrite: bool = False, templates: str | list[str] | tuple[str] = ('pharaoh.default_project',), template_context: dict[str, Any] | None = None, **kwargs)

Instantiates a Pharaoh project instance using either an existing project root (directory containing pharaoh.yaml) or creates a new project.

Parameters:

• project_root – A directory containing a pharaoh.yaml file

• overwrite – If the project directory already contains files, overwrite them.

• templates – The project template(s) to use for creating the Sphinx project files. Maybe be any number of template names or paths to directories.

• template_context – The Jinja rendering context for the selected project templates.

• custom_settings – A path to a YAML file containing settings to overwrite the default project settings.

add_component(component_name: str, templates: str | Path | Iterable[str] | Iterable[Path] = ('pharaoh.empty',), render_context: dict | None = None, resources: list[resource.Resource] | None = None, metadata: dict[str, Any] | None = None, index: int = -1, overwrite: bool = False)

Adds a new component to the Pharaoh project

Example:

from pharaoh.api import FileResource, PharaohProject

proj = PharaohProject(".")
proj.add_component(
    component_name="component_ABC",
    templates=[
        "plugin_abc.template_xyz",              # plugin template
        "path/to/template/directory",           # template directory
        "path/to/template/file/tmpl.pharaoh.py" # template file
    ],
    render_context={"foo": "bar"},
    resources=[FileResource(alias="dlh5_result", pattern="C:/temp/**/*.dlh5")],
    metadata={"some tag": "some value"},
    index=-1,  # append
    overwrite=False
)

Parameters:

• component_name – The name of the component. Must be a valid Python identifier.

• templates –

  A list of component templates to use for creating the components project files. Those may be the template identifier (e.g. plugin_abc.template_xyz) of a registered plugin template, a path to a template directory or a path to a template file (single-file template).

  Since multiple template may be specified, their order matters in cases where different templates create equally-named files, thus templates might overwrite files of the previous templates.

  This enables template-composition, where template designers can chunk their bigger templates into smaller reusable building blocks.

  If omitted, an empty default template is used.

• render_context – The Jinja rendering context for the selected template. The actual template rendering context will have component_name, resources and metadata available under the respective keys.

• resources – A list of Resource instances, defining the component’s resources used in asset scripts

• metadata – A dictionary of metadata that may be used to find the component via PharaohProject.find_component() method.

• overwrite – If True, an already existing component will be overwritten

add_template_to_component(component_name: str, templates: str | Path | Iterable[str], render_context: dict | None = None)

Adds additional templates to an existing component, that may overwrite existing files during rendering.

Parameters:

• component_name – The name of the component. Must be a valid Python identifier.

• templates – The component template(s) to use for creating the components project files. Maybe be the a single or multiple template names or paths to a directory.

• render_context – The Jinja rendering context for the selected template. The actual template rendering context will have component_name, resources and metadata available under the respective keys.

archive_report(dest: str | Path | None = None) → Path

Create an archive from the build folder.

Parameters:

dest – A destination path to create the archive. Relative paths are relative to the project root. If omitted, the filename will be taken from the report.archive_name setting.

Returns:

The path to the archive

property asset_build_dir

property asset_finder: AssetFinder

build_report(catch_errors=True) → int

Builds the Sphinx project and returns the status code.

Parameters:

catch_errors – If True, Sphinx build errors will not raise an exception but return a -1 instead.

find_components(expression: str = '') → list[DictConfig]

Find components by their metadata using an evaluated expression.

The expression must be a valid Python expression and following local variables may be used:

• name: The name of the component

• templates: A list of templates used to render the component

• metadata: A dict of metadata specified

• render_context: The Jinja rendering context specified

• resources: A list of resource definitions

Parameters:

expression –

A Python expression that will be evaluated. If it evaluates to a truthy result, the component name is included in the returned list.

Example: name == "dummy" and metadata.foo in (1,2,3).

A failing evaluation will be treated as False. An empty expression will always match.

generate_assets(component_filters: Iterable[str] = ('.*',)) → list[Path]

Generate all assets by executing the asset scripts of a selected or all components.

All asset scripts are executed in separate parallel child processes (number of workers determined by asset_gen.worker_processes setting; setting 0 executes all asset scripts sequentially in the current process).

Setting asset_gen.script_ignore_pattern determines if a script is ignored.

Putting the comment # pharaoh: ignore at the start of a script will also ignore the file.

Parameters:

component_filters – A list of regular expressions that are matched against each component name. If a component name matches any of the regular expressions, the component’s assets are regenerated (containing directory will be cleared)

get_default_sphinx_configuration(confdir: str | Path)

Provides Sphinx project configurations, that will be dynamically included by the generated conf.py.

get_resource(alias: str, component: str) → Resource

Finds a Resource from a project component by its alias.

Parameters:

• component – The component’s name

• alias – The resource’s alias

get_setting(key: str, default=<object object>, to_container: bool = False, resolve: bool = True)

Gets a setting by its dot-separated name, e.g. “core.debug”.

The settings are preferably taken from environment variables (e.g. PHARAOH.CORE.DEBUG), whereas the values must be TOML compatible values (e.g. “true” for boolean).

If no environment variable is found, the user defined settings in the pharaoh.yaml file in the project root will be used. If the setting is not specified in there or the file is not existing, the Pharaoh default settings are queried.

If the setting is not found, a LookupError is returned unless “default” is set.

Examples:

proj.get_setting("report.title")
proj.get_setting("cloud.metadata", to_container=True, resolve=False) == {
    "author": "${report.author}",
    "title": "${report.title}",
}
proj.get_setting("cloud.metadata", to_container=True, resolve=True) == {
    "author": "loibljoh",
    "title": "Pharaoh Report",
}
proj.get_setting("cloud.metadata.title") == "Pharaoh Report"
proj.get_setting("cloud.metadata") == omegaconf.dictconfig.DictConfig(...)

Parameters:

• key – Dot-separated name of the setting value to return

• default – The returned default value if the key does not exist.

• to_container – If set, recursively converts an OmegaConf config to a primitive Python container type (dict or list).

• resolve – True to resolve all values if “to_container” is set to True

get_settings() → DictConfig

Returns merged settings from all namespaces.

iter_components() → Iterator[omegaconf.DictConfig]

Returns an iterator over all components from a project.

iter_resources(component: str) → Iterator[omegaconf.DictConfig]

Returns an iterator over all resources from a project component.

Parameters:

component – The component’s name

load_settings(namespace: str = 'all')

Loads settings from various namespaces.

default

The default setting in the Pharaoh library

project

The project settings that may be modified by the user

env

The settings defined by environment variables starting with PHARAO

all

Loads all of the above

Parameters:

namespace – The namespace to load settings from. all, default, project or env.

open_report()

Opens the generated report (if possible, e.g. for local HTML reports).

property project_root

put_setting(key: str, value: Any)

Sets a setting by its dot-separated name, e.g. “core.debug”.

This has no effect, if there is an environment variable defining the same setting!

remove_component(filter: str, regex: bool = False) → list[str]

Removes one or multiple existing components.

Parameters:

• filter – A case-insensitive component filter. Either a full-match or regular expression, depending on regex argument.

• regex – If True, the filter argument will be treated as regular expression. Components that partially match the regular expression are removed.

Returns:

A list of component names that got removed

save_settings(include_env: bool = False)

Saves back the project settings to the project YAML file.

Parameters:

include_env – If True, Pharaoh settings that are set via environment variables will be persisted to the project settings YAML file.

property settings_file

property sphinx_report_build

property sphinx_report_project

property sphinx_report_project_components

update_resource(alias: str, component: str, resource: Resource)

Updates a Resource from a project component by its alias.

Parameters:

• component – The component’s name

• alias – The resource’s alias

• resource – The resource’s alias

Asset Generation API Module

Lists all items that can be imported from pharaoh.assetlib.api.

This module contains API functions related to asset generation. When asset scripts are accessing those API functions, the functions can access the project information by themselves.

catch_exceptions(catch: tuple[type[Exception], ...] = (<class 'Exception'>,), reraise: tuple[type[Exception], ...] = (), msg_prefix: str = '', log_exc: bool = True, render_exc: bool = True)

Catch exceptions of given types.

If an exception is caught, the Sphinx report generation will report a warning.

See also

Catching Errors

Parameters:

• catch – The exception types to catch

• reraise – The exception types to re-raise (has precedence over catch)

• msg_prefix – The message prefix to log before the exception message

• log_exc – Whether to log the exception message as a warning

• render_exc – Whether to export the exception traceback as an asset to be included in the report

find_components(expression: str = '')

Find components by their metadata using an evaluated expression.

The expression must be a valid Python expression and following local variables may be used:

• name: The name of the component

• templates: A list of templates used to render the component

• metadata: A dict of metadata specified

• render_context: The Jinja rendering context specified

• resources: A list of resource definitions

Parameters:

expression –

A Python expression that will be evaluated. If it evaluates to a truthy result, the component name is included in the returned list.

Example: name == "dummy" and metadata.foo in (1,2,3).

A failing evaluation will be treated as False. An empty expression will always match.

get_asset_finder() → AssetFinder

Returns the AssetFinder instance for the current project

get_current_component() → str

If executed from within a script that is placed inside a Pharaoh component, the function returns the components name by analyzing the call stack.

get_resource(alias: str, component: str | None = None)

Get a Resource instance by its component name and resource alias.

metadata_context(context_name='', **context) → MetadataContext

Returns a context manager that adds a new stack entry with custom metadata, effectively updating the metadata context assets are exported with.

When the context manager is left, the stack entry is removed again.

See Metadata Stack docs.

Parameters:

• context_name – An optional name for the context in case it must be looked up.

• context – Keywords arguments to specify custom metadata.

register_asset(file: str | Path, metadata: dict | None = None, template: str | None = None, data: BytesIO | None = None, copy2build: bool = False, **kwargs) → Asset | None

Register an asset manually. The file will be copied (if data is None and ‘file’ is a real file) or written (if data is given) to the asset build folder of the current Pharaoh project.

Parameters:

• file – The filename. Must exist even if data is set, to have a filename to store the asset and to automatically determine the template (if not set via template argument).

• metadata – Additional metadata to store on the asset.

• template –

  The template used to render the asset. If omitted, it is inferred by the file extension.

  See also

  Asset Templates

• data – An io.BytesIO instance. Used if the asset is generated in memory and should be stored to disk by this function.

• copy2build –

  If True, the asset will be copied to the asset build directory, even if not referenced in the template.

  Background: Pharaoh stores all assets in the project directory and copies them to the build directory only if copy2build is set to True or on-demand by Pharaoh. For example if an HTML file is rendered using an iframe, the HTML file has to be copied to the build folder where the iframe can later include it.

Returns:

The file path where the asset will be actually stored

register_templating_context(name: str, context: str | Path | dict | list, metadata: dict | None = None, **kwargs)

Register a data context for the build-time templating stage. The data may be given directly as dict/list or via a json or yaml file.

This function is designed to be used within asset scripts, to easily register data you extract from resources for the templating process.

Example:

from pharaoh.assetlib.api import register_templating_context

register_templating_context(name="foo", context={"bar": "baz"})
# will be accessed like this: {{ ctx.local.foo.bar.baz }}

Parameters:

• name –

  The name under which the data context is available inside Jinja templates. Access like this (name: mycontext):

  {% set mycontext = ctx.local.mycontext %}
  

• context – Either a str or Path instance pointing to a json or yaml file, or a dict or list. All data must contain only json-compatible types, otherwise the data cannot be stored.

• metadata – The given context will be internally registered as an asset with following metadata: dict(pharaoh_templating_context=name, **metadata)

• kwargs – Keyword arguments that are mostly (except component) passed to json.dumps(...), in case context is a dict or list.

class AssetFinder(lookup_path: Path)

__init__(lookup_path: Path)

A class for discovering and searching generated assets.

An instance of this class will be created by the Pharaoh project, where lookup_path will be set to report_project/.asset_build.

Parameters:

lookup_path – The root directory to look for assets. It will be searched recursively for assets.

discover_assets(components: list[str] | None = None) → dict[str, list[Asset]]

Discovers all assets by recursively searching for *.assetinfo files and stores the collection as instance variable (_assets).

Parameters:

components – A list of components to search for assets. If None (the default), all components will be searched.

Returns:

A dictionary that maps component names to a list of Asset instances.

get_asset_by_id(id: str) → Asset | None

Returns the corresponding Asset instance for a certain ID.

Parameters:

id – The ID of the asset to return

Returns:

An Asset instance if found, None otherwise.

iter_assets(components: str | Iterable[str] | None = None) → Iterator[Asset]

Iterates over all discovered assets.

Parameters:

components – A list of component names to search. If None (the default), all components will be searched.

Returns:

An iterator over all discovered assets.

search_assets(condition: str, components: str | Iterable[str] | None = None) → list[Asset]

Searches already discovered assets (see discover_assets()) that match a condition.

Parameters:

• condition –

  A Python expression that is evaluated using the content of the *.assetinfo JSON file as namespace. If the evaluation returns a truthy result, the asset is returned.

  Refer to this example assetinfo file to see the available default namespace.

  Example:

  # All HTML file where the "label" metadata ends with "_plot"
  finder.search_assets('asset.suffix == ".html" and label.endswith("_plot")')
  

• components – A list of component names to search. If None (the default), all components will be searched.

Returns:

A list of assets whose metadata match the condition.

class Asset(info_file: Path)

Holds information about a generated asset.

Variables:

• id – An MD5 hash of the asset’s filename, prefixed with “__ID__”. Since a unique suffix is included in the filename, this ID hash is also unique. Can be used to quickly find this Asset instance.

• infofile (Path) – Absolute path to the *.assetinfo file

• assetfile (Path) – Absolute path to the actual asset file

• context (omegaconf.DictConfig) – The content of infofile parsed into a OmegaConf dict.

copy_to(target_dir: Path) → Path

Copy the asset plus info-file.

Parameters:

target_dir – The target directory to copy to. Will be created if it does not exist.

Returns:

True if files were copied, False otherwise (files already exist)

read_bytes() → bytes

Reads the file as bytes

read_json() → dict

Reads the file using a JSON parser.

read_text(encoding: str = 'utf-8') → str

Reads the file as text

read_yaml() → dict

Reads the file using a YAML parser.

asset_groupby(seq: Iterable[Asset], key: str, sort_reverse: bool = False, default: str | None = None) → dict[str, list[Asset]]

Groups an iterable of Assets by a certain metadata key.

During build-time rendering this function will be available as Jinja global function asset_groupby and alias agroupby.

Example:

We have following 4 assets (simplified notation of specified metadata):
Asset[a="1", b="3"]
Asset[a="1", c="4"]
Asset[a="2", b="3"]
Asset[a="2", c="4"]

Grouping by "a":
    asset_groupby(assets, "a")
will yield
    {
        "1": [Asset[a="1", b="3"], Asset[a="1", c="4"]],
        "2": [Asset[a="2", b="3"], Asset[a="2", c="4"]],
    }


Grouping by "b" and default "default":
    asset_groupby(assets, "b", default="default")
will yield
    {
        "3":       [Asset[a="1", b="3"], Asset[a="2", b="3"]],
        "default": [Asset[a="1", c="4"], Asset[a="2", c="4"]],
    }

Parameters:

• seq – The iterable of assets to group

• key – The nested attribute to use for grouping, e.g. “A.B.C”

• sort_reverse – Reverse-sort the keys in the returned dictionary

• default – Sort each item, where “key” is not an existing attribute, into this default group

Returns:

A dictionary that maps the group names (values of A.B.C) to a list of items out of the input iterable

Resources

class FileResource(alias: str, pattern, sort: str = 'descending', *, cachedir: str = '')

Bases: LocalResource

A resource that matches files/directories using a wildcard pattern.

Example:

FileResource(alias="mycsvfile", pattern="C:/temp/*.csv", sort="ascending")

Variables:

• pattern – A pathname pattern. The pattern may contain simple shell-style wildcards. Filenames starting with a dot are special cases that are not matched by ‘*’ and ‘?’ patterns.

• sort –

  descending (default) or ascending. Sorting is done using the natsort library to sort filesystem paths naturally. Example:

  0.txt, 01.txt, 1.txt, 10.txt, 2.txt  # default sorting (ascending)
  0.txt, 01.txt, 1.txt, 2.txt, 10.txt  # natural sorting (ascending)
  

first_match() → Path

Returns the first match for the file pattern, depending on the chosen sort order.

get_files(recursive: bool = True) → list[Path]

Returns all matches for the file pattern, depending on the chosen sort order.

Parameters:

recursive – If recursive is true, the pattern ‘**’ will match any files and zero or more directories and subdirectories.

get_match(index) → Path

Returns the n-th match for the file pattern, depending on the chosen sort order.

last_match() → Path

Returns the last match for the file pattern, depending on the chosen sort order.

locate() → Path

Returns the first match for the file pattern, depending on the chosen sort order.

class CustomResource(alias: str, traits: dict, *, cachedir: str = '')

Bases: Resource

A resource that can hold arbitrary information. It may be used as a temporary replacement for more specific resources that are not yet implemented. For example, it could hold database connection properties or other custom data.

Example:

CustomResource(alias="foo", traits=dict(a=1, b=[2, 3], c=dict(d=[4, 5])))

Variables:

traits – A dict with arbitrary data.

Bases:

class Resource(alias: str, *, cachedir: str = '')

Bases: object

The base class for all resources. Provides serialization/deserialization methods.

class LocalResource(alias: str, *, cachedir: str = '')

Bases: ABC, Resource

Represents a resource that is located on the user’s hard-drive.

abstract locate() → Path

Returns the path to the resource file.

class TransformedResource(alias: str, sources: list[str], *, cachedir: str = '')

Bases: LocalResource, ABC

A resource that is depending on another resource and maybe transforms it dynamically into another resource.

abstract transform(resources: dict[str, Resource])

Executed by Pharaoh before asset generation. Transforms the linked resource into a new one. The resulting resource may be cached or recreated each time.

Matlab Integration API

class Matlab(start_options: str = '-nodesktop')

Matlab engine for Pharaoh asset generation.

Usage:

from pharaoh.assetlib.api import Matlab

eng = Matlab()
with eng:
    out, err = eng.execute_script("myscript.m")
    result, out, err = eng.execute_function("myfunc", [800.0], nargout=1)

__enter__()

Context manager enter. Connects to Matlab.

__exit__(exc_type, exc_val, exc_tb)

Context manager exit. Disconnects from Matlab.

__init__(start_options: str = '-nodesktop')

Matlab engine for Pharaoh asset generation.

Parameters:

start_options – See options at https://de.mathworks.com/help/matlab/ref/matlabwindows.html

connect()

Connects to a Matlab engine. If an engine is running it will be connected, otherwise a new Matlab instance will be started and connected.

property eng: me.MatlabEngine

Returns the matlab engine instance

execute_function(function_name: str, args: list[Any] | None = None, nargout: int = 0, workdir: str | Path | None = None) → tuple[Any, str, str]

Executes a Matlab function

Returns a tuple of result, stdout stream and stderr stream of the function execution.

The shape/type of result depends on the argument ‘nargout’. See below.

Parameters:

• function_name –

  The name of the function to execute.

  The function must be in the Matlab path to execute it.

  Alternatively the workdir argument can be set to the parent directory of the function.

• args – A list of positional input arguments to the function

• nargout –

  The number of output arguments.

  If 0, the result return value of this function will be None.

  If 1, the result return value will be a single value.

  If greater than 1, the result return value will be a tuple containing nargout values.

• workdir – The Matlab working directory to be changed to during function execution. Skipped if None.

execute_script(script_path: str | Path) → tuple[str, str]

Executes a Matlab script.

Returns a tuple of strings containing the stdout and stderr streams of the script execution.

show_gui()

Makes the Matlab GUI visible.

Templating Builtins

Following globals, filters and tests are available during build-time templating.

Jinja2 Builtins

All Jinja2 builtins are available of course:

• List of builtin filters

• List of builtin tests

• List of global functions

Pharaoh Extension

Pharaoh extends the template environment by additional globals and filters, which are described in following chapters.

Globals

Following functions are available in the global templating scope, and can be used like variable, but all callable, e.g. {{ h1("Title") }}.

assert_true(statement: bool, message: str = '')

Wrapper for Python’s assert builtin.

Parameters:

• statement – A boolean statement

• message – The message for the AssertionError, if statement is False.

fglob(pattern: str, root: str = '.') → list[Path]

Returns a list of relative paths of all files relative to a root directory that match a certain pattern.

Examples:

{% set index_files1 = fglob("index_*.rst") %}
{% set index_files2 = fglob("subdir/index_*.rst") %}

Parameters:

• pattern – A glob pattern to match files.

• root – The root directory for discovering files and anchor for relative paths. Default is the current working directory (the parent directory of the currently rendered file).

hrule()

Renders a 1px gray horizontal rule using a raw html directive.

raise_helper(msg)

Raises an Exception with a certain message. The actual name of the Jinja global function is raise.

heading(text: str, level: int) → str

Renders a heading for a certain level.

Shortcuts for this function are automatically generated with names h1…h7.

Examples:

{{ heading("Page Title", 1) }}
{{ h2("Sub-Title") }}

Parameters:

• text – The heading

• level – The heading level. From 1 (top-level) to 7.

rand_id(chars: int | None = None) → str

Returns a unique id for usage in HTML as it is made sure it starts with no number

read_text(file) → str

Reads a file content using utf-8 encoding.

Filters

Variables can be modified by filters. Filters are separated from the variable by a pipe symbol (|) and may have optional arguments in parentheses. Multiple filters can be chained. The output of one filter is applied to the next.

For example, {{ name|trim|title }} will trim whitespace from the variable name and title-case the output.

Filters that accept arguments have parentheses around the arguments, just like a function call. For example: {{ listx|join(', ') }} will join a list with commas.

hasattr_(obj, name)

required(value)

Raises an UndefinedError if the context variable is undefined. The actual name of the Jinja global function is req.

Example:

{{ h1(ctx.local.test.test_name|req) }}

rep(value) → str

Returns repr(value). The actual name of the Jinja global function is repr.

or_default(value, default)

Returns the result of value or default.

Alias: or_d

oc_resolve(value: DictConfig)

Recursively converts an OmegaConf config to a primitive container (dict or list) and returns it.

oc_get(cfg: ~omegaconf.dictconfig.DictConfig, key, default=<object object>)

Returns an object inside an omegaconf.DictConfig container. If the object does not exist, the default is returned.

Example:

{% set plot_title = asset.context|oc_get("plot.title", "") %}

Parameters:

• cfg – The omegaconf.DictConfig container to search

• key – The name of the object to extract, e.g. plot.title.

• default – The default to return if key is not in cfg

exists(path: str) → bool

Returns if path exists.

to_path(path: str) → Path

Returns path as a pathlib.Path instance.

md2html(text)

Converts the Markdown text (CommonMark syntax) to HTML.

Ansible Filters

Jinja2 Ansible Filters is a port of the ansible filters provided by Ansible’s templating engine.

See pypi project page for a listing of all filters.

The filters are useful, but unfortunately not documented. You may refer to the docstrings in the source code.

Plugin API