Catalyst for Simulation Developers
**********************************

This section describes how simulation (and other computational codes) can use
Catalyst.

Building with Catalyst
======================

To use the Catalyst API in any code, the code must be built against an
implementation of the Catalyst API. While one can use any implementation of the
Catalyst API, the stub implementation is probably the easiest to build against
since it doesn't have any external dependencies besides compiler tools.

There are two ways codes can build with Catalyst: using `CMake`_, or using any
build tool like `make`_.

Using CMake
-----------

If your code already uses CMake as the build system generator, then to use
Catalyst APIs, you simply need to find the Catalyst install using ``find_package``
and the link against the ``catalyst::catalyst`` target. This is done as follows:

.. code-block:: cmake
    :linenos:

    # Find the Catalyst install.
    #
    # The version is optional but recommended since it lets you choose
    # the compatibility version. The only supported value currently is 2.0
    #
    # REQUIRED ensures that CMake raises errors if Catalyst is not found
    # properly.

    find_package(catalyst 2.0 REQUIRED)


    # Your simulation will have an executable (or a library) that
    # houses the main-loop in which you'll make the Catalyst API falls.
    # You need to link that executable (or the library) target with Catalyst.
    # This is done as follows (where simulation_target must be replaced by the
    # name of the correct executable (or library) target.

    target_link_library(simulation_target
      PRIVATE catalyst::catalyst)

Now, when you run ``cmake`` on your simulation code, a new cache variable
``catalyst_DIR`` can be set to the directory containing the file ``catalyst-config.cmake``
to help CMake find where you built Catalyst. That file can be found in either the
Catalyst build directory or the Catalyst install directory.


Using `make` (or similar)
-------------------------

If not using CMake as the build system generator for your simulation code, it is
still easy to make it aware of Catalyst. You simply need to pass the include
path i.e. the location where the Catalyst headers are available, and the
location and library to link against.

In a typical Catalyst install at location, ``CATALYST_INSTALL_PREFIX``, these are:

* Include path: ``<CATALYST_INSTALL_PREFIX>/include/catalyst-2.0``
* Library path: ``<CATALYST_INSTALL_PREFIX>/lib``
* Library:      ``<CATALYST_INSTALL_PREFIX>/lib/libcatalyst.so``

Using ``gcc``, for example, this translates to the following command-line:

.. code-block:: bash

    gcc test_driver.c -I<CATALYST_INSTALL_PREFIX>/include/catalyst-2.0 <CATALYST_INSTALL_PREFIX>/lib/libcatalyst.so.3

Catalyst API
============

Catalyst API is used by simulations to invoke Catalyst for co-processing. To use
the Catalyst API, one must include the ``catalyst.h`` header file.

catalyst_initialize
-------------------

.. code-block:: c

  enum catalyst_status catalyst_initialize(const conduit_node* params);

This function must be called once to initialize Catalyst. Metadata that can be
used to configure the initialize is provided using a ``params`` pointer.

The catalyst will attempt to load the implementation named using
``params["catalyst_load/implementation"]``. If not specified, but the
``CATALYST_IMPLEMENTATION_NAME`` environment variable is, it will be used. If
no implementation is named, a default implementation using the stub functions
will be used.

If an implementation is named, it will be loaded at runtime using ``dlopen``
(or the platform equivalent) by searching the nodes specified under the
``params["catalyst_load/search_paths"]`` node. Next, the paths specified by
the ``CATALYST_IMPLEMENTATION_PATHS`` (using ``;`` as a separator on Windows
and ``:`` otherwise) will be searched. Finally, the ``catalyst`` directory
beside ``libcatalyst`` will be searched. Once found, it will be loaded and
inspected for compatibility. If it is compatible, the implementation will be
loaded and made available. The return code indicates the error received, if
any.

The search priority of the ``CATALYST_IMPLEMENTATION_`` environment variables
may be made first by setting teh ``CATALYST_IMPLEMENTATION_PREFER_ENV``
environment variable to a non-empty value.

catalyst_finalize
-----------------

.. code-block:: c

  enum catalyst_status catalyst_finalize(const conduit_node* params);

This function must be called once to finalize Catalyst. Metadata is passed using
``params`` pointer.

catalyst_execute
----------------

.. code-block:: c

  enum catalyst_status catalyst_execute(const conduit_node* params);

This function is called for every time step as the simulation advances. This is
the call in which the analysis may execute. ``params`` provides metadata as well
as the data generated by the simulation for that time-step.


catalyst_about
--------------

.. code-block:: c

  enum catalyst_status catalyst_about(conduit_node* params);

This function fills up the ``params`` instance with metadata about the Catalyst
library being used.

catalyst_results
----------------

.. code-block:: c

  enum catalyst_status catalyst_results(conduit_node* params);

This function fills up the ``params`` instance with updated parameters values
from the Catalyst implementation side.

Asynchronous Execution Control
------------------------------

Catalyst supports asynchronous execution, where ``catalyst_execute()`` returns
immediately while work is processed in the background. Control is via params,
with no new API functions (preserving ABI stability):

**Query async status** via ``catalyst_about()``:

.. code-block:: c

   conduit_node* about = conduit_node_create();
   catalyst_about(about);
   int enabled = conduit_node_fetch_path_as_int64(about, "catalyst/async/enabled");

**Flush pending work** via params to ``catalyst_execute()``:

.. code-block:: c

   conduit_node* params = conduit_node_create();
   conduit_node_set_path_int64(params, "catalyst/async/flush", 1);
   catalyst_execute(params);  // Waits for all pending work

See :doc:`async_execution` for full details on async mode.

All the above functions use a ``params`` object which is a `conduit_node`_. It is
simply a hierarchical mechanism for describing data and/or metadata including
simulation meshes and fields. Essentially, think of it as a map where keys are
strings called paths and values are either data or pointers to data. What these
keys can be and what they mean is totally up to the Catalyst API implementation
being used.

To create and populate the ``conduit_node`` instance, you use the Conduit ``C`` API.
e.g.

.. code-block:: c

  conduit_node* node = conduit_node_create();
  conduit_node_set_path_int(node, "sim/timestep", 0);
  conduit_node_set_path_double(node, "sim/time", 1.212);
  ...
  conduit_node_destroy(node);

Refer to `Conduit`_ documentation for details of the ``C`` API. [TODO: there are
no docs for Conduit C API upstream].

.. _CMake: https://www.cmake.org

.. _make: https://www.gnu.org/software/make/

.. _conduit_node: https://llnl-conduit.readthedocs.io/en/latest/tutorial_cpp_basics.html

.. _Conduit: https://llnl-conduit.readthedocs.io/en/latest/conduit.html
