The enb command-line tools

enb is friendly with the command line interface (CLI). Two main ways of using the CLI with enb:

Using the enb program from the command line
Setting the values of enb.config.options from the command line when invoking your enb-using scripts.

Each way is explored in the following sections

The enb program

If you installed enb, most likely you can run enb from your command line and access its main CLI. An example output of running enb help (to display usage help) is shown next:

usage: enb [-h] {plugin,show,help} ...

CLI to the experiment notebook (enb) framework (see https://github.com/miguelinux314/experiment-notebook).

Several subcommands are available; use `enb <subcommand> -h` to show help about any specific command.

options:
  -h, --help          show this help message and exit

subcommands:
  Available enb CLI commands.

  {plugin,show,help}
    plugin            Install and manage plugins.
    show              Show useful information about enb and enb projects.
    help              Show this help and exit.

Listing available plugins and templates

The enb library comes packed with several plugins and templates. To access them, one can use enb pluginp. In particular, enb plugin -h shows all available options.

Use enb plugin list to get a list of all available plugins and templates. You can add extra parameters for filtering and/or -v for extra details.

[38;2;40;201;255mShowing 53 plugins[0m[38;2;40;201;255m.[0m
[38;2;40;201;255mYou can add arguments to filter this list, and/or use the --exclude argument.[0m
[38;2;40;201;255mAdd -v for extra information on the listed plugins[0m[38;2;40;201;255m.[0m

[38;2;40;201;255m         analysis-gallery :: [0m[38;2;40;201;255mSelf-contained gallery of data analysis and plotting examples[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m         arithmetic_codec :: [0m[38;2;40;201;255mArithmetic codec (8 bit)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m           basic-workflow :: [0m[38;2;40;201;255mBasic, self-contained example of enb's workflow[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m               bitshuffle :: [0m[38;2;40;201;255mCodec wrapper that applies bitshuffle before another codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      bwt :: [0m[38;2;40;201;255mApplication of the Burrows-Wheeler Transform (BWT)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m               ccsds122x1 :: [0m[38;2;40;201;255mWrappers for CCSDS 122.1 (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m               ccsds124x0 :: [0m[38;2;40;201;255mWrappers for CCSDS 124.0-B-1 (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m           cluster-config :: [0m[38;2;40;201;255mTemplate of a CSV configuration file for enb clusters[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m              colors-dark :: [0m[38;2;40;201;255mColor template for dark background terminals[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m           colors-default :: [0m[38;2;40;201;255mDefault color template for light or dark background terminals[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                  emporda :: [0m[38;2;40;201;255mWrapper for the emporda codec (AC and CCSDS versions)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                  enb.ini :: [0m[38;2;40;201;255mCopy a full enb.ini configuration in the destination project folder[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m entropy-codec-comparison :: [0m[38;2;40;201;255mTemplate for the comparison of entropy codecs[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m               experiment :: [0m[38;2;40;201;255mGeneric experiment template. Run `enb plugin list experiment` for specific experiment templates[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                    fapec :: [0m[38;2;40;201;255mWrappers for FAPEC (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m     file_version_example :: [0m[38;2;40;201;255mTemplate and usage example for the FileVersionTable class for dataset curation[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                     flif :: [0m[38;2;40;201;255mWrapper for the FLIF compressor[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                    fpack :: [0m[38;2;40;201;255mWrapper for the FPACK compressor[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      fpc :: [0m[38;2;40;201;255mFPC codec wrappers[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                    fpzip :: [0m[38;2;40;201;255mWrappers for FPZIP codecs[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      fse :: [0m[38;2;40;201;255mWrappers for FSE codecs. Includes a Huffman-only entropy codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                     hdf5 :: [0m[38;2;40;201;255mCodec wrappers for the h5py library[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                     hevc :: [0m[38;2;40;201;255mWrapper for the HEVC / H.265 codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                  huffman :: [0m[38;2;40;201;255mHuffman codec (8 bit)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m          iraf_photometry :: [0m[38;2;40;201;255mExtra photometry information from image files[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                     jpeg :: [0m[38;2;40;201;255mReference JPEG and JPEG-LS implementation[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                   jpegxl :: [0m[38;2;40;201;255mWrapper for a reference JPEG-XL implementation[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                   kakadu :: [0m[38;2;40;201;255mWrappers for Kakadu JPEG 2000 (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                       lc :: [0m[38;2;40;201;255mLC Framework codec generation and application tools[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                     lcnl :: [0m[38;2;40;201;255mWrappers for LCNL CCSDS 123.0-B-2 (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m     lossless-compression :: [0m[38;2;40;201;255mTemplate for lossless compression experiments[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m        lossy-compression :: [0m[38;2;40;201;255mTemplate for lossy compression experiments[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                    lpaq8 :: [0m[38;2;40;201;255mImplementation of the LPAQ8 algorithm[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      lz4 :: [0m[38;2;40;201;255mWrapper for a LZ4 codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                   marlin :: [0m[38;2;40;201;255mMarlin-based image compressor[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m             matplotlibrc :: [0m[38;2;40;201;255mCopy matplotlib's default rc file into the destination directory[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                   mcalic :: [0m[38;2;40;201;255mWrapper for E. Magli et al.'s M-CALIC codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m            montecarlo-pi :: [0m[38;2;40;201;255mDemo project that approximates pi in a distributed way[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                  montsec :: [0m[38;2;40;201;255mWrapper for the montsec codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                    ndzip :: [0m[38;2;40;201;255mWrapper for ndzip[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m               png-to-raw :: [0m[38;2;40;201;255mCurate a directory of PNG files into raw (fixed-length) format[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m  port-experiment-example :: [0m[38;2;40;201;255mSelf-contained port scanning experiment example[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m               ppm-to-raw :: [0m[38;2;40;201;255mCurate a directory of PPM files into raw (fixed-length) format[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      rle :: [0m[38;2;40;201;255mRun-Length Encoding codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                     spdp :: [0m[38;2;40;201;255mWrapper for the spdp codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                    speck :: [0m[38;2;40;201;255mWrapper for the SPECK codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m              test-codecs :: [0m[38;2;40;201;255mInstall all codec plugins verify their availability[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                     trpx :: [0m[38;2;40;201;255mTrpx (Terse/Prolix) codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      v2f :: [0m[38;2;40;201;255mWrapper of a codec based on V2F forests (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      vvc :: [0m[38;2;40;201;255mWrapper for the VVC / H.266 codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      zfp :: [0m[38;2;40;201;255mWrapper for the zfp library[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                      zip :: [0m[38;2;40;201;255mAssortment of lz-based and bzip-based codecs[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m                     zstd :: [0m[38;2;40;201;255mZstandard library wrapper[0m[38;2;40;201;255m.[0m

[38;2;40;201;255mThe following plugin tags have been defined and can be used for filtering:[0m

[38;2;40;201;255m  - documentation    (  8 plugins)[0m[38;2;40;201;255m Documentation examples referenced in the user manual[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m  - codec            ( 35 plugins)[0m[38;2;40;201;255m Data compression/decompression class definitions[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m  - data compression ( 42 plugins)[0m[38;2;40;201;255m Data compression tools[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m  - template         ( 16 plugins)[0m[38;2;40;201;255m Templates formatteable into the installation dir[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m  - project          ( 11 plugins)[0m[38;2;40;201;255m Project templates, including configuration files[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m  - test             (  1  plugin)[0m[38;2;40;201;255m Plugins for testing purposes[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m  - image            ( 21 plugins)[0m[38;2;40;201;255m Tools for image processing (including compression and analysis)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m  - privative        (  6 plugins)[0m[38;2;40;201;255m Plugins requiring additional privative software[0m[38;2;40;201;255m.[0m

[38;2;40;201;255mRun with -v for authorship and additional information, with -h for full help.[0m

Plugin and template installation

To install a plugin or template, use the following syntax

enb plugin install <plugin-name> <destination-folder>

For example, enb plugin install zip ./codecs/zip_codecs should produce something similar to:

Installing zip into [...]/codecs/zip_codecs...
Building zip into [...]/codecs/zip_codecs...
Plugin 'zip' successfully installed into './codecs/zip_codecs'.

Note

You can add calls to the enb.plugins.install() method within your scripts to automatically install (unless already installed) and import a plugin, e.g.,

enb.plugins.install("zip")
enb.plugins.install(name, target_dir=None, overwrite=False, automatic_import=True)

Install an Installable by name into target_dir.

Parameters:

name – name of the installable (e.g., plugin) to be installed. Run enb plugin list in the CLI to get a list of all available installables.

target_dir – If target_dir is None, it is set to plugins/<plugin_name> by default.

overwrite – If overwrite is False and target_dir already exists, no action is taken.

automatic_import – If True, the installable is imported as a module.

CLI with scripts using enb

If your script includes import enb, then you can pass –option=value arguments when invoking it to set the values of enb.config.options.

In addition, to get a list of all of all available options, you can run your script with the -h options. An example option of this is shown next:

usage: basic_workflow.py [-h] [-v] [--ini [INI ...]] [--cpu_limit CPU_LIMIT]
                         [-f] [-q] [--render_only] [--chunk_size CHUNK_SIZE]
                         [--repetitions REPETITIONS] [--report_wall_time]
                         [--force_sanity_checks]
                         [--selected_columns SELECTED_COLUMNS [SELECTED_COLUMNS ...]]
                         [--progress_report_period PROGRESS_REPORT_PERIOD]
                         [--disable_progress_bar]
                         [--project_root PROJECT_ROOT]
                         [--base_dataset_dir BASE_DATASET_DIR]
                         [--persistence_dir PERSISTENCE_DIR]
                         [--reconstructed_dir RECONSTRUCTED_DIR]
                         [--compressed_copy_dir COMPRESSED_COPY_DIR]
                         [--base_version_dataset_dir BASE_VERSION_DATASET_DIR]
                         [--base_tmp_dir BASE_TMP_DIR]
                         [--external_bin_base_dir EXTERNAL_BIN_BASE_DIR]
                         [--plot_dir PLOT_DIR] [--analysis_dir ANALYSIS_DIR]
                         [--ssh_csv SSH_CSV] [--disable_swap]
                         [--worker_script_name WORKER_SCRIPT_NAME]
                         [--preshutdown_wait_seconds PRESHUTDOWN_WAIT_SECONDS]
                         [--ray_port RAY_PORT]
                         [--ray_port_count RAY_PORT_COUNT]
                         [--no_remote_mount_needed]
                         [--selected_log_level {core,error,warning,message,verbose,informative,debug}]
                         [--default_print_level {core,error,warning,message,verbose,informative,debug}]
                         [--log_level_prefix {True,False}]
                         [--show_prefix_level {core,error,warning,message,verbose,informative,debug}]
                         [--force_live_progress {True,False}]

A number of options can be set via the command line interface, then accessed
via enb.config.options.property_name. All of them are optional, and may be
interpreted differently by enb core modules and client code.

options:
  -h, --help            show this help message and exit

General Options:
  Group of uncategorized options.

  -v, --verbose         Be verbose? Repeat for more. Change at any time to
                        increase the logger's verbosity. (default: 0)
  --ini [INI ...], --extra_ini_paths [INI ...]
                        Additional .ini files to be used to attain file-based
                        configurations, in addition to the default ones
                        (system, user and project). If defined more than once,
                        the last definition sets the list instead of appending
                        to a common list of extra ini paths. (default: [])

Execution Options:
  General execution options.

  --cpu_limit CPU_LIMIT
                        Maximum number of CPUs to use for computation in this
                        machine See
                        https://miguelinux314.github.io/experiment-
                        notebook/cluster_setup.html for details on how to set
                        the resources employed in remote computation nodes.
                        (default: None)
  -f, --force, --overwrite
                        Force calculation of pre-existing results, if
                        available? Note that should an error occur while re-
                        computing a given index, that index is dropped from
                        the persistent support. (default: 0)
  -q, --quick           Perform a quick test with a subset of the input
                        samples? If specified q>0 times, a subset of the first
                        q target indices is employed in most get_df methods
                        from ATable instances (default: 0)
  --render_only, --no_new_results
                        If True, ATable's get_df method relies entirely on the
                        loaded persistence data, no new rows are computed.
                        This can be useful to speed up the rendering process,
                        for instance to try different aesthetic plotting
                        options. Use this option only if you know you need it.
                        (default: False)
  --chunk_size CHUNK_SIZE
                        Chunk size used when running ATable's get_df(). Each
                        processed chunk is made persistent before processing
                        the next one. This parameter can be used to control
                        the trade-off between error tolerance and overall
                        speed. (default: None)
  --repetitions REPETITIONS
                        Number of repetitions when calculating execution
                        times. This value allows computation of more reliable
                        execution times in some experiments, but is normally
                        most representative in combination with -s to use a
                        single execution process at a time. (default: 1)
  --report_wall_time    If this flag is activated, the wall time instead of
                        the CPU time is reported by default by
                        tcall.get_status_output_time. (default: False)
  --force_sanity_checks
                        If this flag is used, extra sanity checks are
                        performed by enb during the execution of this script.
                        The trade-off for rare error condition detection is a
                        slower execution time. (default: False)
  --selected_columns SELECTED_COLUMNS [SELECTED_COLUMNS ...]
                        List of selected column names for computation. If one
                        or more column names are provided, all others are
                        ignored. Multiple columns can be expressed, separated
                        by spaces. (default: None)
  --progress_report_period PROGRESS_REPORT_PERIOD
                        Default minimum time in seconds between progress
                        report updates, when get_df() is invoked and
                        computation is being processed in parallel. (default:
                        1)
  --disable_progress_bar
                        If this flag is enabled, no progress bar is employed
                        (useful to minimize the stdout volume of long-running
                        experiments). (default: False)

Dir Options:
  Options regarding default data directories.

  --project_root PROJECT_ROOT
                        Project root path. It should not normally be modified.
                        (default:
                        /home/miguelinux/Dropbox/desarrollo/experiment-noteboo
                        k.git/enb/plugins/template_basic_workflow_example)
  --base_dataset_dir BASE_DATASET_DIR
                        Directory to be used as source of input files for
                        indices in the get_df method of tables and
                        experiments. It should be an existing, readable
                        directory. (default:
                        /home/miguelinux/Dropbox/desarrollo/experiment-noteboo
                        k.git/enb/plugins/template_basic_workflow_example/data
                        sets)
  --persistence_dir PERSISTENCE_DIR
                        Directory where persistence files are to be stored.
                        (default:
                        /home/miguelinux/Dropbox/desarrollo/experiment-noteboo
                        k.git/enb/plugins/template_basic_workflow_example/pers
                        istence)
  --reconstructed_dir RECONSTRUCTED_DIR
                        Base directory where a copy of the reconstructed
                        versions of data are to be stored. (default: None)
  --compressed_copy_dir COMPRESSED_COPY_DIR
                        Base directory where a copy of the compressed versions
                        of data are to be stored. (default: None)
  --base_version_dataset_dir BASE_VERSION_DATASET_DIR
                        Base dir for versioned folders. (default: None)
  --base_tmp_dir BASE_TMP_DIR
                        Temporary dir used for intermediate data storage. This
                        can be useful when experiments make heavy use of tmp
                        and memory is limited, avoiding out-of-RAM crashes at
                        the cost of potentially slower execution time. The dir
                        is created when defined if necessary. (default:
                        /dev/shm)
  --external_bin_base_dir EXTERNAL_BIN_BASE_DIR
                        External binary base dir. In case a centralized
                        repository is defined at the project or system level.
                        (default: None)
  --plot_dir PLOT_DIR   Directory to store produced plots. (default:
                        /home/miguelinux/Dropbox/desarrollo/experiment-noteboo
                        k.git/enb/plugins/template_basic_workflow_example/plot
                        s/basic_workflow)
  --analysis_dir ANALYSIS_DIR
                        Directory to store analysis results. (default:
                        /home/miguelinux/Dropbox/desarrollo/experiment-noteboo
                        k.git/enb/plugins/template_basic_workflow_example/anal
                        ysis/basic_workflow)

Ray Options:
  Options related to the ray library, used for parallel/distributed
  computing only when --ssh_cluster_csv_path (or, equivalently --ssh_csv)
  are employed.

  --ssh_csv SSH_CSV, --ssh_cluster_csv_path SSH_CSV
                        Path to the CSV file containing a enb ssh cluster
                        configuration. See
                        https://miguelinux314.github.io/experiment-
                        notebook/installation.html. (default: None)
  --disable_swap        If this flag is used, then swap memory will not be
                        allowed by ray. By default, swap memory is enabled.
                        Note that your system may become unstable if swap
                        memory is used (specially a big portion thereof).
                        (default: False)
  --worker_script_name WORKER_SCRIPT_NAME
                        Base name of ray's worker scripts, invoked to run
                        tasks in parallel processes. You don't need to change
                        this unless you want to use custom ray workers.
                        (default: default_worker.py)
  --preshutdown_wait_seconds PRESHUTDOWN_WAIT_SECONDS
                        A wait period can be held before shutting down ray.
                        This allows displaying messages produced by child
                        processes (e.g., stack traces) in case of abrupt
                        termination of enb client code. (default: 0.5)
  --ray_port RAY_PORT   Ray port and first port that need to be open in case a
                        cluster is to be set up. Refer to
                        https://miguelinux314.github.io/experiment-
                        notebook/installation.html for further information on
                        this. (default: 11000)
  --ray_port_count RAY_PORT_COUNT
                        Total number of consecutive ports that can be assumed
                        to be open after `ray_port`. For instance, if
                        `ray_port` is 11000 and `ray_port_count` is 1000, then
                        ports 11000-11999 will be used for parallelization and
                        (if so-configured) enb clusters. (default: 500)
  --no_remote_mount_needed
                        If this flag is used, the calling script's project
                        root path is assumed to be valid AND synchronized
                        (e.g., via NFS). By default, remote mounting via sshfs
                        and vde2 is employed. (default: False)

Logging Options:
  Options controlling what and how is printed and/or logged to files.

  --selected_log_level {core,error,warning,message,verbose,informative,debug}
                        Maximum log level / minimum priority required when
                        printing messages. (default: message)
  --default_print_level {core,error,warning,message,verbose,informative,debug}
                        Selects the default log level equivalent to a regular
                        print-like message. It is most effective when combined
                        with log_print set to True. (default: message)
  --log_level_prefix {True,False}
                        If True, logged messages include a prefix, e.g., based
                        on their priority. (default: True)
  --show_prefix_level {core,error,warning,message,verbose,informative,debug}
                        If True, a prefix indicating the priority level of
                        each message is shown preceding those messages.
                        (default: info)
  --force_live_progress {True,False}
                        If True, the progress panels shown by ATable are
                        forced to be updated live even if the output is not an
                        interactive terminal. Enabling this option will
                        significantly increase the size of any log file to
                        which the output is redirected, because the output
                        will include the periodic screen writes and deletions
                        produced, e.g., every second. (default: False)

Note

*.ini files in the project root are searched for to look for default values for the attributes of enb.config.options. You can copy and modify the default enb.ini template to your project if you would like to use file-based configuration.

Note

You can also modify enb.config.options in your code as in enb.config.options.verbose = 5. If you do, this is applied after the CLI parameter recognition, and therefore overwrites any values set via –option=value.

A set of predefined parameters are available in enb.config.options, the single instance of the enb.config.AllOptions class. When running a script that import enb.config, “-h” can be passed as argument to show a help message with all available options.

All these options (whether provided via the CLI or not) can be read and/or changed with code like the following:

from enb.config import options
print(f"Verbose level: {options.verbose}")