The enb command-line tools
enb
is friendly with the command line interface (CLI). Two main ways of using the CLI
with enb:
Using the enb program from the command line
Setting the values of enb.config.options from the command line when invoking your enb-using scripts.
Each way is explored in the following sections
The enb program
If you installed enb, most likely you can run enb from your command line and access its main CLI. An example output of running enb help (to display usage help) is shown next:
usage: enb [-h] {plugin,show,help} ...
CLI to the experiment notebook (enb) framework (see https://github.com/miguelinux314/experiment-notebook).
Several subcommands are available; use `enb <subcommand> -h` to show help about any specific command.
options:
-h, --help show this help message and exit
subcommands:
Available enb CLI commands.
{plugin,show,help}
plugin Install and manage plugins.
show Show useful information about enb and enb projects.
help Show this help and exit.
Listing available plugins and templates
The enb
library comes packed with several plugins and templates. To access them, one can use enb pluginp.
In particular, enb plugin -h shows all available options.
Use enb plugin list to get a list of all available plugins and templates. You can add extra parameters for filtering and/or -v for extra details.
[38;2;40;201;255mShowing 53 plugins[0m[38;2;40;201;255m.[0m
[38;2;40;201;255mYou can add arguments to filter this list, and/or use the --exclude argument.[0m
[38;2;40;201;255mAdd -v for extra information on the listed plugins[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m analysis-gallery :: [0m[38;2;40;201;255mSelf-contained gallery of data analysis and plotting examples[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m arithmetic_codec :: [0m[38;2;40;201;255mArithmetic codec (8 bit)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m basic-workflow :: [0m[38;2;40;201;255mBasic, self-contained example of enb's workflow[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m bitshuffle :: [0m[38;2;40;201;255mCodec wrapper that applies bitshuffle before another codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m bwt :: [0m[38;2;40;201;255mApplication of the Burrows-Wheeler Transform (BWT)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m ccsds122x1 :: [0m[38;2;40;201;255mWrappers for CCSDS 122.1 (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m ccsds124x0 :: [0m[38;2;40;201;255mWrappers for CCSDS 124.0-B-1 (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m cluster-config :: [0m[38;2;40;201;255mTemplate of a CSV configuration file for enb clusters[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m colors-dark :: [0m[38;2;40;201;255mColor template for dark background terminals[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m colors-default :: [0m[38;2;40;201;255mDefault color template for light or dark background terminals[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m emporda :: [0m[38;2;40;201;255mWrapper for the emporda codec (AC and CCSDS versions)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m enb.ini :: [0m[38;2;40;201;255mCopy a full enb.ini configuration in the destination project folder[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m entropy-codec-comparison :: [0m[38;2;40;201;255mTemplate for the comparison of entropy codecs[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m experiment :: [0m[38;2;40;201;255mGeneric experiment template. Run `enb plugin list experiment` for specific experiment templates[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m fapec :: [0m[38;2;40;201;255mWrappers for FAPEC (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m file_version_example :: [0m[38;2;40;201;255mTemplate and usage example for the FileVersionTable class for dataset curation[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m flif :: [0m[38;2;40;201;255mWrapper for the FLIF compressor[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m fpack :: [0m[38;2;40;201;255mWrapper for the FPACK compressor[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m fpc :: [0m[38;2;40;201;255mFPC codec wrappers[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m fpzip :: [0m[38;2;40;201;255mWrappers for FPZIP codecs[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m fse :: [0m[38;2;40;201;255mWrappers for FSE codecs. Includes a Huffman-only entropy codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m hdf5 :: [0m[38;2;40;201;255mCodec wrappers for the h5py library[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m hevc :: [0m[38;2;40;201;255mWrapper for the HEVC / H.265 codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m huffman :: [0m[38;2;40;201;255mHuffman codec (8 bit)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m iraf_photometry :: [0m[38;2;40;201;255mExtra photometry information from image files[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m jpeg :: [0m[38;2;40;201;255mReference JPEG and JPEG-LS implementation[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m jpegxl :: [0m[38;2;40;201;255mWrapper for a reference JPEG-XL implementation[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m kakadu :: [0m[38;2;40;201;255mWrappers for Kakadu JPEG 2000 (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m lc :: [0m[38;2;40;201;255mLC Framework codec generation and application tools[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m lcnl :: [0m[38;2;40;201;255mWrappers for LCNL CCSDS 123.0-B-2 (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m lossless-compression :: [0m[38;2;40;201;255mTemplate for lossless compression experiments[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m lossy-compression :: [0m[38;2;40;201;255mTemplate for lossy compression experiments[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m lpaq8 :: [0m[38;2;40;201;255mImplementation of the LPAQ8 algorithm[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m lz4 :: [0m[38;2;40;201;255mWrapper for a LZ4 codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m marlin :: [0m[38;2;40;201;255mMarlin-based image compressor[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m matplotlibrc :: [0m[38;2;40;201;255mCopy matplotlib's default rc file into the destination directory[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m mcalic :: [0m[38;2;40;201;255mWrapper for E. Magli et al.'s M-CALIC codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m montecarlo-pi :: [0m[38;2;40;201;255mDemo project that approximates pi in a distributed way[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m montsec :: [0m[38;2;40;201;255mWrapper for the montsec codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m ndzip :: [0m[38;2;40;201;255mWrapper for ndzip[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m png-to-raw :: [0m[38;2;40;201;255mCurate a directory of PNG files into raw (fixed-length) format[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m port-experiment-example :: [0m[38;2;40;201;255mSelf-contained port scanning experiment example[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m ppm-to-raw :: [0m[38;2;40;201;255mCurate a directory of PPM files into raw (fixed-length) format[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m rle :: [0m[38;2;40;201;255mRun-Length Encoding codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m spdp :: [0m[38;2;40;201;255mWrapper for the spdp codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m speck :: [0m[38;2;40;201;255mWrapper for the SPECK codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m test-codecs :: [0m[38;2;40;201;255mInstall all codec plugins verify their availability[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m trpx :: [0m[38;2;40;201;255mTrpx (Terse/Prolix) codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m v2f :: [0m[38;2;40;201;255mWrapper of a codec based on V2F forests (privative)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m vvc :: [0m[38;2;40;201;255mWrapper for the VVC / H.266 codec[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m zfp :: [0m[38;2;40;201;255mWrapper for the zfp library[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m zip :: [0m[38;2;40;201;255mAssortment of lz-based and bzip-based codecs[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m zstd :: [0m[38;2;40;201;255mZstandard library wrapper[0m[38;2;40;201;255m.[0m
[38;2;40;201;255mThe following plugin tags have been defined and can be used for filtering:[0m
[38;2;40;201;255m - documentation ( 8 plugins)[0m[38;2;40;201;255m Documentation examples referenced in the user manual[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m - codec ( 35 plugins)[0m[38;2;40;201;255m Data compression/decompression class definitions[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m - data compression ( 42 plugins)[0m[38;2;40;201;255m Data compression tools[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m - template ( 16 plugins)[0m[38;2;40;201;255m Templates formatteable into the installation dir[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m - project ( 11 plugins)[0m[38;2;40;201;255m Project templates, including configuration files[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m - test ( 1 plugin)[0m[38;2;40;201;255m Plugins for testing purposes[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m - image ( 21 plugins)[0m[38;2;40;201;255m Tools for image processing (including compression and analysis)[0m[38;2;40;201;255m.[0m
[38;2;40;201;255m - privative ( 6 plugins)[0m[38;2;40;201;255m Plugins requiring additional privative software[0m[38;2;40;201;255m.[0m
[38;2;40;201;255mRun with -v for authorship and additional information, with -h for full help.[0m
Plugin and template installation
To install a plugin or template, use the following syntax
enb plugin install <plugin-name> <destination-folder>
For example, enb plugin install zip ./codecs/zip_codecs should produce something similar to:
Installing zip into [...]/codecs/zip_codecs...
Building zip into [...]/codecs/zip_codecs...
Plugin 'zip' successfully installed into './codecs/zip_codecs'.
Note
You can add calls to the enb.plugins.install()
method within your scripts
to automatically install (unless already installed) and import a plugin, e.g.,
enb.plugins.install("zip")
- enb.plugins.install(name, target_dir=None, overwrite=False, automatic_import=True)
Install an Installable by name into target_dir.
- Parameters:
name – name of the installable (e.g., plugin) to be installed. Run enb plugin list in the CLI to get a list of all available installables.
target_dir – If target_dir is None, it is set to plugins/<plugin_name> by default.
overwrite – If overwrite is False and target_dir already exists, no action is taken.
automatic_import – If True, the installable is imported as a module.
CLI with scripts using enb
If your script includes import enb, then you can pass –option=value arguments when invoking it to set the values of enb.config.options.
In addition, to get a list of all of all available options, you can run your script with the -h options. An example option of this is shown next:
usage: basic_workflow.py [-h] [-v] [--ini [INI ...]] [--cpu_limit CPU_LIMIT]
[-f] [-q] [--render_only] [--chunk_size CHUNK_SIZE]
[--repetitions REPETITIONS] [--report_wall_time]
[--force_sanity_checks]
[--selected_columns SELECTED_COLUMNS [SELECTED_COLUMNS ...]]
[--progress_report_period PROGRESS_REPORT_PERIOD]
[--disable_progress_bar]
[--project_root PROJECT_ROOT]
[--base_dataset_dir BASE_DATASET_DIR]
[--persistence_dir PERSISTENCE_DIR]
[--reconstructed_dir RECONSTRUCTED_DIR]
[--compressed_copy_dir COMPRESSED_COPY_DIR]
[--base_version_dataset_dir BASE_VERSION_DATASET_DIR]
[--base_tmp_dir BASE_TMP_DIR]
[--external_bin_base_dir EXTERNAL_BIN_BASE_DIR]
[--plot_dir PLOT_DIR] [--analysis_dir ANALYSIS_DIR]
[--ssh_csv SSH_CSV] [--disable_swap]
[--worker_script_name WORKER_SCRIPT_NAME]
[--preshutdown_wait_seconds PRESHUTDOWN_WAIT_SECONDS]
[--ray_port RAY_PORT]
[--ray_port_count RAY_PORT_COUNT]
[--no_remote_mount_needed]
[--selected_log_level {core,error,warning,message,verbose,informative,debug}]
[--default_print_level {core,error,warning,message,verbose,informative,debug}]
[--log_level_prefix {True,False}]
[--show_prefix_level {core,error,warning,message,verbose,informative,debug}]
[--force_live_progress {True,False}]
A number of options can be set via the command line interface, then accessed
via enb.config.options.property_name. All of them are optional, and may be
interpreted differently by enb core modules and client code.
options:
-h, --help show this help message and exit
General Options:
Group of uncategorized options.
-v, --verbose Be verbose? Repeat for more. Change at any time to
increase the logger's verbosity. (default: 0)
--ini [INI ...], --extra_ini_paths [INI ...]
Additional .ini files to be used to attain file-based
configurations, in addition to the default ones
(system, user and project). If defined more than once,
the last definition sets the list instead of appending
to a common list of extra ini paths. (default: [])
Execution Options:
General execution options.
--cpu_limit CPU_LIMIT
Maximum number of CPUs to use for computation in this
machine See
https://miguelinux314.github.io/experiment-
notebook/cluster_setup.html for details on how to set
the resources employed in remote computation nodes.
(default: None)
-f, --force, --overwrite
Force calculation of pre-existing results, if
available? Note that should an error occur while re-
computing a given index, that index is dropped from
the persistent support. (default: 0)
-q, --quick Perform a quick test with a subset of the input
samples? If specified q>0 times, a subset of the first
q target indices is employed in most get_df methods
from ATable instances (default: 0)
--render_only, --no_new_results
If True, ATable's get_df method relies entirely on the
loaded persistence data, no new rows are computed.
This can be useful to speed up the rendering process,
for instance to try different aesthetic plotting
options. Use this option only if you know you need it.
(default: False)
--chunk_size CHUNK_SIZE
Chunk size used when running ATable's get_df(). Each
processed chunk is made persistent before processing
the next one. This parameter can be used to control
the trade-off between error tolerance and overall
speed. (default: None)
--repetitions REPETITIONS
Number of repetitions when calculating execution
times. This value allows computation of more reliable
execution times in some experiments, but is normally
most representative in combination with -s to use a
single execution process at a time. (default: 1)
--report_wall_time If this flag is activated, the wall time instead of
the CPU time is reported by default by
tcall.get_status_output_time. (default: False)
--force_sanity_checks
If this flag is used, extra sanity checks are
performed by enb during the execution of this script.
The trade-off for rare error condition detection is a
slower execution time. (default: False)
--selected_columns SELECTED_COLUMNS [SELECTED_COLUMNS ...]
List of selected column names for computation. If one
or more column names are provided, all others are
ignored. Multiple columns can be expressed, separated
by spaces. (default: None)
--progress_report_period PROGRESS_REPORT_PERIOD
Default minimum time in seconds between progress
report updates, when get_df() is invoked and
computation is being processed in parallel. (default:
1)
--disable_progress_bar
If this flag is enabled, no progress bar is employed
(useful to minimize the stdout volume of long-running
experiments). (default: False)
Dir Options:
Options regarding default data directories.
--project_root PROJECT_ROOT
Project root path. It should not normally be modified.
(default:
/home/miguelinux/Dropbox/desarrollo/experiment-noteboo
k.git/enb/plugins/template_basic_workflow_example)
--base_dataset_dir BASE_DATASET_DIR
Directory to be used as source of input files for
indices in the get_df method of tables and
experiments. It should be an existing, readable
directory. (default:
/home/miguelinux/Dropbox/desarrollo/experiment-noteboo
k.git/enb/plugins/template_basic_workflow_example/data
sets)
--persistence_dir PERSISTENCE_DIR
Directory where persistence files are to be stored.
(default:
/home/miguelinux/Dropbox/desarrollo/experiment-noteboo
k.git/enb/plugins/template_basic_workflow_example/pers
istence)
--reconstructed_dir RECONSTRUCTED_DIR
Base directory where a copy of the reconstructed
versions of data are to be stored. (default: None)
--compressed_copy_dir COMPRESSED_COPY_DIR
Base directory where a copy of the compressed versions
of data are to be stored. (default: None)
--base_version_dataset_dir BASE_VERSION_DATASET_DIR
Base dir for versioned folders. (default: None)
--base_tmp_dir BASE_TMP_DIR
Temporary dir used for intermediate data storage. This
can be useful when experiments make heavy use of tmp
and memory is limited, avoiding out-of-RAM crashes at
the cost of potentially slower execution time. The dir
is created when defined if necessary. (default:
/dev/shm)
--external_bin_base_dir EXTERNAL_BIN_BASE_DIR
External binary base dir. In case a centralized
repository is defined at the project or system level.
(default: None)
--plot_dir PLOT_DIR Directory to store produced plots. (default:
/home/miguelinux/Dropbox/desarrollo/experiment-noteboo
k.git/enb/plugins/template_basic_workflow_example/plot
s/basic_workflow)
--analysis_dir ANALYSIS_DIR
Directory to store analysis results. (default:
/home/miguelinux/Dropbox/desarrollo/experiment-noteboo
k.git/enb/plugins/template_basic_workflow_example/anal
ysis/basic_workflow)
Ray Options:
Options related to the ray library, used for parallel/distributed
computing only when --ssh_cluster_csv_path (or, equivalently --ssh_csv)
are employed.
--ssh_csv SSH_CSV, --ssh_cluster_csv_path SSH_CSV
Path to the CSV file containing a enb ssh cluster
configuration. See
https://miguelinux314.github.io/experiment-
notebook/installation.html. (default: None)
--disable_swap If this flag is used, then swap memory will not be
allowed by ray. By default, swap memory is enabled.
Note that your system may become unstable if swap
memory is used (specially a big portion thereof).
(default: False)
--worker_script_name WORKER_SCRIPT_NAME
Base name of ray's worker scripts, invoked to run
tasks in parallel processes. You don't need to change
this unless you want to use custom ray workers.
(default: default_worker.py)
--preshutdown_wait_seconds PRESHUTDOWN_WAIT_SECONDS
A wait period can be held before shutting down ray.
This allows displaying messages produced by child
processes (e.g., stack traces) in case of abrupt
termination of enb client code. (default: 0.5)
--ray_port RAY_PORT Ray port and first port that need to be open in case a
cluster is to be set up. Refer to
https://miguelinux314.github.io/experiment-
notebook/installation.html for further information on
this. (default: 11000)
--ray_port_count RAY_PORT_COUNT
Total number of consecutive ports that can be assumed
to be open after `ray_port`. For instance, if
`ray_port` is 11000 and `ray_port_count` is 1000, then
ports 11000-11999 will be used for parallelization and
(if so-configured) enb clusters. (default: 500)
--no_remote_mount_needed
If this flag is used, the calling script's project
root path is assumed to be valid AND synchronized
(e.g., via NFS). By default, remote mounting via sshfs
and vde2 is employed. (default: False)
Logging Options:
Options controlling what and how is printed and/or logged to files.
--selected_log_level {core,error,warning,message,verbose,informative,debug}
Maximum log level / minimum priority required when
printing messages. (default: message)
--default_print_level {core,error,warning,message,verbose,informative,debug}
Selects the default log level equivalent to a regular
print-like message. It is most effective when combined
with log_print set to True. (default: message)
--log_level_prefix {True,False}
If True, logged messages include a prefix, e.g., based
on their priority. (default: True)
--show_prefix_level {core,error,warning,message,verbose,informative,debug}
If True, a prefix indicating the priority level of
each message is shown preceding those messages.
(default: info)
--force_live_progress {True,False}
If True, the progress panels shown by ATable are
forced to be updated live even if the output is not an
interactive terminal. Enabling this option will
significantly increase the size of any log file to
which the output is redirected, because the output
will include the periodic screen writes and deletions
produced, e.g., every second. (default: False)
Note
*.ini files in the project root are searched for to look for default values for the attributes of enb.config.options. You can copy and modify the default enb.ini template to your project if you would like to use file-based configuration.
Note
You can also modify enb.config.options in your code as in enb.config.options.verbose = 5. If you do, this is applied after the CLI parameter recognition, and therefore overwrites any values set via –option=value.
A set of predefined parameters are available in enb.config.options, the single instance
of the enb.config.AllOptions
class.
When running a script that import enb.config, “-h” can be passed as argument to show
a help message with all available options.
All these options (whether provided via the CLI or not) can be read and/or changed with code like the following:
from enb.config import options
print(f"Verbose level: {options.verbose}")