config.yaml specification¶
Terminilogy¶
sectionBy a section we understand either
testingorinstallingsection. A section is a main configuration block which groups together installation or benchmark testing procedures. Asectioncan contain zero or moresteps.Note: Testing section named
installshould contain a configuration for the project installation, compilation or even git cloning. Testing section calledtestcontains configuration for the benchmark testing.stepA step is main unit which can contain
shellcommands, git cloning and more.shellA shell part can contain
bashcommands (multiline line string)
config.yaml example¶
# start of a install section
install:
# first step in the install section
- name: repository-checkout
git:
- url: git@github.com:janhybs/bench-stat.git
# you can also omit shell if there is no need for it
shell: |
echo "By this point, the repository is already cloned"
echo "And checkout out to latest commit"
# seconds step in the install section
- name: compilation-phase
shell: |
cd bench-stat
./configure --prefix=$(pwd)/build
make && make install
# start of a test section
test:
# first step in the test section
- name: testing-phase
shell: |
cd bench-stat
build/O3.out
config.yaml structure¶
Note: keys in [brackets] are optional.
install: # value is list of steps
- name: string # name of the step
[description]: string # description of the step
[enabled]: boolean # default true, if true step is enabled
# will be processed, otherwised will be skipped
[verbose]: boolean # default false, if true shell is started with set-x
[repeat]: int # default 1, number of this step repetition
# (useful for benchmark testing)
[shell]: string # bash commands to be executed
[output]: string # default log+stdout, how should be output
# of the shell be displayed, possible values:
# log - logging to log file only
# stdout - only display output
# log+stdout - combination of both
[container]: string # if set, will execute shell in side container
# value must command(s) which when called will
# start container (docker/singularity), command
# must contain string %s at the end
# %s will be subsituted with a suitable command
#
# examples:
# container: |
# docker run --rm -v $(pwd):$(pwd) -w $(pwd) ubuntu %s
# container: |
# module load singularity
# singularity exec -B /mnt sin.simg %s
[variables]: # complex type, if set, will create build matrix of variables
# detailed explanation below
[collect]: # complex type, if set, will collect results
# detailed explanation below
config.yaml variables specification¶
This field will allow you to create so called build matrix of all possible combinations of the given
variables. It is especially useful when running multiple benchmarks which are basically the same
and only thing which is different are arguments passed to the binary. It this case there is no need
to copy the step in the install section over and over again only to change a single word in shell.
The principle is the same as the build matrix used in a
.travis.yml
You can specify this fields and you can set unlimited amount of variables and their values like this:
variables:
- matrix:
- var-name: [value-1, value-2, value-3]
- foobar: [1, 2, 3, 4]
- test: [cache, io]
The exmaple above will expand to 24 (3 * 4 * 2) individual configurations, variables are available
in the shell field (and in the extra field of a collect field).
A shell field can use these variables with help of placeholders which are in <variable> form,
usage like this:
shell: |
echo "Running test <test> with arguments foobar=<foobar> and var-name=<var-name>"
# the first echo will look like this:
# Running test cache with arguments foobar=1 and var-name=value-1
benchmark/O3.out <test> --foobar=<foobar> -v <var-name>
# the first call of the binary benchmark/O3.out will look like this:
# benchmark/O3.out cache --foobar=1 -v value-1
The value of the variable in a matrix field must be an array. It can be array of strings, ints,
floats, or even dictionaries. The example below will demonstrate usage of dictionaries.
variables:
- matrix:
- foobar:
- foo: 10.65 # the first value
bar: -3.14
- foo: 1.05 # the second value
bar: 42.00
- test: [cache, io]
and usage in shell:
shell: |
echo "Running test <test> with arguments foo=<foobar.foo> and bar=<foobar.bar>"
# the first echo will look like this:
# Running test cache with arguments foo=10.65 and bar=-3.14"
benchmark/O3.out <test> --foo=<foobar> --bar=<var-name>
# the first call of the binary benchmark/O3.out will look like this:
# benchmark/O3.out cache --foo=10.65 --bar=-3.14
There can be multiple matrix fields as well (for when you don’t want all the combinations):
variables:
- matrix:
- benchmark: 01_square_regular_grid
- mesh:
- 1_15662_el
- 2_31498_el
- 4_62302_el
- 8_124498_el
- matrix:
- benchmark: 02_cube_123d
- mesh:
- 1_15786_el
- 2_29365_el
- 3_47367_el
- 4_58803_el
config.yaml collect specification¶
If you specify collect in a step of the install section, CI-HPC framework will automatically
look for the benchmark results in form of json or yaml files. But you have to tell CI-HPC,
what these files are, and how to work with them.
collect:
files: string # value must be a string containing a path specification.
# pathname can be either absolute (like /foo/bar/result.json) or
# relative (like bar/*/*.json), and can contain shell-style wildcards
# double asterisk ** will match any files and zero or
# more directories and subdirectories
# the value is usually something like
# directory/*.json
# more here https://docs.python.org/3.6/library/glob.html
repo: string # path to the repository from which git information is taken
# if set will determine commit, branch and datetime of the current HEAD
[module]: string # a path to the python module which will take care of the parsing and
# storing. There is a generic module which does a decent job, so if
# your result output format can be easily edited, it will work just fine
[move-to]: string # location where matched files can be moved to after the files has
# been processed. This will simply move the files to a location
# so if you have multiple files from single execution with the same name
# they will be overwritten, to avoid that see 'cut-prefix' below
# You can avoiding processing the same results twice.
# (it is recommended to put the files away)
[cut-prefix]: string # if move-to is set, will cut the path prefix of your files
# it is especially useful when your results are located deep structure
# or if they are in a structure, you want to preserve
[extra]: dictionary # after the processing is done, you can add some extra properties on top
# you can even use variables from build matrix.
# for example:
# extra:
# foo: true
# size: <test-size|i>
# will put extra two fields to a document, which is headed to the database
# they will be put in a system field:
# {
# system: {
# foo: true,
# size: 1024,
# ...
# },
# problem: {
# ...
# }
# }
# The second variable size will be that of the typo of integer
# this is bacause |i was specified at the end. All possible
# conversions are:
# |s for string (default)
# |i for integer
# |f for floats
[save-to-db]: boolean # if true, will save the processed results to the DB