gridtest.main package¶

Submodules¶

gridtest.main.check module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

gridtest.main.check.check_tests(testfile, include_private=False, include_classes=True, skip_patterns=None)[source]¶

A wrapper to get_missing_tests, but we return 0 if no new tests are to be added, and 1 otherwise.

Arguments:

testfile (str) : the yaml test file
include_private (bool) : include “private” functions
include_classes (bool) : include classes
skip_patterns (list) : list of test keys (patterns) to exclude

gridtest.main.check.get_missing_tests(testfile, include_private=False, skip_patterns=None, include_classes=True)[source]¶

Given a testing file, load in as a GridRunner, load the module again, and check if new tests need to be generated. Optionally take patterns to skip. If no new tests are added, we return 0. Otherwise, we exit with 1. This is similar to black linting, and is intended for running in CI to pass if a user has written all tests to correpond with their module (akin to a more rigorous coverage tool).

Arguments:

testfile (str) : the yaml test file
include_private (bool) : include “private” functions
skip_patterns (list) : list of test keys (patterns) to exclude

gridtest.main.expand module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

gridtest.main.expand.custom_range(start, stop, by=1.0, precision=2)[source]¶

the range function only accepts integers, and user’s will likely want to provide float. Thus we use custom_range to provide this

Arguments:

start (int or float) : the starting value
stop (int or float) : go up to this value
by (float or int) : increment by this value (default 1.0)
precision (int) : decimals to round to (default 2)

gridtest.main.expand.expand_args(args)[source]¶

Given a grid of arguments, expand special cases into longer lists of arguments. E.g., convert an entry with these keys:

into:

In the case that a grid has a string identifier to point to a key in the lookup, we use that listing of values instead that should already be calculated.

gridtest.main.generate module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

gridtest.main.generate.extract_functions(filename, include_private=False, quiet=False, include_classes=True)[source]¶

Given a filename, extract a module and associated functions with it into a grid test. This means creating a structure with function names and (if provided) default inputs. The user will fill in the rest of the file. The function can be used easily recursively by calling itself to get metadata for a subclass, and passing along the (already imported) module.

Arguments:

filename (str) : a filename or module name to parse
include_private (bool) : include “private” functions
quiet (bool) : suppress additional output
include_classes (bool) : extract classes

gridtest.main.generate.extract_modulename(filename, input_dir=None)[source]¶

Extract a module, file, or relative path for a filename. First

Arguments:

filename (str) : a filename or module name to parse
input_dir (str)an input directory with the recipe, in case
of a local file.

gridtest.main.generate.formulate_arg(arg, default=None)[source]¶: Return a data structure (dictionary) with the argument as key, and a default defined, along with a random value to test.

gridtest.main.generate.generate_tests(module, output=None, include_private=False, force=False, include_classes=True)[source]¶

Generate a test output file for some input module. If an output file is specified and already has existing content, in the case that check is used, we only print section names that have not been written. If check is used and the file doesn’t exist, we print the tests to create to the screen. If an existing file is found and check isn’t used, we only update it with tests not yet written. This functionality is provided so that the user can easily update a testing file without erasing old tests. A “module” input variable can be:

a script path explitly

a directory path with files to be recursively discovered

a module name

By default, if a testing file is provided that already has sections defined, they will not be overwritten (but new sections will be added). If the user wants to produce a new (reset) template, the file should be deleted and generate run freshly.

Arguments:

module (str) : a file, directory, or module name to parse
output (str) : a path to a yaml file to save to
include_private (bool) : include “private” functions
force (bool) : force overwrite existing functions (default False)
include_classes (bool) : extract classes to write tests too

gridtest.main.generate.get_function_typing(func)[source]¶

Given a function that is inspected or otherwise present, return a lookup of arguments with any expected default types. This is done at runtime and done as a check, and done here so we don’t need to install mypy.

Arguments:

func (function) : loaded function to return types for

Returns: lookup dictionary of variable names and types. Return

is in the lookup and corresponds to the value of the return.

gridtest.main.generate.import_module(name)[source]¶

Import module will try import of a module based on a name. If import fails and there are no ., we expect that it could be a script in the present working directory and add .<script>

Arguments:

name (str) : the name of the module to import

gridtest.main.generate.include_function(funcname, func, include_classes=True, include_private=False)[source]¶: A helper to determine if a function (or class) should be included. Returns True for yes, False otherwise.

gridtest.main.grids module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class gridtest.main.grids.Grid(name, params, filename='', refs=None)[source]¶

Bases: object

apply_function(funcname, args)[source]¶

Given a function (a name, or a dictionary to derive name and other options from) run some set of input variables (that are taken by the function) through it to derive a result. The result returned is used to set another variable. If a count is defined, we run the function (count) times and return a list. Otherwise, we run it once.

Arguments:

funcname (str or dict) : the function name or definition
args (dict) : lookup of arguments for the function

generate_references()[source]¶: Given a loaded set of references from other grids (self.refs) load them into the current args space.

get_function(funcname)[source]¶: Given a function name, return it. Exit on error if not found.

unwrap_functions()[source]¶: Given that a function is to be unwrapped, this means that we evaluate it first to generate a list that is used to updated args.

gridtest.main.grids.intersect_args(func, args)[source]¶: Given a loaded function and a dictionary of args, return the overlapping set (those that are allowed to be given to the function

gridtest.main.helpers module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class gridtest.main.helpers.Capturing[source]¶

Bases: list

capture output from stdout and stderr into capture object

set_stderr()[source]¶

set_stdout()[source]¶

gridtest.main.helpers.get_function(module, funcname, args, filename)[source]¶: given a module name, function name, argument, and filename, derive a function, optionally deriving an instance first that it might belong to

gridtest.main.helpers.print_interactive(**kwargs)[source]¶: A helper function to print locals that are relevant to test_basic for the user.

gridtest.main.helpers.test_basic(funcname, module, filename, func=None, args=None, returns=None, interactive=False, metrics=None)[source]¶

test basic is a worker version of the task.test_basic function. If a function is not provided, funcname, module, and filename are required to retrieve it. A function can only be provided directly if it is pickle serializable (multiprocessing would require this). It works equivalently but is not attached to a class, and returns a list of values for [passed, result, out, err, raises]

Arguments:

funcname (str) : the name of the function to import
module (str) : the base module to get the function from
func (Function) : if running serial, function can be directly provided
args (dict) : dictionary of arguments
returns (type) : a returns type to test for
interactive (bool) : run in interactive mode (giving user shell)
metrics (list) : one or more metrics (decorators) to run.

gridtest.main.helpers.test_types(func, args=None, returns=None)[source]¶: Given a loaded function, get it’s types and ensure that they are correct. Returns a boolean to indicate correct/ passing (True)

gridtest.main.substitute module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

gridtest.main.substitute.substitute_args(value, params=None)[source]¶: Given a value, determine if it has variable argument substitutions in the format of {{ args.<name> }} and if so, if the argument is present return the value with the substitution.

gridtest.main.substitute.substitute_func(value, funcs=None)[source]¶

Given a value, determine if it contains a function substitution, and if it’s one an important function (e.g., one from gridtest.helpers) return the value with the function applied.

Arguments:

value (str) : the value to do the substitution for.
funcs (dict) : lookup dictionary of functions to be used

Notes:

A function should be in the format: {% tempfile.mkdtemp %} (global import) or a function in gridtest.func in the format {% tmp_path %}. If arguments are supplied, they should be in the format {% tmp_path arg1=1 arg2=2 %}

gridtest.main.test module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class gridtest.main.test.GridRunner(input_file, **kwargs)[source]¶

Bases: object

failed(tests)[source]¶: Given a test of tests, return True if any are not successful.

get_grids()[source]¶: a grid is a specification under “grids” that can be run to parameterize a set of arguments, optionally run through a function or just generated to have combinations. If a count variable is included, we multiply by that many times.

get_tests(regexp=None, verbose=False, cleanup=True)[source]¶

get tests based on a regular expression.

Arguments:

regexp (str) : if provided, only include those tests that match.

iter_grids()[source]¶

iter_tests()[source]¶

load(input_file)[source]¶: load a testing gridtest file.

print_results(tests)[source]¶: print the results of the tests, meaning that success is in green, and non-success is in red.

run(regexp=None, parallel=True, nproc=None, show_progress=True, verbose=False, interactive=False, name=None, cleanup=True, save=None, save_report=None, save_compact=False, save_metrics=None, report_template='report')[source]¶

run the grid runner, meaning that we turn each function and set of tests into a single test, and then run with multiprocessing. This is the function called by the user that also does filtering of tests based on a regular expression before calling them.

Arguments:

regexp (str) : if supplied, filter to this pattern
parallel (bool) : use multiprocessing to run tasks (default True)
show_progress (bool) : show progress instead of task information
nproc (int) : number of processes to use for parallel testing
verbose (bool) : print success output too
interactive (bool) : interactively debug functions
name (str) : if specified, a name of a test to interact with
cleanup (bool) : cleanup files/directories generated with tmp_path tmp_dir
save (str) : a filepath to save results to (must be json)
save_report (str) : path to folder (not existing) to save a report to
report_template (str) : a template name of a report to generate

run_tests(tests, nproc=9, parallel=True, interactive=False, name=None)[source]¶

run tests. By default, we run them in parallel, unless serial is selected.

Arguments:

parallel (bool) : run tasks in parallel that are able (default is True)
nproc (int) : number of processes to run
cleanup (bool) : clean up files/dir generated with tmp_path, tmp_dir
name (str) : the name of a test to interact with
interactive (bool) : run jobs interactively (for debugging) not available for parallel jobs.

save(testfile)[source]¶: Save the runner.config to an output yaml file.

save_metrics(filename, tests, save_compact=False)[source]¶

save metrics to file. This is the same data as a general results export, but without the results, and without the params.

Arguments:

filename (str) : the json file to save to (must end in json)
tests (gridtest.main.test.GridTest) : the gridtest object
save_compact (bool) : don’t pretty print

save_report(report_dir, report_template)[source]¶: save a runner results to file.

save_results(filename, tests, save_compact=False)[source]¶

save a runner results to file.

Arguments:

filename (str) : the json file to save to (must end in json)
tests (gridtest.main.test.GridTest) : the gridtest object
save_compact (bool) : don’t pretty print

set_name(name=None)[source]¶

set a custom name. If the user provides a name to the GridRunner, this name will be used. Otherwise we use the basename of the input file.

Arguments:

name (str): the name of the input file

success(tests)[source]¶: Given a test of tests, return True if all are successful.

class gridtest.main.test.GridTest(module, name, func=None, filename=None, params=None, verbose=False, cleanup=True, show_progress=True)[source]¶

Bases: object

check_equals(statement)[source]¶: check if a result equals some statement.

check_exists(filename)[source]¶: check if a filename exists.

check_isfalse(statement)[source]¶: check if a statement is false

check_isinstance(instance)[source]¶: check if the result is of a particular type

check_istrue(statement)[source]¶: check if a statement is true.

check_metrics()[source]¶: After runs are complete, given metrics defined in params, parse over the list and look for metric output in the output (and remove)

check_output()[source]¶: Given that self.result is defined, check final output for the test. This works (and is called) after self.run(), OR by the multiprocessing worker that has updated self.result. Each of the actions below does additional parsing of the result, and the client will update self.success to be False if there is an issue.

check_raises(exception)[source]¶: Ensure that running a function raises a particular error. If the function runs successfully, this is considered a failure.

check_returns(value)[source]¶: test that a function returns a particular value. The value might reference an input variable, so we use the args dictionary to substitute

cleanup()[source]¶: Given a list of paths (files or folders) generated by gridtest, clean them up with shutil.rm

get_func()[source]¶: Get the function name, meaning we get the module first. This can also be used for one off (custom) function and module names.

get_funcname()[source]¶: Get the function name, meaning we get the module first.

post_substitute()[source]¶: After a run, sometimes we want to check the result (whatever it is)

reset()[source]¶: reset a test to it’s original state, meaning that original parameters, the result, raises, etc. are reset.

run(interactive=False, cleanup=None)[source]¶

run an isolated test, and store the return code and result with the tester here.

Arguments:

interactive (bool) : run interactively for shell to debug
cleanup (bool) : remove any temporary directories or files (True)

set_params(params)[source]¶: Given params with args that are loaded, making substitutions at the onset of generating the test. Also keep track of any directories / files defined by tmp_path and tmp_dir to clean up after the test is run.

substitute(value)[source]¶: Given an input value, return the appropriate substituted string for it. This means that {{ args.x }} references can reference arguments in params, or {% func %} can refer to a function in the gridtest helpers namespace.

property summary¶: print a summary of the test, including if it is supposed to return, raise, or check existance.

summary_failure()[source]¶

summary_success()[source]¶

class gridtest.main.test.GridTestFunc(func, params=None, verbose=False, show_progress=True)[source]¶

Bases: gridtest.main.test.GridTest

a function can be loaded from within Python with GridTestFunc.

gridtest.main.update module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

gridtest.main.update.update_tests(testfile, include_private=False, skip_patterns=None, include_classes=True)[source]¶

Given a testing file, load in as a GridRunner, load the module again, and update with new tests not found. Optionally take patterns to skip. This is akin to check_tests in check.py, but instead we update the runner and save the file.

Arguments:

testfile (str) : the yaml test file
include_private (bool) : include “private” functions
skip_patterns (list) : list of test keys (patterns) to exclude
include_classes (bool) : include classes in update (True)

gridtest.main.workers module¶

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class gridtest.main.workers.Workers(workers=None, show_progress=False)[source]¶

Bases: object

end()[source]¶

run(tests, cleanup=True)[source]¶

run will execute a test for each entry in the list of tests. the result of the test, and error codes, are saved with the test.

Arguments:

tests (gridtest.main.test.GridTest) : the GridTest object

start()[source]¶

gridtest.main.workers.init_worker()[source]¶

gridtest.main.workers.multi_package(func, kwargs)[source]¶

gridtest.main.workers.multi_wrapper(func_args)[source]¶

gridtest.main package¶

Submodules¶

gridtest.main.check module¶

gridtest.main.expand module¶

gridtest.main.generate module¶

gridtest.main.grids module¶

gridtest.main.helpers module¶

gridtest.main.substitute module¶

gridtest.main.test module¶

gridtest.main.update module¶

gridtest.main.workers module¶

Module contents¶