gridtest
A Basic Script
This tutorial will walk through starting with a minimal Python script (with functions and typing) and:
- Generate a test template
- Customize the template
- Run tests.
If you haven’t installed gridtest, you should do this first.
Generate Testing Template
Let’s say that we have this script, called “script.py”
# These are functions in my script
# Typing is here, so Python
def add(one, two):
"""add will add two numbers, one and two. There is no typing here"""
return one + two
def add_with_type(one: int, two: int) -> int:
"""add_with_type will add two numbers, one and two, with typing for ints."""
return one + two
def hello(name):
"""print hello to a name, with no typing"""
print(f"hello {name}!")
def hello_with_default(name="Dinosaur"):
"""print hello to a name with a default"""
print(f"hello {name}!")
def hello_with_type(name: str) -> None:
"""print hello to a name, with typing"""
print(f"hello {name}!")
The first step is to generate your testing template, which needs to be done once, and will generate a yaml file that you can fill in as a template.
$ gridtest generate script.py gridtest.yml2
Extracting add from script
Extracting add_with_type from script
Extracting hello from script
Extracting hello_with_default from script
Extracting hello_with_type from script
The first argument is the input for the generate command, and this can be a filename, a folder name (that might contain multiple scripts) or a python module string (.e.g, requests.get). The second argument is the gridtest output file that will be produced with your tests. After you finish, the output template file (gridtest.yml) will have a list of tests that you can add values for. You can delete sections that aren’t relevant, or copy paste new entries to each list for another testing case. The base template looks like this:
script:
filename: /home/vanessa/Desktop/Code/gridtest/examples/basic/script.py
tests:
script.add:
- args:
one: null
two: null
script.add_with_type:
- args:
one: null
two: null
script.hello:
- args:
name: null
script.hello_with_default:
- args:
name: Dinosaur
script.hello_with_type:
- args:
name: null
As you can see, each function is discovered, and arguments (and any defaults) are provided.
Customize
You can then open the file in a text editor, and add arguments to each. If your library uses typing, the typing will be checked at testing time, and it’s not specified here. You’ll generally want to fill in args for each testing condition (or leave null for None). For example, we might want to change:
script.add:
args:
- one: null
- two: null
to instead be:
script.add:
args:
- one: 1
- two: 2
To test adding 1+2. You should look at the complete options to templates for details on…
This means that we can edit our script from this:
script:
filename: /home/vanessa/Desktop/Code/gridtest/examples/basic/script.py
tests:
script.add:
- args:
one: null
two: null
script.add_with_type:
- args:
one: null
two: null
script.hello:
- args:
name: null
script.hello_with_default:
- args:
name: Dinosaur
script.hello_with_type:
- args:
name: null
to be something more reasonable to test:
script:
filename: /home/vanessa/Desktop/Code/gridtest/examples/basic/script.py
tests:
script.add:
- args:
one: 1
two: 2
returns: 3
- args:
one: 1
two: null
raises: TypeError
script.add_with_type:
- args:
one: 1
two: 2
returns: 3
script.hello:
- args:
name: Vanessa
script.hello_with_default:
- args:
name: Dinosaur
script.hello_with_type:
- args:
name: 1
For typing, given that a function uses typing, that will be tested. For example, the last function “hello_with_type” will not be successful.
Test
Finally, you’ll have your test file, and an environment where you want to test. You can run tests like this:
$ gridtest test gridtest.yml
[6/6] |===================================| 100.0%
Name Status Summary
________________________________________________________________________________________________________________________
script.add.0 success returns 3
script.add.1 success raises TypeError
script.add_with_type.0 success returns 3
script.hello.0 success
script.hello_with_default.0 success
script.hello_with_type.0 success
6/6 tests passed
And since gridtest looks for the gridtest.yml, you can just do:
$ gridtest test
Verbose
You can print a little more output about successes or failures with --verbose
$ gridtest test --verbose
[6/6] |===================================| 100.0%
Name Status Summary
________________________________________________________________________________________________________________________
script.add.0 success returns 3
script.add.1 success Exception: TypeError raised as desired. raises TypeError
script.add_with_type.0 success returns 3
script.hello.0 success hello Vanessa!
script.hello_with_default.0 success hello Dinosaur!
script.hello_with_type.0 success success key set to false, expected failure.
6/6 tests passed
Or you can filter to a regular expression (pattern) to only run a subset of tests:
$ gridtest test --pattern script.add
[3/3] |===================================| 100.0%
Name Status Summary
________________________________________________________________________________________________________________________
script.add.0 success returns 3
script.add.1 success raises TypeError
script.add_with_type.0 success returns 3
3/3 tests passed
Debugging
Now let’s say there is an error in a script. Let’s randomly raise an exception:
def hello(name):
"""print hello to a name, with no typing"""
raise Exception('ruhroh')
If we run tests again, we see a failure with an unexpected exception:
$ gridtest test gridtest.yml
[6/6] |===================================| 100.0%
Name Status Summary
________________________________________________________________________________________________________________________
script.add.0 success returns 3
script.add.1 success raises TypeError
script.add_with_type.0 success returns 3
script.hello.0 failure ruhroh Unexpected Exception: Exception.
script.hello_with_default.0 success
script.hello_with_type.0 success
5/6 tests passed
Adding Interactivity
If we add the --interactive flag, it’s going to allow us to cycle through
every single test and press Control+d to jump to the next test:
$ gridtest test gridtest.yml --interactive
[script.add:1/6] |=====|-----------------------------| 16.7%
Gridtest interactive mode! Press Control+D to cycle to next test.
Variables
func: <function add at 0x7fe4c0a44200>
module: <module 'script' from '/home/vanessa/Desktop/Code/gridtest/examples/basic/script.py'>
args: {'one': 1, 'two': 2}
returns: 3
How to test
passed, error = test_types(func, args, returns)
result = func(**args)
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]:
But that’s not really what we want - we know the failing test is script.hello.0
so let’s run the tests, but only this particular test for interactive:
$ gridtest test gridtest.yml --interactive --name script.hello
[script.add:1/6] |=====|-----------------------------| 16.7% False
[script.add:2/6] |===========|-----------------------| 33.3% False
[script.add_with_type:3/6] |=================|-----------------| 50.0% False
[script.hello:4/6] |=======================|-----------| 66.7% True
Gridtest interactive mode! Press Control+D to cycle to next test.
Variables
func: <function hello at 0x7fc118e8c680>
module: <module 'script' from '/home/vanessa/Desktop/Code/gridtest/examples/basic/script.py'>
args: {'name': 'Vanessa'}
returns: None
How to test
passed, error = test_types(func, args, returns)
result = func(**args)
Python 3.7.4 (default, Aug 13 2019, 20:35:49)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]:
In the example above, we show that the function, module, and arguments for the test are loaded, and you are shown how to run the tests. For example, here is how we would inspect arguments and then test typing:
In [1]: args
Out[1]: {'name': 'Vanessa'}
In [2]: passed, error = test_types(func, args, returns)
In [3]: passed
Out[3]: True
In [4]: error
Out[4]: []
And then run the actual test to trigger the full error:
In [6]: result = func(**args)
hello Vanessa!
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
~/Desktop/Code/gridtest/gridtest/main/helpers.py in <module>
----> 1 result = func(**args)
~/Desktop/Code/gridtest/examples/basic/script.py in hello(name)
18 """print hello to a name, with no typing"""
19 print(f"hello {name}!")
---> 20 raise Exception('ruhroh')
21
22 def hello_with_default(name="Dinosaur"):
Exception: ruhroh
At this point, you can interact with your arguments or the function to debug further.
Specifying Names
In the case of a file with multiple tests (the typical case) You can also specify the name of the test you want to interact with:
$ gridtest test --interactive --name script.add
For the above, this would interact with all tests that start with script.add. If you want to limit to the script.add module, you might want to do:
$ gridtest test --interactive --name script.add.
Or a specific indexed text for the module:
$ gridtest test --interactive --name script.add.0
You might next want to browse tutorials available.