gridtest

Getting Started

Grids

GridTest isn’t just for testing! In fact, you can write a file of grid specifications that can be loaded and used for parameterization, or your own custom functions. The description here will follow example the files in the grids examples folder. Here is a peek at the top of this file, where we define two grids:

mygrids:
  grids:

    # A grid that will generate 10 empty set of arguments
    generate_empty:
      count: 10

    # A grid that will generate each cross of x and y (total of 9)
    generate_matrix:
      args:
        x: [1, 2, 3] 
        y: [1, 2, 3]

Loading via a GridRunner

Let’s say that we have a grids.yml file with grids defined. We can load with gridtest easily via the GridRunner:

from gridtest.main.test import GridRunner
runner = GridRunner('grids.yml')
# [gridtest|grids.yml]

We can easily generate the grids as follows:

runner.get_grids()
{'generate_empty': [grid|generate_empty],
 'generate_matrix': [grid|generate_matrix],
 'generate_lists_matrix': [grid|generate_lists_matrix],
 'generate_by_min_max': [grid|generate_by_min_max],
 'generate_by_min_max_twovars': [grid|generate_by_min_max_twovars]}

Loading as a Grid

A Grid is a first class citizen, so you can also generate without the GridRunner, either via a json object or yaml you’ve loaded:

from gridtest.main.grids import Grid

grid = Grid(name="mygrid", params={"args": {"one": [1, 11, 111], "two": 2}})
[grid|mygrid]

Cached Loading

By default, grids are generated on demand, meaning that the argument sets (optionally derived by functions you have provided under params) are generated at this point in time. For example:

for argset in grid:
    print(argset)

{'one': 1, 'two': 2}
{'one': 11, 'two': 2}
{'one': 111, 'two': 2}

This also means that, by default, your grid.argsets will be empty when you instantiate it.

grid = Grid(name="mygrid", params={"args": {"one": [1, 11, 111], "two": 2}})
grid.argsets
[]

This is done intentionally because it can sometimes take a lot of memory to store a long list of argument sets. However, if you want to generate the argument sets on init, just set “cache” to True in your params:

grid = Grid(name="mygrid", params={"cache": True, "args": {"one": [1, 11, 111], "two": 2}})

You should then be able to see the argsets ready for your use!

grid.argsets
[{'one': 1, 'two': 2}, {'one': 11, 'two': 2}, {'one': 111, 'two': 2}]

Grid Functions

What if you want to generate a long list of items for an argument, and it would be crazy to type them out? This is what the functions specification for a grid is for. You would provide your functions, each associated with an argument, under the “functions” section. As an example, let’s say we want to run random.choice, and select from a list defined under the argument “choices”:

import random
grid = Grid(name="mygrid", params={"functions": {"pid": random.choice}, "args": {"seq": [[1,2,3,4,5,6,7]]}})

The above would look like this in yaml:

grids:
  mygrid:
    args:
      seq: [[1,2,3,4,5,6,7]]
    functions:
      pid: random.choice

And we would generate our argument sets:

list(grid)
[{'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 2}]

This also points out another important point - if you were to provide a list for an argument, it would be parameterized. If you want the entire list treated as one variable, then define it within another list as shown above. The above grid says “generate the argument pid by using random.choice to select from seq.” How does the grid know to match seq to random.choice? It’s a known keyword argument! Another insight here is that although we define a sequence argument, we don’t actually need it, we really only care about the result (pid). But let’s say we want to generate 10 x a pid, how do we do that? We add a count to the params:

grid = Grid(name="mygrid", params={"count": 10, "functions": {"pid": random.choice}, "args": {"seq": [[1,2,3,4,5,6,7]]}})
list(grid)

[{'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 5},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 1},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 3},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 7},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 5},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 7},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 1},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 4},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 1},
 {'seq': [1, 2, 3, 4, 5, 6, 7], 'pid': 3}]

These values are being generated by an iterator and not saved to any single list in memory, so although it seems very rundant to see the same list more than once, it shouldn’t serve significant issues with respect to memory.

Viewing on the Command Line

You can also preview the grids generated from the command line. First you might want to list all grids defined in a file:

$ gridtest gridview grids.yml
generate_empty
generate_matrix
generate_lists_matrix
generate_by_min_max
generate_by_min_max_twovars

and then inspect a specific named grid:

$ gridtest gridview grids.yml generate_empty
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}

If you want to export your entire grid to json, possibly to read in as-is later and use, add –export:

$  gridtest gridview grids.yml generate_by_min_max --export exported.json 

cat exported.json
[
    {
        "x": 0.0
    },
    {
        "x": 2.0
    },
    {
        "x": 4.0
    },
    {
        "x": 6.0
    },
    {
        "x": 8.0
    }
]

That’s not a really interesting grid, but you get the gist. Each of the grids listed above will be explained below in more detail.

Writing Grids

Let’s start with the most basic of grids, which are those that don’t import any special functions.

The header should have a named section for the grids that you want to define (e.g. mygrids) and then a grids index, under which we will write each of our named grids. Here is the start:

mygrids:
  grids:
  ...

Then each grid is added as a named section below that:

mygrids:
  grids:

    # A grid that will generate 10 empty set of arguments
    generate_empty:
      count: 10

The full content of this file will be written to the grids.yml example here. Each example grid is discussed below. For each example, you can preview the grid in the terminal with gridtest gridview or obtain the grid by instantiating the GridRunner as shown above.

Empty

It could be that you need to generate empty lists of arguments for a function, in which case the count variable will be useful to you. Here is how to specify a grid that will generate 10 empty set of arguments

generate_empty:
  count: 10

The result comes out to be:

$ gridtest gridview grids.yml generate_empty
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}

Parameterize Variables

Let’s say that we have two variables, x and y, and we want to generate a grid of all possible combinations for a listing of each. That would look like this:

    generate_matrix:
      args:
        x: [1, 2, 3] 
        y: [1, 2, 3]

And the resulting grid will have 3x3 or 9 total combinations of x and y:

$ gridtest gridview grids.yml generate_matrix
{'x': 1, 'y': 1}
{'x': 1, 'y': 2}
{'x': 1, 'y': 3}
{'x': 2, 'y': 1}
{'x': 2, 'y': 2}
{'x': 2, 'y': 3}
{'x': 3, 'y': 1}
{'x': 3, 'y': 2}
{'x': 3, 'y': 3}

Parameterize Lists

If you want to do similar but instead have a list of values be paramaterized, just specify a list of lists instead.

    generate_lists_matrix:
      args:
        x: [[1, 2, 3], [4, 5, 6]] 
        y: [[1, 2, 3], [4, 5, 6]]

The result will have 2x2 or 4 entries:

$ gridtest gridview grids.yml generate_lists_matrix
{'x': [1, 2, 3], 'y': [1, 2, 3]}
{'x': [1, 2, 3], 'y': [4, 5, 6]}
{'x': [4, 5, 6], 'y': [1, 2, 3]}
{'x': [4, 5, 6], 'y': [4, 5, 6]}

Here is an easier way to check the count:

$ gridtest gridview grids.yml generate_lists_matrix --count
4 argument sets produced.

Range of Values

For most use cases, you’ll want to generate a list of values over a range. You can do that with min and max and (optionally) by that defaults to 1.

    generate_by_min_max:
      args:
        x:
          min: 0
          max: 10
          by: 2

$ gridtest gridview grids.yml generate_by_min_max
{'x': 0.0}
{'x': 2.0}
{'x': 4.0}
{'x': 6.0}
{'x': 8.0

Here is an example with two variables:

    generate_by_min_max_twovars:
      args:
        x:
          min: 0
          max: 10
          by: 2
        y:
          min: 10
          max: 20
          by: 2

$ gridtest gridview grids.yml generate_by_min_max_twovars 
{'x': 0.0, 'y': 10.0}
{'x': 0.0, 'y': 12.0}
{'x': 0.0, 'y': 14.0}
{'x': 0.0, 'y': 16.0}
{'x': 0.0, 'y': 18.0}
{'x': 2.0, 'y': 10.0}
{'x': 2.0, 'y': 12.0}
{'x': 2.0, 'y': 14.0}
{'x': 2.0, 'y': 16.0}
{'x': 2.0, 'y': 18.0}
{'x': 4.0, 'y': 10.0}
{'x': 4.0, 'y': 12.0}
{'x': 4.0, 'y': 14.0}
{'x': 4.0, 'y': 16.0}
{'x': 4.0, 'y': 18.0}
{'x': 6.0, 'y': 10.0}
{'x': 6.0, 'y': 12.0}
{'x': 6.0, 'y': 14.0}
{'x': 6.0, 'y': 16.0}
{'x': 6.0, 'y': 18.0}
{'x': 8.0, 'y': 10.0}
{'x': 8.0, 'y': 12.0}
{'x': 8.0, 'y': 14.0}
{'x': 8.0, 'y': 16.0}
{'x': 8.0, 'y': 18.0}

Logically there are the previous number of tests, but squared.

$ gridtest gridview grids.yml generate_by_min_max_twovars --count
25 argument sets produced.

Grids with Functions

What does it mean to use a function?

You can map functions to parametrs in a grid, meaning that the values for the parameters will be generated by the function. For example, let’s use the function random.choice to dynamically generic a grid of parameters. The grid below will call random.choice ten times (count is set to 10) across the sequence of values [1, 2, 3], which is an input key word argument to random choice.

...
  grids:
    random_choice:
      count: 10
      functions: 
        pid: random.choice
      args:
         seq: [[1, 2, 3]]

Notice that the sequence argument input is a list of lists, and this is because we want the entire list to be treated as an argument. If we run this from it’s respective file, we get a result with 10 argument sets:

$ gridtest gridview grids-with-function.yml random_choice
{'seq': [1, 2, 3], 'pid': 2}
{'seq': [1, 2, 3], 'pid': 2}
{'seq': [1, 2, 3], 'pid': 2}
{'seq': [1, 2, 3], 'pid': 1}
{'seq': [1, 2, 3], 'pid': 3}
{'seq': [1, 2, 3], 'pid': 1}
{'seq': [1, 2, 3], 'pid': 3}
{'seq': [1, 2, 3], 'pid': 3}
{'seq': [1, 2, 3], 'pid': 2}
{'seq': [1, 2, 3], 'pid': 3}

Grids with Custom Functions

If you are using a script for any of your grids that isn’t a system installed module, then (akin to a standard gridtest) it needs to be included under a section header that is named by the relevant module, and that includes the filename to import. For example, the file script.py in the present working directory has a function, get_pokemon_id that I want to use. Here is how I’d write the recipe:

script: filename: script.py grids:

# A grid that will generate 10 random values using a custom function
generate_pids:
  functions: 
    pid: script.get_pokemon_id
  count: 10 ```

Now let’s run it!

$ gridtest gridview grids-with-function.yml generate_pids
{'pid': '125'}
{'pid': '780'}
{'pid': '508'}
{'pid': '566'}
{'pid': '803'}
{'pid': '513'}
{'pid': '854'}
{'pid': '405'}
{'pid': '639'}
{'pid': '353'}

Akin to the previous example, since we’ve provided a function to pass our grid arguments into, the results are returned. This is how gridtest can use a grid specified under a test to generate a list of values for an argument.

Grids with Unwrapped Functions

Let’s say that we have some function that produces a list of lists, and we want to use that as a list to be parameterized for another grid. First, here is our function:

And here is how we might define it in the grid:

    # Generate a list of lists, intended to be unwrapped and used for another grid
    unwrapped_grid:
      functions:
        numbers: 
          func: script.generate_numbers
          unwrap: true

The “unwrap” serves to take the list, and unwrap it so that each result can be used separately. If we generate the grid, the parameterization is done with any additional arguments. Since we don’t have any, we get a list of 10 inputs, each of length 10

$ gridtest gridview grids-with-function.yml unwrapped_grid 
{'numbers': [82, 82, 82, 82, 82, 82, 82, 82, 82, 82]}
{'numbers': [52, 52, 52, 52, 52, 52, 52, 52, 52, 52]}
{'numbers': [41, 41, 41, 41, 41, 41, 41, 41, 41, 41]}
{'numbers': [25, 25, 25, 25, 25, 25, 25, 25, 25, 25]}
{'numbers': [43, 43, 43, 43, 43, 43, 43, 43, 43, 43]}
{'numbers': [64, 64, 64, 64, 64, 64, 64, 64, 64, 64]}
{'numbers': [43, 43, 43, 43, 43, 43, 43, 43, 43, 43]}
{'numbers': [39, 39, 39, 39, 39, 39, 39, 39, 39, 39]}
{'numbers': [27, 27, 27, 27, 27, 27, 27, 27, 27, 27]}
{'numbers': [64, 64, 64, 64, 64, 64, 64, 64, 64, 64]}

We can also ask to just look at the numbers parameter, as it was defined before we produced the parameter matrix above (it’s a list of lists)

$ gridtest gridview grids-with-function.yml unwrapped_grid --arg numbers --count
Variable numbers has length 10.

$ gridtest gridview grids-with-function.yml unwrapped_grid --arg numbers
[[32, 32, 32, 32, 32, 32, 32, 32, 32, 32], [70, 70, 70, 70, 70, 70, 70, 70, 70, 70], [75, 75, 75, 75, 75, 75, 75, 75, 75, 75], [86, 86, 86, 86, 86, 86, 86, 86, 86, 86], [86, 86, 86, 86, 86, 86, 86, 86, 86, 86], [50, 50, 50, 50, 50, 50, 50, 50, 50, 50], [92, 92, 92, 92, 92, 92, 92, 92, 92, 92], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [80, 80, 80, 80, 80, 80, 80, 80, 80, 80], [35, 35, 35, 35, 35, 35, 35, 35, 35, 35]]

We can also add another parameter (e.g., defining two for length doubles the results)

    # Generate a list of lists, intended to be unwrapped and used for another grid
    unwrapped_grid:
      args:
        length: [10, 20]
      functions:
        numbers: 
          func: script.generate_numbers
          unwrap: true

Regardless, we could then reference this variable with ref for another grid. For example, here we want to

    sum_unwrapped_numbers:
      ref:
        numbers: unwrapped_grid.numbers
      functions: 
        total: script.dosum

And then we generate a grid with the result.

$ gridtest gridview grids-with-function.yml sum_unwrapped_numbers
{'numbers': [90, 90, 90, 90, 90, 90, 90, 90, 90, 90], 'total': 900}
{'numbers': [72, 72, 72, 72, 72, 72, 72, 72, 72, 72], 'total': 720}
{'numbers': [88, 88, 88, 88, 88, 88, 88, 88, 88, 88], 'total': 880}
{'numbers': [21, 21, 21, 21, 21, 21, 21, 21, 21, 21], 'total': 210}
{'numbers': [37, 37, 37, 37, 37, 37, 37, 37, 37, 37], 'total': 370}
{'numbers': [96, 96, 96, 96, 96, 96, 96, 96, 96, 96], 'total': 960}
{'numbers': [40, 40, 40, 40, 40, 40, 40, 40, 40, 40], 'total': 400}
{'numbers': [37, 37, 37, 37, 37, 37, 37, 37, 37, 37], 'total': 370}
{'numbers': [26, 26, 26, 26, 26, 26, 26, 26, 26, 26], 'total': 260}
{'numbers': [2, 2, 2, 2, 2, 2, 2, 2, 2, 2], 'total': 20}

What to do with Grids?

You might just be using grids inline to go with your tests. However, grids are useful to many things:

Next you might want to read about how gridtest grids can be used in tests.