Using Tox

Getting started with Tox

We are using the sample project from the Crafting Test Drieven Software with Python book.

Step 1. Install tox

$ pip install tox
.............

Step 2: Create tox.ini

Create tox.ini file at the project root directory.

# FILE: tox.ini
[tox]
setupdir = ./src

[testenv]
deps =
   flaky == 3.7.0
   pytest
   pytest-bdd == 3.4.0
   pytest-cov
   pytest-benchmark == 3.2.3
commands =
   pytest --cov=contacts {posargs}

1. The setupdir option instructs tox which directory to use to build and install the package from. This is the directory where setup.py or pyproject.toml is located. Current directory is used as default.

2. The testenv section defines default settings for tox environment.

3. The deps option instructs tox to install listed dependencies to the environment. In our example, tox will install flaky, pytest, pytest-bdd version 3.4.0, pytest-cov and pytest-benchmark.

4. The commands option instructs tox_ which commands to execute after the environment is created, acitvated and the project package is installed into it. By default no commands are executed.

5. {posargs} tells tox to include the command line argiments found after the -- command line, e.g. tox -- m load will execute only test cases makred as load.

Step 3: Run tox

$ tox
GLOB sdist-make: C:\Sandbox\projects\statuspage\tox\practice\src\setup.py
python recreate: C:\Sandbox\projects\statuspage\tox\practice\.tox\python
python installdeps: flaky == 3.7.0, pytest, pytest-bdd == 3.4.0, pytest-cov, pytest-benchmark
python inst: C:\Sandbox\projects\statuspage\tox\practice\.tox\.tmp\package\1\contacts-0.0.0.zip
python installed: atomicwrites==1.4.0,attrs==21.2.0,colorama==0.4.4,contacts @ file:///C:/Sandbox/projects/statuspage/tox/practice/.tox/.tmp/package/1/contacts-0.0.0.zip,coverage==6.2,flaky==3.7.0,glob2==0.7,iniconfig==1.1.1,Mako==1.1.6,MarkupSafe==2.0.1,packaging==21.3,parse==1.19.0,parse-type==0.5.2,pluggy==1.0.0,py==1.11.0,py-cpuinfo==8.0.0,pyparsing==3.0.6,pytest==6.2.5,pytest-bdd==3.4.0,pytest-benchmark==3.4.1,pytest-cov==3.0.0,six==1.16.0,toml==0.10.2,tomli==2.0.0
python run-test-pre: PYTHONHASHSEED='176'
python run-test: commands[0] | pytest --cov=contacts
================================ test session starts =================================
platform win32 -- Python 3.10.0, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
cachedir: .tox\python\.pytest_cache
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: C:\Sandbox\projects\statuspage\tox\practice
plugins: flaky-3.7.0, bdd-3.4.0, benchmark-3.4.1, cov-3.0.0
collected 26 items

tests\acceptance\test_delete_contact.py .                                       [  3%]
tests\acceptance\test_list_contacts.py ..                                       [ 11%]
benchmarks\test_persistence.py .                                                [ 15%]
tests\acceptance\test_adding.py ..                                              [ 23%]
tests\functional\test_basic.py ...                                              [ 34%]
tests\functional\test_main.py .                                                 [ 38%]
tests\unit\test_adding.py ......                                                [ 61%]
tests\unit\test_application.py .......                                          [ 88%]
tests\unit\test_flaky.py .                                                      [ 92%]
tests\unit\test_persistence.py ..                                               [100%]

---------- coverage: platform win32, python 3.10.0-final-0 -----------
Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
.tox\python\Lib\site-packages\contacts\__init__.py      51      0   100%
.tox\python\Lib\site-packages\contacts\__main__.py       0      0   100%
------------------------------------------------------------------------
TOTAL                                                   51      0   100%



----------------------------------------------------- benchmark: 1 tests ----------------------------------------------------
Name (time in us)          Min         Max      Mean    StdDev    Median      IQR  Outliers  OPS (Kops/s)  Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------
test_loading          335.3000  9,861.9000  449.3980  604.9698  385.9000  75.8000
1;19        2.2252     249           1
-----------------------------------------------------------------------------------------------------------------------------

Legend:
Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
OPS: Operations Per Second, computed as 1 / Mean
===Flaky Test Report===

test_appender passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
================================= 26 passed in 1.66s =================================
______________________________________ summary _______________________________________
python: commands succeeded
congratulations :)

1. Line #2 - tox is building the project package.

2. Line #3 - tox creates Python virtual enviornment.

3. Line #4 - tox installs depenencies, listed in tox.ini.

4. Line #8 - tox runs commands, isted in tox.ini.

5. Lines #9 - #53 are output from the command execution

6. Lines from line #54 are summary report from tox execution.

Manage multiple environments

# FILE: tox.ini
[tox]
setupdir = ./src
envlist = py39, py310

[testenv]
deps =
    pytest
    pytest-bdd == 3.4.0
    pytest-cov == 2.10.1
commands =
    pytest --cov=contacts --benchmark-skip {posargs}

[testenv:py36]
deps =
    pytest == 4.6.11
    pytest-bdd == 3.4.0
    pytest-cov == 2.10.1
    flaky == 3.7.0
    pytest-benchmark == 3.2.3
    pytest-cov == 2.10.1
commands =
    pytest --cov=contacts --benchmark-skip {posargs}

1. The envlist option tells tox which environments to build and execute.

2. The testenv section defines default environment settings.

3. The testenv:py36 section defines settings for the Python 3.6 environment.

Using Travis with Tox

# FILE: tox.ini
[tox]
setupdir = ./src

[testenv]
usedevelop = true
deps =
   flaky == 3.7.0
   pytest
   pytest-bdd == 3.4.0
   pytest-cov
   pytest-benchmark == 3.2.3
commands =
   pytest --cov=contacts {posargs}

1. Add the usedevelop option to instruct tox to not build and isntall source distribution, but to use develop setup instead (-e option of pip).

Modify travis.yml:

script:
   - "tox"

install:
   - "pip install travis-tox"

Further reading

• Tox documentation