pytest-bdd

Behavior-driven Development (BDD) by automating Gherkin test specs using Pytest for test coverage analysis, data-driven tests, and localization verification

Overview

Testing Python
Tools for Debugging Python code
Pytest
- Manual install
Run individual test
- Test coverage
View test code
Alternative runs
“test” folder contents
Pytest
pytest-bdd
- Hooks in conftest.py
- Gherkin feature files
Python step_defs
- Parameters
Alternatives
References
More about Python

After following this hands-on tutorial manually, you would be able to add to your resume:

Automated testing of BDD based on Gherkin using Python-based pytest-bdd installed using a Bash script. Integrated libraries for test coverage analysis, data-driven tests, and localization verification.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Testing Python

Being a dynamic language, errors in Python code can appear only when run rather than when compiled.

PROTIP: Create a test .py file to go with each py file.

There are several libraries to support testing:

unittest is built into Python interpreter:
https://docs.python.org/3/library/unittest.html
```
python -m unittest test_unittest.py -f -b --locals
```
-f stops run on the first error/failure.

-b buffers output for display on on unsuccessful runs.

–locals shows local variables in tracebacks.

Run all tests:
```
python -m 
```
https://www.youtube.com/watch?v=HKTyOUx9Wf4
- VIDEO by Socratica (Ulka Simone Mohanty) developers watch even though they know Python because her deadpan delivery of admonishments is so entertaining. “I can almost feel her stilleto heels digging into my chest as she makes a point.”
- VIDEO: Python Tutorial: Unit Testing Your Code with the unittest Module Aug 16, 2017 by Corey Schafer
pytest needs to be installed:
```
pip3 install pytest
```
Pytest-bdd is a plug-in to Pytest.

Applicable to both:

Name test .py files beginning with “test”.
Name all test classes in code with a name beginning with “test”.
Tests are not run from top to bottom, so each test needs to be stand-alone.
Define asserts to determine if pass or fail.
To do stuff before the tests:

@classmethod
   def setupClass(cls)
       print('in setupClass')
 
   @classmethod
   def tearDownClass(cls)
       print('in tearDownClass')

Tools for Debugging Python code

Python Tutor - visualize how the Python interpreter reads and executes your code
DiffChecker - compares two sets of text and shows you which lines are different
Debugging in Python - steps you can take to try to debug your program
Mocking of API end-points when the actual service is not available.

Pytest

Tutorials to learn about basic concepts of Pytest:

Andrew Knight (@automationpanda, AutomationPanda.com) gradually presents, in a logic sequence and with quizzes 9 videos at Applitools’ Test Automation University (TAU). YOUTUBE: 2 min. intro

Matt Harrison held live classes on OReilly.com in 2021.

## Automation of install and run

My value-add here is writing a Bash script (below) that automatically installs what is needed and runs the test on a public website under test:

https://github.com/wilsonmar/tau-pytest-bdd

That is a modified version of a fork of Andrew’s repo at https://github.com/AndyLPK247/tau-pytest-bdd, to avoid future possible breaking changes in or disappearance of Andrew’s upstream repo). (I occassionally sync with it and reconcile changes in my forked version.)

For automation I built a Bash script based on coding techniques described at: https://wilsonmar.github.io/bash-scripts

Install pytest
```
pip3 install pytest
```

Highlight and copy this command: TODO:

... bash https://github.com/wilsonmar/tau-pytest-bdd.sh

import file_ab_session as fas
def test_add_function_given_two_arguments():
 RESULT = fas.add(2,3)
 EXPECTED_RESULT = 5
 assert RESULT == EXPECTED_RESULT

Note the parameters at the end of the command above:

-v -V

Open a Terminal.
Navigate to a folder and paste the command to execute it.

If you performed the above, click here to run what was installed.

Manual install

Alternately, the text below describes what the install script above does so you can do it manually instead:

Clone the repo from Andrew:

git clone https://github.com/AndyLPK247/tau-pytest-bdd.git
Fork it using hub command

git remote add upstream https://github.com/AndyLPK247/tau-pytest-bdd.git
Be inside virtualenv

Install pre-requsities referencing file pipfile:

pipenv install

The response:

Installing dependencies from Pipfile.lock (f55e24)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 19/19 — 00:00:07
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

NOTE: This is instead of running individual “pip install” commands:

pip install pyenv
pip install -U pytest
pip install -U pytest-bdd
pip install -U pytest-cov    # to generate code coverage reports
pip install -U pytest-xdist  # to run tests in parallel

Alternately, after git cloning, install: https://github.com/hchasestevens/fault-localization

Activate

pipenv shell

The response should be a change to the prompt, such as:
(tau-pytest-bdd) bash-5.0$

Launching subshell in virtual environment…
bash-5.0$  . /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA/bin/activate
(tau-pytest-bdd) bash-5.0$

Run individual test

The script runs a specific test while at the repo’s root folder:

pipenv run python -m pytest

The response begins with:

Creating a virtualenv for this project…
Pipfile: /Users/wilson_mar/gits/wilsonmar/tau-pytest-bdd/Pipfile
Using /usr/local/bin/python3 (3.7.6) to create virtualenv…
⠏ Creating virtual environment...Already using interpreter /usr/local/opt/python/bin/python3.7
Using base prefix '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7'
New python executable in /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA/bin/python3.7
Also creating executable in /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA/bin/python
Installing setuptools, pip, wheel...
done.
Running virtualenv with interpreter /usr/local/bin/python3
✔ Successfully created virtual environment! 
Virtualenv location: /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA
/Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA/bin/python: No module named pytest

The script runs a specific test while at the repo’s root folder:
```
pipenv run python -m pytest test_web_steps.py
```
A new window pops up and disappears.

The response on the Terminal (with … elipses replacing long strings):

======...====== test session starts ======...======
platform darwin -- Python 3.7.6, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /Users/wilson_mar/gits/wilsonmar/tau-pytest-bdd/tests/step_defs
plugins: bdd-3.1.0
collected 2 items                                 
 
test_web_steps.py .                             ...[100%]
 
======...====== 2 passed in 23.64 seconds ======...
(tau-pytest-bdd) bash-5.0$ pipenv run python -m pytest test_web_steps.py
======...======  test session starts ===========...====== 
platform darwin -- Python 3.7.6, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /Users/wilson_mar/gits/wilsonmar/tau-pytest-bdd/tests/step_defs
plugins: bdd-3.1.0
collected 2 items
 
test_web_steps.py .                             ...[100%]
 
======...======  2 passed in 19.80 seconds ======...====== 
(tau-pytest-bdd) bash-5.0$

Test coverage

After the test finishes, look at the test log/report and test coverage report.

View test code
Additionally, some want to install files to enable reference from within IDEs such as PyCharm, Eclipse, VSCode, etc.
- VIDEO: Productive pytest with PyCharm by Brian Okken (@brianokken/Github:okken)
- VIDEO: Kris Jordan 1 hour “An Introductory Tutorial to Unit Testing your Python Functions with Pytest and Visual Studio Code”
- VIDEO: “Automated Testing in Python with pytest, tox, and GitHub Actions”
Alternative runs
To run all tests defined:
```
pipenv run python -m pytest
```
To run all “web” tests (not “api” tests):
```
pipenv run python -m pytest -k "web"
```

“test” folder contents

In the root of the repo is file cucumbers.py file, which defines the subject being tested (a basket of cucumbers).

In your mind, substitute the word “cucumber” with a “batch of invoices”, or whatever else your own application manages.

In the repo is a “test” folder, which in an integrated Agile team would be inside the source code repo of the app being tested.

### Folder structure for Pytest-bdd

Under the sample test folder, folders “features” and “step_defs” is for use by the pytest-bdd framework.

“pytest-bdd” is used here because it is not a standalone framework like alternative Behave. Pytest-bdd is a plugin for pytest (https://docs.pytest.org/en/latest) and all of its features and other plugins.

Like other BDD frameworks, pytest-bdd test scenarios are written within “.feature” files using the Gherkin language which uses specific vocabulary. Keywords in Gherkin can be in several spoken languages.

Since Gherkin can be userstood by non-technical people, it is a common way to communicate specifications among “Three Amigos” (business analysts, testers, developers).

A frameworks for BDD (“black box testing”) is very different from traditional testing frameworks like unittest and pytest which specify specific CSS markers coded by developers.

However, the combination provides a separation of concerns between test cases and test code. Gherkin steps may also be reused by multiple scenarios.

BTW, Within step_defs, file init.py (with no content) is for Python 3.3 and earlier * to look for submodules inside that directory.

PROTIP: In your editor, open a feature file in one pane and its associated py file in another pane. Better yet, if you have two monitors, have “features” folder in one and “step_defs” files in another.

Pytest

Pytest introduces fixtures that sets up objects needed for testing, such as a SMTP port for sending email.

Fixtures can have scope to run once or multiple times.

Pytest can be augmented with plug-ins for code coverage, Flask integration, etc.

https://docs.pytest.org/en/latest/

https://automationpanda.com/python/

https://automationpanda.com/2018/09/27/book-review-pytest-quick-start-guide/

https://pragprog.com/book/bopytest/python-testing-with-pytest

VIDEO: Automated testing with pytest and fixtures</a> at PyGotham 2017

https://pytest-bdd.readthedocs.io/en/latest/index.html?#hooks Pytest-BDD – Hooks

https://github.com/AndyLPK247/tau-pytest-bdd/tree/chapter-9 GitHub Repo – Chapter 9

https://pytest-bdd.readthedocs.io/ Pytest-BDD Documentation

https://automationpanda.com/2017/03/14/python-testing-101-pytest/ Automation Panda - Python Testing 101: pytest

https://automationpanda.com/2018/10/22/python-testing-101-pytest-bdd/ Automation Panda - Python Testing 101: pytest-bdd

https://github.com/AndyLPK247/behavior-driven-python GitHub Repo – Python BDD Test Framework Examples

pytest-bdd

Pytest-bdd is a plug-in to Pytest.

https://github.com/pytest-dev/pytest-bdd

https://pytest-bdd.readthedocs.io/en/latest/

https://github.com/AndyLPK247/tau-pytest-bdd/tree/chapter-2

Packt Book: “Pytest quick start guide” by Bruno Olivera

Book: “Python Testing with pytest” by Brian Okken

Hooks in conftest.py

To share common steps, fixtures, and BDD hooks between test modules.

per-directory hooks

Shared steps video

See https://docs.pytest.org/en/2.7.3/plugins.html

Gherkin feature files

Name each of the various features to be tested as a “feature” file.

For pytest-bdd, a “features” folder is added to contain files with names ending with “.feature”. Such files are in the Gherkin language.

@web duckduckgo at the top of the file enable features defined to be referenced by all scenarios by a filter. See this video.

Numbers are in quotes so that the parser can recognize where parameter substitution can occur.

This video</strong> shows the use of scenario outlines that references example tables.

Feature files are not runnable as a program. So each step definition in Gherkin (Given, When, and Then) is “glued” to a Python function.

Python step_defs
For each step definition, specify a fixture decorated by a matching string in a Pytest step definition module to associate with Python code.

A “step_defs” folder contains files containing Python code. The top line of each file starts with:

See https://testautomationu.applitools.com/behavior-driven-python-with-pytest-bdd/chapter4.html
```
from pytest_bdd import scenario, given, when, then
from cucumbers import CucumberBasket
from pytest bdd import parsers
from functools import partial
```
A fixture statement (with @) decorates functions.

The @scenarios fixture creates functions for all … within the feature.
```
@scenarios('../features/cucumbers.feature')
```
That takes the place of specific code:
```
@scenario("...")
def test_add('../features/cucumbers.feature','Add ... to a basket'):
   pass
```
```
@scenario("...")
def test_remove('../features/cucumbers.feature','Remove ... from a basket'):
   pass
```
```
@given("The basket has 2 tasks")
```
@when(“…”)

@then(“…”)

The above steps can be re-used, which enables more rapid test development.

Parameters

Instead of static numbers and text strings, values can be replaced with parameters such as:
```
@then(parsers.cfparse('the basket contains "{total:Number}" cucumbers', extra_types=dict(Number=int)))
```
This means that references within features are within single quotes.

Alternatives

pytest-bdd test scenarios are written in Gherkin “.feature” files using plain language. Thus, it is a BDD test framework that is similar to Behave, Cucumber, and SpecFlow. For Python there is also radish, which extends Cucumber with constants and scenario loops.

“The Cucumber Book” by Matt Wynne and Asiak Hellesoy.

Implement BDD with TDD: Using Python, Behave, and Mocking

Book “BDD in Action” by John Ferguson Smart.

Not used much anymore is “lettuce”.

References

https://itnext.io/common-python-security-problems-ffedbae7b11c

Abhishake Gupta’s pyTest at https://github.com/letspython3x/code_examples

More about Python

This is one of a series about Python:

Wilson Mar

pytest-bdd

Testing Python

Tools for Debugging Python code

Pytest

Manual install

Run individual test

Test coverage

View test code

Alternative runs

“test” folder contents

Pytest

pytest-bdd

Hooks in conftest.py

Gherkin feature files

Python step_defs

Parameters

Alternatives

References

More about Python

You might also enjoy (View all posts)