Wilson Mar bio photo

Wilson Mar


Calendar YouTube Github


Behavior-driven Development (BDD) by automating Gherkin test specs using Pytest for test coverage analysis, data-driven tests, and localization verification

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean


After following this hands-on tutorial manually, you would be able to add to your resume:

Automated testing of BDD based on Gherkin using Python-based pytest-bdd installed using a Bash script. Integrated libraries for test coverage analysis, data-driven tests, and localization verification.

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

Testing Python

Being a dynamic language, errors in Python code can appear only when run rather than when compiled.

PROTIP: Create a test .py file to go with each py file.

There are several libraries to support testing:

  1. unittest is built into Python interpreter:

    python -m unittest test_unittest.py -f -b --locals

    -f stops run on the first error/failure.

    -b buffers output for display on on unsuccessful runs.

    –locals shows local variables in tracebacks.

    Run all tests:

    python -m 


  2. pytest needs to be installed:

    pip3 install pytest

    Pytest-bdd is a plug-in to Pytest.

Applicable to both:

  • Name test .py files beginning with “test”.

  • Name all test classes in code with a name beginning with “test”.

  • Tests are not run from top to bottom, so each test needs to be stand-alone.

  • Define asserts to determine if pass or fail.

  • To do stuff before the tests:

   def setupClass(cls)
       print('in setupClass')
   def tearDownClass(cls)
       print('in tearDownClass')

Tools for Debugging Python code

  • Python Tutor - visualize how the Python interpreter reads and executes your code

  • DiffChecker - compares two sets of text and shows you which lines are different

  • Debugging in Python - steps you can take to try to debug your program

  • Mocking of API end-points when the actual service is not available.


Tutorials to learn about basic concepts of Pytest:

Andrew Knight (@automationpanda, AutomationPanda.com) gradually presents, in a logic sequence and with quizzes 9 videos at Applitools’ Test Automation University (TAU). YOUTUBE: 2 min. intro

Matt Harrison held live classes on OReilly.com in 2021.

## Automation of install and run

My value-add here is writing a Bash script (below) that automatically installs what is needed and runs the test on a public website under test:


That is a modified version of a fork of Andrew’s repo at https://github.com/AndyLPK247/tau-pytest-bdd, to avoid future possible breaking changes in or disappearance of Andrew’s upstream repo). (I occassionally sync with it and reconcile changes in my forked version.)

For automation I built a Bash script based on coding techniques described at: https://wilsonmar.github.io/bash-scripts

  1. Install pytest

    pip3 install pytest
  2. Highlight and copy this command: TODO:

    ... bash https://github.com/wilsonmar/tau-pytest-bdd.sh
    import file_ab_session as fas
    def test_add_function_given_two_arguments():
     RESULT = fas.add(2,3)

    Note the parameters at the end of the command above:

    -v -V

  3. Open a Terminal.
  4. Navigate to a folder and paste the command to execute it.

If you performed the above, click here to run what was installed.

Manual install

Alternately, the text below describes what the install script above does so you can do it manually instead:

  1. Clone the repo from Andrew:

    git clone https://github.com/AndyLPK247/tau-pytest-bdd.git

  2. Fork it using hub command

    git remote add upstream https://github.com/AndyLPK247/tau-pytest-bdd.git

  3. Be inside virtualenv

  4. Install pre-requsities referencing file pipfile:

    pipenv install

    The response:

    Installing dependencies from Pipfile.lock (f55e24)…
      🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 19/19 — 00:00:07
    To activate this project's virtualenv, run pipenv shell.
    Alternatively, run a command inside the virtualenv with pipenv run.

    NOTE: This is instead of running individual “pip install” commands:

    pip install pyenv
    pip install -U pytest
    pip install -U pytest-bdd
    pip install -U pytest-cov    # to generate code coverage reports
    pip install -U pytest-xdist  # to run tests in parallel

    Alternately, after git cloning, install: https://github.com/hchasestevens/fault-localization

  5. Activate

    pipenv shell

    The response should be a change to the prompt, such as:
    (tau-pytest-bdd) bash-5.0$

    Launching subshell in virtual environment…
    bash-5.0$  . /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA/bin/activate
    (tau-pytest-bdd) bash-5.0$ 

    Run individual test

  6. The script runs a specific test while at the repo’s root folder:

    pipenv run python -m pytest

    The response begins with:

    Creating a virtualenv for this project…
    Pipfile: /Users/wilson_mar/gits/wilsonmar/tau-pytest-bdd/Pipfile
    Using /usr/local/bin/python3 (3.7.6) to create virtualenv…
    ⠏ Creating virtual environment...Already using interpreter /usr/local/opt/python/bin/python3.7
    Using base prefix '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7'
    New python executable in /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA/bin/python3.7
    Also creating executable in /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA/bin/python
    Installing setuptools, pip, wheel...
    Running virtualenv with interpreter /usr/local/bin/python3
    ✔ Successfully created virtual environment! 
    Virtualenv location: /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA
    /Users/wilson_mar/.local/share/virtualenvs/tau-pytest-bdd-YNf2NFbA/bin/python: No module named pytest
  7. The script runs a specific test while at the repo’s root folder:

    pipenv run python -m pytest test_web_steps.py
  8. A new window pops up and disappears.
  9. The response on the Terminal (with … elipses replacing long strings):

    ======...====== test session starts ======...======
    platform darwin -- Python 3.7.6, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
    rootdir: /Users/wilson_mar/gits/wilsonmar/tau-pytest-bdd/tests/step_defs
    plugins: bdd-3.1.0
    collected 2 items                                 
    test_web_steps.py .                             ...[100%]
    ======...====== 2 passed in 23.64 seconds ======...
    (tau-pytest-bdd) bash-5.0$ pipenv run python -m pytest test_web_steps.py
    ======...======  test session starts ===========...====== 
    platform darwin -- Python 3.7.6, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
    rootdir: /Users/wilson_mar/gits/wilsonmar/tau-pytest-bdd/tests/step_defs
    plugins: bdd-3.1.0
    collected 2 items
    test_web_steps.py .                             ...[100%]
    ======...======  2 passed in 19.80 seconds ======...====== 
    (tau-pytest-bdd) bash-5.0$ 

    Test coverage

  10. After the test finishes, look at the test log/report and test coverage report.

    View test code

  11. Additionally, some want to install files to enable reference from within IDEs such as PyCharm, Eclipse, VSCode, etc.

    Alternative runs

  12. To run all tests defined:

    pipenv run python -m pytest
  13. To run all “web” tests (not “api” tests):

    pipenv run python -m pytest -k "web"

“test” folder contents

In the root of the repo is file cucumbers.py file, which defines the subject being tested (a basket of cucumbers).

In your mind, substitute the word “cucumber” with a “batch of invoices”, or whatever else your own application manages.

In the repo is a “test” folder, which in an integrated Agile team would be inside the source code repo of the app being tested.

### Folder structure for Pytest-bdd

Under the sample test folder, folders “features” and “step_defs” is for use by the pytest-bdd framework.

“pytest-bdd” is used here because it is not a standalone framework like alternative Behave. Pytest-bdd is a plugin for pytest (https://docs.pytest.org/en/latest) and all of its features and other plugins.

Like other BDD frameworks, pytest-bdd test scenarios are written within “.feature” files using the Gherkin language which uses specific vocabulary. Keywords in Gherkin can be in several spoken languages.

Since Gherkin can be userstood by non-technical people, it is a common way to communicate specifications among “Three Amigos” (business analysts, testers, developers).

A frameworks for BDD (“black box testing”) is very different from traditional testing frameworks like unittest and pytest which specify specific CSS markers coded by developers.

However, the combination provides a separation of concerns between test cases and test code. Gherkin steps may also be reused by multiple scenarios.

BTW, Within step_defs, file init.py (with no content) is for Python 3.3 and earlier * to look for submodules inside that directory.

  1. PROTIP: In your editor, open a feature file in one pane and its associated py file in another pane. Better yet, if you have two monitors, have “features” folder in one and “step_defs” files in another.


Pytest introduces fixtures that sets up objects needed for testing, such as a SMTP port for sending email.

Fixtures can have scope to run once or multiple times.

Pytest can be augmented with plug-ins for code coverage, Flask integration, etc.





VIDEO: Automated testing with pytest and fixtures</a> at PyGotham 2017

https://pytest-bdd.readthedocs.io/en/latest/index.html?#hooks Pytest-BDD – Hooks

https://github.com/AndyLPK247/tau-pytest-bdd/tree/chapter-9 GitHub Repo – Chapter 9

https://pytest-bdd.readthedocs.io/ Pytest-BDD Documentation

https://automationpanda.com/2017/03/14/python-testing-101-pytest/ Automation Panda - Python Testing 101: pytest

https://automationpanda.com/2018/10/22/python-testing-101-pytest-bdd/ Automation Panda - Python Testing 101: pytest-bdd

https://github.com/AndyLPK247/behavior-driven-python GitHub Repo – Python BDD Test Framework Examples


Pytest-bdd is a plug-in to Pytest.




Packt Book: “Pytest quick start guide” by Bruno Olivera

Book: “Python Testing with pytest” by Brian Okken

Hooks in conftest.py

To share common steps, fixtures, and BDD hooks between test modules.

per-directory hooks

Shared steps video

See https://docs.pytest.org/en/2.7.3/plugins.html

Gherkin feature files

  1. Name each of the various features to be tested as a “feature” file.

    For pytest-bdd, a “features” folder is added to contain files with names ending with “.feature”. Such files are in the Gherkin language.

    @web duckduckgo at the top of the file enable features defined to be referenced by all scenarios by a filter. See this video.

    Numbers are in quotes so that the parser can recognize where parameter substitution can occur.

    This video</strong> shows the use of scenario outlines that references example tables.

    Feature files are not runnable as a program. So each step definition in Gherkin (Given, When, and Then) is “glued” to a Python function.

    Python step_defs

  2. For each step definition, specify a fixture decorated by a matching string in a Pytest step definition module to associate with Python code.

    A “step_defs” folder contains files containing Python code. The top line of each file starts with:

    See https://testautomationu.applitools.com/behavior-driven-python-with-pytest-bdd/chapter4.html

    from pytest_bdd import scenario, given, when, then
    from cucumbers import CucumberBasket
    from pytest bdd import parsers
    from functools import partial

    A fixture statement (with @) decorates functions.

    The @scenarios fixture creates functions for all … within the feature.


    That takes the place of specific code:

    def test_add('../features/cucumbers.feature','Add ... to a basket'):
    def test_remove('../features/cucumbers.feature','Remove ... from a basket'):
    @given("The basket has 2 tasks")



    The above steps can be re-used, which enables more rapid test development.


    Instead of static numbers and text strings, values can be replaced with parameters such as:

    @then(parsers.cfparse('the basket contains "{total:Number}" cucumbers', extra_types=dict(Number=int)))

    This means that references within features are within single quotes.


pytest-bdd test scenarios are written in Gherkin “.feature” files using plain language. Thus, it is a BDD test framework that is similar to Behave, Cucumber, and SpecFlow. For Python there is also radish, which extends Cucumber with constants and scenario loops.

“The Cucumber Book” by Matt Wynne and Asiak Hellesoy.

Implement BDD with TDD: Using Python, Behave, and Mocking

Book “BDD in Action” by John Ferguson Smart.

Not used much anymore is “lettuce”.



Abhishake Gupta’s pyTest at https://github.com/letspython3x/code_examples

More about Python

This is one of a series about Python:

  1. Python install on MacOS
  2. Python install on MacOS using Pyenv
  3. Python install on Raspberry Pi for IoT

  4. Python tutorials
  5. Python Examples
  6. Python coding notes
  7. Pulumi controls cloud using Python, etc.
  8. Jupyter Notebooks provide commentary to Python

  9. Python certifications

  10. Test Python using Pytest BDD Selenium framework
  11. Test Python using Robot testing framework
  12. Testing AI uses Python code

  13. Microsoft Azure Machine Learning makes use of Python

  14. Python REST API programming using the Flask library
  15. Python coding for AWS Lambda Serverless programming
  16. Streamlit visualization framework powered by Python
  17. Web scraping using Scrapy, powered by Python
  18. Neo4j graph databases accessed from Python