Cucumber testing

Gherkin user specs drive Selenium tests for behavior-driven development (BDD) within Agile

Overview

Cucumber is an open-source server program that tests applications based on specification of that system’s features (behavior) written in the “Gherkin” language.

Cucumber is often used for what some call “User Acceptance Testing” (UAT) to determine if the system built should be accepted based on user specs defined in Gherkin. When written before development, Cucumber enables BDD (Behavior Driven Development).

Business features

PROTIP: Defining a full scope of features enables features to have a better naming conventions.

Examples of business functions are:

Login admin for admin menu.
Create user (registration).
Login user.
Logout.
Password recovery.
Delete account.
Share photos or videos.
Send a friend request.
Approve requests.

Additionally, each feature may have some error recovery features.

PROTIP: It’s better NOT to include the name of the app under test as part of the feature name in order to make the files more reusable with other apps.

PROTIP: Identify negative test cases (those designed to fail in order to test error handling).

Gherkin language

Gherkin is used for “Black box testing” where the person executing the test does not get into the internal engineering of the app.

The Gherkin language is designed to be a “natural” (English-like readable) language designed to be understandable by business people (unlike coding in Java, Go, Python, etc. which are written for reading by specially trained programmers.)

Gherkin code can be edited with any text editor.

“Having conversations is more important than capturing conversations is more important than automating conversations.

VIDEO: An Introduction to BDD (with Cynefin) by Liz Keogh (@lunivore) says BDD is an analysis technique. Her example [4:18]:

Feature: Addition
   In order to avoid mistakes
   As a math idiot
   I want to be told the sum of two numbers
      Given I have entered 50 into the calculator
      And I have added 70 into the calcualator
      When I add the numbers together
      Then the result should be 120 on the screen

An example of a feature file (with .feature file suffix) which describes a single feature:

# language: en
Feature: Site Search
Scenario:
  Given We navigate to the homepage
  When We search for the word agile
  Then The results for the search will be displayed
  Examples:
    |foo|
    |bar|

The Gherkin language is simple. It doesn’t require matching parentheses and semicolons at the end of each line.

Indentation is used to define structure. Like Python and Bash files, pound signs are read as comments and not parsed.

The sample above does not need to begin with a language specification because English is the default. Gherkin can be written in 60 languages. Feature files using the Norwegian language (“no”) begin with “Egenskap” instead of “Feature” in a header line:

# language: no
Egenskap: Summering

Each feature can consists of one or more scenarios. Each scenario consists of steps specified by “Given”, “When”, and “Then” keywords.

Rules of Gherkin “Treetop” grammar are defined in the Gherkin language standard at , see https://cucumber.io/docs/reference

Some call the Gherkin code “living” specifications because there is an automated way to keep specifications in sync with what the system created.

Thus, Cucumber bridges the “communication gap between domain experts and developers”. Cucumber enables text about what business people require to be executed as tests.

The behavior desired by a system are defined by Gherkin specifications that are executable.

Alternately, some testing utilities (such as est.ai, testim.ai, etc.) create Gherkin by automatically scanning apps to identify page objects.

Capybara

Capybara is the part of Cucumber that acts like a user on a web browser. It is for “integration testing”. The Rails tools simulates how a user would interact with a website. Capybara puts applications “through its paces” by simulating a user and browser performing the steps of the scenarios – interacting with the app to receive pages, parse the HTML, and submit forms, just as a real user would.

By storing features in files along with different scenarios of feature use composed of many steps, and storing Ruby code in separate files containing step definitions that tests each type of step, Cucumber and Capybara automatically test the behavior of the SaaS app.

Execution

The Gherkin interpreter reads text written like spoken language.

When the Gherkin interpreter encounters a keyword such as “navigate” and “search”, Cucumber looks into a subdirectory specifically called “step_definitions”.

Feature files can be executed by several software testing platforms:

Ruby on Rails
Selenium (Java)
PicoContainer
Spring Framework
Watir for Ruby and WatiN for dot Net

See https://github.com/cucumber/cucumber/wiki/Gherkin

The cucumber server can be started using a command such as:

cucumber -f pretty -f html -o report.html

NOTE: An example of cucumber’s HTML output:

In the US, green is used on street lights to indicate “pass”. Cucumbers are green. Thus the name of the software. Gherkins are small pickled cucumbers.

For fancier formatting:

https://www.npmjs.com/package/multiple-cucumber-html-reporter
http://toolsqa.com/selenium-cucumber-framework/cucumber-reports/
https://www.tutorialspoint.com/cucumber/cucumber_reports.htm
Image from https://www.npmjs.com/package/cucumber-html-report

Send results to “status of done” dashboards such as SonarQube, Grafana, etc.

Send run information to Hygieia or other DevOps dashboards.

runTest.java

package Annotation; 
 
import org.junit.runner.RunWith; 
import cucumber.junit.Cucumber; 
 
@RunWith(Cucumber.class) 
@Cucumber.Options(format = {"pretty", "html:target/cucumber"}) 
 
public class runTest { }

Installation

There is a Bash script that installs the various Cucumber modules on a Mac:

https://github.com/wilsonmar/mac-setup

Cucumber consists of a set of software modules. When “cucumber” is specified among TEST_TOOLS, several modules need to be installed with it.

Selenium

mvn

Different execution frameworks

Execution frameworks provide assertions and verifications. Assertions determine whether each step succeeded. Verifications determine whether ???

Modules to install

Cucumber’s GitHub library provides code in programming languages using dependency management and verification frameworks for each language:

Programming script	Dependency manager	Verification framework
C#	nuget	NUnit and MSTest
Java	maven, gradle, ant	JUnit or TestNG
Javascript (in NodeJs)	-	-
Perl (abbreviated as 3P)	-	-
PHP	-	-
Python	pip	Behave, Nose
Ruby	-	Spock or RSpec

For Java environments, see https://drive.google.com/drive/folders/0B5v_nInLNoqualhZWHk3eGFfd0k

Modules to run

./script_name.sh features/registration.feature --format html --expand

The arguments specify what features are run, referenced by the “$@” below:

#!/bin/sh
bundle install
rake db:migrate
rake db:test:prepare
rake cucumber "$@"

Many Agile teams have embraced use of Gherkin language in a Behavior Driven Development (BDD) approach to software development.

BDD workflow

Because Gherkin is a language common to both business people and technical programmers, those who know what needs to occur and developers who write programs. Thus, Gherkin enables collaboration among “Three Amigos”:

Business Analyst who understands user needs and concerns
Developer who will do the work and make it real
Tester who discover problems before work starts

BDD is focused on what to test, not how to test.

PROTIP: A key part of BDD is the collaborative discussion among the stakeholders rather than silos not talking with each other.

BDD works at a higher level than unit tests of the TDD (Test Driven Development) approach to software creation. As with TDD, the BDD approach builds software incrementally, by writing feature files that fail before writing code that makes the tests pass. In fact, it is an anti-pattern to write Gherkin after the software.

The “driven” part of BDD means that business people can specify what the computer should do in a way that can be actually verified. It’s more rigorous that the traditional approach of writing “requirements” documents (in Microsoft Word).

The team has a mechanism to define and verify what “done” really means.

This is a big time saver to business people because the automation enables everything previously defined to be verified again and again after even the most misicule of changes. This enables agility with quality.

THIS presents a good comparison between Behavioral and Functional statements (which reference HTTP codes).

When specs are automatically (Cucumber, etc.), specifications remain up-to-date.

Dan North (@tastapod in London UK) and Liz Keogh created a BDD framework, JBehave,[1] followed by a story-level BDD framework for Ruby called RBehave[9] which was later integrated into the RSpec project. https://relishapp.com/rspec The group also worked with David Chelimsky, Aslak Hellesøy (@aslak_hellesoy Norwegian in London), and others on “The RSpec Book: Behaviour Driven Development with RSpec, Cucumber, and Friends”. The first story-based framework in RSpec was later replaced by Cucumber mainly developed by Aslak Hellesøy.

Other frameworks include JBehave, JDave, Easyb, etc.

SpecFlow for .NET

SpecFlow is Cucumber for .NET environments. To install the SpecFlow.Nunit package, see Jason Roberts (@RobertsJason) Automated Acceptance Testing with SpecFlow and Gherkin for Visual Studio and C# coding [2:51] 28 Oct 2013

Step definitions

The mapping between each step of the scenario defined in the Gherkin feature file and the code of each function to be executed is defined in steps definition file stores.

public void goToFacebook() { 
   driver = new FirefoxDriver(); 
   driver.navigate().to("https://www.facebook.com/"); 
} 
@When "^user logs in using Username as \"([^\"]*)\" and Password as \"([^\"]*)\"$"
public void I_enter_Username_as_and_Password_as(String arg1, String arg2) {
   driver.findElement(By.id("email")).sendKeys(arg1);
   driver.findElement(By.id("pass")).sendKeys(arg2);
   driver.findElement(By.id("u_0_v")).click(); 
} 
@Then"^login should be unsuccessful$" 
public void validateRelogin() { 
   if(driver.getCurrentUrl().equalsIgnoreCase(
      "https://www.facebook.com/login.php?login_attempt=1&lwv=110")){ 
         System.out.println("Test Pass");
   } else { 
      System.out.println("Test Failed"); 
   } 
   driver.close(); 
}

@When “^user logs in using Username as "([^"])" and Password as "([^"])"$”

Annotations to avoid duplicates

Under src/test/java, create an “outline.feature” file such as:

Feature: annotation 
#This is how background can be used to eliminate duplicate steps 
 
Background: 
   User navigates to Facebook Given 
   I am on Facebook login page 
 
#Scenario with AND 
Scenario: 
   When I enter username as "TOM"
   And I enter password as "JERRY" 
   Then Login should fail 
 
#Scenario with BUT 
Scenario: 
   When I enter username as "TOM" 
   And I enter password as "JERRY" 
   Then Login should fail 
   But Relogin option should be available

End-to-end

QUESTION: How is Gherkin different than Rules specified in rule-based systems?

PicoContainer
Salesforce
Spring Framework

Selenium

How to test is the job of Selenium.

The first line of a Selenium program:

require 'selenium-webdriver'
driver = Selenium::WebDriver.for :firefox

Selenium makes use of webdriver to control web browsers.

On Windows:

    java -jar selenium-server-standalone-2.35.0.jar -Dwebdriver.chrome.driver="D:\dev\chromedriver.exe"

So Selenium is more suited to “end to end” testing, for example with nightwatch.js.

Karma is a test runner designed for unit tests, so it is much harder to achieve end to end tests on it, you can add a phantomjs launcher, but it is not the same as real browser tests with Selenium. I think both of them can run any js testing framework if you have an adapter… Mocha, jasmine, qunit, etc…

In Visual Studio, highlight a scenario line and right-click to Generate step definitions.

Mocha is used for unit testing. It’s said that Mocha has better asynchronous support than Jasmine. Jasmine is used for end-to-end testing.

The JUnit plug-in for Cucumber called “cucumber-html-report” tracks scenarios and steps tested and produces HTML reports.

Alternatives

Jest, being from Facebook, at https://facebook.github.io/jest/ is focused on JavaScript-based apps built using Facebook’s React.

Install Cucumber server

There are several ways to obtain a running instance of Cucumber, listed from easiest to most difficult:

A) You don’t need a local machine if you use the cloud service

B) Pull an image from Docker Hub within a Google Compute or AWS cloud instance.

C) Use the Dockerfile to build your own Docker image containing JMeter.

D) Use Homebrew to install JMeter natively on you Mac.

E) Download installer to install locally.

CAUTION: If you are in a large enterprise, confer with your security team before installing. They often have a repository such as Artifactory or Nexus where installers are available after being vetted and perhaps patched for security vulnerabilities.

Create within your internal cloud, Google Cloud, Amazon EC2, Microsoft Azure, etc. a VM instance of an Ubuntu server. 4 GB RAM and 10 GB drive is the minimum.

A sample command to create a Google Cloud instance:

gcloud beta compute --project "attiopinfosys" instances create "cucumber-1" --zone "us-central1-f" --machine-type "n1-standard-1" --subnet "default" --maintenance-policy "MIGRATE" --service-account "776279044010-compute@developer.gserviceaccount.com" --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring.write","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --min-cpu-platform "Automatic" --tags "http","https","web","http-server","https-server" --image "ubuntu-1604-xenial-v20171026a" --image-project "ubuntu-os-cloud" --boot-disk-size "10" --boot-disk-type "pd-standard" --boot-disk-device-name "cucumber-1"

Use Dockerfile to image

Install Git and Docker.
Run the Dockerfile at

https://raw.githubusercontent.com/wilsonmar/Cucumber/master/Dockerfile

Its contains:
```
FROM ubuntu:16.04
RUN apt-get update
RUN apt-get update && apt-get install -y \
  default-jre \
  default-jdk \
  git \
  maven 
 
RUN mvn -version
RUN git clone https://github.com/cucumber/cucumber-jvm.git --depth=1
CMD ls
RUN cd cucumber-jvm/examples/java-calculator --depth=1 && mvn test
```
The above provides commands to install Cucumber within a blank Docker container.

cucumber-jvm/examples/java-calculator is a simple sample app, which is replaced with a real app in the real world.

Use Docker image
Install Docker.
```
 docker pull graze/cucumber-rest-bdd
```

Run the image:

docker run --rm -v $(pwd):/opt/src -e endpoint=http://server/ graze/cucumber-rest-bdd

Binary install

Cucumber server software was written in the Ruby programming language. Ruby is the “R” in RSpec, which defines the language grammar.

So Cucumber is supplied as a Ruby gem.

Install the Ruby interpreter.
Declare that our Ruby app depends on this gem and use Bundler to install it.
```
gedit Gemfile
... add 'gem
bundle install
```

Feature files

Cucumber executes files named with a .feature file extension.

Within that file, the keyword “Feature” is on the first line.

PROTIP: Some teams find it useful to hold a feature brainstorm session to identify as many features as they can think of right away, to get all ideas “on the table” before diving into more detail.

Scenarios

Scenarios are examples of expected system behaviors. Each scenario describes a particular situation that uses the feature.

Sentences not starting with any of these keywords are treated as comment documentation. A fanciful example[1]:

Feature: Brush teeth
   In order to stay pain free
   As a human
   I want to brush my teeth

There are other keywords:

Background provides setup setups common to several scenarios.
Given to set-up initial state
Scenario describes a concrete example illustrating a business rule. It consists of a list of steps.

Scenario Outline

When to perform action(s)
Then for the end-state to check
And adds additional items as the keyword above it.
But is a counter-example on undesired side-effects

Background - to avoid repeating givens in all scenarios.

PROTIP: Instead of English words, French, Japanese, and other words can be used with installation of the target translation. This is because Gherkin is known as a Domain-Specific-Language (DSL).

Helpers in IDEs

SpecFlow is an extension installed into Visual Studio. SpecFlow features are

SpecFlow Feature File
SpecFlow Hooks (event binding)
SpecFlow Step Definition

User stories

A User Story is a low fidelity (inexact) description of a use case.

From https://saascourse.blogspot.com/2012/03/feature-add-med.html

Feature: user can add a medication

Scenario: add a medication
   Given I am on the patient medications screen
   When I follow "add new medication"
   Then I should be on the "New Medication" page
   When I fill in "Med Name" with "Aspirin"
   And I select "Route Oral"
   And I press "Return"
   Then the list of matched medications should be displayed in the list
   And I should see Aspirin Tablets

Steps generally

Given
Perform action(s)
Check end state

Key websites

https://cucumber.io/school has 3 hours of videos for $199 either Java or Ruby
http://cukenfest.cucumber.io/ Cukenfest conf. was in London April 17-10
https://cucumber.io/docs/reference
https://github.com/cucumber
https://github.com/cucumber/cucumber/wiki/A-Table-Of-Content
https://en.wikipedia.org/wiki/Cucumber_(software)
selenium-users@googlegroups.com

Video Training

Mukesh otwani at https://learn-automation.com created a series of videos about Cucumber

Matt Wynne and Aslak Hellesøy authored The Cucumber Book and “Cucumber Recipies” from Pragmatic Press.

https://pragprog.com/titles/hwcuc/source_code
https://www.tutorialspoint.com/cucumber/cucumber_overview.htm

Greg Bee takes issue with their REST API example containing HTTP GET and JSON, then presents a more readable version with explanations:

http://gregbee.ch/blog/effective-api-testing-with-cucumber

From 2012

https://saascourse.blogspot.com/2012/03/cucumber.html

Ripon Al Wasim (https://twitter.com/riponalwasim”>@riponalwasim</a>) released through Packt on 12/23/15 the “Mastering Selenium Testing Tools” video course. It is also available via Udemy and LinkedIn/Lynda.

Thomas Sundberg

Kevin Skoglund release the “Ruby: Testing with RSpec” video course also available via LinkedIn/Lynda.

Others:

https://github.com/cucumber/cucumber/wiki/Cucumber-Backgrounder
http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html
http://dannorth.net/2011/01/31/whose-domain-is-it-anyway/
http://elabs.se/blog/15-you-re-cuking-it-wrong
https://github.com/intuit/karate

Using the Gherkin Language to Write User Stories Tutorial by Reval Govender

An Introduction to BDD by Konstantin Kudryashov of Inviqa

Behaviour Driven Development - Dan North Was Right + FREE CHEAT SHEET Development That Pays

Test Driven Development vs Behaviour Driven Development + FREE CHEAT SHEET Development That Pays

What is BDD? What is Behavior Driven Development? Development That Pays

Gherkin Best Practices by Jared

Interview with Dan North on Behavior-Driven Development Vladimir Gitlevich

Beauty in Code 2019, 6 of 6 — Dan North: “BDD Is Not About Testing”

Introduction to BDD with Cucumber Conygre

What is BDD (example with Cucumber)? QAShahin

Cucumber Framework || BDD || Data Table to List Data Structure</a> Rahul Rathore

BDD all the way down by Enrico Campidoglio FooCafe

[5:10] BDD vs TDD (explained) by Llewellyn Falco

Specification
Feedback
Regression
Granularity - info when tests fail

Wilson Mar

Cucumber testing

Overview

Business features

Gherkin language

Capybara

Execution

Installation

Different execution frameworks

Modules to install

Modules to run

BDD workflow

SpecFlow for .NET

Step definitions

Annotations to avoid duplicates

End-to-end

Selenium

Alternatives

Install Cucumber server

Use Dockerfile to image

Use Docker image

Binary install

Feature files

Tags

Scenarios

Helpers in IDEs

User stories

Key websites

Video Training

Wilson Mar

Cucumber testing

Business features

Gherkin language

Capybara

Execution

Installation

Different execution frameworks

Modules to install

Modules to run

BDD workflow

SpecFlow for .NET

Step definitions

Annotations to avoid duplicates

End-to-end

Selenium

Alternatives

Install Cucumber server

Use Dockerfile to image

Use Docker image

Binary install

Feature files

Tags

Scenarios

Helpers in IDEs

User stories

Key websites

Video Training

You might also enjoy (View all posts)