Wilson Mar bio photo

Wilson Mar

Hello!

Calendar YouTube Github

LinkedIn

Cucumber testing

Gherkin user specs drive Selenium tests for behavior-driven development (BDD) within Agile

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

Cucumber is an open-source server program that tests applications based on specification of that system’s features (behavior) written in the “Gherkin” language.

Cucumber is often used for what some call “User Acceptance Testing” (UAT) to determine if the system built should be accepted based on user specs defined in Gherkin. When written before development, Cucumber enables BDD (Behavior Driven Development).

Business features

PROTIP: Defining a full scope of features enables features to have a better naming conventions.

Examples of business functions are:

  • Login admin for admin menu.
  • Create user (registration).
  • Login user.
  • Logout.
  • Password recovery.

  • Delete account.

  • Share photos or videos.
  • Send a friend request.
  • Approve requests.

Additionally, each feature may have some error recovery features.

PROTIP: It’s better NOT to include the name of the app under test as part of the feature name in order to make the files more reusable with other apps.

PROTIP: Identify negative test cases (those designed to fail in order to test error handling).

Gherkin language

Gherkin is used for “Black box testing” where the person executing the test does not get into the internal engineering of the app.

The Gherkin language is designed to be a “natural” (English-like readable) language designed to be understandable by business people (unlike coding in Java, Go, Python, etc. which are written for reading by specially trained programmers.)

Gherkin code can be edited with any text editor.

“Having conversations is more important than capturing conversations is more important than automating conversations.

VIDEO: An Introduction to BDD (with Cynefin) by Liz Keogh (@lunivore) says BDD is an analysis technique. Her example [4:18]:

Feature: Addition
   In order to avoid mistakes
   As a math idiot
   I want to be told the sum of two numbers
      Given I have entered 50 into the calculator
      And I have added 70 into the calcualator
      When I add the numbers together
      Then the result should be 120 on the screen

An example of a feature file (with .feature file suffix) which describes a single feature:

# language: en
Feature: Site Search
Scenario:
  Given We navigate to the homepage
  When We search for the word agile
  Then The results for the search will be displayed
  Examples:
    |foo|
    |bar|

The Gherkin language is simple. It doesn’t require matching parentheses and semicolons at the end of each line.

Indentation is used to define structure. Like Python and Bash files, pound signs are read as comments and not parsed.

The sample above does not need to begin with a language specification because English is the default. Gherkin can be written in 60 languages. Feature files using the Norwegian language (“no”) begin with “Egenskap” instead of “Feature” in a header line:

# language: no
Egenskap: Summering

Each feature can consists of one or more scenarios. Each scenario consists of steps specified by “Given”, “When”, and “Then” keywords.

Rules of Gherkin “Treetop” grammar are defined in the Gherkin language standard at , see https://cucumber.io/docs/reference

Some call the Gherkin code “living” specifications because there is an automated way to keep specifications in sync with what the system created.

Thus, Cucumber bridges the “communication gap between domain experts and developers”. Cucumber enables text about what business people require to be executed as tests.

The behavior desired by a system are defined by Gherkin specifications that are executable.

Alternately, some testing utilities (such as est.ai, testim.ai, etc.) create Gherkin by automatically scanning apps to identify page objects.

Capybara

Capybara is the part of Cucumber that acts like a user on a web browser. It is for “integration testing”. The Rails tools simulates how a user would interact with a website. Capybara puts applications “through its paces” by simulating a user and browser performing the steps of the scenarios – interacting with the app to receive pages, parse the HTML, and submit forms, just as a real user would.

By storing features in files along with different scenarios of feature use composed of many steps, and storing Ruby code in separate files containing step definitions that tests each type of step, Cucumber and Capybara automatically test the behavior of the SaaS app.

Execution

The Gherkin interpreter reads text written like spoken language.

When the Gherkin interpreter encounters a keyword such as “navigate” and “search”, Cucumber looks into a subdirectory specifically called “step_definitions”.

Feature files can be executed by several software testing platforms:

  • Ruby on Rails
  • Selenium (Java)
  • PicoContainer
  • Spring Framework
  • Watir for Ruby and WatiN for dot Net

See https://github.com/cucumber/cucumber/wiki/Gherkin

The cucumber server can be started using a command such as:

cucumber -f pretty -f html -o report.html

NOTE: An example of cucumber’s HTML output:

In the US, green is used on street lights to indicate “pass”. Cucumbers are green. Thus the name of the software. Gherkins are small pickled cucumbers.

For fancier formatting:

  • https://www.npmjs.com/package/multiple-cucumber-html-reporter
  • http://toolsqa.com/selenium-cucumber-framework/cucumber-reports/
  • https://www.tutorialspoint.com/cucumber/cucumber_reports.htm
  • Image from https://www.npmjs.com/package/cucumber-html-report

Send results to “status of done” dashboards such as SonarQube, Grafana, etc.

Send run information to Hygieia or other DevOps dashboards.

runTest.java

package Annotation; 
 
import org.junit.runner.RunWith; 
import cucumber.junit.Cucumber; 
 
@RunWith(Cucumber.class) 
@Cucumber.Options(format = {"pretty", "html:target/cucumber"}) 
 
public class runTest { }

Installation

There is a Bash script that installs the various Cucumber modules on a Mac:

https://github.com/wilsonmar/mac-setup

Cucumber consists of a set of software modules. When “cucumber” is specified among TEST_TOOLS, several modules need to be installed with it.

Selenium

mvn

Different execution frameworks

Execution frameworks provide assertions and verifications. Assertions determine whether each step succeeded. Verifications determine whether ???

Modules to install

Cucumber’s GitHub library provides code in programming languages using dependency management and verification frameworks for each language:

Programming
script		Dependency
manager		Verification
framework
C# nuget NUnit and MSTest
Java maven, gradle, ant JUnit or TestNG
Javascript (in NodeJs)- -
Perl (abbreviated as 3P)- -
PHP--
Python pip Behave, Nose
Ruby - Spock or RSpec

For Java environments, see https://drive.google.com/drive/folders/0B5v_nInLNoqualhZWHk3eGFfd0k

Modules to run

./script_name.sh features/registration.feature --format html --expand

The arguments specify what features are run, referenced by the “$@” below:

#!/bin/sh
bundle install
rake db:migrate
rake db:test:prepare
rake cucumber "$@"

Many Agile teams have embraced use of Gherkin language in a Behavior Driven Development (BDD) approach to software development.

BDD workflow

Because Gherkin is a language common to both business people and technical programmers, those who know what needs to occur and developers who write programs. Thus, Gherkin enables collaboration among “Three Amigos”:

  • Business Analyst who understands user needs and concerns
  • Developer who will do the work and make it real
  • Tester who discover problems before work starts

BDD is focused on what to test, not how to test.

PROTIP: A key part of BDD is the collaborative discussion among the stakeholders rather than silos not talking with each other.

cucumber-bdd-545x412-54158.jpg

BDD works at a higher level than unit tests of the TDD (Test Driven Development) approach to software creation. As with TDD, the BDD approach builds software incrementally, by writing feature files that fail before writing code that makes the tests pass. In fact, it is an anti-pattern to write Gherkin after the software.

The “driven” part of BDD means that business people can specify what the computer should do in a way that can be actually verified. It’s more rigorous that the traditional approach of writing “requirements” documents (in Microsoft Word).

The team has a mechanism to define and verify what “done” really means.

This is a big time saver to business people because the automation enables everything previously defined to be verified again and again after even the most misicule of changes. This enables agility with quality.

THIS presents a good comparison between Behavioral and Functional statements (which reference HTTP codes).

When specs are automatically (Cucumber, etc.), specifications remain up-to-date.

Dan North (@tastapod in London UK) and Liz Keogh created a BDD framework, JBehave,[1] followed by a story-level BDD framework for Ruby called RBehave[9] which was later integrated into the RSpec project. https://relishapp.com/rspec The group also worked with David Chelimsky, Aslak Hellesøy (@aslak_hellesoy Norwegian in London), and others on “The RSpec Book: Behaviour Driven Development with RSpec, Cucumber, and Friends”. The first story-based framework in RSpec was later replaced by Cucumber mainly developed by Aslak Hellesøy.

Other frameworks include JBehave, JDave, Easyb, etc.

SpecFlow for .NET

SpecFlow is Cucumber for .NET environments. To install the SpecFlow.Nunit package, see Jason Roberts (@RobertsJason) Automated Acceptance Testing with SpecFlow and Gherkin for Visual Studio and C# coding [2:51] 28 Oct 2013

Step definitions

The mapping between each step of the scenario defined in the Gherkin feature file and the code of each function to be executed is defined in steps definition file stores.

public void goToFacebook() { 
   driver = new FirefoxDriver(); 
   driver.navigate().to("https://www.facebook.com/"); 
} 
@When "^user logs in using Username as \"([^\"]*)\" and Password as \"([^\"]*)\"$"
public void I_enter_Username_as_and_Password_as(String arg1, String arg2) {
   driver.findElement(By.id("email")).sendKeys(arg1);
   driver.findElement(By.id("pass")).sendKeys(arg2);
   driver.findElement(By.id("u_0_v")).click(); 
} 
@Then"^login should be unsuccessful$" 
public void validateRelogin() { 
   if(driver.getCurrentUrl().equalsIgnoreCase(
      "https://www.facebook.com/login.php?login_attempt=1&lwv=110")){ 
         System.out.println("Test Pass");
   } else { 
      System.out.println("Test Failed"); 
   } 
   driver.close(); 
}

@When “^user logs in using Username as "([^"])" and Password as "([^"])"$”

Annotations to avoid duplicates

Under src/test/java, create an “outline.feature” file such as:

Feature: annotation 
#This is how background can be used to eliminate duplicate steps 
 
Background: 
   User navigates to Facebook Given 
   I am on Facebook login page 
 
#Scenario with AND 
Scenario: 
   When I enter username as "TOM"
   And I enter password as "JERRY" 
   Then Login should fail 
 
#Scenario with BUT 
Scenario: 
   When I enter username as "TOM" 
   And I enter password as "JERRY" 
   Then Login should fail 
   But Relogin option should be available

End-to-end

QUESTION: How is Gherkin different than Rules specified in rule-based systems?

  • PicoContainer
  • Salesforce
  • Spring Framework

Selenium

How to test is the job of Selenium.

The first line of a Selenium program:

require 'selenium-webdriver'
driver = Selenium::WebDriver.for :firefox

Selenium makes use of webdriver to control web browsers.

On Windows:

    java -jar selenium-server-standalone-2.35.0.jar -Dwebdriver.chrome.driver="D:\dev\chromedriver.exe"

So Selenium is more suited to “end to end” testing, for example with  nightwatch.js.

Karma is a test runner designed for unit tests, so it is much harder to achieve end to end tests on it, you can add a phantomjs launcher, but it is not the same as real browser tests with Selenium. I think both of them can run any js testing framework if you have an adapter… Mocha, jasmine, qunit, etc…

In Visual Studio, highlight a scenario line and right-click to Generate step definitions.

Mocha is used for unit testing. It’s said that Mocha has better asynchronous support than Jasmine. Jasmine is used for end-to-end testing.

The JUnit plug-in for Cucumber called “cucumber-html-report” tracks scenarios and steps tested and produces HTML reports.

Alternatives

Jest, being from Facebook, at https://facebook.github.io/jest/ is focused on JavaScript-based apps built using Facebook’s React.

Install Cucumber server

There are several ways to obtain a running instance of Cucumber, listed from easiest to most difficult:

A) You don’t need a local machine if you use the cloud service

B) Pull an image from Docker Hub within a Google Compute or AWS cloud instance.

C) Use the Dockerfile to build your own Docker image containing JMeter.

D) Use Homebrew to install JMeter natively on you Mac.

E) Download installer to install locally.

CAUTION: If you are in a large enterprise, confer with your security team before installing. They often have a repository such as Artifactory or Nexus where installers are available after being vetted and perhaps patched for security vulnerabilities.

  1. Create within your internal cloud, Google Cloud, Amazon EC2, Microsoft Azure, etc. a VM instance of an Ubuntu server. 4 GB RAM and 10 GB drive is the minimum.

    A sample command to create a Google Cloud instance:

    gcloud beta compute --project "attiopinfosys" instances create "cucumber-1" --zone "us-central1-f" --machine-type "n1-standard-1" --subnet "default" --maintenance-policy "MIGRATE" --service-account "776279044010-compute@developer.gserviceaccount.com" --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring.write","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --min-cpu-platform "Automatic" --tags "http","https","web","http-server","https-server" --image "ubuntu-1604-xenial-v20171026a" --image-project "ubuntu-os-cloud" --boot-disk-size "10" --boot-disk-type "pd-standard" --boot-disk-device-name "cucumber-1"

    Use Dockerfile to image

  2. Install Git and Docker.

  3. Run the Dockerfile at

    https://raw.githubusercontent.com/wilsonmar/Cucumber/master/Dockerfile

    Its contains:

    FROM ubuntu:16.04
RUN apt-get update
RUN apt-get update && apt-get install -y \
  default-jre \
  default-jdk \
  git \
  maven 
 
RUN mvn -version
RUN git clone https://github.com/cucumber/cucumber-jvm.git --depth=1
CMD ls
RUN cd cucumber-jvm/examples/java-calculator --depth=1 && mvn test

    The above provides commands to install Cucumber within a blank Docker container.

    cucumber-jvm/examples/java-calculator is a simple sample app, which is replaced with a real app in the real world.

    Use Docker image

  4. Install Docker.

     docker pull graze/cucumber-rest-bdd

  5. Run the image:

    docker run --rm -v $(pwd):/opt/src -e endpoint=http://server/ graze/cucumber-rest-bdd

Binary install

Cucumber server software was written in the Ruby programming language. Ruby is the “R” in RSpec, which defines the language grammar.

So Cucumber is supplied as a Ruby gem.

  1. Install the Ruby interpreter.

  2. Declare that our Ruby app depends on this gem and use Bundler to install it.

    gedit Gemfile
... add 'gem
bundle install

Feature files

Cucumber executes files named with a .feature file extension.

Within that file, the keyword “Feature” is on the first line.

PROTIP: Some teams find it useful to hold a feature brainstorm session to identify as many features as they can think of right away, to get all ideas “on the table” before diving into more detail.

Tags

The team would then be in a position to assign various tags to group features in order to tease out duplicates and to classify each feature as:

  • @HappyPath
  • @ErrorPath
  • @EdgeCase

Tags are “arbitrary” in that they can be about any category.

@ignore is recognized as a feature not to be executed during testing.

@important

@wip identify work-in-process that need more attention.

@humanexecuted identify features that are executed manually.

@weekly, @hourly, @daily, @yearly, etc. identify when they are executed.

Other tags, for example:

  • Public (before login)
  • Login Authentication
  • Navigation
  • Purchase
  • Post-sale

Ideally, each feature should be independent of and isolated from others. So a feature can include several steps to perform.

The team then can dive into estimating and elaboration.

Scenarios

Scenarios are examples of expected system behaviors. Each scenario describes a particular situation that uses the feature.

Sentences not starting with any of these keywords are treated as comment documentation. A fanciful example[1]:

Feature: Brush teeth
   In order to stay pain free
   As a human
   I want to brush my teeth

There are other keywords:

  • Background provides setup setups common to several scenarios.

  • Given to set-up initial state

  • Scenario describes a concrete example illustrating a business rule. It consists of a list of steps.

Scenario Outline

  • When to perform action(s)

  • Then for the end-state to check

  • And adds additional items as the keyword above it.

  • But is a counter-example on undesired side-effects

Background - to avoid repeating givens in all scenarios.

PROTIP: Instead of English words, French, Japanese, and other words can be used with installation of the target translation. This is because Gherkin is known as a Domain-Specific-Language (DSL).

Helpers in IDEs

SpecFlow is an extension installed into Visual Studio. SpecFlow features are

  • SpecFlow Feature File
  • SpecFlow Hooks (event binding)
  • SpecFlow Step Definition

User stories

A User Story is a low fidelity (inexact) description of a use case.

From https://saascourse.blogspot.com/2012/03/feature-add-med.html

Feature: user can add a medication

Scenario: add a medication
   Given I am on the patient medications screen
   When I follow "add new medication"
   Then I should be on the "New Medication" page
   When I fill in "Med Name" with "Aspirin"
   And I select "Route Oral"
   And I press "Return"
   Then the list of matched medications should be displayed in the list
   And I should see Aspirin Tablets

Steps generally

  • Given
  • Perform action(s)
  • Check end state

Key websites

  • https://cucumber.io/school has 3 hours of videos for $199 either Java or Ruby
  • http://cukenfest.cucumber.io/ Cukenfest conf. was in London April 17-10
  • https://cucumber.io/docs/reference
  • https://github.com/cucumber
  • https://github.com/cucumber/cucumber/wiki/A-Table-Of-Content
  • https://en.wikipedia.org/wiki/Cucumber_(software)
  • selenium-users@googlegroups.com

Video Training

Mukesh otwani at https://learn-automation.com created a series of videos about Cucumber

Matt Wynne and Aslak Hellesøy authored The Cucumber Book and “Cucumber Recipies” from Pragmatic Press.

  • https://pragprog.com/titles/hwcuc/source_code
  • https://www.tutorialspoint.com/cucumber/cucumber_overview.htm

Greg Bee takes issue with their REST API example containing HTTP GET and JSON, then presents a more readable version with explanations:

  • http://gregbee.ch/blog/effective-api-testing-with-cucumber

From 2012

  • https://saascourse.blogspot.com/2012/03/cucumber.html

Ripon Al Wasim (https://twitter.com/riponalwasim”>@riponalwasim</a>) released through Packt on 12/23/15 the “Mastering Selenium Testing Tools” video course. It is also available via Udemy and LinkedIn/Lynda.

Thomas Sundberg

Kevin Skoglund release the “Ruby: Testing with RSpec” video course also available via LinkedIn/Lynda.

Others:

  • https://github.com/cucumber/cucumber/wiki/Cucumber-Backgrounder
  • http://benmabey.com/2008/05/19/imperative-vs-declarative-scenarios-in-user-stories.html
  • http://dannorth.net/2011/01/31/whose-domain-is-it-anyway/
  • http://elabs.se/blog/15-you-re-cuking-it-wrong
  • https://github.com/intuit/karate

Using the Gherkin Language to Write User Stories Tutorial by Reval Govender

An Introduction to BDD by Konstantin Kudryashov of Inviqa

Behaviour Driven Development - Dan North Was Right + FREE CHEAT SHEET Development That Pays

Test Driven Development vs Behaviour Driven Development + FREE CHEAT SHEET Development That Pays

What is BDD? What is Behavior Driven Development? Development That Pays

Gherkin Best Practices by Jared

Interview with Dan North on Behavior-Driven Development Vladimir Gitlevich

Beauty in Code 2019, 6 of 6 — Dan North: “BDD Is Not About Testing”

Introduction to BDD with Cucumber Conygre

What is BDD (example with Cucumber)? QAShahin

  1. Cucumber Framework || BDD || Data Table to List Data Structure</a> Rahul Rathore

BDD all the way down by Enrico Campidoglio FooCafe

[5:10] BDD vs TDD (explained) by Llewellyn Falco

  1. Specification
  2. Feedback
  3. Regression
  4. Granularity - info when tests fail

Cucumber testing was published on .