Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

No waiting for test automation


Overview

I have a dream …

where developers get their test scripts not by waiting for manual coding, but by

fig test code gen v01 998x568.pptx

  1. test code generation programs referencing

  2. a specification database

  3. also used to create client applications making calls to servers.

  4. All this done by a toolchain of Continuous Integration/Continous Deployment.

  5. The specifications can be specified manually or generated from programs which scan programming code to extract specifications.

  6. Variations in the generated code are defined in a control file.

  7. This control file can be edited manually using a text editor and manually referencing the specifications,

  8. adjusting for issues observed by the generator.

  9. Results from test generator run include timings of transactions and size of resources processed on the client and

  10. metrics from monitoring of servers.

  11. Analysis of these results are made in light of the original specifications.

  12. Insights from human or artificial intelligence doing the analysis would influence what testing code to generate in subsequent rounds.

What took us so long?

I think that when software vendors focus on tools for testers, they may miss or dismiss the idea of eliminating them.

The automation discusseed here has the objective of “testers” being developers themselves – developers of test code generation programs.

The audience for this page are developers who want to move faster.

Client applications

Let’s for now focus on examples of clients making REST API calls.

  • Yahoo Weather

  • Google Weather

  • Twitter http://www.ibm.com/developerworks/library/x-twitterREST/index.html

  • https://developer.salesforce.com/page/REST_API

Tutorials on APIs include:

  • http://www.restapitutorial.com/

Specification databases

Several databases for APIs have grown up over the years:

Specifications for REST API (such as Swagger) are more amenable as the basis for generating code if they are complete specifications of how client computers interact with servers.

Specifications about screen elements may be too complex or don’t provide enough information for use in code generation at this point in time.

  Swagger RAML API-Blueprint
Format: JSON YAML Markdown
Sponsor: Reverb Mulesoft Apiary
Initial: July, 2011 Sep, 2013 April, 2013
Approach: Bottom-up Top-down Top-down

Other alternatives include:

RAML

RAML (at RAML.org) can reuse WADL (pronounced “waddle”) introspection logic of Restlet Framework resources developed by Java (SE/EE, Google AppEngine, OSGi, GWT, Android) REST API developers using the Restlet Studio and the APISpark cloud managed by Reslet based in France with an office in Palo Alto.

Reslet (in Oct. 2015) bought Czech Filip Kolařík’s DHC (Dev HTTP Client) Chrome Add-in and https://www.sprintapi.com/dhcs.html to test and debug web APIs

Swagger2RAML converts Swagger JSON to RAML YAML.

API Description Languages: Which One Is Right For Me? Aug. 15, 2014 by Laura dot Heritage at soa.com (Akana, formerly SOA Software).

WADL Example

Let’s use the sample yahoo-news-call.wadl.xml file from the Wikipedia page defining one. Open the file using an XML reader app.

  1. The service endpoint is specified by:

    
    <resources base="http://api.search.yahoo.com/NewsSearchService/V1/">
    

Test code

There are several languages used by test script processors:

JavaScript LoadRunner Example

An example of the JavaScript code generated to emulate a client retrieving an index.html file:

  web.url(
    {
      name : 'index.html',
      url : '{pEndPoint}',
      resource : 0,
      recContentType : 'text/html',
      referer : '',
      snapshot : 't1.inf',
      mode : 'HTML'
    }
  );

“{pEndPoint}” is the run parameter containing an End Point such as:

http://api.search.yahoo.com/NewsSearchService/V1/

The “mode: HTML” specifies processing by LoadRunner to scan the html returned and issue requests for links to CSS, JavaScript, images, etc.

(Images specified within CSS are not retrieved this way)

Java Selenium Example

TODO:

Generator Control File

We use the yml format that can be read by humans and computers. Ruby programmers are used to such files.

Since Swagger users may prefer JSON formatting.

  1. Input Specification database

  2. Output Test code language

  3. Authentication.

  4. Call Method (GET, PUT, etc.).

  5. Authorization (such as tokens in the HTTP header).

  6. Response code (if not standard 200).

  7. Repetitions.

  8. Negative tests (to force 400 to return).

  9. Data errors (bad attribute names, bad attribute value).

CI/CD Toolchain

Test code generation programs

I have examples of code generation programs written in Python, PHP, Java, Scala, and even VBScript in Excel files.