Wilson Mar bio photo

Wilson Mar

Hello. Hire me!

Email me Calendar Skype call 310 320-7878

LinkedIn Twitter Gitter Google+ Youtube

Github Stackoverflow Pinterest

A robot sees what’s on your screen and clicks on it

Gitter

Overview

opencv-sikulix-v01-192x133.png

This is a deep dive from the innnards up:

OpenCV (Open Computer Vision) is a utility used by many Artificial Intelligence packages such as self-driving cars to understand images. It is used to take pictures, recognize objects in the pictures, and the coordinates of objects it recognizes in pictures.

SikuliX uses OpenCV to find the location of a picture provided to OpenCV. SikuliX uses the location returned from OpenCV to mouse click on the screen, or type as if a user is using a physical keyboard.

SikuliX does that and other methods based on a script in its own IDE or in response to an API called by a JavaScript scripting engine managed by Java, as organized by Selenium. Alternately, calls from Python code organized within the Robot Framework.

Optionally, Tesseract provides OCR (Optical Character Recognition) to extract text from pictures.

All this to make you more “productive” at playing games. ;)

The benefit of repeating some action repeatedly (and not get bored or distracted) extends to:

  • Videographers making screencast videos in multiple takes
  • Salespeople use it to power their live demos.
  • Developers and testers going through a sequence of actions to reach a particular point in the UI needing debugging.

Here’s how it works: A picture is taken of your screen(s). and the text and location of objects recognized in the picture is fed to your program to click or type. Your program then takes another round of picture taking and visual object recognition to ensure the results are what are intended.

This uses a different, more generalized, approach than packages such as HP UFT which depended on finding identifiers in the DOM (Document Object Model) behind each internet browser web page.

Websites

http://sikulix.com is the new web page.

(Ignore abandoned sites:
http://sikulix.weebly.com/quickstart.html and
https://sikulix.wordpress.com)

http://sikulix-2014.readthedocs.io/en/latest/
has the most detail, which are repeated several places:

Installer bits are obtained from Ubuntu
https://launchpad.net/sikuli

https://github.com/wilsonmar/sikuli-scripts

Social

Gitter channel at
https://gitter.im/RaiMan/SikuliX-2014

https://answers.launchpad.net/sikuli

Versions

There are three versions:

Detailed docs are at
http://sikulix-2014.readthedocs.io/en/latest/basicinfo.html#sikulix-system-specifics</a

Manual install

The steps to make it work:

  1. Install JRE
  2. Install Python
  3. Install Robot Framework (which uses Python)
  4. Install the IDE you want to use (IntelliJ)
  5. Download SikuliX
  6. Create a Git folder to hold your scripts.

Instructions below are based on run on Mac 10.12 (Sierra) referencing http://sikulix.com/quickstart/

Install-1.1.0

  1. If this is your first time:
    create a folder to hold the SikuliX installer:

    On a Mac:

    cd ~
    mkdir sikulix-1.1.0-install
    cd ~/sikulix-1.1.0-install
    

    (Capitalization is distracting)

    Otherwise, begin anew by removing the folder loaded by the installer:

    
    cd ~/Library
    cd "Application Support"
    rm -rf /Sikulix 
    cd ..
    
    
    The installer creates folders 
    to contain the Sikulix.app
    for use on Macs.
    
    On Windows: in the folder Sikulix in the folder pointed to by the environment variable %APPDATA%
    
    On Linux: in ~/.Sikulix
    
    
  2. As noted in this page, download the last stable version at https://launchpad.net/sikuli/sikulix/1.1.0/+download/sikulixsetup-1.1.0.jar

    sikulixsetup-1.1.0.jar is dated 2015-10-06, 3 MB

    NOTE: Code is at https://github.com/RaiMan/SikuliX-2014

    The next generation is active on
    https://github.com/RaiMan/SikuliX2

  3. Click the “md5” for the MD5 hash (a33616bac6d4f44785b89a02b110a0f8) to verify download integrity.

  4. Move the file to your skilux folder.

  5. Expand the installer:

    
    java -jar sikulixsetup-1.1.0.jar
    

    This creates file:
    SikuliX-1.1.1-SetupLog. (which reflects what goes to the Terminal) and an empty SetupStuff folder.

  6. Open the txt file and refresh it to see progress.

  7. Click checkboxes for Jython and for Tesseract in the UI pop-up, then Setup Now.

    sikulix 1 1 mac setup1 1974x1028.png
    (Click to open in full new screen)

  8. Click Setup Now for:

    sikulix-1-1-setup2 976x816
    (Click to open in full new screen)

  9. Click Yes for:

    sikulix-1 1-setup3 1424x294
    (Click to open in full new screen)

  10. Click No to pop-ups about Mac native libs (jar’s) previously installed into /Users/___/Library/Application Support/Sikulix/SikulixDownload_201610061722/…

  11. Click Yes to download a stand-alone version of Python (No to get Jython 2.5 due to 2.7 UTF-8 warning):

    sikulix-1 1 jython 856x368
    (Click to open in full new screen)

  12. BLAH: This error in the console stopped me from going further

    [debug (12/20/16 8:05:24 AM)] RunSetup: downloaded: tessdata-eng
    Exception in thread "main" java.lang.NullPointerException
      at org.sikuli.setup.RunSetup.main(RunSetup.java:845)
    
    
    PROTIP: We are forced to use nightly builds.
    
    
  13. Delete the installer folder.

Install-1.1.1 Dev build

  1. Create a version-specific installer folder:

    On a Mac:

    cd ~
    mkdir sikulix-1.1.1-install
    cd ~/sikulix-1.1.1-install
    
  2. Download from http://nightly.sikuli.de/ righ-click and select Save link As … the link that says:

    sikulixsetup….jar

    At the time of writing, the URL is:

    https://oss.sonatype.org/content/groups/public/com/sikulix/sikulixsetup/1.1.1-SNAPSHOT/sikulixsetup-1.1.1-20170316.001623-90-forsetup.jar

  3. Run the file:

    java -jar sikulixsetup-1.1.0.jar

  4. Check boxes as described above and install. The result:

    sikulix-hello-working-269x156

  5. Click “OK” to the “Hallo” pop-up.

    Note different files are downloaded for Windows vs. Mac:

    • sikulixjython.jar
    • sikulixlibsmac.jar
    • sikulix.jar
    • sikulixapi.jar

  6. In Finder, move sikulix.app to your user’s Applications folder.

  7. Still in Finder, Move to Trash the installer file sikulixsetup-1.1.0.jar

After Install

QUESTION: How to verify a good install and use the program?

only in case of problems it might be necessary to visit this folder:

  • Extensions folder
  • Lib folder contains the supporting stuff for Jython and JRuby
  • SikulixDownloads
  • SikulixDownloads_versionstamp contains
  • SikulixLibs_versionstamp contains native libraries
  • SikulixStore
  • SikulixTesseract

  1. Open the SikuliX IDE app from the Applications folder.

    sikulix starting popup 198x154

    then…

    sikulix-1 1 1-ide-landing-1022x699

    (Click to open in full new screen)

SikuliX Scripting

  1. Create a Git folder to hold scripts. In real life, you should create a folder containing the name of the app under test.

    On a Mac:

    cd ~
    cd gits
    cd wilsonmar
    mkdir sikulix-scripts
    cd ~/sikulix-scripts
    

    Into this folder the SikuliX IDE saves scripts with the file extension “.sikuli”.

  2. Press # to insert a comment

  3. Press Ctrl+S to save the file as a “.sikuli” file.

    PROTIP: A keyboard press is quicker than moving the mouse.

    Take a picture

  4. Create a small picture where you want Sikuli to click or type.

    This can be done by clicking the “Take screenshot” icon, positioning the mouse to the top-left corner, drag to the lower-right corner, then release the mouse.

    Or press a hot-key (Command + Shift + 2).

    QUESTION: If a dialog disappears with mouse motions, press the keystroke to take a picture of the screen, then use a photo editing program to cut out an area, and paste the little picture into the IDE.

    SikuliX Actions and Methods

  5. Open an app:

    PathFirefox = r"C:\Program Files (x86)\Mozilla Firefox\firefox.exe"
    App.open(PathFirefox)
    

    PROTIP: Instead of the hassle of using double slashes to specify special characters, put an 4 in front of calls to take the backslash and quotes as part of the word/sentence.

  6. Define variables:

    imageExpected1=3;
    
  7. Define action:

    click(SearchImageButton)
    wait(imageExpected1)
    type("hello")
    
  8. Type the verification actions.

  9. Save a sample script with comments after # character:

    On a Mac:

    http://doc.sikuli.org/tutorials/helloworld/helloworld-mac.html

    On Windows:

    http://doc.sikuli.org/tutorials/helloworld/helloworld-win.html

  10. Run the file.

    More actions

    http://doc.sikuli.org/genindex.html is an alphabetical index of all methods and command options.

    JavaDocs of classes are at
    http://nightly.sikuli.de/docs/index.html

    https://github.com/RaiMan/SikuliX-2014/tree/master/StuffContainer/testScripts
    has several sikulix scripts.

SikuliX Commands

SikuliX can be called from a Terminal/Run command line by batch scripts.

SikuliX recognize both one-character flags and longer options

sikulix –args -i

-t or –test sikuli-file

-r or –run sikuli-file

-s or –stderr

SikulixAppData folder

Tesseract

Originally from HP, @theRaySmith at Google says in 2016 Tesseract includes LSTM (machine learning with convolutional and deep belief networks).

BTW, the word tesseract means a representaton of a 4-dimensional cube (https://www.wikiwand.com/en/Tesseract). The word is used for the name of the library because the library works with multiple directional 3D lines.

  • https://github.com/gulakov/tesseract-ocr-sample
  • http://blog.ayoungprogrammer.com/2012/11/tutorial-installing-tesseract-ocr-30202.html/
  1. Install using Homebrew

    brew install tesseract

Selenium

More on IoT

This is one of a series on IoT:

  1. IoT Acronymns and Abbreviations

  2. IoT Apprentice school curriculum
  3. IoT use cases
  4. IoT reminders prevent dead mobile battery
  5. IoT ceiling dumper

  6. IoT text to speech synthesis
  7. IoT AWS button
  8. Intel IoT
  9. IoT Raspberry hardware
  10. IoT Raspberry installation

  11. IoT Clouds
  12. Samsung IoT Cloud

  13. Predix basics
  14. Predix installation
  15. Predix services
  16. Predix programming