Wilson Mar bio photo

Wilson Mar

Hello!

Calendar YouTube Github

LinkedIn

Use the OpenCV to recognize a portion of your screen and have Selenuium Python click on a portion of it

US (English)   Norsk (Norwegian)   Español (Spanish)   Français (French)   Deutsch (German)   Italiano   Português   Estonian   اَلْعَرَبِيَّةُ (Egypt Arabic)   Napali   中文 (简体) Chinese (Simplified)   日本語 Japanese   한국어 Korean

Overview

This is a deep dive from the innnards up:

NOTE: Content here are my personal opinions, and not intended to represent any employer (past or present). “PROTIP:” here highlight information I haven’t seen elsewhere on the internet because it is hard-won, little-know but significant facts based on my personal research and experience.

OpenCV (Open Computer Vision) is a utility used to recognize objects in the pictures, and the coordinates of objects it recognizes in pictures. Its recent adoption of Machine Learning (DNN by Aleksandr Rybnikov) has increased its accuracy. It’s used by many Artificial Intelligence apps such as self-driving cars to understand images.

Why?

The benefit of repeating some action repeatedly (and not get bored or distracted) extends to:

  • Videographers making screencast videos in multiple takes
  • Salespeople use it to power their live demos.
  • Developers and testers going through a sequence of actions to reach a particular point in the UI needing debugging.

Here’s how it works:

  1. A picture is taken of your screen(s).
  2. The text and location of objects recognized in the picture is fed to your program to click or type.
  3. Your program then takes another round of picture taking
  4. visual object recognition to ensure the results are what are intended.

This uses a different, more generalized, approach than packages such as HP UFT which depended on finding identifiers in the DOM (Document Object Model) behind each internet browser web page.

Versions

Sikuli was started somewhen in 2009 as an open-source research project at the User Interface Design Group at MIT by Tsung-Hsiang Chang and Tom Yeh.

RaiMan created an IDE (using Java) and got APIs for Java, Python, and Ruby. His SikuliX2 (version 2.0.1 as of ) at
https://github.com/RaiMan/SikuliX2

https://github.com/RaiMan/Sikulix2opencv

  • https://github.com/RaiMan/SikuliX1/releases

  • https://github.com/RaiMan/SikuliX1/wiki/ZZZ-Bug-Fixes

  • https://github.com/RaiMan/SikuliX1/wiki/macOS-Linux:-Support-Libraries-for-OpenCV-4

Others have created installers:

A) https://github.com/wernight/docker-sikulix

B) https://github.com/dhoer/chef-sikulix

C) Download and compile it locally*

exec java -Xmx512M -Dfile.encoding=UTF-8 -Dsikuli.FromCommandLine -jar /usr/share/sikulix/sikulix.jar “$@”

Instructions below are based on Mac 10.14 (Sierra) referencing http://sikulix.com/quickstart.

All in a Terminal:

### Pre-requisites

  1. Install JRE 8.
  2. Install the Java language IDE you want to use (Eclipse or IntelliJ).
  3. Install Eclipse or Robot Framework (which uses Python).
  4. Install Python 3.7.

    OpenCV install

  5. Get the latest version of OpenCV:

    brew install opencv
    ==> Pouring opencv-4.1.2.mojave.bottle.1.tar.gz
    Warning: opencv dependency gcc was built with a different C++ standard
    library (libstdc++ from clang). This may cause problems at runtime.
    🍺  /usr/local/Cellar/opencv/4.1.2: 753 files, 228MB
    

    Tesseract install

    Sikuli IDE from Raiman

  6. To avert Issues in GitHub, disable ClearType in Windows Settings.

  7. Navigate to a folder:

  8. Download the ready-to-use jar (Java Archive):

    curl -O https://launchpad.net/sikuli/sikulix/2.0.1/+download/sikulix-2.0.1.jar
  9. Invoke the Sikuli IDE:

    java -jar sikulix-2.0.1.jar</pre> The IDE should appear: sikuli-2 0 1-ide-menu Alternate command line parameters. -v or -d 3 sets the debug level. -c redirects output from IDE message area to the command line. The IDE running redirect all debug output to a file named SikulixDebug.txt in your home folder. ### Take a picture Create a small picture where you want Sikuli to click or type:
  10. In the IDE, Click “Take a picture” or press a hot-key (Command + Shift + 2).
  11. Drag the upper-left to the lower-right, then release the mouse.

    QUESTION: If a dialog disappears with mouse motions, press the keystroke to take a picture of the screen, then use a photo editing program to cut out an area, and paste the little picture into the IDE.

    Script

  12. Type the debug level inline:

    Debug.on(3) # switch debugging on with level 3
    ...
    Debug.off() # switch debugging off
    

    # inserts a comment.

  13. Press Ctrl+S to save the file as a “.sikuli” file.

    PROTIP: A keyboard press is quicker than moving the mouse.

    SikuliX Actions and Methods

  14. Open an app:

    PathFirefox = r"C:\Program Files (x86)\Mozilla Firefox\firefox.exe"
    App.open(PathFirefox)
    

    PROTIP: Instead of the hassle of using double slashes to specify special characters, put a 4 in front of calls to take the backslash and quotes as part of the word/sentence.

  15. Define variables:

    imageExpected1=3;
    
  16. Define action:

    click(SearchImageButton)
    wait(imageExpected1)
    type("hello")
    
  17. Type the verification actions.

  18. Save a sample script with comments after # character:

    On a Mac:

    http://doc.sikuli.org/tutorials/helloworld/helloworld-mac.html

    On Windows:

    http://doc.sikuli.org/tutorials/helloworld/helloworld-win.html

  19. Run the file.

    More actions

    http://doc.sikuli.org/genindex.html is an alphabetical index of all methods and command options.

    JavaDocs of classes are at
    http://nightly.sikuli.de/docs/index.html

    https://github.com/RaiMan/SikuliX-2014/tree/master/StuffContainer/testScripts
    has several sikulix scripts.


  1. Type S and double-click the SikuliX IDE app to open it.

    sikulix starting popup 198x154

    The IDE should appear:

    sikulix-1 1 1-ide-landing-1022x699
    (Click to open image in full new screen)

The installer installs its own version of Python.

  1. On a Mac, create a version-specific installer folder on your $HOME folder:

    cd ~
    mkdir sikulix-1.1.2-install
    cd    sikulix-1.1.2-install
    
  2. Download from http://nightly.sikuli.de right-click and select Save link As … the link that says:

    sikulixsetup….jar

    At the time of writing, the URL is:

    https://oss.sonatype.org/content/groups/public/com/sikulix/sikulixsetup/1.1.2-SNAPSHOT/sikulixsetup-1.1.2-20170316.001623-90-forsetup.jar

    By the time you read this, the date in the link will change because the code is actively worked on.

  3. Click the “md5” for the MD5 hash (a33616bac6d4f44785b89a02b110a0f8) to verify download integrity using whatever tool you use.

  4. Expand the installer (in Downloads):

    java -jar sikulixsetup-1.1.2.jar</pre> This creates file:
    SikuliX-1.1.2-SetupLog. (which reflects what goes to the Terminal) and an empty SetupStuff folder.
  5. Open the txt file and refresh it to see progress.

  6. Click checkboxes for Jython and for Tesseract in the UI pop-up, then Setup Now.

    sikulix 1 1 mac setup1 1974x1028.png
    (Click to open in full new screen)

  7. Click Setup Now for:

    sikulix-1-1-setup2 976x816
    (Click to open in full new screen)

  8. Click Yes for:

    sikulix-1 1-setup3 1424x294
    (Click to open in full new screen)

  9. Click No to pop-ups about Mac native libs (jar’s) previously installed into /Users/___/Library/Application Support/Sikulix/SikulixDownload_201610061722/…

  10. Click Yes to download a stand-alone version of Python (No to get Jython 2.5 due to 2.7 UTF-8 warning):

    sikulix-1 1 jython 856x368
    (Click to open in full new screen)

    Success!

  11. Click 3 the boxs OK to the pop-up:

    sikulix-hello-working-269x156

  12. Click “OK” to the “Hallo” pop-up.

    Clean up installers

  13. In Finder, Move to Trash the installer file sikulixsetup-1.1.0.jar

    After Install

    QUESTION: How to verify a good install and use the program?

  14. In Finder, move sikulix.app to your user’s Applications folder.

  15. In case you’re curious, view the folders (not necessary unless you run into problems)

    • Extensions folder
    • Lib folder contains the supporting stuff for Jython and JRuby
    • SikulixDownloads
    • SikulixDownloads_versionstamp contains
    • SikulixLibs_versionstamp contains native libraries
    • SikulixStore
    • SikulixTesseract

    Note different files are downloaded for Windows vs. Mac:

    • sikulixjython.jar
    • sikulixlibsmac.jar
    • sikulix.jar
    • sikulixapi.jar

Sample scripts

  1. PROTIP: Search for Sikulix scripts within GitHub:

    Here’s one:

    https://github.com/lowkorn/SikuliX

  2. Click the Example folder, JavaCompile.sikuli, then JavaCompile.py.

  3. If you intend on updating this repository and contributing changes back to the author, sign-in to your own account and fork the repository under your own account.

  4. PROTIP: Use Git to store Sikuli scripts. Install a Git client.

  5. PROTIP: Create a Git folder to hold files associated with GitHub.

    On a Mac:

    cd ~
    mkdir gits
    cd gits
    
  6. Create a folder to hold sikuli repositories created when cloning from others:

    cd ~/gits
    mkdir sikulix
    cd sikulix
    

    You are now in the folder where you clone scripts from other repositories on GitHub.

  7. Create a folder to hold repositories from lowkorn:

    mkdir lowkorn
    cd lowkorn
    
  8. Clone one I’ve found:

    git clone https://github.com/lowkorn/SikuliX
    

    Response:

    Cloning into 'SikuliX'...
    remote: Counting objects: 34, done.
    remote: Total 34 (delta 0), reused 0 (delta 0), pack-reused 34
    Unpacking objects: 100% (34/34), done.
    

    Repositories cloned locally you can later update with changes made, using commands such as:

    git remote add upstream https://github.com/lowkorn/SikuliX
    git remote -v
    git pull upstream master
    

    Response:

    remote: Counting objects: 34, done.
    remote: Total 34 (delta 0), reused 0 (delta 0), pack-reused 34
    Unpacking objects: 100% (34/34), done.
    From https://github.com/lowkorn/SikuliX
            * branch            master     -> FETCH_HEAD
            * [new branch]      master     -> upstream/master
    

    Alternately, to do a two-stage sync per https://help.github.com/articles/syncing-a-fork/

    git fetch upstream
    git checkout master
    git merge upstream/master
    
  9. On GitHub.com, sign-in to your own account and create a repository (named sikulix-scripts in this example) with a READM.md file.

  10. Clone your own sikulix-scripts repo to your laptop (in this example, substitute “wilsonmar” with your own account name):

    On a Mac:

    cd ~/gits/sikulix
    mkdir wilsonmar
    cd wilsonmar
    git clone https://github.com/wilsonmar/sikulix-scripts
    cd ~/sikulix-scripts
    

    Into this folder the SikuliX IDE should read and save scripts.

SikuliX Commands

SikuliX can be called by batch scripts from a Terminal/Run command line.

SikuliX recognize both one-character flags and longer options:

sikulix –args -i

-t or –test sikuli-file

-r or –run sikuli-file

-s or –stderr

Videos

jobiaj

  1. Intro and setup 13 Nov 2015

https://www.youtube.com/watch?v=3LRm3fJQ6AU

  1. The Basics shows how to (in Windows) open the Calculator and click a calculation.

  2. Conditional Automation

  3. Conditional Looping

  4. Game Example

Others:

https://www.youtube.com/watch?v=8y41vKeGM9U 5 Apr 2016

https://www.youtube.com/watch?v=I-UYoezac4Q

https://www.youtube.com/watch?v=3LRm3fJQ6AU 24 Jul 2016

https://www.youtube.com/watch?v=H2cK85z_8bs Keywords and Misc Stuff</a> 17 Nov 2016

Playlist of Sikuli scripting (no sound)

Sample scripts

  1. Navigate to where you can open a .sikulix file

    sikulix open sample

  2. Switch to an internet browser:

    woodplastic has a Windows cmd file to invoke a jar file

    wait(Pattern("1437608642600.png").similar(0.90),15)
    click(Pattern("1437608666644.png").similar(0.90).targetOffset(82,125))
    



  3. Let me know if you find others.

  4. PROTIP: Additional Sikuli scripts from the internet:

  5. Let me know if you find others.

Alternatives

Java

https://github.com/lowrin/sikulix-test/blob/master/src/main/java/Main.java

How to use Sikuli Java API - Part 2

Jython

https://github.com/saint1729/SikuliX-Jython

Robot framework

https://github.com/jaredfin/SikuliXRobotLibrary

Scala

https://github.com/haradakazuy02/SikulixScript

Cucumber: https://github.com/dilcom/sikulix_formatter

Fitnesse: https://github.com/sergebug/SikuliX-fixture

Ruby

https://github.com/rssdev10/sikulix-ide-templates

Resources

http://www.praqma.com/stories/automating-gui-tasks-with-sikuli-and-jenkins/

Social

Gitter channel at
https://gitter.im/RaiMan/SikuliX-2014

https://answers.launchpad.net/sikuli

Websites

http://sikulix.com is the new web page.

Detailed docs are at
http://sikulix-2014.readthedocs.io/en/latest/basicinfo.html#sikulix-system-specifics</a (with source at https://github.com/RaiMan/SikuliX-2014-Docs)

(Ignore abandoned sites:
http://sikulix.weebly.com/quickstart.html and
https://sikulix.wordpress.com)

http://sikulix-2014.readthedocs.io/en/latest/
has the most detail, which are repeated several places:

Installer bits are obtained from Ubuntu
https://launchpad.net/sikuli

https://github.com/wilsonmar/sikuli-scripts

  1. In your Mac’s Finder, get to Applications folder by pressing shift+command+A.

More on IoT

This is one of a series on IoT:

  1. IoT Acronymns and Abbreviations on Quizlet

  2. IoT Home Assistant system

  3. IoT Apprentice school curriculum
  4. IoT use cases
  5. IoT reminders prevent dead mobile battery
  6. IoT barn feeder

  7. IoT text to speech synthesis
  8. IoT AWS button
  9. Intel IoT
  10. IoT Raspberry hardware
  11. IoT Raspberry installation

  12. IoT Clouds
  13. Samsung IoT Cloud

NOTE: Pages about GE’s Predix have been removed.