A robot sees what’s on your screen and clicks on it
This is a deep dive from the innnards up:
OpenCV (Open Computer Vision) is a utility used by many Artificial Intelligence packages such as self-driving cars to understand images. It is used to take pictures, recognize objects in the pictures, and the coordinates of objects it recognizes in pictures.
SikuliX uses OpenCV to find the location of a picture provided to OpenCV. SikuliX uses the location returned from OpenCV to mouse click on the screen, or type as if a user is using a physical keyboard.
Optionally, Tesseract provides OCR (Optical Character Recognition) to extract text from pictures.
All this to make you more “productive” at playing games. ;)
The benefit of repeating some action repeatedly (and not get bored or distracted) extends to:
- Videographers making screencast videos in multiple takes
- Salespeople use it to power their live demos.
- Developers and testers going through a sequence of actions to reach a particular point in the UI needing debugging.
Here’s how it works: A picture is taken of your screen(s). and the text and location of objects recognized in the picture is fed to your program to click or type. Your program then takes another round of picture taking and visual object recognition to ensure the results are what are intended.
This uses a different, more generalized, approach than packages such as HP UFT which depended on finding identifiers in the DOM (Document Object Model) behind each internet browser web page.
http://sikulix.com is the new web page.
(Ignore abandoned sites:
Installer bits are obtained from Ubuntu
There are three versions:
1.1.0 final (available since 2015-10-07)
I’ve had installation errors with this version on my Mac.
There is an entry on 2015-10-07 in Maven Central:
but I haven’t tried to install it.
SikuliX-2014 (version 1.1.x) from
This is the one that worked for me.
Install instructions below.
SikuliX2 in 2017 at
The steps to make it work:
- Install JRE
- Install Python
- Install Robot Framework (which uses Python)
- Install the IDE you want to use (IntelliJ)
- Download SikuliX
- Create a Git folder to hold your scripts.
Instructions below are based on run on Mac 10.12 (Sierra) referencing http://sikulix.com/quickstart/
If this is your first time:
create a folder to hold the SikuliX installer:
On a Mac:
cd ~ mkdir sikulix-1.1.0-install cd ~/sikulix-1.1.0-install
(Capitalization is distracting)
Otherwise, begin anew by removing the folder loaded by the installer:
cd ~/Library cd "Application Support" rm -rf /Sikulix cd ..
The installer creates folders to contain the Sikulix.app for use on Macs. On Windows: in the folder Sikulix in the folder pointed to by the environment variable %APPDATA% On Linux: in ~/.Sikulix
As noted in this page, download the last stable version at https://launchpad.net/sikuli/sikulix/1.1.0/+download/sikulixsetup-1.1.0.jar
sikulixsetup-1.1.0.jar is dated 2015-10-06, 3 MB
NOTE: Code is at https://github.com/RaiMan/SikuliX-2014
The next generation is active on
Click the “md5” for the MD5 hash (a33616bac6d4f44785b89a02b110a0f8) to verify download integrity.
Move the file to your skilux folder.
Expand the installer:
java -jar sikulixsetup-1.1.0.jar
This creates file:
SikuliX-1.1.1-SetupLog. (which reflects what goes to the Terminal) and an empty SetupStuff folder.
Open the txt file and refresh it to see progress.
Click checkboxes for Jython and for Tesseract in the UI pop-up, then Setup Now.
Click Setup Now for:
Click Yes for:
Click No to pop-ups about Mac native libs (jar’s) previously installed into /Users/___/Library/Application Support/Sikulix/SikulixDownload_201610061722/…
Click Yes to download a stand-alone version of Python (No to get Jython 2.5 due to 2.7 UTF-8 warning):
BLAH: This error in the console stopped me from going further
[debug (12/20/16 8:05:24 AM)] RunSetup: downloaded: tessdata-eng Exception in thread "main" java.lang.NullPointerException at org.sikuli.setup.RunSetup.main(RunSetup.java:845)
PROTIP: We are forced to use nightly builds.
Delete the installer folder.
Install-1.1.1 Dev build
Create a version-specific installer folder:
On a Mac:
cd ~ mkdir sikulix-1.1.1-install cd ~/sikulix-1.1.1-install
Download from http://nightly.sikuli.de/ righ-click and select Save link As … the link that says:
At the time of writing, the URL is:
Run the file:
java -jar sikulixsetup-1.1.0.jar
Check boxes as described above and install. The result:
Click “OK” to the “Hallo” pop-up.
Note different files are downloaded for Windows vs. Mac:
In Finder, move sikulix.app to your user’s Applications folder.
Still in Finder, Move to Trash the installer file sikulixsetup-1.1.0.jar
QUESTION: How to verify a good install and use the program?
only in case of problems it might be necessary to visit this folder:
- Extensions folder
- Lib folder contains the supporting stuff for Jython and JRuby
- SikulixDownloads_versionstamp contains
- SikulixLibs_versionstamp contains native libraries
Open the SikuliX IDE app from the Applications folder.
Create a Git folder to hold scripts. In real life, you should create a folder containing the name of the app under test.
On a Mac:
cd ~ cd gits cd wilsonmar mkdir sikulix-scripts cd ~/sikulix-scripts
Into this folder the SikuliX IDE saves scripts with the file extension “.sikuli”.
Press # to insert a comment
Press Ctrl+S to save the file as a “.sikuli” file.
PROTIP: A keyboard press is quicker than moving the mouse.
Take a picture
Create a small picture where you want Sikuli to click or type.
This can be done by clicking the “Take screenshot” icon, positioning the mouse to the top-left corner, drag to the lower-right corner, then release the mouse.
Or press a hot-key (Command + Shift + 2).
QUESTION: If a dialog disappears with mouse motions, press the keystroke to take a picture of the screen, then use a photo editing program to cut out an area, and paste the little picture into the IDE.
SikuliX Actions and Methods
PathFirefox = r"C:\Program Files (x86)\Mozilla Firefox\firefox.exe" App.open(PathFirefox)
PROTIP: Instead of the hassle of using double slashes to specify special characters, put an 4 in front of calls to take the backslash and quotes as part of the word/sentence.
click(SearchImageButton) wait(imageExpected1) type("hello")
Type the verification actions.
Save a sample script with comments after # character:
On a Mac:
Run the file.
http://doc.sikuli.org/genindex.html is an alphabetical index of all methods and command options.
JavaDocs of classes are at
has several sikulix scripts.
SikuliX can be called from a Terminal/Run command line by batch scripts.
SikuliX recognize both one-character flags and longer options
sikulix –args -i
-t or –test sikuli-file
-r or –run sikuli-file
-s or –stderr
Originally from HP, @theRaySmith at Google says in 2016 Tesseract includes LSTM (machine learning with convolutional and deep belief networks).
BTW, the word tesseract means a representaton of a 4-dimensional cube (https://www.wikiwand.com/en/Tesseract). The word is used for the name of the library because the library works with multiple directional 3D lines.
Install using Homebrew
brew install tesseract
More on IoT
This is one of a series on IoT:
- IoT Apprentice school curriculum
- IoT use cases
- IoT reminders prevent dead mobile battery
- IoT text to speech synthesis
- IoT AWS button
- Intel IoT
- IoT Raspberry hardware
- IoT Clouds
- Predix basics
- Predix installation
- Predix services
- Predix programming