A robot sees what’s on your screen and clicks on it
- Sample scripts
- SikuliX Commands
- Use IDE for SikuliX Scripting
- More on IoT
This is a deep dive from the innnards up:
OpenCV (Open Computer Vision) is a utility used by many Artificial Intelligence packages such as self-driving cars to understand images. It is used to take pictures, recognize objects in the pictures, and the coordinates of objects it recognizes in pictures.
SikuliX uses OpenCV to find the location of a picture provided to OpenCV. SikuliX uses the location returned from OpenCV to mouse click on the screen, or type as if a user is using a physical keyboard.
Optionally, Tesseract provides OCR (Optical Character Recognition) to extract text from pictures.
All this to make you more “productive” at playing games. ;)
The benefit of repeating some action repeatedly (and not get bored or distracted) extends to:
- Videographers making screencast videos in multiple takes
- Salespeople use it to power their live demos.
- Developers and testers going through a sequence of actions to reach a particular point in the UI needing debugging.
Here’s how it works: A picture is taken of your screen(s). and the text and location of objects recognized in the picture is fed to your program to click or type. Your program then takes another round of picture taking and visual object recognition to ensure the results are what are intended.
This uses a different, more generalized, approach than packages such as HP UFT which depended on finding identifiers in the DOM (Document Object Model) behind each internet browser web page.
http://sikulix.com is the new web page.
(Ignore abandoned sites:
Installer bits are obtained from Ubuntu
Sikuli was started somewhen in 2009 as an open-source research project at the User Interface Design Group at MIT by Tsung-Hsiang Chang and Tom Yeh.
There are three versions:
1.1.0 final (available since 2015-10-07)
I’ve had installation errors with this version on my Mac.
There is an entry on 2015-10-07 in Maven Central:
but I haven’t tried to install it.
SikuliX-2014 (version 1.1.x) from
This is the one that worked for me.
Install instructions below.
NOTE: Code is at https://github.com/RaiMan/SikuliX-2014
SikuliX2 in 2017 at
This is supposed to exist side-by-side with previous versions.
C) Download and compile it locally*
exec java -Xmx512M -Dfile.encoding=UTF-8 -Dsikuli.FromCommandLine -jar /usr/share/sikulix/sikulix.jar “$@”
D) Manual install (see below)
- Install JRE 8 or 9
- Install the Java language IDE you want to use (Eclipse or IntelliJ)
- Install Eclipse or Robot Framework (which uses Python)
- Download SikuliX
The installer installs its own version of Python.
Install-1.1.2 Dev build
Instructions below are based on run on Mac 10.12 (Sierra) referencing http://sikulix.com/quickstart/
On a Mac, create a version-specific installer folder on your $HOME folder:
cd ~ mkdir sikulix-1.1.2-install cd sikulix-1.1.2-install
Download from http://nightly.sikuli.de right-click and select Save link As … the link that says:
At the time of writing, the URL is:
By the time you read this, the date in the link will change because the code is actively worked on.
Click the “md5” for the MD5 hash (a33616bac6d4f44785b89a02b110a0f8) to verify download integrity using whatever tool you use.
Expand the installer (in Downloads):
java -jar sikulixsetup-1.1.2.jar</pre> This creates file:
SikuliX-1.1.2-SetupLog. (which reflects what goes to the Terminal) and an empty SetupStuff folder.
Open the txt file and refresh it to see progress.
Click checkboxes for Jython and for Tesseract in the UI pop-up, then Setup Now.
Click Setup Now for:
Click Yes for:
Click No to pop-ups about Mac native libs (jar’s) previously installed into /Users/___/Library/Application Support/Sikulix/SikulixDownload_201610061722/…
Click Yes to download a stand-alone version of Python (No to get Jython 2.5 due to 2.7 UTF-8 warning):
Click 3 the boxs OK to the pop-up:
Click “OK” to the “Hallo” pop-up.
Clean up installers
In Finder, Move to Trash the installer file sikulixsetup-1.1.0.jar
QUESTION: How to verify a good install and use the program?
In Finder, move sikulix.app to your user’s Applications folder.
In case you’re curious, view the folders (not necessary unless you run into problems)
- Extensions folder
- Lib folder contains the supporting stuff for Jython and JRuby
- SikulixDownloads_versionstamp contains
- SikulixLibs_versionstamp contains native libraries
Note different files are downloaded for Windows vs. Mac:
PROTIP: Search for Sikulix scripts within GitHub:
Click the Example folder, JavaCompile.sikuli, then JavaCompile.py.
If you intend on updating this repository and contributing changes back to the author, sign-in to your own account and fork the repository under your own account.
PROTIP: Use Git to store Sikuli scripts. Install a Git client.
PROTIP: Create a Git folder to hold files associated with GitHub.
On a Mac:
cd ~ mkdir gits cd gits
Create a folder to hold sikuli repositories created when cloning from others:
cd ~/gits mkdir sikulix cd sikulix
You are now in the folder where you clone scripts from other repositories on GitHub.
Create a folder to hold repositories from lowkorn:
mkdir lowkorn cd lowkorn
Clone one I’ve found:
git clone https://github.com/lowkorn/SikuliX
Cloning into 'SikuliX'... remote: Counting objects: 34, done. remote: Total 34 (delta 0), reused 0 (delta 0), pack-reused 34 Unpacking objects: 100% (34/34), done.
Repositories cloned locally you can later update with changes made, using commands such as:
git remote add upstream https://github.com/lowkorn/SikuliX git remote -v git pull upstream master
remote: Counting objects: 34, done. remote: Total 34 (delta 0), reused 0 (delta 0), pack-reused 34 Unpacking objects: 100% (34/34), done. From https://github.com/lowkorn/SikuliX * branch master -> FETCH_HEAD * [new branch] master -> upstream/master
Alternately, to do a two-stage sync per https://help.github.com/articles/syncing-a-fork/
git fetch upstream git checkout master git merge upstream/master
On GitHub.com, sign-in to your own account and create a repository (named sikulix-scripts in this example) with a READM.md file.
Clone your own sikulix-scripts repo to your laptop (in this example, substitute “wilsonmar” with your own account name):
On a Mac:
cd ~/gits/sikulix mkdir wilsonmar cd wilsonmar git clone https://github.com/wilsonmar/sikulix-scripts cd ~/sikulix-scripts
Into this folder the SikuliX IDE should read and save scripts.
SikuliX can be called by batch scripts from a Terminal/Run command line.
SikuliX recognize both one-character flags and longer options:
sikulix –args -i
-t or –test sikuli-file
-r or –run sikuli-file
-s or –stderr
Use IDE for SikuliX Scripting
In your Mac’s Finder, get to Applications folder by pressing shift+command+A.
Type S and double-click the SikuliX IDE app to open it.
The IDE should appear:
Next, let’s get some sample files the program can work with.
Press # to insert a comment.
Press Ctrl+S to save the file as a “.sikuli” file.
PROTIP: A keyboard press is quicker than moving the mouse.
Take a picture
Create a small picture where you want Sikuli to click or type.
This can be done by clicking the “Take screenshot” icon, positioning the mouse to the top-left corner, drag to the lower-right corner, then release the mouse.
Or press a hot-key (Command + Shift + 2).
QUESTION: If a dialog disappears with mouse motions, press the keystroke to take a picture of the screen, then use a photo editing program to cut out an area, and paste the little picture into the IDE.
SikuliX Actions and Methods
PathFirefox = r"C:\Program Files (x86)\Mozilla Firefox\firefox.exe" App.open(PathFirefox)
PROTIP: Instead of the hassle of using double slashes to specify special characters, put an 4 in front of calls to take the backslash and quotes as part of the word/sentence.
click(SearchImageButton) wait(imageExpected1) type("hello")
Type the verification actions.
Save a sample script with comments after # character:
On a Mac:
Run the file.
http://doc.sikuli.org/genindex.html is an alphabetical index of all methods and command options.
JavaDocs of classes are at
has several sikulix scripts.
- Intro and setup 13 Nov 2015
The Basics shows how to (in Windows) open the Calculator and click a calculation.
https://www.youtube.com/watch?v=8y41vKeGM9U 5 Apr 2016
https://www.youtube.com/watch?v=3LRm3fJQ6AU 24 Jul 2016
https://www.youtube.com/watch?v=H2cK85z_8bs Keywords and Misc Stuff</a> 17 Nov 2016
Navigate to where you can open a .sikulix file
Switch to an internet browser:
bpteam’s Firefox browser opens https://sinoptik.ua and search on Windows within a try-except block.
vincent-tygron/SikuliX contains a full suite of scripts for an app
https://github.com/vincent-tygron/sikulix-160429 using defined functions
Excedrin provides several variables to play Windows game Clicker Heroes at
https://github.com/windows-2048/The-Windows-Intel-Fastest-Mouse-Clicker also at
git clone git://git.code.sf.net/p/fast-mouse-clicker-pro/code fast-mouse-clicker-pro-code
Let me know if you find others.
PROTIP: Additional Sikuli scripts from the internet:
Let me know if you find others.
- boter50 clicks http://www.pianotiles.org/ which provides audio feedback The script has a java unit test with eclipse perfs settings
More on IoT
This is one of a series on IoT:
- IoT Apprentice school curriculum
- IoT use cases
- IoT reminders prevent dead mobile battery
- IoT text to speech synthesis
- IoT AWS button
- Intel IoT
- IoT Raspberry hardware
- IoT Clouds
- Predix basics
- Predix installation
- Predix services
- Predix programming