Using Sikuli & Selenium in VS to Detect the Undetectable

While doing GUI automation, We can identify few limitations of the selenium web driver.

• Selenium web driver does not support flash objects.
• Selenium web driver automate only web applications.

Other than these web driver limitations,some times you may have to write automation scripts, while you do not have easy access to GUI’s internal code.

Sikuli is an open source tool for test automation, which provides good support to overcome above limitations. It controls GUI components by identifying objects using image recognition.

Following script shows how to do sikuli+selenium script in the visual studio IDE.

• First, You need to create unit test project and you have to add few NuGet Packages which are described below.

Go to File—>New—>Project

In the Left Panel, Under Visual C# select Test

Then select unit test project and give suitable  project name and location and click OK to create the project.

• Then need to add NuGet packages that required to do sikuli+selenium automation

To do that, Right click on solution name of solution explorer and select Manage NuGet Packages for Solution

• Install NuGet following packages
• Selenium.WebDriver
• Selenium.Support
• Selenium.WebDriver.ChromeDriver

If do Sikuli development using C# and .NET,there are three different NuGet packages

• Sikuli Integrator/Sikuli Module
• Sikuli4Net
• Sikuli Sharp

(*Sikuli4Net is the mostly commonly used one and adding it to the project would be sufficient for this example)

Let’s add some code now

In the following simple code,

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using Sikuli4Net.sikuli_UTIL;
using Sikuli4Net.sikuli_REST;

namespace UnitTestProject1
{
    [TestClass]
    public class UnitTest1
    {
        private APILauncher launcher = new APILauncher(true);
        [TestMethod]
        public void TestMethod1()
        {
            launcher.Start();
            IWebDriver driver = new ChromeDriver();
            driver.Navigate().GoToUrl("https://www.google.lk/");
            Screen screen = new Screen();
            Pattern pattern_Searchtex = new Pattern("C:\\Users\\pushpah\\Desktop\\gsearch.JPG");
            Pattern pattern_searchButton = new Pattern("C:\\Users\\pushpah\\Desktop\\searchbtn.JPG");
            screen.Wait(pattern_Searchtex, 50);
            screen.Type(pattern_Searchtex, "python", KeyModifier.NONE);
            screen.Wait(pattern_searchButton, 100);
            screen.Click(pattern_searchButton);
            driver.Quit();
            launcher.Stop();
        }
    }
}

• It navigates to google search page and compare a predefined image (image of the search text box – stored in “C:\Users\pushpah\Desktop\gsearch.JPG” ) within google search page objects, using image recognition techniques.
• After identifying the search text box, it types the word “python” in it.
• Next it compares the search button image with google search page objects  and clicks on the “Google Search” button.

You can run the test by right click inside a test case method and select “Run Tests” or using test explorer.

The key factor here is the ability to recognize objects in web page reliably, which is generally not easier to do with any automation scripting technique. Further similar approach can be used to capture windows based applications as well. Possibly these techniques may work in other platform applications such as on Linux or Mac.