Fully Automated Acceptance Testing

Automated testing of Titanium seems to be a hot topic in the community. We’ve seen several talks on this topic on all 3 tiConfs. Today we have a guest blog by Steven Senior about fully automated acceptance testing. A lengthy blog, but it really takes you from start to end!

Introdution

A few years ago I developed my first mobile application. This was a standard native iOS app written in Objective C and built using XCode. That project was relatively successful but the development and subsequent support of the app were painful due to a number of mistakes on my part. I recently made the switch to Appcelerator Titanium which is a wonderful cross platform development framework.

For my current project using this framework I was determined to correct all of the mistakes I had made on that first iOS app. For example, code control using subversion linked to an issue tracker, offline access etc. Somewhere towards the top of this list was automated testing. As a solo developer, testing was a completely manual process for me. Launch the app, open all the views/windows I need to test, click all the buttons etc etc.

There has to be an easier way right? Well there is. In this post I want to show you how you can implement a robust, fully automated acceptance testing suite for your own Titanium apps.

Testing Options

There are a few tools I evaluated when investigating the various options:

  1. ti-mocha – Simple and reliable support for mocha testing with Appcelerator’s Titanium SDK by Tony Lukasavage
  2. ti-jasmine – A jasmine (Javascript testing) implementation for Appcelerator Titanium by Bill Dawson
  3. ticalabash – A CLI plugin for Titanium Mobile for running Calabash tests by AppersonLabs

The first two have limitations (based on my very limited experience evaluating them):

  • Can’t interact with the UI. All you can do is check whether UI elements exist and their size and position
  • You can end up duplicating a lot of code in your app to make your tests work
  • There’s no easy way to run all the tests in your app end to end from the command line, as the tests are defined in the controllers

ticalabash (which may actually itself use ti-mocha) solves all of these problems.

Calabash, cucumber and gherkin

Calabash, cucumber and gherkin are the tools you will become familiar with. ticalabash implements these tools for Titanium.

Cucumber

Links:

Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain-specific language (DSL) and serves as documentation, automated tests and development-aid – all rolled into one format. Cucumber’s plain text DSL is called Gherkin. Cucumber runs the Gherkin files.

Gherkin

Links:

Gherkin is the language you use to describe your tests. It is structured as follows: Features -> Scenarios -> Step Definitions. There are a few conventions.

  • Single Gherkin source file contains a description of a single feature
  • Source files have .feature extension

The Gherkin parser divides the input into features, scenarios and steps. When you run the feature the trailing portion (after the keyword) of each step is matched to a Ruby code block called Step Definitions.

A Gherkin source file usually looks like this

Feature: Some terse yet descriptive text of what is desired
  Textual description of the business value of this feature
  Business rules that govern the scope of the feature
  Any additional information that will make the feature easier to understand

  Scenario: Some determinable business situation
     Given I have 22 cucumbers in my belly
       And some other precondition
      When some action by the actor
       And some other action
       And yet another action
      Then some testable outcome is achieved
       And something else we can check happens too

   Scenario: A different situation
       ...

In which:

  • First line starts the feature.
  • Lines 2–4 are unparsed text, which is expected to describe the business value of this feature.
  • Line 6 starts a scenario.
  • Lines 7–13 are the steps for the scenario.
  • Line 15 starts next scenario and so on.

Features

It’s up to you to decide what the features are. There is a discussion about what these should say here – Business value and MMF

Scenarios

When you’re writing a new scenario, the recommendation is that you start with the formulation of the desired outcome. Write the Then steps first. Then write the When step to discover the action/operation and finally write the Given steps that need to be in place in order for the When/Then to make sense

Note: one thing to note about the scenarios you create in your .feature file is that the iOS app is terminated between each of them. This is important because it means you need to do everything in a single scenario. A scenario can’t rely on steps taken in a previous scenario.

Step Definitions

The significance of the adverb (Given/When/Then) used is for humans only and has no significance when Cucumber is registering or looking for Step Definitions.

The adverb chosen should drive what type of work you do inside your Step Definitions – see Given-When-Then

But basically…

  • Given – put the system in a known state before the user (or external system) starts interacting with the system (in the When steps)
  • When – describe the key action the user performs
  • Then – observe outcomes

Step definitions are defined in Ruby files under

features/step_definitions/*_steps.rb

All Step definitions are loaded (and defined) before Cucumber starts to execute the plain text. It does not matter what you call the files. There is no mapping of a feature file name and a step definition file name. The matching is done on the adverb (Given, When, Then, And, But) in the features file.

Here is a simple step definition example, which would cover step 7 in the above features file.

Given /^I have (\d+) cucumbers in my belly$/ do |cukes|
  # Some Ruby code here
 
  # Some Calabash code here to interact with UI
  touch("button: index:1")
end

Calabash

Links:

Calabash enables you to write and execute automated acceptance tests of mobile apps. Calabash is cross-platform, supporting Android and iOS native apps. Calabash could be compared to Selenium WebDriver. However, it is important to realize that interacting with a web app from a desktop computer is vastly different than interacting with a native app using a touch screen. Calabash provides APIs that are specialized to native apps running on touch screen devices. Calabash does the work to interact with the iOS / Android device to allow testing. These calabash calls are done in Gherkin DSL Step Definition files

ticalabash

ticalabash is the Titanium CLI plugin that will let you run the tests you’ve created using the tools above against your Titanium mobile app. ticalabash is currently Beta software and as such information is sparse. Some official information that I am aware of can be found here:

Install ticalabash

As this is currently beta software, this was a bit of a pain to get working, due to these issues

  1. OSX Mountain Lion shipping with Ruby version 1.8.7 and this requiring 1.9+
  2. A bug in the ticalabash node package meaning the -cal scheme was never built
  3. cucumber profiles missing from ticalabash version 1.0.6

These are the fixes I used to overcome these issues:

OSX Mountain Lion shipping with Ruby version 1.8.7 and this requiring 1.9+

Install rbenv

Follow Basic GitHub Checkout section of rbenv installation

Install ruby version 2.0.0p481

rbenv install 2.0.0p481

Set ruby version to 2.0.0p481

rbenv global ruby 2.0.0p481
rbenv shell ruby 2.0.0p481

remove symlink to ruby 1.8.7

ls -l /usr/bin/ruby
/usr/bin/ruby -> /System/Library/Frameworks/Ruby.framework/Versions/1.8/Ruby
sudo rm /usr/bin/ruby

remove gem symlink

ls -l /usr/bin/gem
/usr/bin/gem -> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/gem
sudo rm /usr/bin/gem

Install ticalabash

npm -g install ticalabash

This must install the gems using ruby version 2 and not 1.8.7. You will see the ruby version used in the output. If not then

cd /usr/lib/node_modules/ticalabash/
bundle install

A bug in the ticalabash node package meaning the -cal scheme was never built

Follow the steps I documented here The project does not contain a -cal target

cucumber profiles missing from ticalabash version 1.0.6

See this issue for full details – Support cucumber profiles

cd /usr/lib/node_modules/ticalabash/lib

Edit run_ios.js

change:

exec('cucumber',null, { cwd: 'build/iphone' }, function(data) {

to:

exec('cucumber', ['-p', 'ios'], { cwd: 'build/iphone' }, function(data) {

Edit run_android.js

change:

exec('calabash-android', ['run', path.join(apkPath, apkName)], null, function(output) {

to:

exec('calabash-android', ['run', path.join(apkPath, apkName), '-p', 'android'], null, function(output) {

Wrapper scripts to run tests

I found it helpful to create the following scripts in the features folder to workaround some of the issues I encountered. You don’t need to do this, but the rest of this post assumes you have.

run_ios.sh

app=<>

if [ "$#" -ne 3 ]; then
    echo
    echo "Usage: run_ios.sh  <6.0/6.1/7.1> "
    echo
    exit 1
fi

if [ $1 = "iphone" ]; then
    platform=iphone
elif [ $1 = "ipad" ]; then
    platform=ipad
else
    echo
    echo "1st parameter must be one of: iphone/ipad"
    echo
    exit 1
fi

if [ $2 = "6.0" -o $2 = "6.1" -o $2 = "7.1" ]; then
    os_ver=$2
else
    echo
    echo "2nd parameter must be one of: 6.0/6.1/7.1"
    echo
    exit 1
fi

if [ $3 = "stop_simulator" ]; then
    no_stop=0
elif [ $3 = "no_stop_simulator" ]; then
    no_stop=1
else
    echo
    echo "3rd parameter must be one of: stop_simulator/no_stop_simulator"
    echo
    exit 1
fi

cd ${app}
rm -rf build/iphone

echo
echo "***** Running: ti build --platform=iphone --device-family=$platform --sim-type=$platform --sim-version $os_ver --build-only --quiet"
ti build --platform=iphone --device-family=$platform --sim-type=$platform --sim-version $os_ver --build-only --quiet
echo

cd build/iphone
# required due to https://github.com/appersonlabs/TiCalabash/issues/10
calabash-ios setup

cd ${app}
cp cucumber.yml build/iphone/

echo

if [ $platform = "iphone" ]; then
    echo "***** Running: NO_STOP=$no_stop DEVICE=iphone DEVICE_TARGET=\"iPhone Retina (3.5-inch) - Simulator - iOS $os_ver\" ti calabash --platform=ios"
    NO_STOP=$no_stop DEVICE=iphone DEVICE_TARGET="iPhone Retina (3.5-inch) - Simulator - iOS $os_ver" ti calabash --platform=ios    
    echo
elif [ $platform = "ipad" ]; then
    echo "***** Running: NO_STOP=$no_stop DEVICE=ipad DEVICE_TARGET=\"iPad - Simulator - iOS $os_ver\" ti calabash --platform=ios"
    NO_STOP=$no_stop DEVICE=ipad DEVICE_TARGET="iPad - Simulator - iOS $os_ver" ti calabash --platform=ios    
    echo
else
    echo
    echo "Unknown platform: $platform specified"
    echo
    exit 1
fi

echo
echo +--------------------------------------------+
echo + Any screenshots are in the build directory +
echo +--------------------------------------------+
echo

Note: when you run the script the “calabash-ios setup” command may prompt you to choose your target from a list of targets. If it does, press return to accept the default.

Note: the DEVICE_TARGET specifies which simulator you want to run the tests on. You can get a list of known devices by running the following:

instruments -s devices

Change the DEVICE_TARGET values in the script as you see fit.

run_android.sh

app=<>

#Check for a running Genymotion emulator
running_gvms=`"/Applications/Genymotion Shell.app/Contents/MacOS/genyshell" -c "devices list" | grep -c "On"`

if [ ${running_gvms} -eq 0 ]
then
  echo "No running Genymotion VMs. Start one and rerun this script. Exiting"
  echo
  exit 1
fi

unset ANDROID_HOME
export ANDROID_HOME=<>

cd ${app}
rm -rf test_servers
ti calabash --platform=android

echo
echo +--------------------------------------------+
echo + Any screenshots are in the build directory +
echo +--------------------------------------------+
echo

Before you run this script, you must make sure the following file exists with this content:

cat build/android/bin/calabash_settings

{"keystore_location":"/Library/Application Support/Titanium/mobilesdk/osx/3.2.3.GA/android/dev_keystore","keystore_password":"tirocks","keystore_alias":"tidev"}

Change the keystore_location based on where you installed the Titanium SDK.

Note: you must start your GenyMotion VM/Android emulator before running run_android.sh as it will not be started for you, unlike the iOS Simulator.

Create your scenarios, features, and step definitions

The quickest way is to:

  1. git clone this repo https://github.com/calabash/x-platform-example
  2. recursively copy the features directory into your app root folder
  3. copy config/cucumber.yml into your app root folder

The features and step definitions from x-platform-example were written for a sample application and will obviously not work with your app. Nevertheless now is a good time to run ticalabash to make sure it is all working.

cd <<your app root>>/features

./run_ios.sh <iphone/ipad> <6.0/6.1/7.1> <stop_simulator/no_stop_simulator>
./run_android.sh

The tests will fail as expected, but you should see your app appear in the simulator/emulator. Now you need to start writing your own features and step definitions. The structure of the code in the x-platform-example is (I believe) best practice and DRY in terms of its use of Page Objects, so you should try and stick to that. It’s beyond the scope of this post to tell you how to write these, but the links I have provided will get you going.

At some point, in your step definitions you are going to have to add Calabash commands to interact with your UI. However, you don’t know what they are, so you have to have a way of exploring your app with Calabash…

Calabash console

ios

cd <<your app root>>/features
./run_ios.sh iphone 7.1 no_stop_simulator

--let any tests finish and return you to the bash prompt

calabash-ios console

--you should now be on the irb prompt

android

cd <<your app root>>/build/android/bin
calabash-android console <>.apk

--you should now be on the irb prompt

start_test_server_in_background

Now you can start querying and interacting with your app via the irb console. These resources will get you started:

IOS: iOS Calabash Query Syntax
Android: Android Calabash Query Syntax

The names of the elements are different in iOS and Android. This is why in the features folder you have subdirectories for ios and android. In the pages directory you create class objects for each platform to cater for this. Study the x-platform-example to learn how it was implemented.

When starting out, it’s useful to query everything:

irb(main):002:0> query("*")

When you’ve got the commands you need from the console, copy them into your step definitions and they will then be run before your very own eyes, next time you run ticalabash.

Conclusion

Thorough regression testing of your app is both an essential step in the development process, and a necessity when updating your app once it is out in the wild. Often this is neglected by individual developers or small teams, who instead divert their resource and time into developing the app itself. With ticalabash and the cucumber/gherkin framework, we now have a simple to use yet fully automated acceptance testing framework for titanium apps. Taken together with real world beta testing, you can have the confidence that updates you make will not break your app. Although resources and tutorials for ticalabash are presently in short supply, hopefully this post has shown you how you can practically start to use it in your own apps.

Comments