Automated testing of Titanium seems to be a hot topic in the community. We’ve seen several talks on this topic on all 3 tiConfs. Today we have a guest blog by Steven Senior about fully automated acceptance testing. A lengthy blog, but it really takes you from start to end!
Introdution
A few years ago I developed my first mobile application. This was a standard native iOS app written in Objective C and built using XCode. That project was relatively successful but the development and subsequent support of the app were painful due to a number of mistakes on my part. I recently made the switch to Appcelerator Titanium which is a wonderful cross platform development framework.
For my current project using this framework I was determined to correct all of the mistakes I had made on that first iOS app. For example, code control using subversion linked to an issue tracker, offline access etc. Somewhere towards the top of this list was automated testing. As a solo developer, testing was a completely manual process for me. Launch the app, open all the views/windows I need to test, click all the buttons etc etc.
There has to be an easier way right? Well there is. In this post I want to show you how you can implement a robust, fully automated acceptance testing suite for your own Titanium apps.
Testing Options
There are a few tools I evaluated when investigating the various options:
- ti-mocha – Simple and reliable support for mocha testing with Appcelerator’s Titanium SDK by Tony Lukasavage
- ti-jasmine – A jasmine (Javascript testing) implementation for Appcelerator Titanium by Bill Dawson
- ticalabash – A CLI plugin for Titanium Mobile for running Calabash tests by AppersonLabs
The first two have limitations (based on my very limited experience evaluating them):
- Can’t interact with the UI. All you can do is check whether UI elements exist and their size and position
- You can end up duplicating a lot of code in your app to make your tests work
- There’s no easy way to run all the tests in your app end to end from the command line, as the tests are defined in the controllers
ticalabash (which may actually itself use ti-mocha) solves all of these problems.
Calabash, cucumber and gherkin
Calabash, cucumber and gherkin are the tools you will become familiar with. ticalabash implements these tools for Titanium.
Cucumber
Links:
Cucumber lets software development teams describe how software should behave in plain text. The text is written in a business-readable domain-specific language (DSL) and serves as documentation, automated tests and development-aid – all rolled into one format. Cucumber’s plain text DSL is called Gherkin. Cucumber runs the Gherkin files.
Gherkin
Links:
Gherkin is the language you use to describe your tests. It is structured as follows: Features -> Scenarios -> Step Definitions. There are a few conventions.
- Single Gherkin source file contains a description of a single feature
- Source files have .feature extension
The Gherkin parser divides the input into features, scenarios and steps. When you run the feature the trailing portion (after the keyword) of each step is matched to a Ruby code block called Step Definitions.
A Gherkin source file usually looks like this
Feature: Some terse yet descriptive text of what is desired Textual description of the business value of this feature Business rules that govern the scope of the feature Any additional information that will make the feature easier to understand Scenario: Some determinable business situation Given I have 22 cucumbers in my belly And some other precondition When some action by the actor And some other action And yet another action Then some testable outcome is achieved And something else we can check happens too Scenario: A different situation ...
In which:
- First line starts the feature.
- Lines 2–4 are unparsed text, which is expected to describe the business value of this feature.
- Line 6 starts a scenario.
- Lines 7–13 are the steps for the scenario.
- Line 15 starts next scenario and so on.
Features
It’s up to you to decide what the features are. There is a discussion about what these should say here – Business value and MMF
Scenarios
When you’re writing a new scenario, the recommendation is that you start with the formulation of the desired outcome. Write the Then steps first. Then write the When step to discover the action/operation and finally write the Given steps that need to be in place in order for the When/Then to make sense
Note: one thing to note about the scenarios you create in your .feature file is that the iOS app is terminated between each of them. This is important because it means you need to do everything in a single scenario. A scenario can’t rely on steps taken in a previous scenario.
Step Definitions
The significance of the adverb (Given/When/Then) used is for humans only and has no significance when Cucumber is registering or looking for Step Definitions.
The adverb chosen should drive what type of work you do inside your Step Definitions – see Given-When-Then
But basically…
- Given – put the system in a known state before the user (or external system) starts interacting with the system (in the When steps)
- When – describe the key action the user performs
- Then – observe outcomes
Step definitions are defined in Ruby files under
features/step_definitions/*_steps.rb
All Step definitions are loaded (and defined) before Cucumber starts to execute the plain text. It does not matter what you call the files. There is no mapping of a feature file name and a step definition file name. The matching is done on the adverb (Given, When, Then, And, But) in the features file.
Here is a simple step definition example, which would cover step 7 in the above features file.
Given /^I have (\d+) cucumbers in my belly$/ do |cukes| # Some Ruby code here # Some Calabash code here to interact with UI touch("button: index:1") end
Calabash
Links:
Calabash enables you to write and execute automated acceptance tests of mobile apps. Calabash is cross-platform, supporting Android and iOS native apps. Calabash could be compared to Selenium WebDriver. However, it is important to realize that interacting with a web app from a desktop computer is vastly different than interacting with a native app using a touch screen. Calabash provides APIs that are specialized to native apps running on touch screen devices. Calabash does the work to interact with the iOS / Android device to allow testing. These calabash calls are done in Gherkin DSL Step Definition files
ticalabash
ticalabash is the Titanium CLI plugin that will let you run the tests you’ve created using the tools above against your Titanium mobile app. ticalabash is currently Beta software and as such information is sparse. Some official information that I am aware of can be found here:
- TiCalabash: Fully automated Acceptance Testing @ TiConf EU 2014
- TiCalabash and TiMocha: The keys to Better & More Stable Titanium Apps
Install ticalabash
As this is currently beta software, this was a bit of a pain to get working, due to these issues
- OSX Mountain Lion shipping with Ruby version 1.8.7 and this requiring 1.9+
- A bug in the ticalabash node package meaning the -cal scheme was never built
- cucumber profiles missing from ticalabash version 1.0.6
These are the fixes I used to overcome these issues:
OSX Mountain Lion shipping with Ruby version 1.8.7 and this requiring 1.9+
Install rbenv
Follow Basic GitHub Checkout section of rbenv installation
Install ruby version 2.0.0p481
rbenv install 2.0.0p481
Set ruby version to 2.0.0p481
rbenv global ruby 2.0.0p481 rbenv shell ruby 2.0.0p481
remove symlink to ruby 1.8.7
ls -l /usr/bin/ruby /usr/bin/ruby -> /System/Library/Frameworks/Ruby.framework/Versions/1.8/Ruby sudo rm /usr/bin/ruby
remove gem symlink
ls -l /usr/bin/gem /usr/bin/gem -> /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/gem sudo rm /usr/bin/gem
Install ticalabash
npm -g install ticalabash
This must install the gems using ruby version 2 and not 1.8.7. You will see the ruby version used in the output. If not then
cd /usr/lib/node_modules/ticalabash/ bundle install
A bug in the ticalabash node package meaning the -cal scheme was never built
Follow the steps I documented here The project does not contain a -cal target
cucumber profiles missing from ticalabash version 1.0.6
See this issue for full details – Support cucumber profiles
cd /usr/lib/node_modules/ticalabash/lib
Edit run_ios.js
change:
exec('cucumber',null, { cwd: 'build/iphone' }, function(data) {
to:
exec('cucumber', ['-p', 'ios'], { cwd: 'build/iphone' }, function(data) {
Edit run_android.js
change:
exec('calabash-android', ['run', path.join(apkPath, apkName)], null, function(output) {
to:
exec('calabash-android', ['run', path.join(apkPath, apkName), '-p', 'android'], null, function(output) {
Wrapper scripts to run tests
I found it helpful to create the following scripts in the features folder to workaround some of the issues I encountered. You don’t need to do this, but the rest of this post assumes you have.
run_ios.sh
app=<> if [ "$#" -ne 3 ]; then echo echo "Usage: run_ios.sh <6.0/6.1/7.1> " echo exit 1 fi if [ $1 = "iphone" ]; then platform=iphone elif [ $1 = "ipad" ]; then platform=ipad else echo echo "1st parameter must be one of: iphone/ipad" echo exit 1 fi if [ $2 = "6.0" -o $2 = "6.1" -o $2 = "7.1" ]; then os_ver=$2 else echo echo "2nd parameter must be one of: 6.0/6.1/7.1" echo exit 1 fi if [ $3 = "stop_simulator" ]; then no_stop=0 elif [ $3 = "no_stop_simulator" ]; then no_stop=1 else echo echo "3rd parameter must be one of: stop_simulator/no_stop_simulator" echo exit 1 fi cd ${app} rm -rf build/iphone echo echo "***** Running: ti build --platform=iphone --device-family=$platform --sim-type=$platform --sim-version $os_ver --build-only --quiet" ti build --platform=iphone --device-family=$platform --sim-type=$platform --sim-version $os_ver --build-only --quiet echo cd build/iphone # required due to https://github.com/appersonlabs/TiCalabash/issues/10 calabash-ios setup cd ${app} cp cucumber.yml build/iphone/ echo if [ $platform = "iphone" ]; then echo "***** Running: NO_STOP=$no_stop DEVICE=iphone DEVICE_TARGET=\"iPhone Retina (3.5-inch) - Simulator - iOS $os_ver\" ti calabash --platform=ios" NO_STOP=$no_stop DEVICE=iphone DEVICE_TARGET="iPhone Retina (3.5-inch) - Simulator - iOS $os_ver" ti calabash --platform=ios echo elif [ $platform = "ipad" ]; then echo "***** Running: NO_STOP=$no_stop DEVICE=ipad DEVICE_TARGET=\"iPad - Simulator - iOS $os_ver\" ti calabash --platform=ios" NO_STOP=$no_stop DEVICE=ipad DEVICE_TARGET="iPad - Simulator - iOS $os_ver" ti calabash --platform=ios echo else echo echo "Unknown platform: $platform specified" echo exit 1 fi echo echo +--------------------------------------------+ echo + Any screenshots are in the build directory + echo +--------------------------------------------+ echo
Note: when you run the script the “calabash-ios setup” command may prompt you to choose your target from a list of targets. If it does, press return to accept the default.
Note: the DEVICE_TARGET specifies which simulator you want to run the tests on. You can get a list of known devices by running the following:
instruments -s devices
Change the DEVICE_TARGET values in the script as you see fit.
run_android.sh
app=<> #Check for a running Genymotion emulator running_gvms=`"/Applications/Genymotion Shell.app/Contents/MacOS/genyshell" -c "devices list" | grep -c "On"` if [ ${running_gvms} -eq 0 ] then echo "No running Genymotion VMs. Start one and rerun this script. Exiting" echo exit 1 fi unset ANDROID_HOME export ANDROID_HOME=< > cd ${app} rm -rf test_servers ti calabash --platform=android echo echo +--------------------------------------------+ echo + Any screenshots are in the build directory + echo +--------------------------------------------+ echo
Before you run this script, you must make sure the following file exists with this content:
cat build/android/bin/calabash_settings {"keystore_location":"/Library/Application Support/Titanium/mobilesdk/osx/3.2.3.GA/android/dev_keystore","keystore_password":"tirocks","keystore_alias":"tidev"}
Change the keystore_location based on where you installed the Titanium SDK.
Note: you must start your GenyMotion VM/Android emulator before running run_android.sh as it will not be started for you, unlike the iOS Simulator.
Create your scenarios, features, and step definitions
The quickest way is to:
- git clone this repo https://github.com/calabash/x-platform-example
- recursively copy the features directory into your app root folder
- copy config/cucumber.yml into your app root folder
The features and step definitions from x-platform-example were written for a sample application and will obviously not work with your app. Nevertheless now is a good time to run ticalabash to make sure it is all working.
cd <<your app root>>/features ./run_ios.sh <iphone/ipad> <6.0/6.1/7.1> <stop_simulator/no_stop_simulator> ./run_android.sh
The tests will fail as expected, but you should see your app appear in the simulator/emulator. Now you need to start writing your own features and step definitions. The structure of the code in the x-platform-example is (I believe) best practice and DRY in terms of its use of Page Objects, so you should try and stick to that. It’s beyond the scope of this post to tell you how to write these, but the links I have provided will get you going.
At some point, in your step definitions you are going to have to add Calabash commands to interact with your UI. However, you don’t know what they are, so you have to have a way of exploring your app with Calabash…
Calabash console
ios
cd <<your app root>>/features ./run_ios.sh iphone 7.1 no_stop_simulator --let any tests finish and return you to the bash prompt calabash-ios console --you should now be on the irb prompt
android
cd <<your app root>>/build/android/bin calabash-android console <>.apk --you should now be on the irb prompt start_test_server_in_background
Now you can start querying and interacting with your app via the irb console. These resources will get you started:
IOS: iOS Calabash Query Syntax
Android: Android Calabash Query Syntax
The names of the elements are different in iOS and Android. This is why in the features folder you have subdirectories for ios and android. In the pages directory you create class objects for each platform to cater for this. Study the x-platform-example to learn how it was implemented.
When starting out, it’s useful to query everything:
irb(main):002:0> query("*")
When you’ve got the commands you need from the console, copy them into your step definitions and they will then be run before your very own eyes, next time you run ticalabash.
Conclusion
Thorough regression testing of your app is both an essential step in the development process, and a necessity when updating your app once it is out in the wild. Often this is neglected by individual developers or small teams, who instead divert their resource and time into developing the app itself. With ticalabash and the cucumber/gherkin framework, we now have a simple to use yet fully automated acceptance testing framework for titanium apps. Taken together with real world beta testing, you can have the confidence that updates you make will not break your app. Although resources and tutorials for ticalabash are presently in short supply, hopefully this post has shown you how you can practically start to use it in your own apps.
Comments
Terry Morgan
This looks great. I haven’t read the whole article in detail yet but I was wondering if there are Node.js alternatives to Calabash & Gherkin? I see there’s a Node port of Cucumber (https://github.com/cucumber/cucumber-js) and it would be great to cut Ruby out of the equation completely, if possible, given the rest of the Ti ecosystem is all JS and that’s what most Ti devs will be familiar with.
Andrew McElroy
I tried to cut ruby out entirely with TiCucumber, but it was really too much of an undertaking to do alone. I ultimately launched ( with matt apperson) ticalabash which is a titanium CLI hook that makes using this seamless. It still does have a ruby dependency, but at this point it is able to just use the system ruby.
see: https://github.com/appersonlabs/TiCalabash
and: http://www.slideshare.net/sophrinix/ticalabash-fully-automated-testing-ticonf-eu-2014
Terry Morgan
I’ve created slightly updated versions of the two bash scripts mentioned above that run ticalabash. Aside from ios8 / xcode6 compatibility, I’ve also altered them to accept the path to the project dir as a param rather than hardcoding it, which made the scripts slightly more flexible for me.
Android
https://gist.github.com/shodanuk/c379f1db916ec7cbf129
ios
https://gist.github.com/shodanuk/540cb5309f4a3fab7f12