A few months ago one of my colleagues, Bhalin Ramabhadran, approached me and asked if I could automate Ghost Inspector tests with each deployment to the QA server for a particular project. At first, I said: “yeah, doesn’t seem to be a big request, no problem”.
I began bashing away on my keyboard doing some basic integration, which was simply a curl request for a particular test suite in Ghost Inspector. This worked well for us for about a month, when we realised it might not be the best solution, since we were executing a suite of tests on every deployment. We discussed having a way to determine which tests we should run on each deployment.
When the Inviqa Dev Day call for papers opened, Bhalin suggested I talk about it to our peers. I was very apprehensive about this, as I didn’t think we had enough content to talk about. It was a simple problem with a simple solution, and we hadn’t even added in the new feature of controlling which tests to run during deployment.
I decided the only way I would talk about this is if I made the integration more generic and configurable. Something our colleagues could use in other projects. The integration had to be a portable and transferable plugin that others can use with little configuration. My obvious choice was to develop a Capistrano plugin that was easy to install.
I set off again, this time porting the code I had written into a packaged Ruby gem for Capistrano. This was basically a direct copy at this stage, more of a proof of concept prototype, but I still had one problem to solve, which was controlling which tests to run during deployment.
Ghost Inspector offers a great API for remotely triggering your tests, however the identifiers of the tests and suites are pretty long and forgettable IDs. These IDs are 25 characters long, so they were useless for manually triggering each test.
What I needed was a mapping, so I decided the best place to hold all this information was a simple YAML file that can be used for all configurations, including API Key, global enable/disable switch and the mapping of test and suite IDs to memorable names like “homepage”. This allowed us to specify on the command line which test/tests to run, by providing a comma-separated list of tests and/or suites defined in the YAML file.
At this point, Bhalin asked if we could get the plugin to auto-rollback to a previous version when any tests failed. I said, “yeah, sure, let me work this one out too”. Back to the laptop, I bashed on the keyboard for a few more hours and finally released version 0.1.0. The initial response from my peers was incredible, they saw a real benefit to this and in contributing in many ways.
One colleague had pointed out to me on Twitter that the auto-rollback feature was dangerous, as we need to be careful when dealing with database migrations. The effects on such an action could be catastrophic depending on the site and setup. His concerns were quickly diminished when he realised we could disable the auto-rollback feature by setting the global rollback flag to false in the YAML configuration file or in each of the specific stage files, i.e staging.rb or production.rb
At this point my colleague and in-house Google Analytics expert, Grant Kemp, asked for some Google Analytics (GA) integration. We had a few conversations on this and how we envisioned it would work, and I began working on version 0.2.0, which used the Google Analytics Measurement Protocol to record page views and events for deployments.
We used events in GA to record the errors of Ghost Inspector test runs, showing which step in the test failed and logging that error message to GA. This could then be used in GA to quantify the success of deployments with the Ghost Inspector test runs.
This was working well at first, before we realised we needed more data in GA. We then set off to use custom dimensions in GA to record which test was run, along with the server and domain information. We also included the revision of the deployed code to allow us to track which failed tests were against each deployed code version.
One of the most useful features we added was tracking which Jira tickets were deployed with the code, allowing us to analyse which new features or bug fixes were causing the failed tests. There is no special Jira integration here, we simply extract the Jira ticket number from each commit message in the GIT log.
This feature does require your commit message to be prefixed with the Jira ticket number manually, but you should be doing this anyway. This is also configurable and doesn’t specificallyrequire Jira, you can use any ticketing tool you prefer as long as you configure the plugin to look for a particular pattern.
After a few tweaks and bug fixes, we finally had a stable working version of the plugin. Until another colleague, Matt Curran, came along and asked for a few changes for a project he was working on. His project had too many GA accounts and was unable to use the GA integration, sine this would only work with a single GA account. He made a few changes to make GA an optional feature, submitted a PR, and we now have a finally stable version of 1.0.1.
Matt’s changes also allowed us to set the run order of the plugin during deployment. This was extremely useful, as by default we were running Ghost Inspector after the default `deploy` task. This is fine for small sites, however large applications like Magento may have the requirement for setup scripts and cache flushing to be run first before we execute any Ghost Inspector tests.
First thing you need to do is install the Ruby gem for your application to use. Run the following command in your terminal.
Now that we have the Ruby gem installed, we need to tell Capistrano to use this in our application. To do this, add the following line to the top of your `deploy.rb` file as part of your Capistrano application.
At the end of your `deploy.rb` file you need to define the run order of your Ghost Inspector tests.
You can change the run order by changing deploy for any other task in your deployment run list. For example, you may have a task of `run_scripts` that is executed after deployment for database migration. In this case you would change the above line to:
There is a main configuration file used for Capistrano::Ghostinspector. This is a simple YAML file that allows for nested configuration. The first thing you need to do is create your YAML configuration file (gi_config.yaml) in the Capistrano folder with the following format. The name of the file is important and must be in the root Capistrano folder for your application.
You can obtain your API key, suite ID and test ID from your Ghost Inspector console. At the bottom right of the suite page you will see API Access e.g. api.ghostinspector.com/v1/suites/<SUITE ID IS HERE>/execute/?apiKey=<API KEY ID IS HERE>
Add as many suites or tests as you like, the name you give your test isn't important but you should make it easy to remember when executing the test.
By default the ghost inspector execution is enabled, you can disabled this for all stages by setting `gi_enabled: false` in the YAML file. Alternatively you can change this on a per-stage basis by setting the appropriate variable in your correct stage file i.e staging.rb
This ensures that Ghost Inspector is not automatically run when accidentally triggered via the command line.
The rollback feature is enabled by default, you can disabled this for all stages by setting rollback: false in the YAML file. Alternatively, you can change this on a per stage basis by setting the appropriate variable in each of your specific stage files i.e staging.rb or production.rb
Configure Start URL
Ghost Inspector has a nice feature that allows you to dynamically alter the start URL for your test. This allows you to reuse the same tests across multiple environments, i.e staging.mysite.com, uat.mysite.com, www.mysite.com. This feature utilises the `domain` variable set in your specific staging file. Some of the projects at Inviqa already use this variable and no further configuration is required. For older projects you may set this manually. i.e. for staging you might have:
and production might have:
Failure to set the domain in any stage will revert the tests to be only run against the URL you defined in Ghost Inspector.
Google Analytics tracking
The Google Analytics property must be inserted into the ga_property in order to log deployments and errors. Simply update your YAML to include this ga_property: "UA-XXXXXXXX-1". The Google Analytics feature also has a ga_enabled flag in your YAML file, which must be true to successfully run. To disable the Google Analytics tracking either set the ga_enabled to be false, or your can disable Google Analytics in each stage by setting the following:
Google Analytics uses Custom Dimensions as outlined in the Google Measurement Protocol documentation. When you define a new custom dimension in Google Analytics, you are given a new dimension index. Default accounts have 20 available indexes, whereas premium accounts have 200. The ga_custom_1 property is used to define the custom dimension for the testname and ga_custom_2 is used to define the Jira tickets*. If you do not set the ga_custom_1 or ga_custom_2 properties then the default index of 1 & 2 will be used.
*Jira tickets are extracted from the git log during the deployment. For this reason it can only track the tickets where you have correctly assigned the ticket number and identifier to the commit message. i.e.
Update `jira_project_code' with your project prefix.
Run a particular test when deploying to staging:
Run multiple tests when deploying to staging:
Run a particular suite when deploying to staging:
Run multiple suites when deploying to staging:
Run default test
You can set your default tests/suites to run in each stage, e.g. you might want to run a certain test suite in production only, but have other tests running in staging. You can now set this in your stage.rb file using the two flags.
i.e production.rb might look like this:
and your staging.rb file might have the following:
As you can see the two variables `gi_default_suite` and `gi_default_test` can also take a comma-separated list to run.
This is by no means the only solution for automating your Ghost Inspector tests. There are ways to automate the tests from within your continuous integration server. There is a great blog post by Ghost Inspector’s founder Justin Klemm on doing just that.
If you would like to read more on how to fully utilise the Google Analytics integration, my colleague Grant Kemp has written a fantastic article on how to do just that. You can read the article on his blog.
If you would like to get involved with the development of the plugin you can fork the repository on GitHub and submit a pull request.
About the author
Steven Richardson is a Software Engineer for Session Digital and has over 6 years’ experience in eCommerce development for fashion, retail, jewellery, food and drink, and travel industries. Steven is also a Magento Certified Developer. When not working on the next, innovative feature for Session Digital’s many clients, Steven enjoys photography, sports, and spending time with his family.
This article was originally published under Session Digital, which unified with Inviqa in June 2016. For more information about the unification visit https://inviqa.com/new-era.