This post will emphasise continuous integration from a Test Automation Engineer’s point of view and explain the benefits that are achievable with it. Continuous integration is a game-changer for increasing the effectiveness of Test Automation at every step in the software development pipeline. Continuous integration adoption brings new opportunities into the testing process. For many, the cornerstones in that process are the cucumber reports.

Continuous Integration was initially a development methodology, automating the process of building software.  However, this is improved where Test Automation Engineers commit changes to test automation source code into the same, shared repository. The benefit is that it enables teams to find and resolve issues as early as possible in the development process – the tests run as soon as the code is ready. Integrating tests like this has quickly become a best practice method, because of how much it improves efficiency. 

What tool should you use?

There are many tools that teams can use to implement continuous integration. One of the most popular ways to implement this method is by using Jenkins, which is an open-source automation tool written in Java. It has a super extensive plugin ecosystem. Its architecture lends itself to extensibility and innovation. You will also find many options when it comes to features, variety overall and the support of third-party tools. If your task includes extracting the result of Cucumber tests, you definitely should try the Neuro Cucumber reports plugin.

Using Jenkins, test engineers can spend less time and effort on testing or managing test automation. Since Jenkins is free, easy to use and user-friendly, test engineers are able to get up and running quickly. They can schedule and run test automation cases when they are ready or based on triggers, and then dedicate their time to other tasks. 

Also Read: The Power of JIRA for Project Management

Why Jenkins?

Jenkins provides both test summaries and information on test results as cucumber reports. On the test result “graph”, users can see how test results look like over time and gain more information on how to move forward. Test summaries give specific information on the number of tests, how long they took to execute and other information that can play a role in total productivity. 

Understanding the cause of a test failure is critically important for test engineers. Jenkins provides details on a test failure to give testers more detailed information. Using Jenkins, you can easily access to the details on a test failure in one click. Once you navigate to these details you will find the error or failure message and information about the test failure. This gives test engineers the opportunity to sort out complex issues they may not have seen before testing.

Jenkins is a great choice if you are ready to jump into test automation. It works well with the continuous integration methodology and helps your team complete the testing process more quickly and produce beautiful cucumber reports easily. There is a reason why Jenkins is one of the most popular continuous integration tools with test engineers! It packs everything that companies needs to support test automation into one highly functional tool.

Also Read: What is value stream management?

Automating testing empowers your team and allows them to be more confident in the work that they do because they can quickly see the results. When you couple that with the fact that Jenkins continues to evolve to keep up with the industry, its a no-brainier to go this route.

You might also be interested in our blogs on jira dashboards, xray dashboards, or zephyr dashboards.

In this blog post we will focus on Xray Test Management which is a JIRA plugin that allows teams to manage all test items as JIRA issues. We will discuss the testing lifecycle activities and break down how to effectively use Xray entities and features to plan, prepare, execute and report on the testing performed.

Test planning and design

An important aspect of test management is being able to plan effectively and structure tests in test plans and cycles in accordance with the overall project schedule.

• Let’s review the Xray entities, which provide the flexibility for users to perform the following:
  Create Tests to validate specific JIRA requirements, which then can be linked to provide end-to-end traceability.
• Add Preconditions to tests to ensure initial conditions are met, for example for a login test, a precondition can be that a user has been invited and has already registered on the system.
• Sort and organise tests within Test Sets, which essentially can be used as a collection of tests such as ‘Login tests’.
• Create a Test Plan and import Tests or Test Sets into this plan to capture the current version of these tests, this can be used for fortnightly sprints or System Integration Testing (SIT) for instance.
• Create a Test Execution which contains a list of Tests or Test Sets to be executed along with detailed steps, expected results and recorded execution status for each test. Typically Test Executions are linked from a Test Plan, however they can also be created in an ad-hoc manner.

Testers are usually delegated functional areas to script test cases for, as an example a tester may be required to analyse requirements for logging into a system. They will then script test cases in Xray specifically around the Login functionality, covering both positive and negative scenarios and ensuring the acceptance criteria has been met. These tests can then be grouped into a Login test set and order them in a logical fashion.

The test manager who oversees planning can then create specifically a SIT test plan for example, and import the Login test set as well as others, which contain the tests to build out sufficient test coverage for the system under test.

Test execution

Multiple test executions can be created from this SIT test plan, which aligns with the sprint deliverables to map out what is being tested in Cycles 1, 2 and 3 over the coming months. This is to ensure all tests are executed and there is room for defects to be identified, fixed and re-tested in subsequent cycles.

The overall SIT test plan will show the total number of test executions per test across the cycles, but the overall status will always reflect the latest version of the test executed. For instance, if a specific test fails in Cycle 1, is blocked in Cycle 2 and passes in Cycle 3, then the test plan will detail there were 3 test executions for this test and show an overall status of PASS in the test plan.

Xray provides all the bells and whistles for test executions, allowing the tester to execute tests with detailed steps and expected results, along with recording any comments, evidence/attachments, outcomes and raise associated defects in JIRA.

Also Read: The Power of JIRA for Project Management

It also supports creating both manual and automated tests, with the ability to specify tests in Cucumber BDD style language, to hook into test automation frameworks. Xray 5.0 was released in July this year, which now supports key major testing features such as parameterized tests, data-driven testing and compatibility with the Robot Framework v4.0. These tests can then also be linked to continuous integration (CI) systems such as Jenkins or GitLab.

One of the key benefits of this Xray is encapsulating requirement, test and defect entities all within JIRA and therefore having that full traceability across the project. This transparency is imperative for auditability which is vital for organisations running large complex projects.

Test Reporting

Xray provides test managers the ability to perform MI reporting with its out-of-the-box reports and gadgets, covering some key areas such as:

• Overall requirements coverage.
• Test run summary.
• Test coverage.
• Test plans metrics by status.
• Test sets list by status.
• Test executions list by status.
  
  

These gadgets can be added and sorted on a dashboard, as well as setting filters against each gadget. This allows users to present an overall dashboard to senior stakeholders on key testing metrics.

Also Read: What is value stream management?

A couple of limitations we have encountered with the Xray reporting features, is that there is only one evolution type chart (i.e. data over time) and no ability to analyse current data and forecast or predict upcoming trends. Currently Xray only has a Test Evolution gadget that shows test execution status trends for a group of Tests over a given time frame. It would be useful for test managers to view other evolution type charts for example, test scripts created over time or unresolved test executions across priority over time.

Neuro for Xray

Over at Dragonfly HQ, we’ve worked with multiple test management tools and plugins and given all the features and benefits outlined earlier on Xray, we’ve decided to take it one step further and that’s one of the main reasons why we built neuro.

Neuro is our engineering and quality management platform that turns insight into action, integrating with multiple systems including JIRA, Xray, Zephyr and Jenkins – removing the need to configure and manage reporting dashboards across each system separately. What this means for neuro users is having a single source of truth and being able to produce an overarching report capturing JIRA issues, defects, test execution and automation build outcomes all in the one consolidated view.

Some other key features of neuro are insightful out-of-the-box dashboards (including multiple evolution type charts), being able to enable predictions for charts over time, or to set project goals to closely track key milestones. For example, we can track a UAT Test completion milestone by visualising work throughput over time when constrained by a specific end date and identify when action is required to ensure the milestone is achieved.

Stay on top of your testing with Xray and neuro, by discovering neuro’s other key features and how it can drive efficiency, increase value and accelerate your growth by talking to our team today.

You might also be interested in our blogs on jira dashboards, and zephyr dashboards.