Test Automation From the Ground Up By Vasily Shishkin and Yaron Kottler

Introduction

Test automation promises many attractive benefits such as running multiple tests overnight at a click of a button, eliminating mindless work, increasing test coverage, and reducing the cost of testing. So the question is, how to get started?

Deliberation of Merit

The first order of business then is to be sure that test automation is the right decision in your particular case. The initial automation effort can be expensive, especially if the systems have not been designed with test automation in mind (very few of them are…). Before considering automation make sure you can positively respond to ALL the following per-requisites:

• The systems or applications are relatively stable with no significant redesigns expected in the foreseeable future.
• The organization has a solid test process in place.
• The Organization has the right type of person or team in place.
• The Organization has a defined scope for an initial automation effort.
• There is an understanding by senior management that test automation is an investment, not a short term solution.
• There is a return on investment (ROI) analysis demonstrating at least a 50% ROI for a set period.

For an effort to be considered for test automation, it should result in a positive ROI. Since the expected lifespan of testing a system is not always known or can be hard to predict, an attempt at a comprehensive ROI calculation at an early stage will likely miscalculate an actual ROI. A short term ROI estimate (between one to three years) is a safer and easier calculation to make, typically utilizing at least some of the following factors:

• Length of time to run a predefined test set manually.
• Cost and availability of support resources necessary for manual test execution.
• Number of times such test sets are expected to run during the agreed upon period.
• Expected effort for establishing automation infrastructure (mapping objects, reusable functions…).
• Expected effort for establishing test automation coverage utilizing above infrastructure.
• Expected maintenance effort factor between each version or cycles.
• Expected effort for executing automated tests and investigating results.
• The value of enabling manual testers to focus mostly on testing new functionality.
• Estimated value of shortening a test cycle by X days.
• Estimated value of increased test coverage consistency.
• Estimated value of knowledge retention.
• Estimated value of executing additional test types which couldn’t be executed earlier.

A ROI calculation is not enough though; as indicated above you must also ensure that you have a solid test process in place including things like scope, strategy, test types and test levels. Simply put, automated garbage is still garbage, so if you haven’t been able to positively answer the entire automation per-requisites list above, start there. Also, keep in mind that test automation will almost never replace the first round of manual testing of new functionality, but is rather targeted at reducing the cost and risks associated with regression testing. There are cases in which a progressive automation approach enables utilizing test automation for both regression testing and testing of new functionality but it is not recommended for organizations just getting started with test automation.

Once your foundation and preliminary ROI calculations are in place, it is time to run a test automation proof of concept (POC) project.

Test Automaton Proof of Concept (POC)

Phase	Duration (small project)	Customer Involvement
Determining Scenarios and Learning the Application	1 Day	2-3 Hours
Writing Up the Test Plan	2-3 Days (depends on revisions)	1-4 Hours (Depends on the infrastructure of the customer)
Scripting	1-3 Weeks	5 Hours – to answer whatever questions come up during scripting
Testing	1-3 Weeks (depends on issues found)	Best Case scenario: (no problems, testers have access to server hardware) – 3 Hours Worst Case scenario: (lots of problems, server hardware has to be monitored by customer): practically same as tester
Final Report	1 Day (compile and Present)	1-2 Hours for the presentation

A POC project should start with defining the immediate goals it should meet. While doing so, keep in mind that the POC’s main purpose is not actual automation implementation, but is rather intended to help determine what kind of methods and resources would be required for a successful automation implementation. It is recommended that the POC project take no longer than three to four weeks. It should have a balance between the need to discover the main automation challenges and demonstrating feasibility with the need to support relatively quick decision making. In addition to organization-specific goals, your list of POC goals should include at least some of the following:

• Verifying test automation feasibility.
• Exposing technical blocks and challenges.
• Experimenting with potential solutions and workarounds.
• Further refining cost estimates and ROI calculations.
• Experimenting with a number of test automation tools
• Finalizing a tool selection or at least supporting a selection.
• Experimenting with a number of test automation approaches (KDT, DDT, TDD, BDD, Progressive automation…) and finalizing an approach or supporting a selection.
• Helping establish test automation context and an appropriate state of mind at your organization.
• Defining a detailed 6 month implementation plan with 30, 60, 90 and 180 day targets.

Once your POC goals are in place, start the POC project by educating yourself on available automation approaches and their advantages. Selection of test automation approaches should include reviewing at least some of the following factors:

• Proficiency of current personnel with each approach and technical tools associated with it.
• Cost of training for each approach.
• Depth of knowledge of the system under test required for each approach.
• Number of personnel who have BOTH the skills required for the approach and the knowledge of the system under test.

When an approach is selected, next consider the tools you will need. Some of the important factors are:

• Compatibility with the technologies present in the system under test.
• Compatibility with the technologies planned to be utilized.
• Compatibility with the test management systems in place.
• Market prevalence of the tool ( a wider user base facilitates finding experienced users, support groups, and other network effect factors).
• Quality of tool vendor support.
• Tool usability.
• Tool features that simplify script maintenance.
• Cost of the tool.

Once the tools have been chosen, the next step in a POC project is to establish a detailed Test Automation Implementation Plan. The plan should include a list of reusable business and utility functions to be tested, as well as a detailed automation breakdown. The plan should allow for the distribution of workload, estimation of the timeline, and tracking of progress.

The last topic to consider is technical guidelines, such as: object mapping structures, creation and maintenance processes, and coding conventions.

When all of the preliminary work is finished, it is time to actually implement the selected scenarios on the tools of choice. From these scenarios, a list of encountered and predicted challenges should be generated and solutions for said challenges evaluated. If the challenges prove to be insurmountable, the POC conclusion is that there is no test automation feasibility. Otherwise, the information gathered should enable meeting the POC goals and enable more accurate recalculation of the ROI. If a revised ROI is unacceptable, another POC with an alternative approach or toolset can be performed. Alternatively, this could indicate that test automation might not be a good idea for this system.

Test automation maintenance

If the POC is completed successfully, and the decision to pursue automation is reached, the concern of automation maintenance should become the leading factor throughout your test automation implementation as it will be the major factor in delivering positive ROI. To insure efficiencies, it is important to build a framework that supports minimal and easy maintenance.

A good maintenance process follows this general outline:

1. Review the list of known changes in the system and conduct a gap analysis.
2. Create an action list of things to update such as:
   • New objects to map.
   • Existing object maps to update.
   • Object maps to remove.
   • API calls to revise.
   • Changes in system logic to accommodate.
3. Inform the personnel of the workflow and data set changes.
4. Upgrade scripts and infrastructure as necessary.
5. Conduct a manual sanity test to verify that the environment is operational.
6. Run a full suite of tests and investigate failures to understand if issues are system related or automation related.
7. In case issues are automation related, fix the automation as needed and add such issues to future gap analysis.

Once maintenance has been completed successfully, automation infrastructure and automation scenarios should be developed to support additional test coverage.

Summary

In this paper, we outline the basic steps of getting started with test automation at your organization. While each organization has different needs and technologies we find that just about every organization can benefit from utilizing test automation and that the above step by step guide will provide a good framework for getting started.

Examples of good candidates

One last thing to keep in mind. When looking at test automation, the following list of areas typically benefit from test automation:

• Regression tests that are repeated often – a sanity test is a good example of that.
• Tests of stable functionality that is not expected to change much.
• Tests that can be completed automatically with no human intervention.
• Tests which are expensive to run manually.
• Tests which require multiple user roles to execute.

Contact the Authors

Yaron Kottler: yaronk@qualitestgroup.com
Vasily Shishkin: vasilys@qualitestgroup.com

Tip of the day : Finding a software Testing Job

here are many people who would like to get software testing jobs, but they are unsure about how to approach it. This may seem like a dream job where people get to test software including games as part of their work. Most people do not realize the requirements that are needed to get into software testing jobs.

You firstly need to understand a little bit about how software testing works. When software is created it goes through a product life cycle which has a number of different stages. These include things such as specifications, design, coding and user acceptance. When it comes to software testing there is a requirement of an in-depth knowledge of both coding and software design. To get the best software testing job you need strong coding skills and experience with product design.

If you don’t have very much experience in these areas then it is up to you to either attend some type of training course or learn more on your own. This requires you to learn how to code and also how to do your own bug testing. When you have a lot of experience doing this then you will be much more regarded for software testing jobs than if you applied straight out of university.

Even if you have a computer science or computer engineering degree you may not have the necessary experience to apply for software testing jobs. Often in the computer industry it is very important for people to have experience. Experience often counts for much more than qualifications when it comes to hiring for a job. This is true when it comes to software testing jobs.

When you are applying for such jobs you want to make it clear to the employer of the experience you have in this area so that they can get a good idea of how much you really know. When you’re applying for software testing jobs you will be going up against a number of other people who may have a lot of experience and this can be difficult to compete against. That is why it is important for you to learn as much as you can in your own time and study about coding and bug testing yourself.

You can find a number of software testing books and coding books available to buy that can help you learn more about this area. But when it comes down to getting software testing jobs it is important to have hands-on experience rather than just know theory you read in a book. If you can demonstrate to an employer experience you have had in the software testing industry or even experience you have learnt yourself then this will go a long way to getting you good software testing jobs.

Even if you are not successful applying for software testing jobs the first time, you should keep trying and keep building on your experience. When you apply for a lot of software testing jobs you will get a feel for what type of person they are looking for and also be able to learn from the past interviews as to the best way to present yourself for software testing jobs

Source : http://ejobhub.org/software-testing-jobs/

So you want to be a Software Tester?

Some people decide they want to be a ‘software tester’. Perhaps they know someone who does this, or have seen the role in real life or in a movie or in a book. Or have experienced the result of inadequate testing. Perhaps the concept just ‘sings’ to them. Some of these people have no past in software development, and some have development experience but want to move to the testing area.

And then there are people, like me, who just ‘fall into’ the role. I was doing electrical engineering and we were working on a product which required some programming. As the only person on the team who knew anything about programming, I was ‘volunteered’. And did well, so that the programming area of the business was interested in me. The job offered ‘some programming and some testing’.

Ewww, testing. But at least I would get to do some programming, so I accepted. Turned out there never was any programming, but who cares; I found out I loved testing, and just as important, was really good at it.

Let me burst your bubble here. Testing is not about finding bugs. Oh sure, you will find them, and it is just as satisfying to the tester to find bugs as it was to the developer to generate the code. Testing is about evaluating the quality of the product. If you don’t find serious bugs, the quality can be considered high, and if you do, the quality can be considered low. And you document the bugs found so that the quality can be improved to the desired level.

In fact, testing contributes to Quality Assurance, which in general has as a goal, to

1) Prevent the generation of bugs (through processes, reviews, design tools and coding tools).

2) Discover any bugs which do get in as early as possible (via reviews and testing)

The later in the process you discover a bug, the more trouble and cost to fix (or patch around) it. Thus testing and Quality Assurance are valuable positions which may be attractive to you.
There are a lot of jobs out there for testers. Experienced testers.

So how do you become a Software Tester? One presumes if you already are a tester, you have everything you need to continue testing or move to a new test position. If you are not a tester, then you need to get the credentials which can get you into that testing position.

First of all, there is Education. Learn about what testing is and various ways it can be done. Check out books, online classes, and perhaps courses at local schools. Best is if the education includes hands on experience doing actual testing, but even ‘book learning’ has its place.

Next, there is Certification. ASTQB offers an internationally recognized Software Test certification, the CTFL (Certified Tester, Foundation Level), along with more advanced certifications which are beyond the scope of this article. This is a multiple choice test which can be studied for in a classroom setting (expensive) or online. Note this by itself will not get you a job, but it should give you an advantage.

Finally, and most importantly, there is Experience. This is a hard one. You need experience to get the job, and you need the job to get experience. Classic Catch 22. So maybe you will luck out and find an entry level test position. Don’t hold your breath. You need to be proactive.

Experience testing is experience testing. Look for it everywhere. If there is a class in testing with an actual testing lab, grab it. Volunteer for extra projects through your school, or for unpaid positions (by nonprofit organizations) which involve testing. If you can get an internship, that is of enormous benefit (several of our intern testers went on to test careers with our company and others when they graduated).

If none of that is practical, then look for classes or jobs with a lot of programming. Any time you program, you need to do testing. Treat the testing part of this activity as the ‘most important’ part. Do it and document it in a professional manner; these may provide evidence of your skills to potential employers. Plus, if you get an entry level programming job with a company worth staying with, you might be able to ‘move’ into the testing position you desire.

Finally, do testing as a ‘hobby’. There are not many computerized devices, web sites or software programs out there which have ‘no’ bugs. So, test them. Record the bugs in a professional manner. You may even be able to submit the bug reports to the company which produced them. And there are ‘cloud testing’ organizations out there which accept testers without experience to do real testing, and pay for it (usually per valid bug found).

So you can see, that in today’s market at least, getting to be a Software Tester is not a straight and broad road. But there are paths to that goal, even if they may be narrow and twisty.