Open Source Testing – The 2020 Survey

Tricentis is a software testing company founded in 2007, it provides software testing automation and software quality assurance products for enterprise software.

Being a commercial tool vendor I was a little surprised at their commitment to open source testing initiatives. A brief inspection of their website would tell you that they contribute to popular testing frameworks such as  Flood and SpecFlow, supporting 100,000’s of users around the world.


The Open Source Testing Survey

Tricentis conducted a global survey consisting of around 2000 people, focusing mostly on test professionals. The overall objective was to gather data regarding the use of open-source testing tools within the testing community.

The results appear to be mostly without surprise but for this article, I have decided to cover the findings that I personally found most interesting.


Years of experience in the field

Testing, as a distinct role, is fairly recent, with the results of the survey supporting that. Of those asked, less than 15% had more than 15 years of experience in the field. On the flip side, it also goes a long way to verifying that testing is a career that can span over a whole career.

The largest single demographic was between 5-10 years which was also around the time I was introduced to the field. I have since seen the number of testers per team and project increase.

Over 30% of those surveyed had less than 5 years of experience which clearly indicated that there is a considerable amount of individuals who are choosing software testing as a viable career path

Within the next 5-10 years, the number of people with 15, 20+ years of experience as a tester will no doubt increase. This will increase the competition for roles similar to the levels we see in the developer space, but it will also increase the opportunity for newer testers to work with vastly experienced testers which can only help the industry move forward.


Open-Source Testing Tools

When looking into the use of tools and languages it’s important to understand that a good testing approach will use a number of tools and languages, it’s always better to use the right tool over the tool you’re comfortable with. Taking that into the account the survey did allow for multiple answers.

Absolutely no surprises here with Selenium being the most popular. I was a little surprised however with TestNG having double the usage when compared to JUnit, being that they are fairly similar I feel either is an excellent choice to support your test runs.

Selenium can often get a lot of bad press but the results of this survey show it’s a valuable skill to have and won’t be going anywhere any time soon.


Java has been the most popular language with regards to test automation for some time. This will obviously be in part due to the general popularity Java still has as a programming language, the level of support is almost infinite. It’s still the language I would recommend for someone looking to create an automation framework.

On the other hand, It’s good to see other languages being reasonably close behind, this will only increase the number of those looking to skill up in the field. Being versed in multiple languages is ideal but if you are comfortable with any of the top four, it puts you in a very good position.


Use of BDD and Expertise 

With regards to the level of expertise, this is not an easy one to measure, most people felt comfortable putting themselves within the proficient bracket which is most likely more than enough for most of the mid-senior roles on the market.

I feel the line between proficient and expert may not always be directly related to the tester’s skill but could also relate to how effectively Behaviour Driven Development has been implemented. Experience implementing an efficient BDD practice that provides value would certainly give me the confidence to regard myself as an expert.


It was no surprise to me to see cucumber as the most widely used BDD framework, it’s almost become synonymous with Behavior Driven Development. I was a little surprised by the lead it had over tools such as Specflow and Serenity as they seem to be growing in popularity recently.

This probably has something to do with Java and JavaScript being the two most popular programming languages for test automation being that cucumber supports them both.



Test Approaches

Almost 80% of those surveyed said that they do some form of automated functional testing, which although I’m not overly surprised by, is still reassuring to see. Manual testing was second with exploratory testing third. I tend to group these together and I feel a good test approach should always include manual exploratory testing.

I do expect testers to be given more responsibility with regards to Load and Security testing over the coming years. As testing shifts towards the left and testers become more and more technical, the need for us to be multi-skilled is more relevant than ever.

The findings below corroborate these views with around 80% saying they are at least frequently writing automated tests with only 4% claiming they never write automated tests.

I personally feel there is still space a good tester who does not write code but they are very few and far between so you have to be absolute rockstar manual tester in order to survive in the current climate.


I found this to be a very interesting question and one that resonated with me quite a bit. This can often feel like the final hurdle and is often left at least partially incomplete. It did give me some reassurances that it’s a common issue.

I have, however, seen an improvement in this area with the recent resurgence of DevOps so we can only hope this is less of an issue going forward.


Load/Performance Testing

Traditionally speaking, Load and in most cases Performance testing have been carried out towards the very end of a release, Functional testing had been successfully integrated into the sprint but Load and Performance testing were done at the end if even done at all.

In my time working in the industry, I have seen a huge shift away from such behaviour where load/performance testing is now being included within the pipelines which have allowed teams to get relevant feedback a lot sooner.

Around 60% of those asked say that performance and Load testing is carried out in their organisation. This on its own may not mean a huge deal to a Tester until you look at the next graph.


This is where the results start to get interesting and become relevant. It would appear that testers now form the bulk of those actually carrying out the load/performance testing, almost double that of the next.

With over 60% of companies now doing such tests and within that over 50% of those doing the work being testers, it’s fairly clear to me it has to become part of a testers skillset going forward if it already is not.


When asked which tool was the most popular I was not surprised to see it was JMeter but I was a little surprised to see how far ahead it was when compared to other popular performance testing tools such as Gatling.

I was however very surprised to see Selenium so high up being used as a performance testing tool



Some of the results of the survey were slightly surprising whilst a number of them were genuinely reassuring. I feel it gives a clear indication of the continued trend of testers requiring a more rounded technical skillset. It was also interesting to see that there a number of clear front runners with regards to testing tools and languages.

For full details of the survey and watch the full webinar follow the link to the official survey website

Please follow and like us:

Leave a Reply

Your email address will not be published. Required fields are marked *