Developers are frustrated around the testing code for security and functionality and are increasingly incorporating automation and machine learning to ease workloads, according to results from an annual survey on software development trends from GitLab.

The is about the continuing problem that developers face around testing the past few years, with a majority of respondents saying delays due to code testing and review process was a frequent source of delay in the development process.

One specific piece of feedback from a customer noted that “testing delays everything.” Another complaint was that their software delivery teams passed testing responsibilities to their quality assurance employees in lieu of writing end-to-end testing suites, something they said has led to “very long” bottlenecks when shipping code to production. Other complaints highlighted how their employees do not like reviewing code and find it to be “a chore.”

It is perhaps unsurprising that automation is being steadily incorporated into more of the software development process. However, there are complications, with developers expressing frustration about the technical limitations and lack of practical automation options for parts of the code testing process.

“The strongest light at the end of the testing tunnel may be found in the use of artificial intelligence/machine learning,” the report states, noting that adoption of such tools has more than doubled over the past year and a substantial number of their customers say it is the most important skill they could learn for their future careers.

The sentiments point to growing acceptance within the developer community that security, like software development, is an iterative and continuous process. While “DevSecOps” has been around for decades, it’s clear that many organizations have yet to integrate the concept in part or in whole. TJ Jermoluk, CEO of Beyond Identity, who works to build passwordless identity and authentication services into the software updating process noted:

“The nature of a zero-trust system is that security is continuous and it’s checked all the time. You have to move from being bound to checking security at the perimeter of things to checking it at everything…at every single point where any form of transaction is done, whether its access to a database or an application or checking in source code.”

One of the biggest changes from previous years is around the adoption of Kubernetes, the open-source platform for automating cloud-based containers, workloads and services that can also be used to conduct end-to-end code testing and review. Other tools like static and dynamic attack surface testing saw big jumps in use as well.

The survey was conducted on 4,294 GitLab customers. While it drew from multiple industries, disciplines and regions, the most common respondent was male (81%), a software developer or engineer (41%) who was located in Asia (50%).