Sleuth wants to use AI to measure software developer productivity

0

As knowledge workers, including software engineers, have shifted to remote work during the pandemic, executives have expressed concern that productivity will take a hit. The evidence is mixed on this, but in the software industry in particular, remote work has exacerbated many of the challenges employees were already facing. According to a 2021 Garden survey, the majority of developers found slow feedback loops during the software development process to be a source of frustration, second only to difficult communication between teams and functional groups. Seventy-five percent said the time they spend on specific tasks is wasted time, suggesting it could be used for more strategic purposes.

Seeking a solution to boost developer productivity, three former Atlassian employees – Dylan Etkin, Michael Knighten and Don Brown – co-founded Sleuth, a tool that integrates with existing software development tool chains to provide information to measure effectiveness. Sleuth today announced that it has raised $22 in Series A funding led by Felicis with participation from Menlo Ventures and CRV, which CEO Etkin says will be dedicated to product development and the expansion of the Sleuth workforce (particularly the engineering and sales teams).

“With the pandemic-induced avalanche of remote work, the need for developers, managers, and executives to understand and communicate engineering efficiency has increased dramatically,” Etkin told TechCrunch via e -mail. “Developers, who are no longer in the same room, need a way to coordinate deployments and a quick way to find out when a deployment has gone wrong. Managers need an unobtrusive way to find out proactively on the bottlenecks affecting their teams.Executives need an unobtrusive way to understand the organization-wide impact of their initiatives and investments.Sleuth takes care of understanding and communicating the efficiency of offline engineering and makes it digestible by all.

Etkin, Knighten and Brown were colleagues at Atlassian, where they claim to have helped the company’s engineering organizations move from releasing software every nine months to releasing daily. Etkin was an architect on the Jira team before becoming head of development at Bitbucket and StatusPage, while Knighten and Brown served as vice president of product and architect/team lead, respectively.

At Atlassian, which has grown from 50 to more than 5,000 employees when the Sleuth co-founders worked there, Etkin says it’s become “crystal clear” that many engineering teams haven’t. quantitative way to measure effectiveness – and that this gap could hold them back from growing and improving.

“Measurement of engineering effectiveness is a known, important, and growing problem that is now being addressed. As every business invests more in software engineering, the need for visibility into engineering effectiveness s intensified,” Etkin said. “However, measuring effectiveness has always been very difficult for a multitude of reasons, namely the complexity of the tools, the lack of access to data, and the use of questionable proxy measures that have bred micromanagement and mistrust.”

Sleuth’s solution is the DevOps Research and Assessment (DORA) metric, an emerging standard used by developer teams to measure the time it takes to deploy code, the average time it takes for a service to bounce back from an outage, and how often it takes to deploy code. which a team’s patches lead to post-deployment issues. DORA was born out of a Google academic research team, which between 2013 and 2017 surveyed over 31,000 engineers about DevOps practices to identify key differentiators between “low performers” and “elite”.

Sleuth isn’t the only platform that uses DORA metrics to quantify productivity. LinearB, Jellyfish and Athenian are among the competing solutions that have adopted the DORA standard. But Etkin says its competitors don’t “fully or accurately” follow these metrics.

“Sleuth is unique…because we use deployment tracking to model how engineers ship their work from concept to launch,” he explained. “Precise modeling of how engineers move through their pre-production and production environments and how they interact with issue trackers, CI/CDs, error trackers and metrics allows Sleuth to create a fully automated view of a team’s DORA metrics and their Engineering Efficiency.”

Sleuth uses AI to try to determine a team’s baseline change failure rate (i.e. the percentage of changes that resulted in degraded services) and average recovery time – two of the four DORA metrics – from existing systems such as Datadog and Sentry. The platform can automatically determine when a metric is outside that baseline, Etkin explains, and even automate steps in the development process to potentially improve the metric.

From Sleuth’s project dashboard, individual teams can track their DORA metrics. An organization-wide dashboard reveals trends across projects and teams.

“Customers just point Detective to to … error data and Detective lets engineers know when they’ve pushed those metrics into a failing range. Using AI to determine these values ​​means engineers can focus on their work without needing to understand every metric in their system or what ‘normal’ looks like for each.”

Detective

Tracking DORA metrics with Sleuth.

DORA metrics aren’t the ultimate solution, of course. They can be a hindrance when an organization’s focus on them becomes overwhelming. As Sagar Bhujbal, VP of Technology at Macmillan Learning, told InfoWorld in a recent article, “Developer productivity shouldn’t be measured by the number of errors, delivery delays, or incidents. This causes unnecessary angst with development teams who are always under pressure to deliver more capabilities faster and better.”

Etkin agrees, pointing out that engineering leaders must avoid the temptation to micromanage.

“Engineering is a creative business, and engineers are more like artists than assembly line workers,” Etkin said. Engineering managers need to… track the right metrics [and] follow them with precision [but also] give engineers the tools they need to improve metrics.”

Detective clients range from companies like Atlassian to startups like Launchdarkly, Puma, Matillion, and Monte Carlo. Etkin claims the platform has tracked nearly a million deployments and taken over a million automated actions on behalf of developers. He declined to reveal revenue figures when asked, but said Sleuth, which has 12 employees, grew 700% last year with a “very healthy” margin and cash flow.

Share.

Comments are closed.