See How Well AI Agents Use Your Software

Microsoft's new open-source tool, DevAx, lets you test and measure how effectively AI coding agents can actually use your software products.

Microsoft has released a new open-source tool designed to help developers see their software through the eyes of an AI. According to the announcement reported by Slashdot, the project, named DevAx, addresses a growing challenge for engineering teams: understanding how effectively AI coding agents can interact with their products. As developers increasingly rely on AI assistants to write and manage code, the software these agents use—from command-line interfaces (CLIs) to software development kits (SDKs)—needs to be optimized for them. The problem, as Microsoft frames it, is that an agent's ability to discover and correctly use a tool can be unpredictable. Performance varies widely based on the agent, its configuration, and the design of the software it's trying to use. DevAx aims to replace this guesswork with a systematic way to measure and improve how well software works for these new, non-human developers.

The DevAx framework functions as an evaluation engine that simulates how an AI agent would approach a given development task using a specific tool. It provides a structured way to define tasks, run them against different AI models, and score the results based on success, efficiency, and robustness. For example, a team could use DevAx to test whether an AI agent can successfully use their CLI to provision a new piece of infrastructure or query a database via their SDK. The tool automates these test runs, allowing developers to benchmark performance across different versions of their software or different AI models. By generating concrete metrics, it transforms the abstract problem of "AI-friendliness" into a quantifiable engineering goal. This allows teams to identify specific points of friction, such as ambiguous documentation, inconsistent command syntax, or unhelpful error messages that might confuse an AI agent even if they are clear to a human.

This release is part of a broader industry shift toward treating AI agents as a primary user group for developer tools. For years, the focus has been on "Developer Experience" (DX), which prioritizes clear documentation, intuitive APIs, and helpful tooling for human engineers. Microsoft's tool formalizes a new concept: "Agent Experience" (AX). Just as companies compete on the quality of their DX, they will soon compete on the quality of their AX. This trend is driven by the rapid adoption of tools like GitHub Copilot and the emergence of more autonomous agents that perform complex software engineering tasks. If an agent cannot effectively use a particular cloud service's API or a specific open-source library, its human user is likely to switch to an alternative that works more seamlessly. DevAx is one of the first major open-source efforts to provide a standard for measuring this new, critical layer of usability.

For founders, CTOs, and engineering leads, the immediate takeaway is to begin evaluating your products from an agent's perspective. Integrating a tool like DevAx into a testing pipeline can provide a significant competitive advantage by ensuring your software is ready for the next wave of AI-driven development. Teams should ask whether their APIs are self-describing, if their error messages provide enough context for an LLM to self-correct, and if their documentation is structured for machine readability. Looking ahead, we can expect "AX optimization" to become a standard practice, similar to performance testing or security scanning. As AI agents become more capable, the tools that are easiest for them to use will see the fastest adoption. Microsoft's DevAx provides an early framework for what will likely become an essential part of the modern software development lifecycle.

As AI agents become common in development workflows, software that is difficult for them to use will be at a disadvantage. DevAx provides a standardized way for engineering teams to benchmark, debug, and improve their APIs, SDKs, and CLIs for discoverability and usability by these non-human users.

Companies whose developer tools are easily adopted by AI agents will gain a significant competitive edge. By optimizing for AI-driven development with tools like DevAx, businesses can increase product adoption, reduce friction for users who rely on AI assistants, and position their ecosystems as future-proof.