A New Tool Explains Why Your AI Agent Acts

Dataiku's new tool, Kiji Inspector, helps developers see inside the "black box" to understand why AI agents make certain choices.

AI company Dataiku has released Kiji Inspector, a new open-source tool designed to solve a critical problem for developers: understanding why an AI agent chooses to use a specific tool. In a presentation at InfoQ, Dataiku’s Hannes Hapke explained that as AI agents become more complex, simply reading their “chain-of-thought” reasoning is no longer enough to ensure they are behaving as expected. These text-based explanations can be superficial or even misleading. Kiji Inspector aims to open the “black box” of agent decision-making, giving engineers a direct view into the model’s internal logic at the moment it decides to act, providing a more reliable way to debug and monitor these increasingly autonomous systems.

Kiji Inspector works by applying a technique from the field of mechanistic interpretability called Sparse Autoencoders (SAEs). In simple terms, an SAE is a type of neural network trained to find the most important, fundamental features within a massive amount of data. Kiji Inspector applies SAEs directly to the large language model’s internal activations—the raw numerical data flowing between its layers—at the precise moment it generates a “decision token” to call a tool. To access this internal data efficiently, the tool uses a modified version of the popular vLLM inference engine. By isolating the specific internal features that caused the tool-use decision, developers can move beyond the agent's own narrative and see the direct, causal reasons for its actions.

This tool arrives at a time when AI explainability is shifting from a research concept to a practical necessity. Early methods for understanding AI focused on analyzing inputs and outputs, but this approach is insufficient for complex, multi-step agents that can interact with external systems. Kiji Inspector represents a move toward a deeper level of analysis that examines the internal mechanics of the model itself. This is crucial as agents are deployed in high-stakes environments like finance, healthcare, and infrastructure management, where an unexpected or incorrect tool use could have serious consequences. The ability to audit an agent's decision-making process is becoming a core requirement for building trust and ensuring safety.

For CTOs, developers, and security teams, Kiji Inspector offers a tangible path toward building more robust and predictable agentic systems. By using the tool for production monitoring, teams can create workflows to flag when an agent makes a decision based on unusual or undesirable internal reasoning, even if the final outcome appears correct. This allows for proactive debugging and helps prevent future failures. As an open-source project, it invites community contribution and wider adoption, potentially setting a new standard for how developers ensure their AI agents are not just effective, but also transparent and trustworthy. The next step will be to see if these deep-inspection techniques become integrated into mainstream AI development platforms.

For developers building agentic systems, understanding tool selection is critical for debugging, security, and reliability. Kiji Inspector provides a mechanism to move from unreliable chain-of-thought logs to direct, causal analysis of a model's internal state, enabling more robust AI agent behavior in production.

As companies deploy AI agents in mission-critical roles, unpredictable behavior creates significant risk. Tools like Kiji Inspector reduce this risk by enabling teams to audit agent decisions, ensuring they operate safely and as intended, which builds trust and prevents costly operational failures.