TRACER-AI: A Multi-Layer Explainable Framework for Prompt Injection, Agent Goal Hijacking, and Tool Misuse Detection in Agentic AI Systems

Abstract

Large language model (LLM) agents extend generative models with planning, memory, and external tool access, but this capability creates a security path in which untrusted content can alter instructions, hijack an agent\'s operational goal, and trigger harmful tool actions. This paper proposes TRACER-AI, a four-layer explainable defense-in-depth framework that combines (i) semantic prompt-injection detection, (ii) continuous goal-integrity monitoring, (iii) contextual tool-risk control, and (iv) structured explainable security decisions. The framework is designed around the attack progression prompt injection -> goal hijacking -> tool misuse rather than treating prompt filtering as the only enforcement boundary. A dynamic risk score fuses prompt-injection probability, goal deviation, tool risk, and contextual anomaly before action execution. A controlled proof-of-concept evaluation was conducted on a 3,500-case synthetic adversarial testbed containing benign interactions and five attack families: direct prompt injection, indirect prompt injection, goal hijacking, tool misuse, and chained attacks. The held-out test set comprised 1,050 cases with previously unseen attack wording and benign security-text decoys. The standalone prompt detector achieved 0.679 accuracy, 0.575 F1-score, and 0.760 ROC-AUC, illustrating the weakness of relying on prompt detection alone under distribution shift. In contrast, the full TRACER-AI configuration achieved a 96.4% attack detection rate, reduced attack success rate to 3.6%, preserved 99.0% benign task success, and limited false positives to 1.0% in the controlled testbed. The results support the central hypothesis that agent security benefits from multiple independent checkpoints spanning instruction intake, goal continuity, and execution-time tool authorization. The study also maps the framework to contemporary agentic-AI security guidance and benchmark research, and provides a reproducible experimental protocol for subsequent validation on AgentDojo, InjecAgent, AgentDyn, and domain-specific agent benchmarks.

Introduction

Agentic AI systems go beyond traditional chat-based LLMs by planning tasks, using external tools, accessing files, calling APIs, and performing actions that can change system states. While this capability provides significant benefits, it also introduces new security risks because vulnerabilities can affect not only generated text but the entire action pipeline. One of the major threats is prompt injection, where malicious instructions hidden in user inputs or external sources such as webpages, emails, and documents can manipulate an AI agent’s behavior.

Existing defenses often focus only on detecting malicious prompts, but they may fail when attackers alter the agent’s goals or misuse authorized tools. Therefore, a stronger security approach requires continuous monitoring of the user’s original objective, detection of unauthorized goal changes, and verification of tool usage before execution.

The paper introduces TRACER-AI, a defense-in-depth security framework designed to protect agentic AI systems through four major layers:

1. Prompt and context inspection – Detects potentially malicious instructions and separates trusted user commands from untrusted external information.
2. Goal-integrity monitoring – Maintains a protected representation of the original user goal and identifies unauthorized changes.
3. Execution-time tool-risk assessment – Evaluates whether requested tool actions are safe and authorized before execution.
4. Explainability generation – Provides structured evidence explaining why a security decision was made.

The framework addresses several limitations of existing approaches by combining prompt detection, goal monitoring, tool authorization, and transparent decision tracking rather than depending on a single security mechanism.

Main Contributions:

• Develops a threat model connecting prompt injection, goal hijacking, and tool misuse into a complete attack chain.
• Proposes a four-layer TRACER-AI architecture for continuous agent protection.
• Introduces a dynamic risk score combining:
  • Prompt injection probability
  • Goal deviation
  • Tool authorization risk
  • Contextual anomalies
• Provides explainable security decisions using evidence-based traces.
• Evaluates the framework using a controlled dataset of 3,500 attack and normal cases.
• Creates an evaluation approach compatible with agent-security benchmarks such as AgentDojo, InjecAgent, AgentDyn, and NetInjectBench.

Literature Review Summary:

Previous research established the foundation for agentic AI through technologies such as ReAct and Toolformer, which enable LLMs to reason and interact with external tools. However, these capabilities also create security challenges.

Research on prompt injection shows that attackers can exploit untrusted external content to manipulate AI agents. Existing defenses include:

• Spotlighting – Adds provenance information to untrusted content.
• StruQ – Separates trusted instructions from external data.
• SecAlign – Improves model resistance through alignment methods.
• PromptArmor – Detects and removes injected instructions.
• CaMeL – Uses system-level separation and capability restrictions.

Other studies focus on tool-use safety. Approaches such as ToolEmu, R-Judge, and AgenTRIM highlight the need for evaluating and controlling AI actions, especially when agents can access sensitive systems.

Security benchmarks such as AgentDojo, InjecAgent, AgentDyn, and NetInjectBench demonstrate that modern AI security evaluation must consider complete workflows rather than only prompt classification.

Research Gap:

Existing solutions usually protect only one stage of the AI pipeline, such as prompt filtering or tool authorization. They do not fully address the continuous transition from:
malicious instruction → goal modification → unsafe action execution.

TRACER-AI fills this gap by monitoring the complete attack lifecycle and providing an auditable security trail.

Threat Model:

The framework protects:

• User objectives
• Sensitive information
• Agent memory
• Tool credentials
• External systems accessed through tools
• Integrity of agent actions

The main attacker capabilities considered are:

• Direct prompt injection
• Indirect prompt injection through external data
• Goal hijacking
• Unauthorized tool usage
• Multi-stage chained attacks

Security Requirements:

TRACER-AI aims to provide:

• Instruction provenance: External data should not gain authority over user commands.
• Goal continuity: The original task objective must remain protected.
• Execution-time control: Every tool action must be checked before execution.
• Defense in depth: Failure of one security layer should not compromise the system.
• Utility preservation: Legitimate tasks should continue without excessive blocking.
• Explainability: Security decisions should include clear evidence and reasoning.

Conclusion

This study presented TRACER-AI, a multi-layer explainable security framework for protecting tool-enabled agentic AI against prompt injection, agent goal hijacking, and tool misuse. The framework reframes agent compromise as an attack progression that can be interrupted at multiple independent checkpoints. Prompt analysis estimates injection risk; goal-integrity monitoring detects objective drift; tool control validates authorization and context immediately before execution; and the explainability layer records the evidence behind each intervention. A controlled 3,500-case proof-of-concept evaluation showed why this layered design is valuable. The prompt detector alone generalized imperfectly to unseen wording and produced substantial benign overblocking. The full TRACER-AI configuration reduced attack success to 3.6% while preserving 99.0% benign task success on the held-out controlled test set. These results do not establish universal robustness, but they demonstrate the architecture\'s central principle: security signals observed at different stages of the agent action pipeline can compensate for one another and improve the security-utility balance. Future work should prioritize four directions. First, the framework should be evaluated end-to-end on AgentDojo, InjecAgent, AgentDyn, and emerging domain-specific benchmarks. Second, goal integrity should be upgraded from surface similarity to provenance-aware semantic entailment that distinguishes legitimate task refinement from malicious replacement. Third, tool risk shouldSXwSS incorporate parameter-level information flow, data sensitivity, identity, and least privilege. Fourth, adaptive attacks should be constructed against the complete risk aggregator to assess whether a knowledgeable adversary can keep each individual signal below threshold while still causing harmful action. These extensions would move TRACER-AI from a controlled proof of concept toward a deployable reference monitor for trustworthy agentic AI.

References

[1] OWASP GenAI Security Project, “OWASP Top 10 for Agentic Applications for 2026,” Dec. 9, 2025. [2] A. Vassilev, A. Oprea, A. Fordyce, H. Anderson, X. Davies, and M. Hamin, “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations,” NIST AI 100-2 E2025, National Institute of Standards and Technology, Mar. 2025, doi: 10.6028/NIST.AI.100-2e2025. [3] H. Booth, W. Fisher, R. Galluzzo, and J. Roberts, “Accelerating the Adoption of Software and Artificial Intelligence Agent Identity and Authorization,” NIST NCCoE Concept Paper, Feb. 5, 2026. [4] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection,” Proc. 16th ACM Workshop on Artificial Intelligence and Security (AISec), pp. 79–90, 2023, doi: 10.1145/3605764.3623985. [5] Y. Liu, G. Deng, Y. Li, K. Wang, Z. Wang, X. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, L. Y. Zhang, and Y. Liu, “Prompt Injection Attack against LLM-Integrated Applications,” arXiv:2306.05499, 2023. [6] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing Reasoning and Acting in Language Models,” arXiv:2210.03629, 2022. [7] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language Models Can Teach Themselves to Use Tools,” arXiv:2302.04761, 2023. [8] X. Liu et al., “AgentBench: Evaluating LLMs as Agents,” arXiv:2308.03688, 2023. [9] Y. Ruan et al., “Identifying the Risks of LM Agents with an LM-Emulated Sandbox (ToolEmu),” arXiv:2309.15817, 2023; presented at ICLR 2024. [10] Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents,” arXiv:2403.02691, 2024. [11] E. Debenedetti, J. Zhang, M. Balunovi?, L. Beurer-Kellner, M. Fischer, and F. Tramèr, “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents,” arXiv:2406.13352, 2024. [12] K. Hines, G. Lopez, M. Hall, F. Zarfati, Y. Zunger, and E. Kiciman, “Defending Against Indirect Prompt Injection Attacks With Spotlighting,” arXiv:2403.14720, 2024. [13] S. Chen, J. Piet, C. Sitawarin, and D. Wagner, “StruQ: Defending Against Prompt Injection with Structured Queries,” arXiv:2402.06363, 2024; USENIX Security, 2025. [14] S. Chen et al., “SecAlign: Defending Against Prompt Injection with Preference Optimization,” arXiv:2410.05451, 2024; ACM CCS, 2025. [15] T. Yuan et al., “R-Judge: Benchmarking Safety Risk Awareness for LLM Agents,” arXiv:2401.10019, 2024. [16] E. Debenedetti, I. Shumailov, T. Fan, J. Hayes, N. Carlini, D. Fabian, C. Kern, C. Shi, A. Terzis, and F. Tramèr, “Defeating Prompt Injections by Design,” arXiv:2503.18813, 2025. [17] T. Shi et al., “PromptArmor: Simple yet Effective Prompt Injection Defenses,” arXiv:2507.15219, 2025. [18] H. Li, R. Wen, S. Shi, N. Zhang, and C. Xiao, “AgentDyn: A Dynamic Open-Ended Benchmark for Evaluating Prompt Injection Attacks of Real-World Agent Security System,” arXiv:2602.03117, 2026. [19] R. Betser, S. Bose, A. Giloni, C. Picardi, S. Padakandla, and R. Vainshtein, “AgenTRIM: Tool Risk Mitigation for Agentic AI,” arXiv:2601.12449, 2026. [20] [R. K. Shayoni, M. F. Shoaib, S. M. A. Hossain, and M. F. Mridha, “NetInjectBench: Benchmarking Indirect Prompt Injection in Tool-Using Large Language Model Agents for Network Operations,” arXiv:2607.10490, 2026.

Copyright

Copyright © 2026 Pallavi Singh, Khushboo Gupta, Pratibha Singh. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.