Grant controls depend on separation of duties. An AI agent can compress prepare, review and approve into one automated step. This guide covers how that fails and how to test that it does not.
Part of the AI Security for Research Institutions guide series.
Research institutions increasingly use AI in grant administration, proposal preparation, compliance workflows, reporting, and document review.
That creates a subtle security problem.
Many grant controls depend on separation of duties.
One person prepares a disclosure.
Another reviews it.
A compliance office validates it.
An authorized official approves a submission.
An AI agent can compress those steps into an automated workflow.
That creates productivity.
It can also create a control failure if the same AI identity can prepare, modify, validate, and effectively approve a transaction.
A conflict-of-interest workflow may involve:
If an AI agent participates in several of these steps, the institution must ask whether the workflow still preserves the required separation of duties.
This is a threat model, not a claim that a specific AI agent has already bypassed a particular institution's COI process.
The risk exists because autonomous software can execute multiple workflow stages faster than a human reviewer may realize.
Imagine an agent responsible for identifying incomplete disclosures.
It discovers an apparent conflict.
It drafts a mitigation action.
It updates the record.
It then triggers the “review” step using the same service identity.
The system may technically record a review.
But governance intent has been defeated.
The critical question is:
Was the second step actually independent?
This is the AI version of the classic separation-of-duties problem.
Research funding processes contain high-value information:
An AI workflow with access to this information becomes a concentrated target.
A compromised agent might not need to steal a database password.
It may already have legitimate access to complete its assigned task.
The attack can therefore begin with manipulation of the workflow rather than compromise of the identity layer.
NSF has explicitly warned that non-public proposal and review information must not be uploaded to non-approved generative AI systems. NSF states that such disclosure can breach confidentiality and undermine the integrity of merit review.
This illustrates a broader institutional principle:
AI data handling is part of grant compliance.
A research office therefore needs to know which AI services can process proposal or review data, what those systems retain, and what controls prevent unauthorized disclosure.
Consider an AI grants assistant with access to a proposal-management system.
An attacker finds a way to inject instructions into a document or message the agent will process.
The injected content tells the agent to:
If the system has broad permissions, one manipulation can become a multi-stage event.
The exact scenario is illustrative. The security principle is real:
the blast radius is determined by the workflow's authority.
A practical classification model might separate:
Public: published calls, public deadlines, public guidance.
Internal: administrative procedures, non-public operational information.
Confidential: proposal drafts, budgets, investigator disclosures.
Restricted: protected reviewer information, sensitive sponsor material, participant-related research information, or data governed by specific use restrictions.
AI systems should inherit those classifications.
A general-purpose assistant should not become the default processing location for Restricted content simply because it is convenient.
Human-in-the-loop controls often look good on paper.
But a reviewer cannot meaningfully govern an AI workflow if the interface hides:
A useful approval gate exposes the decision context.
Otherwise, the human becomes an approval mechanism without independent visibility.
For each AI-enabled grants workflow, measure:
These fields can become part of the institution's AI system inventory.
A model can pass a generic jailbreak test and the workflow can still fail.
The more useful testing sequence is:
Can malicious input enter the workflow?
Can it influence the agent's objective?
Can it access restricted grant data?
Can it invoke an approval-related tool?
Can it bypass separation of duties?
Can it export sensitive information?
This is why research AI security needs workflow-level testing, not only prompt testing.
The institution also needs to consider integrity risks.
NSF's policies recognize that AI-assisted tools can introduce fabrication, falsification, and plagiarism risks in funded research activities.
That matters because a compromised AI workflow can alter more than a security record.
It may influence the content or authenticity of a proposal, report, or research submission.
Security controls therefore protect both the system and the integrity of the scientific funding process.
The goal is not to remove AI from grant administration.
The goal is to preserve the controls that existed before AI automated the workflow.
Automation should not collapse separation of duties.
An agent should not approve its own work.
A convenient service account should not become universal authority.
Sensitive grant information should not flow into AI systems without an explicit data-use decision.
HexTyx tests AI agents against live, LLM-backed applications that use synthetic data in controlled environments. Our record covers 28 domain applications. These are not research-institution deployments and no real participants or studies were involved, but the failure patterns map directly onto the boundaries in this guide.
An agent can manufacture the evidence of its own approval. In a synthetic agent-to-agent handoff app (Relay), a forged handoff claiming an admin-worker identity was accepted in the unhardened configuration. The hardened configuration rejected the same attempt at a signature-verification gate. In a synthetic banking app (Vault), a $5,000 transfer executed on a claimed confirmation with no real confirmation behind it.
Both are the separation-of-duties failure described above, in a different domain. The control that worked was verification the agent could not supply for itself.
Run the free playground scanner against your own AI application, or read the other guides in this series.