Partial report, v0.1. This edition covers three named models, Claude Haiku 4.5, DeepSeek V4 Flash and DeepSeek V4 Pro, and seven anonymised peers. A full re-run of every model on one frozen scanner build is planned. Read the limitations before quoting any number.
Key findings
- On the same eight attack modules, DeepSeek V4 Flash was easier to push off-script than Claude Haiku 4.5 in this harness. The widest gap was output-format leakage: an average success rate of 81% for V4 Flash against 34.5% for Haiku with default settings and 23% with protective settings on.
- DeepSeek V4 Pro repeatedly described itself as Claude. It did so in two separate modules while we were calling it as a DeepSeek model through a paid DeepSeek key. Section DeepSeek said it was Claude covers what we checked and what it does and does not mean.
- Hardening Haiku helped a lot, but not completely. Two modules (reward hacking and indirect prompt injection) fell to 0%. Format leakage fell by about a third and was still the largest remaining gap.
- Across nine models, many-shot jailbreaking showed the widest spread. Some models resisted every attempt (0 of 12) and one complied with all 12.
- A single tool-abuse probe worked on five of eight models. It asked the agent to run an unrestricted wildcard search across customer records. Two models resisted it.
- Several early "clean" results were wrong because the testing was wrong, not because the model was safe. We found and fixed detector bugs, including one that hid 15 failed requests behind a perfect score. Section What the testing found about itself lists them.
What was tested
HexTyx attacks deliberately vulnerable mock AI applications: a tool-using support agent (ATS-4) and a multi-source agent that reads documents, email and web content (ATS-6). The model behind each application is swapped between runs, so the only variable is the model. A finding counts only when the scanner confirms the behaviour, for example a planted marker string appearing in the output, or a real tool call carrying attacker-chosen arguments.
Two sets of runs feed this report:
- Eight modules, two models. Claude Haiku 4.5 (
claude-haiku-4-5-20251001, default and hardened settings) against DeepSeek V4 Flash (deepseek/deepseek-v4-flash, default settings), early September 2026.
- Five modules, eight models. Agentic tool abuse, output-format leakage, many-shot jailbreak, indirect prompt injection and reward hacking, run against each model in turn, August 2026. DeepSeek was V4 Pro (
deepseek/deepseek-v4-pro) in this run.
Both DeepSeek models were reached through OpenRouter. The two runs therefore used different DeepSeek models, and figures from one run should not be read as figures for the other.
Head-to-head: Claude Haiku 4.5 vs DeepSeek V4 Flash
Success rate is the share of attack payloads for which the scanner confirmed the target behaviour. Lower is better for the defender. Each figure is the average across the target applications the module ran on.
| Module |
Haiku 4.5 (default) |
Haiku 4.5 (hardened) |
DeepSeek V4 Flash (default) |
| Output-format leakage |
34.5% |
23% |
81% |
| Reward hacking |
3% |
0% |
23.5% |
| Indirect prompt injection |
12.5% |
0% |
17% |
| Many-shot jailbreak |
0% |
0% |
8.5% |
| RAG poisoning |
0% |
0% |
4% |
| Virtualization framing |
0% |
3.5% |
0% |
| Adversarial suffix |
0% |
0% |
0% |
| Token smuggling |
0% |
0% |
0% |
Where V4 Flash differed. Most of the gap sits in three modules: output-format leakage (81% against 34.5%), reward hacking (23.5% against 3%) and many-shot jailbreak (8.5% against 0%). Indirect prompt injection was closer, at 17% against 12.5%.
Where they matched. Neither model showed any weakness to adversarial suffixes or token smuggling in this harness.
What the V4 Pro run showed. The five-module run used the larger DeepSeek V4 Pro and gave a more detailed picture of how it fails:
- Output-format leakage. Twelve of 13 payloads were flagged, but 10 of those 12 had byte-identical responses. V4 Pro sidestepped the requested format by calling a knowledge-base search tool and returning the raw result. That is one strong behaviour, not ten independent failures.
- Indirect prompt injection. Three of 12 payloads succeeded, and every one had the same signature. The model answered the user's real question correctly, then silently appended the injected marker after a separator, with no comment. A reader would have no reason to suspect anything happened. The same model did name and reject one other hiding technique (white-on-white text), so the gap is which techniques it notices.
- Reward hacking. Four of 17 probes were confirmed genuine on manual review. Three produced harmful instructions under a framing, and one was the counterfactual probe described below. The most serious was a fiction-framed request that produced actionable multi-step intrusion guidance. We are not publishing details.
DeepSeek said it was Claude
During the five-module run, DeepSeek V4 Pro identified itself as Claude, and it did so more than once:
- In an output-format leakage test, a structured response filled its model-name field with
"Claude (Anthropic)".
- In a many-shot jailbreak test, a separate response again described itself as Claude.
Two independent modules rule out a one-off. We then checked the obvious explanation, a routing mistake that sent requests to a Claude model. OpenRouter billed tokens against the DeepSeek key we had configured, and the target application's code contains no hard-coded identity. The model itself produced the claim.
What this does not show. A model's statement about what it is does not establish where it came from. Models routinely repeat names that appear in their training data, and we draw no conclusion about how DeepSeek was trained. It is two observations in one harness, not a controlled test.
Why it still matters for security.
- Self-reported identity is not provenance. If an audit log, a compliance record or a routing decision relies on the model's own answer to "which model are you?", it can be wrong. Record the model identifier returned by the provider's API.
- An unreliable identity claim is an impersonation risk. A model that asserts a trusted identity under pressure can lend credibility to a request it should not.
- It is not unique to one model. At least one other model in this test also made inaccurate claims about itself. Claude Sonnet 5 identified itself correctly throughout.
The full re-run will add a fixed identity probe, repeated several times per model, so we can say how often each model gets it wrong.
What hardening changed
Only Haiku was run with the hardened profile, so this section says nothing about DeepSeek.
Switching on the protective settings took reward hacking and indirect prompt injection to 0%, which is a full win on those two. Format leakage dropped from roughly 38% and 31% on the two targets to 23% and 23%. It stayed the biggest weak point. One module, virtualization framing, went the other way, from 0% to 7% on one target. That is small enough to be run-to-run variation, and we are watching it.
Five modules across nine models
The table below uses counts of confirmed findings out of the payloads in each module. Models A to G are anonymised, and the letters were assigned at random. Haiku is shown where its count was recorded in the same format.
| Model |
Agentic tool abuse (of 15) |
Format leakage (of 13) |
Many-shot (of 12) |
Indirect injection (of 12) |
Reward hacking (of 17) |
| Model A |
1 |
10 |
6 |
0 |
2 |
| Model B |
1 |
7 |
12 |
12 |
4 |
| Model C |
0 |
9 |
0 |
0 |
1 |
| Model D |
0 |
7 |
0 |
0 |
1 |
| Model E † |
2 |
5 |
9–10 |
2 |
0 |
| Model F † |
0 |
9 |
10 |
2 |
1 |
| Model G † |
1 |
7 |
1 |
0 |
2 |
| DeepSeek V4 Pro |
1 |
12 |
0 |
3 |
4 |
| Claude Haiku 4.5 |
not recorded |
not recorded |
0 |
not recorded |
not recorded |
† These models were scanned before several detector fixes landed and have not been re-run. Treat their numbers as provisional. Reward-hacking counts include only manually confirmed findings and leave out borderline or unreviewed ones. Format-leakage counts leave out cases where the model sidestepped into an unrelated tool call.
Patterns that held across models
- Many-shot jailbreak is where models differ most. The range runs from 0 to 12 of 12. Four of the nine models, including Haiku and DeepSeek V4 Pro, resisted every attempt, at every shot count and in every attack vector.
- The wildcard-search probe is widely effective. Five of eight models complied with an unrestricted wildcard query over customer records, in several different styles. Two resisted, and the eighth was flagged for a different tool-abuse pattern.
- One reward-hacking probe beat most models. It is a counterfactual self-exposure probe. Five models failed it in a row, and three held.
- The most reliable signal is a literal marker. A planted marker string appearing in the output was the least ambiguous evidence in every round. Softer, interpretive judgments produced the false positives and false negatives.
What the testing found about itself
We treat scanner bugs as findings, because a wrong detector produces a convincing wrong result.
- A perfect score that tested nothing. The first DeepSeek V4 Pro agentic run showed 0 of 15 findings, which looked like a model that resisted a probe two others had failed. In fact every request had returned an authentication error, and the error was logged at a level nobody saw. The run took 30 seconds. The valid re-run took 260.
- Refusals that were misread as compliance. The same family of bug turned up in seven files: models that quoted an attack in order to refuse it were flagged as having complied. It affected apostrophe handling, past-tense phrasing and several other refusal styles.
- Evidence thrown away. An early version of the scanner discarded the target's ground-truth data, and a later one truncated every stored response to 500 characters. Both are fixed.
- A false negative. A response with the marker verbatim inside a tracked field was reported as clean, because of a nested-structure parsing bug.
Each of these was found, fixed and re-verified against real model output before the numbers in this report were accepted.
Limitations
- Mock applications, not production systems. The numbers describe model behaviour inside the HexTyx test harness. They do not rank models by overall security.
- Small samples. Each module has 12 to 17 payloads. Compare counts, and do not read percentages as precise.
- The two DeepSeek runs used different models, V4 Flash and V4 Pro, and neither has been repeated. Differences between them, such as many-shot jailbreak at 8.5% against 0 of 12 and indirect injection at 17% against 3 of 12, cannot yet be split into a model difference and ordinary run-to-run variation.
- The detectors were developed against Haiku. The five modules were tuned on Haiku responses. Later fixes widened coverage to other models' phrasing, but a bias in Haiku's favour or against it can't be ruled out.
- The V4 Flash format-leakage figure was not checked against raw transcripts. Haiku's was.
- Not all models were re-run on the final scanner. The three early models are marked †.
- Haiku has no five-module counts in this format, and there is no hardened DeepSeek run for either model.
- OpenRouter serves models behind a slug. We recorded the slug requested, not the model identifier each response returned, so the exact build behind each run is not independently confirmed.
What comes next
The full report will re-run every model on one frozen scanner build, with identical protocol, version-pinned model names and a hardened-profile run for each. It will also publish confidence intervals. The coverage estimates in the 2026 benchmark are analyst estimates, and this testing is the measured companion to them. You can run the same scanner against your own AI application in the free playground.
Frequently Asked Questions
Which models were tested?
Ten models from eight providers. Claude Haiku 4.5, DeepSeek V4 Flash and DeepSeek V4 Pro are named. The other seven are anonymised as Model A to Model G.
Why are most models anonymised?
This is a partial report. Several of those models were scanned before later scanner fixes and have not been re-run, so naming them would attach provisional numbers to specific vendors. The full report will name every model with its exact version.
Does this mean DeepSeek is less secure than Claude Haiku 4.5?
Only in a narrow sense. In these test applications, DeepSeek V4 Flash was more often pushed off-script than Haiku on several modules. The samples are small, the applications are deliberately vulnerable mock systems, and the detectors were developed against Haiku. It is not a ranking of overall model security.
Can I run the same tests on my own application?
Yes. The HexTyx scanner runs the same attack modules against your AI endpoint. The free playground scanner is the quickest way to start.
Did DeepSeek really say it was Claude?
In two separate test modules, DeepSeek V4 Pro did. The response named "Claude (Anthropic)" as the model. We checked that requests were billed to the DeepSeek key and that the application does not hard-code an identity. This is not evidence of how DeepSeek was trained. It shows that a model's claim about its own identity should not be trusted as a record of which model ran.
Test your own agent before an attacker does
Run the free playground scanner against your AI application, or read the related guides.
Related Guides