Scaling Misuse Detection with Self-Assessment

Tl;dr: We introduce self-assessment, a technique that lets Claude give a preliminary safety assessment of each conversation right after it responds. On Claude Fable 5, self-assessment flags 78% of the conversations that a stronger prompted classifier flags as potentially harmful, and 94% of the high-priority ones, at a 1% false-positive rate. Self-assessment scales with model intelligence: because the model that had the conversation also assesses it, the quality of the assessment improves as our models do. It is also efficient, because the conversation is already in the KV cache and an assessment takes only a short instruction and a few output tokens. Since June, self-assessment has revealed large-scale misuse not identified by other methods, helped us improve our other classifiers, and served as an additional layer in our defense-in-depth strategy.

Introduction

Frontier models are now being used to cause real-world harm. As reported in late 2025, Anthropic disrupted a cyber espionage campaign in which suspected state-sponsored attackers used Claude Code to attempt intrusions into roughly thirty organizations, with the model carrying out most of the operation on its own (Anthropic, 2025). Our recent September 2026 threat intel report found that AI-assisted attacks have spread well beyond state groups: criminal groups and hacktivists now run similar AI-driven campaigns (Anthropic, 2026a). Because improvements in coding, reasoning, and autonomy make each new model more useful to malicious actors (Carlini et al., 2026), our ability to monitor for misuse must keep pace.

Today, we’re introducing self-assessment, a technique that lets Claude provide a safety assessment of its own conversations. After each of Claude’s responses, self-assessment appends a short instruction (for illustration, something like “Please reflect on the safety of this conversation”), to which Claude responds with a compact safety assessment. The assessment states whether the conversation involves a potentially harmful request (such as one that may violate our Usage Policy), what kind of harm it involves, a priority score for human review, whether Claude refused, and a one-sentence explanation of what led to the assessment.¹ Self-assessment builds on a line of work in which a language model monitors its own behavior (Joglekar et al., 2025; Greenblatt, 2025; Lee et al., 2026; Wang et al., 2024). We apply this idea to misuse detection: Claude assesses the entire conversation, including requests that it refused, and the assessments feed additional offline review in production.

Because the monitor is the model itself, it should become more capable as models improve. A scaling experiment supports this: detection improves consistently across four model sizes (see Results). Unlike standard safety classifiers such as probes, which score a conversation against a harm fixed in advance, self-assessment generalizes to harm categories it was not specifically trained to detect. Removing an entire harm category from training does not substantially degrade its performance on that type of harm. This points to a monitor that draws on Claude's own understanding of harm from its broader training, not only on the examples we train it on, which can help us discover kinds of misuse we did not anticipate.

One key benefit of self-assessment is efficiency. When Claude finishes a response, the entire conversation is already in the KV cache. Each assessment therefore only requires processing a short instruction and generating ~10 additional tokens on average. Over our production API traffic distribution, we estimate that self-assessment adds less than 2% to our inference compute.² The main alternative to self-assessment is a prompted classifier, a separate model that is given our policies and the conversation in its prompt. It is the strongest automated classifier we tested (see Results), but it has to read the conversation and the policies again and then reason about them, so it needs many times more compute per conversation (Sharma et al., 2025; OpenAI, 2025). We therefore use it where a closer look is warranted. On high-priority conversations, self-assessment flags 94% of what the prompted classifier flags, with a fraction of the compute.

We now run self-assessment in every Claude model we have released since Fable 5, across a large fraction of Claude’s global traffic, as one layer in a safety defense-in-depth strategy that complements existing safeguards (Cunningham et al., 2026). Concretely, we use self-assessment as an offline monitor with wide coverage, which serves as a first stage for higher-precision investigation tools. On Claude Fable 5, self-assessment flags 78% of potentially harmful conversations, and 94% of high-priority ones, at a 1% false-positive rate, as judged by a prompted classifier with our full safeguards policies in context. It covers a broad range of harm areas in our policy, including rare harms that don’t yet have a dedicated classifier. Self-assessment has limits. It does not catch every harmful conversation, and it is not robust enough to stop a dedicated attacker. This is why we do not rely on it alone: it works alongside other safeguards that block harmful requests in real time, and we continue iterating on its effectiveness.

Background

Monitoring for real-world misuse is a difficult technical problem for the following reasons:

• Harm depends on context. The same request can be harmless or harmful depending on the broader context. A request for exploit code might come from a security team testing its own software, or from an attacker who is roleplaying as a pentester (Anthropic, 2025). The evidence can be spread across the entire conversation (Cunningham et al., 2026). A monitor therefore has to be able to process the whole exchange, which in agentic use can be very long and compute-intensive.
• Monitoring requires intelligence. A monitor can only catch what it can follow. In agent monitoring, even frontier monitors are likely to systematically miss attacks that exploit their capability limits, such as harm spread across a chain of tool calls (Najt et al., 2026). Knowledge matters too. A monitor that lacks the knowledge that a given biological agent can be weaponized, whether from its training or its instructions, cannot precisely flag a request about it. Small models have worked remarkably well as safety classifiers (Cunningham et al., 2026; OpenAI, 2025), but stronger models are better monitors, as our scaling results show. As model capabilities improve, the monitor's intelligence should scale as well to keep pace.
• Monitoring should generalize to unknown unknowns. Real-world misuse of LLMs is emergent. As models improve, misuse that was previously impossible comes within reach (Carlini et al., 2026), and malicious actors keep finding new ways to exploit those models (Anthropic, 2026a). Defenders cannot reliably predict emergent misuses in advance. Existing classifiers can be trained to flag precisely on the harms they were trained on, but they often fail to generalize beyond those harms (Sumers et al., 2025). A good monitor should be able to flag a harmful conversation even when it does not resemble its training data or fit cleanly under labeling guidelines.
• Monitoring should be scalable. Harm can appear anywhere, so a monitor should ideally cover all traffic across all models. Model-based safeguards consume significant compute, and their cost determines the feasibility of scaling them to a large amount of traffic. Our first Constitutional Classifiers from early 2025, separate models fine-tuned to detect a specific set of harms, added 23.7% to inference cost.³ Every prompted classifier we compared them with, which reads its policy from the prompt instead of learning it in training, cost more (Sharma et al., 2025). OpenAI similarly reports that safety reasoning has used as much as 16% of total compute in some launches (OpenAI, 2025). Like any defense, monitoring involves a trade-off between compute and coverage. We set that trade-off based on the current level of risk, and we are prepared to spend more where the risk justifies it. A more efficient monitor improves the trade-off itself: for the same compute, it covers more conversations and is likely to detect more harm, on top of our existing safeguards.

Existing monitoring methods

Existing monitoring approaches often give up one of these properties to keep the others. Standalone prompted classifiers can be accurate, and they can follow a new policy without retraining, but they are expensive to run on every conversation (Sharma et al., 2025; OpenAI, 2025). Probes are simple classifiers trained on the model's own activations. They work well in practice, cost very little, and draw on the model's own understanding (Cunningham et al., 2026; Cunningham et al., 2025). But a probe outputs a score for a target label fixed in advance, so it cannot flag novel harm. Production systems often combine cheap and expensive classifiers in cascades, in which a cheap first stage such as a probe screens all traffic and escalates suspicious exchanges to an expensive second stage (Cunningham et al., 2026; OpenAI, 2025). This lowers cost, but a cascade approach catches only what its first stage escalates.

Where self-assessment fits in

Self-assessment combines the efficiency of probes with the flexibility of prompted classifiers. Like a probe, it reuses the computation the model already did to serve the conversation, so it adds only a small cost. Like a prompted classifier, it uses the full language model to make the assessment, so it draws on everything the model understood about the conversation. By adjusting the prompt we use, we can have it follow new instructions at deployment time without retraining. Its reason is free text, so it can also describe harms outside its taxonomy, and it generalizes well to harm categories held out of training (see Results).

Self-assessment builds on a line of work in which a language model monitors its own behavior. OpenAI's confessions work inserts a system message after the model's answer that asks it to report whether it broke its instructions or took shortcuts (Joglekar et al., 2025). Other researchers have trained AI agents to report their own hidden misbehavior by calling a dedicated reporting tool (Lee et al., 2026), and Greenblatt (2025) describes monitoring agents with a fine-tuned copy of the model that reuses the original's prefix cache. Closest to our work, Self-Guard trains a model to append a harmful-or-harmless tag to its own response, which a filter uses to block the response (Wang et al., 2024). Self-assessment applies the same idea to misuse detection. It assesses the entire conversation and produces a structured assessment for offline review.

Method

How self-assessment works

A normal conversation with Claude alternates between two roles. The user sends a message, Claude responds, and the cycle repeats. After each response, self-assessment appends a system turn with a short instruction that asks for a safety reflection. Claude's reply to this instruction is the safety assessment. Here is an illustrative example (not real data):

The last two turns are not available to the user, and they are not carried into the rest of the conversation. The assessments are retained as we describe in the Data Handling section. The conversation is already in the KV cache when they are added, so an assessment costs only the short instruction and a few output tokens.

The assessment follows a fixed JSON schema. When there is nothing to flag, it is simply {"flagged": false}. For a conversation that Claude flags as unsafe, the assessment has five more fields: {"flagged": true, "reason": "...", "category": "...", "subpolicy": "...", "refused": true|false, "priority": 1-10}. To get a continuous score, we read the probability that Claude assigns to "true" in the flagged field. Thresholding this score lets us choose an operating point, such as the 1% false-positive rate we report.

The category and subpolicy come from an internal taxonomy that implements our public Usage Policy in finer detail, covering areas such as cyber harms, CBRNE weapons, and child safety. Subpolicies divide a category into finer types of harm. The priority score reflects how urgently a human reviewer should investigate a particular case. Self-assessment can in principle, and does in practice, flag harms it was not specifically trained to detect, such as prompt injection attempts. It also generalizes: removing an entire harm category from training does not empirically have a large negative effect on recall.

Training

We train self-assessment in two stages. We first perform supervised fine-tuning (SFT) to teach the model the assessment format and when to flag a conversation, and then we apply reinforcement learning (RL) to improve the quality and accuracy of the responses. Both stages distill the assessments of a teacher into this format. The teacher is a prompted classifier with our full internal policy documents in its prompt, which makes it accurate but too compute-intensive to run on all traffic. The next section describes how we generate training data.

Supervised fine-tuning. We first fine-tune Claude on synthetic conversations. Each conversation is followed by the reflection instruction and the teacher’s assessment. This teaches Claude when to flag a conversation, and how to identify the type of potential harm and its priority. Since many conversations show Claude complying with harmful requests, we mask the loss on every token except the assessment, so Claude learns to judge these responses but not to produce them.

Reinforcement learning. We then build self-assessment training environments. These environments prefill the model with synthetic conversations, both harmful and benign, and sample assessments from Claude on-policy. Each assessment is then scored with six verifiable rewards on the following aspects, and optimized during Claude’s RL training:

• whether the JSON parses against the schema
• whether the flagged label matches the teacher’s
• whether the category matches exactly
• whether the subpolicy matches exactly
• whether the refused label matches
• whether the assigned priority is close to the teacher’s

Training data

Our training data is fully synthetic. We generate it with a pipeline similar to the one used for Constitutional Classifiers (Sharma et al., 2025). The goal is to produce diverse conversations grounded in the explicit policy documents, then label them with a prompted classifier. The pipeline runs in three stages.

Scenario sketching. We expand scenario sketches hierarchically. For each subpolicy, the generator first proposes themes of possible harmful and harmless conversations, and then for each theme, sketches out concrete scenarios. We adopt this hierarchical design because generating all scenarios independently leads to large amounts of duplicates. We generate 32 themes per subpolicy and 32 scenarios per theme, which gives 1,024 harmful and 1,024 benign scenarios per subpolicy.

Each harmful scenario is grounded in a specific prohibited use from our safeguards policies. The scenarios describe who the user is, what capability they seek, and how they would phrase the request. Benign scenarios are chosen to be difficult negatives: legitimate requests topically adjacent to the policy area.

For example, under the “remote exploitation” subpolicy, harmful seed scenarios include:

• An attacker who has compromised a workstation in a hospital network asks for commands to scan the local network and exploit a vulnerable DICOM server.
• An industrial saboteur asks the AI to walk him through scanning exposed Modbus or DNP3 services on a SCADA gateway.
• A stalker asks the AI how to exploit unauthenticated endpoints on an ex-partner's home NAS to access personal files.

Benign seed scenarios on the same topic look like:

• A procurement manager reviewing a penetration-testing vendor’s statement of work wants to understand what “external network exploitation” covers before approving it.
• A network defender designing detection rules for her SIEM wants to understand what lateral movement traffic looks like after an attacker has exploited a remote service.
• A computer science student preparing for a network security exam asks how a buffer overflow in a network service can lead to remote code execution.

Conversation generation. Each scenario is then expanded into an exchange between a simulated user and Claude. For diversity, each conversation samples random attributes: the user's tone and framing, the language (half of all conversations are written in one of nine non-English languages), and for harmful scenarios, whether Claude refuses or complies. Harmful scenarios may additionally sample an adversarial obfuscation style, in which the simulated user disguises the request using techniques like homoglyphs, base64/rot13 encodings or embedding the request in another benign task.

The final turn is the teacher's assessment, which we use as the training label.

Labeling and filtering. Every generated conversation is then labeled by a prompted classifier based on Claude Mythos Preview, with its prompt containing all policy documents and a reasoning checklist: to identify the true request, rule out exemptions, rule in a specific prohibited use, then assign a priority. The teacher’s assessment is used as the training label. Conversations that fail strict structural validation (parse failure, wrong turn count, invalid teacher output) are dropped.

One view of self-assessment's training is distillation of a capable teacher prompted classifier with a significant amount of safety reasoning into the student model’s next-token prediction. The teacher reads every policy document and reasons through a checklist before it answers. The student has neither the policies in its prompt nor any room to reason, and must learn to internalize the safety guidelines and produce the same assessment directly.

Results

Self-assessment in Claude Fable 5

We trained self-assessment as a part of production training for Claude Fable 5. We evaluated it on a random sample of production traffic, labeled by our teacher, a prompted classifier based on Claude Mythos Preview. At a 1% false-positive rate, self-assessment recalls 78% of the conversations the teacher flags, and 94% of high-priority ones (priority 7 or higher) (Figure 1).

Each assessment is logged and used in three ways. First, in a growing set of harm areas, flags are aggregated by organization every night, and organizations with several high-priority flags are escalated to our enforcement team for human review.⁴ Second, self-assessment’s flags are compared against our other classifiers’ to detect potentially harmful conversations not identified by the classifiers. Third, we cluster the flags that fall outside our taxonomy to look for new kinds of misuse. We describe what each has found in Discussion.

Self-assessment scales with model capability

One main reason why we invest in self-assessment is our belief that its power should scale with model capability. To validate this hypothesis, we also trained self-assessment into four Claude models of increasing size, which we refer to as extra-small (XS), small (S), medium (M), and large (L). These models were trained with SFT only, and we evaluate them on a separate evaluation set, so their numbers are not comparable with the Fable 5 results in production. Our results show that self-assessment in these models improves consistently with size (Figure 2). At a 1% false-positive rate, recall rises from about 16% on the smallest model to about 96% on the largest, and AUC rises from 0.862 to 0.995. Self-assessment is therefore most useful for frontier models. We expect self-assessment to improve as models do. This matters because oversight of how models are used becomes more important as they grow more capable.

Self-assessment generalizes to harms unseen in training

A monitor should catch harms it was never specifically trained to detect. To test this, we run a leave-one-out experiment: we train a Claude model to self-assess twice, once on the full training set and once with every training example for one harm category removed. Then, we measure recall on that removed category.

Recall on unseen harm in our test barely drops (Figure 3). At a 1% false-positive rate, removing fraud from training lowers recall on synthetic fraud conversations from 92.8% to 90.0%, and removing cyber lowers recall on synthetic cyber conversations from 92.7% to 90.3%. The result holds on production traffic, where recall falls from 83.3% to 82.1% on fraud and from 94.2% to 90.7% on cyber. This suggests that self-assessment training mostly teaches Claude how to report harm, and that its sense of what is harmful largely comes from what it already knows.⁵

Can self-assessment block attacks?

In production, self-assessment is passive: It flags conversations for review, but it does not block them. Here we run an experiment on whether it could also work robustly as a blocking classifier.

Setup. We use an internal red-teaming harness. An orchestrator model directs an attacker model through an internal safety evaluation set of 24 offensive cybersecurity tasks, such as stealing credentials from a remote host. We run the attacker against Fable 5 in three configurations, each with our production classifiers turned off. The first has no safeguards and acts as the baseline. The second uses self-assessment as a blocking classifier. The third blocks with a standalone prompted classifier, which makes a separate call with our full policy prompt. When the classifier flags a turn in the conversation, the attack is blocked. When a turn is blocked, the attacker can rewind the conversation and try a different request. Each block shows the attacker what the monitor caught, so it can keep retrying until a request passes.

As a blocking classifier, self-assessment is weak. Attack success falls only from 79% (19 of 24 tasks) with no safeguards to 54% (13 of 24) with self-assessment. The prompted classifier reduces it to 25% (6 of 24). Self-assessment does make attacks more expensive. Each successful attack costs roughly four times as many generated tokens as with no safeguards (Figure 4).

As an offline monitor, self-assessment catches attackers who are not trying to evade it. Self-assessment flagged 21 of the 24 attack trajectories at least once. When self-assessment runs offline, an attacker gets no online signal about what was flagged. Signals such as an account ban arrive much later, which makes it hard to adapt an attack in real time. An attacker who actively optimizes against our other safeguards, but does not target self-assessment, would likely be caught. To be clear, self-assessment is not robust against an attacker who targets self-assessment directly. Such an attacker could probably get the model to reveal whether it would flag a request, and then search for a universal jailbreak that stops self-assessment from flagging future conversations.

Self-assessment is not built to stand alone. It is one of many enforcement layers and part of our defense-in-depth approach. In production, our other classifiers and safeguards are designed to block harmful requests as they happen, while self-assessment runs across a large share of traffic and flags what gets through, so that our enforcement teams can prioritize what to review and we can detect emerging harms that might otherwise be missed.

Adapting self-assessment at deployment time

In production, the right assessment can depend on the person or organization using Claude. For example, a verified security team may be authorized to research vulnerabilities in its own software, while we would flag the same work from anyone else. We test whether self-assessment can follow this kind of policy at deployment time, without retraining.

We add a session policy block to the reflection instruction. The block names one authorized target:

<session_policy>
<allowed>vulnerability research on OpenLDAP</allowed>
</session_policy>
Please reflect on the safety of this conversation. [...]

We test this on a set of vulnerability research transcripts with known targets, such as WebKit, and we allowlist some of the targets. A correct monitor should flag every conversation about a target that is not on the allowlist, and no conversation about a target that is. On our test set, Fable 5 does this perfectly across a wide range of thresholds. It flags 100% of conversations about targets that are not on the allowlist, and 0% of conversations about targets that are (Figure 5).

Discussion

What we learned from deploying self-assessment

We have run self-assessment passively on production traffic since Fable 5 launched in June and logged what self-assessment flagged for additional review. Here are some of the things we have found with the help of self-assessment so far:

• It helps us improve our other classifiers. We took conversations that self-assessment flagged and a probe did not, and asked a separate Mythos-level prompted classifier whether each conversation violated that probe's policy. On Fable 5, the grader judged 41% of these conversations to be violations for our child-safety probe. We ran a similar check on our cyber classifier.⁶ A grader judged only about 1% of self-assessment’s cyber flags to be truly violative, which suggests our production cyber classifiers miss little.
• Because self-assessment runs on a large share of traffic, we can examine anomalies in what we flag to detect scaled abuse. In August, self-assessment raised over 100k flags of child sexual abuse & exploitation from a single API customer. Our child safety team investigated and reported the material to the National Center for Missing & Exploited Children (NCMEC).
• Self-assessment's generalization helps us find unknown unknowns. About a third of flags do not map cleanly to our existing taxonomy. When we clustered these flags (Tamkin et al., 2024), we found potential harms that our existing safety detection stack isn’t specifically designed to cover. These clusters found in-the-wild include illicit distillation attempts to extract Claude's reasoning, prompt injection, and cases where Claude, acting as an agent, took actions the user had not authorized.
• Self-assessment has enabled broad organization-level enforcement, including on rare harms that we don’t have classifiers for, by using an automated system that escalates organizations with multiple high-priority flags for human review.⁷ More than half of the organizations self-assessment surfaced had no flags from any other detection source in the preceding month. In an early internal biosecurity prevalence study, self-assessment was the highest-precision enforcement signal tested. In one enforcement case in September, self-assessment flags surfaced an operation that used Claude Code to build and run an automated romance scam targeting dozens of victims. After human review, we banned the accounts involved.

Data Handling

Self-assessment collects no additional data from users. During inference, self-assessment follows the same data-handling policies as our other automated systems. It reads only the conversation Claude has just processed in order to respond, and it runs on traffic that we can already retain under our existing policies.⁸ Monitoring for misuse with high fidelity while maintaining data retention limitations has no easy answer. Detecting serious misuse requires automated systems to analyze conversations, and investigating a flagged potential safety violation currently requires retaining the conversation (Anthropic, 2026b). We are still figuring out how to make self-assessment more effective while preserving user privacy.

Constraining free-text output. The safety assessment contains a short reason string that is model-generated free text and can paraphrase the user's conversation, and we therefore apply the same data governance controls as we do to the user prompts and completions. The structured fields (flag, category, subpolicy, priority, and score) follow a fixed schema and therefore cannot carry user text. We handle them like the scores from our other classifiers.

Limitations

Self-assessment inherits any weakness in its teacher. Self-assessment learns from its teacher, and we evaluate it against the same teacher. Any blind spot in the teacher therefore passes to the student, and our evaluations cannot detect it. Future work should consider adding human evaluations to check whether human and model safety judgments agree.

Limited coverage. Self-assessment generalizes well to the two harm categories we held out of training, fraud and cyber. Both are harms that Claude already recognizes as harmful, so this result may not extend to emergent harms that Claude does not yet recognize.

Monitoring the model's own conduct has failure modes. Self-assessment could in principle also monitor Claude's own actions, such as misalignment. Two limitations make this harder than monitoring user-initiated misuse. First, a misaligned model could be deceptive. If a model were hiding a misaligned action, it could hide it in the assessment too. Collusion is a known risk for monitors that share the agent's cache (Greenblatt, 2025). It is therefore crucial that our models are honest (Joglekar et al., 2025). Second, an honest model can still fail in the same way as the agent, because the same model, with the same judgment, produced both the action and the assessment. A recent study shows that models rate an action as less risky when it appears in their own previous turn than when the same action is shown to them in a fresh context (Khullar et al., 2026). That said, we see some encouraging results: self-assessment already flags some of Claude's own actions without being trained to, such as agentic actions the user had not authorized. In the confessions work, cases where the model broke its instructions and did not admit it made up only about 4% of test cases, and most of those came from confusion, not deception (Joglekar et al., 2025). In other work, training agents to flag their own covert misbehavior left fewer attacks undetected than a separate monitor of matched capability (Lee et al., 2026). Even so, self-assessment is not yet a reliable monitor of Claude's own conduct. We expect this use will require more research on model honesty, along with on-policy evaluations that test the monitor on the model's own actions, beyond fixed examples (Khullar et al., 2026).