Moderate AI-Generated Content: Checking LLM Output

You ship a chatbot, a copilot, or an agent that writes text straight into a product surface. The model's reply is untrusted text. It reaches a user's screen and nobody reads it first.

You can moderate AI-generated content the same way you moderate user posts: send the text to POST /moderation, get JSON back, and decide what the product does next. The response contains twelve boolean category flags and one overall flagged value. That is the whole contract.

What the response does not contain matters just as much. There are no scores, no probabilities and no confidence values. There are no thresholds to tune. Every policy decision — block, regenerate, route to review, log — lives in your application as ordinary code. That moves LLM output moderation from a calibration project into a routing problem, which is a much easier thing to review, test and explain to the rest of the company.

If you have not wired up POST /moderation yet, start with add a content moderation API in an afternoon for the endpoint, authentication and request schema, then come back here to design the policy layer on top of it.

Why model output needs the same check as user content

A generated reply is text. It can carry the same kinds of harm as anything a user types: toxicity, hate, harassment, self-harm, adult content, violence, drugs, weapons, PII, spam, minor safety and profanity. A model that has been aligned to avoid these topics still produces them sometimes, and a model that has not been aligned produces them whenever the prompt nudges it.

In a chat product, both directions matter. The user's prompt and the assistant's reply are both strings you can send to the same endpoint. You do not need a second vendor, a second SDK or a second contract for the outbound side.

The integration shape is deliberately small. It is a text-in, JSON-out HTTP call. If your stack can make a request to https://api.cautionlabs.com/moderation, it can moderate a model reply — Node, Python, Go, a Lambda, a queue worker, whatever already sits between your model call and your client.

One constraint shapes the design: every check is its own request. The API documents no batch endpoint and no webhooks, and no rate-limit headers to plan around. You call it inline, or you call it from a worker that already has the text in hand. Plan for one request per check and the rest of the architecture stays simple.

Where the moderation call goes in an AI request path

A chat turn has two text boundaries worth guarding. They use the same endpoint but different policy.

Checking the incoming prompt

The prompt is the cheapest place to stop a bad turn. Screen the user's text before it reaches the model, and you avoid paying for a generation that you would only throw away.

For most products the prompt check can be lighter than the reply check. A prompt that trips spam or profanity might be a warning; a prompt that trips self_harm might route to a different response path entirely. Those are your decisions. The API returns the flags and nothing else.

Checking the generated reply before it renders

The reply check is the one that protects the user. Send the assembled reply text before it is written to the thread, rendered in the UI, or emitted over your streaming channel.

Here is the call in full. One header, one field:

curl -X POST https://api.cautionlabs.com/moderation \
  -H "Authorization: Bearer $CAUTIONLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "Here is the reply the model just generated."}'

And the shape of what comes back:

{
  "categories": {
    "toxic": false,
    "profanity": false,
    "hate": false,
    "harassment": false,
    "self_harm": false,
    "adult": false,
    "violence": false,
    "drugs": false,
    "weapons": false,
    "pii": false,
    "spam": false,
    "minor": false
  },
  "flagged": false
}

Now the application branch. This snippet checks the reply, gates rendering on flagged, and maps each category to an action your team chose. The comment on the policy object is the point of the whole article:

// The API reports flags. Everything below is your application's decision.
const POLICY = {
  block: ['hate', 'harassment', 'self_harm', 'adult', 'minor'],
  review: ['toxic', 'violence', 'drugs', 'weapons', 'pii'],
  log: ['profanity', 'spam'],
};

async function checkReply(replyText) {
  const res = await fetch('https://api.cautionlabs.com/moderation', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.CAUTIONLABS_API_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ text: replyText }),
  });

  if (!res.ok) {
    // Your application picks the fallback: hold the reply, show a generic
    // message, or surface the failure. The API does not decide this.
    return { action: 'hold', reason: 'moderation_call_failed' };
  }

  const { categories, flagged } = await res.json();
  if (!flagged) return { action: 'send' };

  const matching = (list) => list.filter((c) => categories[c] === true);

  const blocked = matching(POLICY.block);
  if (blocked.length > 0) {
    return { action: 'replace_with_fallback', categories: blocked };
  }

  const reviewed = matching(POLICY.review);
  if (reviewed.length > 0) {
    return { action: 'queue_for_review', categories: reviewed };
  }

  return { action: 'send_and_log', categories: matching(POLICY.log) };
}

Call it between generation and rendering:

const result = await checkReply(completion);
if (result.action === 'send' || result.action === 'send_and_log') {
  render(completion);
} else {
  renderFallback(); // your product's fallback path
}

Note what the code does not do. It does not compare a number against a cutoff, because there is no number. It reads booleans and runs a switch.

What the response actually contains

The response object has exactly two required fields: categories and flagged.

Twelve category booleans

categories is an object with twelve required boolean properties, named exactly:

• toxic
• profanity
• hate
• harassment
• self_harm
• adult
• violence
• drugs
• weapons
• pii
• spam
• minor

Each is true or false. There is nothing else in the object — no per-category score, no explanation string, no span offsets.

That list is fixed. There are no extra categories to enable and no custom categories to define. Your policy layer maps each documented name to an action and leaves the rest alone.

The overall flagged value

flagged is a boolean that is true when any category is flagged. It is the single value you branch on when you want one simple gate in front of rendering.

Two uses, in practice:

• Gate on flagged first. If it is false, send the reply. This is the fast path and it covers the overwhelming majority of generations in a healthy product.
• Read categories second. Once flagged is true, the individual booleans tell you which rule fired, and that determines whether you block, rewrite, regenerate or queue for review.

Neither value tells you how confident the model was. There is no score, no probability distribution and no confidence number anywhere in the response. That is why there is nothing to tune: no cutoff to move up or down when reviewers complain, no calibration job, no per-locale threshold drift. You change behavior by changing code, and the change shows up in a diff review like any other.

Designing behavior around boolean flags

Because the output is binary, all of the judgment sits in your application. That is more work than flipping a threshold and less work than maintaining one.

Per-category handling

A single blanket action for every flag is easy to write and hard to defend. Users notice when a harmless profanity flag triggers the same response as a harassment flag.

Map categories to actions instead. A reasonable starting split:

• Block and replace. Categories where the product should never display the text at all. Swap in a canned fallback or attempt a safe regeneration.
• Hold for review. Categories where a human should look before the reply goes out or before the conversation continues.
• Send and log. Categories where display is acceptable but the event should land in your trust-and-safety pipeline for trend analysis.
• Ignore. Some flags may be acceptable on some surfaces — a weapons flag is noise in a cookbook assistant and serious in a teen-facing chat product. Decide this per product, not per company.

Write the mapping down as a constant, as in the snippet above. It becomes the artifact your trust-and-safety reviewers, legal team and support leads can all argue about, which is exactly where that argument should happen.

When to log, review or regenerate

A few patterns that hold up in production AI chatbot content moderation:

• Regenerate once, then fall back. If flagged is true, retry the generation with a stricter system prompt or different sampling settings, then check again. If the second attempt is also flagged, show the fallback. Two calls on a bad turn is a cheap price for a reply the user actually wants.
• Log the flags, not the text, when you can. The categories object is enough to build dashboards and alerts without keeping prompt or reply content around.
• Never let a flag silently vanish. Every true should produce a log line, a counter increment, or an action. Flags that are returned and discarded are worse than no check, because they create the appearance of coverage.
• Keep the fallback human. A blocked reply should read as a product decision, not an error. "I can't help with that — here's what I can do instead" beats a red banner.

The API itself does none of this. It returns flags. Your code decides.

How to moderate AI-generated content across long prompts and replies

Length is where request accounting gets real, so here is the documented behavior in full.

A request covers up to 3,000 characters. Texts of up to 10,000 characters are checked in 3,000-character pieces, one request each, and each started piece counts as one request.

Two practical consequences for AI pipelines:

• Assemble or truncate streaming output before you send it. A token stream is not a request body. Buffer the reply until it is complete, or cut it at a known boundary, then moderate the resulting string. Checking half a sentence and then appending more text to the screen defeats the purpose.
• Budget by piece, not by turn. A long, document-grounded reply may span several pieces. Cloud usage is $0.40 USD per 1,000 requests, so piece count is the number that matters in your cost model, not conversation count.

You can see the current usage numbers on the pricing page.

Privacy: cloud API versus on-premises for AI pipelines

Model output can contain whatever the user just pasted in — support tickets, medical questions, internal documents. Where the moderation call runs is a privacy decision, not just an infrastructure one.

Cloud API

The cloud API is a hosted service at https://api.cautionlabs.com, authenticated with a Cloud Key. Per the cloud terms, your text is not stored by the hosted API. The API reference documents authentication, the request and response schema, and the error codes.

On-premises model

Some teams cannot send prompts or model output to a hosted service at all. For those, the same model runs in a container on your own infrastructure. The container makes no outbound requests and has no telemetry. You call it at POST /moderate on your own network, default http://localhost:8000, with the same {"text": "..."} body.

A private (on-premises) key does not work with the cloud API. Keep the two key types separate in your configuration and your secret stores so nobody swaps them by accident.

On-premises is a one-time payment of $299 USD per model, with unlimited installs and requests for that model. The first model is Argus 1. If you want the container running before you refactor your pipeline, deploy on-premises content moderation with Docker walks through the registry login, the image and the local endpoint. The on-premises guide covers requirements, GPU cloud deployment and air-gapped installs.

FAQ

Does the API return confidence scores?

No. The response contains twelve boolean category flags and one overall flagged boolean. There are no scores, probabilities or confidence values, and therefore no thresholds to tune.

Can I send a batch of replies in one call?

No. There is no batch endpoint. Each text is its own POST /moderation request, and in a chat application the prompt and the reply are two separate calls.

Does it detect personal data?

pii is one of the twelve documented category flags, and it returns true or false like the others. The response does not return matched spans or the kind of data found, so your application decides what to do when it is true.

Do I need a separate call for the prompt and the reply?

Yes. They are two distinct texts and each is its own request. You can apply different policy to each, which is usually what you want: user input and model output carry different risk.

What happens to text longer than one piece?

Texts of up to 10,000 characters are checked in 3,000-character pieces, one request each, and each started piece counts as one request.

Add the check to your response path

The smallest useful implementation is three things: one call on the assistant's reply, one branch on flagged, and one policy rule for each of the twelve categories. That is enough to stop the reply you would be embarrassed to ship, and it leaves a clear place to add prompt-side checks, regeneration and review queues later.

Start in the API reference for authentication and the full schema, try a sample in the playground, and read the on-premises guide if your data cannot leave your network.

Then send your assistant's next reply to POST /moderation, branch on the overall flagged value, and map one policy rule to each of the twelve category flags. The flags are the API's job. The judgment is yours.