Content Moderation API: Add Moderation in an Afternoon

What a Content Moderation API Actually Returns

Adding a content moderation API to an existing product is a one-afternoon job if you know two things before you write any code: exactly what the endpoint returns, and who owns the policy decision. With CautionLabs, the endpoint returns flags. Your application decides what those flags mean.

That split is the whole design. A moderation model that hands you a float forces you to invent a cutoff, defend it in code review, and revisit it whenever the model changes. A model that hands you booleans moves the policy conversation out of the model and into your application, where your product context actually lives.

Booleans, not confidence scores

Every successful response from the CautionLabs cloud API contains thirteen booleans: twelve per-category flags and one overall result.

• Twelve keys inside a categories object, each true or false.
• One top-level flagged boolean that is true when any category is flagged.

There are no scores, no probabilities, and no confidence values anywhere in the response. There are no thresholds to tune either, because there is nothing numeric to compare against a cutoff. If you were planning a pipeline built around score > 0.7, that branch does not exist here.

The upside is that deterministic booleans are trivial to test. A fixture is a string and an expected object. A regression is a diff. There is no floating-point tolerance to argue about, and nothing to re-tune when the model behind the API is updated.

The twelve categories and the overall flagged value

The categories object is always present and always contains the same twelve keys:

• toxic
• profanity
• hate
• harassment
• self_harm
• adult
• violence
• drugs
• weapons
• pii
• spam
• minor

Treat the key names as a stable contract. They are snake_case, they are safe to switch on, and they arrive in every response regardless of what you sent — a clean message comes back with twelve false values, not a shortened object.

The flagged boolean is a convenience for the common case where every category leads to the same outcome. It is true when any category is flagged, so a product that rejects everything suspicious can branch on flagged alone and ignore the per-category detail. A product that treats categories differently — quarantining something with a pii flag while quietly dropping spam — reads the individual keys.

That is the entire response surface. Everything below is about wiring it up.

Your First Content Moderation API Request in Five Minutes

Authentication with a cloud API key

The cloud API is served from https://api.cautionlabs.com. Authentication is bearer token: put a cloud API key from the dashboard in the Authorization header of every request.

A private on-premises key does not work with the cloud API. If you are running the model on your own infrastructure, that key belongs to the local container, not to the hosted endpoint.

The POST /moderation request and response

There is one endpoint: POST /moderation. The request body is JSON with a single required field.

• text — the string to check. Required. Minimum 1 character, maximum 10,000.

That is all you send.

curl https://api.cautionlabs.com/moderation \
  -H "Authorization: Bearer $CAUTIONLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"text": "the message you want to check"}'

The same call in Node.js:

const res = await fetch("https://api.cautionlabs.com/moderation", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.CAUTIONLABS_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ text: userMessage }),
});

const data = await res.json();

The success payload has exactly two keys:

{
  "categories": {
    "toxic": true,
    "profanity": false,
    "hate": false,
    "harassment": true,
    "self_harm": false,
    "adult": false,
    "violence": false,
    "drugs": false,
    "weapons": false,
    "pii": false,
    "spam": false,
    "minor": false
  },
  "flagged": true
}

What matters for integration is that the keys and types never change: categories is always an object of twelve booleans, and flagged is always a boolean. Send your own text through the playground to see the exact values for a specific string.

Turning Flags Into Product Decisions

Block, review or allow

Three outcomes cover most products: publish, hold for a human, or drop. Write the branch in the order that matches your risk appetite, and be explicit about it.

const { categories, flagged } = data;

if (categories.spam) {
  return drop("spam");            // never reaches a human
}

if (categories.pii) {
  return quarantine("pii");       // hidden pending review
}

if (flagged) {
  return queueForReview();        // anything else that tripped
}

return publish();

Order matters. Checking the categories you handle differently before falling back to flagged keeps the fast path fast — a clean message costs one boolean check, and a message that trips four categories still resolves to a single decision.

If your application only cares about one signal — spam in a marketplace, harassment in a comments thread — you do not need to read the other eleven keys. Branch on the one you care about and ignore the rest. The per-category solutions pages describe the individual detection surfaces, including toxicity detection and spam detection if those are your only concerns.

What to log and store

Log the flags next to the metadata you already keep. A useful row is small:

• your content id and user id
• a timestamp
• the twelve category booleans, or the subset you act on
• the decision you made (publish, queue, drop)

The API returns flags, not a narrative. It will not tell you why a text was flagged, and the cloud terms state that your text is not stored. That means the flags you log are the only record of the moderation decision, which is exactly why you should log them at the moment you make the call rather than reconstructing the state later from your own queue tables.

Two practical notes. First, log the raw booleans rather than only your derived decision where you can afford the space — it lets you re-run policy retroactively over historical traffic without calling the API again. Second, do not invent a severity score by counting how many categories are true. The API does not rank categories, and your logs should not pretend it did.

Sizing, Requests and Cost

The 3,000-character request unit

text accepts 1 to 10,000 characters. A single request covers up to 3,000 characters, and a text is checked in 3,000-character pieces, with each started piece counting as one request.

The practical consequence: the number that matters for your budget is pieces, not messages. If you accept long-form content — articles, support tickets, imported documents — measure your average text length before you estimate volume, because your request count is driven by length, not by the number of users who hit submit.

Moderating many texts without a batch endpoint

There is no batch endpoint. You cannot post an array of texts and get an array back. The API also sends no webhooks, so results are not delivered asynchronously — every result arrives in the response to the request that asked for it.

That means fan-out belongs in your code, inside the request/response cycle you already have. A concurrency-limited loop is enough:

async function moderateAll(texts, limit = 8) {
  const queue = [...texts];
  const results = [];

  async function worker() {
    while (queue.length) {
      const text = queue.shift();
      const res = await fetch("https://api.cautionlabs.com/moderation", {
        method: "POST",
        headers: {
          Authorization: `Bearer ${process.env.CAUTIONLABS_API_KEY}`,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ text }),
      });
      results.push({ text, data: await res.json() });
    }
  }

  await Promise.all(Array.from({ length: limit }, worker));
  return results;
}

Keep limit modest and tunable, and pair each result with its input text as shown — completions arrive out of order and you will need the mapping. If a batch is large enough that it would blow past your own request timeout, move it to a background job and write results into your own queue. The API will not do that for you.

Cloud pricing is straightforward: $0.40 USD per 1,000 requests, usage-based, with no cost dimension other than request count. The full breakdown lives on the pricing page.

One thing the documentation does not publish: rate limits. There are no documented rate limits, no 429 responses, and no rate-limit headers to read. Design your concurrency for your own infrastructure's comfort rather than for a published number.

Errors You Should Handle on Day One

Failures return JSON with two fields: a code from a fixed set, and a human-readable message. Branch on code, log the message alongside it, and never show the raw message to an end user.

The five codes and their statuses:

• invalid_request — 400. The body was malformed or text was missing, empty, or outside the accepted range. This is a bug in your caller, not a runtime condition to retry. Alert on it.
• unauthorized — 401. The credential was missing or not accepted. Check that the header is present and the key is current.
• insufficient_credits — 402. The account is out of credits. Surface this to whoever owns billing, and consider a circuit breaker so you fail closed — holding content — rather than letting flagged text through because the moderation call could not run.
• wrong_key_type — 403. Worth its own branch. A wrong key type has a dedicated code rather than being folded into a generic 401, which makes it diagnosable: the cloud API expects a cloud API key from the dashboard, and an on-premises private key does not work with it.
• internal_error — 500. The request failed on the server side. This is the one case where a bounded retry with backoff is reasonable, ideally with jitter and a cap.

A small handler keeps this tidy:

if (!res.ok) {
  const { code, message } = await res.json();

  switch (code) {
    case "internal_error":
      return retryWithBackoff();
    case "insufficient_credits":
      return failClosed("moderation_unavailable");
    default:
      logger.error({ code, message });
      return failClosed("moderation_error");
  }
}

Decide what "fail closed" means for your product before you need it. For a comments section it might mean holding the comment. For a draft-creation flow it might mean blocking submit with a retryable message.

Cloud API or On-Premises Model?

The same moderation model is available two ways, and the choice is mostly about your constraints rather than the output format — both return flags.

Hosted REST endpoint

The cloud API is the default: no infrastructure, bearer auth with a cloud API key, POST /moderation at https://api.cautionlabs.com, and usage-based pricing at $0.40 USD per 1,000 requests. Choose it when you want to ship this afternoon.

Running the container on your own infrastructure

The on-premises option runs on your own hardware. The first model is Argus 1, model id argus_1, and the setup is four steps:

1. Buy a model.
2. Create a private key in the dashboard.
3. Log in to the CautionLabs registry with that key as the password (docker login registery.cautionlabs.com) and run the image. The registry checks the license on every download.
4. Call POST /moderate on your own network — the default is http://localhost:8000, with a body of {"text": "..."}.

Two properties make this viable for regulated and offline environments: the model container makes no outbound requests and has no telemetry. Nothing about your text leaves the machine it runs on.

Licensing is a one-time payment of $299 USD per model, with unlimited installs and unlimited requests, and upgrades to that model included.

A short checklist for the decision:

• Choose on-premises if you need data residency, an air-gapped deployment, or a hard guarantee that text never leaves your network.
• Choose the cloud API if you would rather not run GPUs, manage images, or own upgrades.
• If both are viable, start with the cloud API and treat the on-premises path as a migration you can make later — the flags and the branch logic on your side stay the same.

Full requirements and deployment details, including GPU cloud options, are in the on-premises guide.

Frequently Asked Questions

Does the API return confidence scores? No. A response contains twelve category booleans and one overall flagged boolean. There are no scores, probabilities, or confidence values, and nothing numeric to threshold.

Is there a batch endpoint or webhooks? No. POST /moderation accepts one text per request, and the API sends no webhooks. Fan-out and result handling happen in your application.

Are there thresholds I can tune? No. There are no thresholds anywhere in the API. The model returns booleans, and your application owns every policy decision built on top of them.

Can I use my on-premises private key with the cloud API? No. A private on-premises key does not work with the cloud API. Use a cloud API key from the dashboard for https://api.cautionlabs.com.

How is long text handled? text accepts 1 to 10,000 characters. One request covers up to 3,000 characters, and each started 3,000-character piece counts as one request.

Does the documentation publish rate limits? No. The documentation does not describe rate limits, 429 responses, or rate-limit headers.

Try It Before You Integrate

You can validate the whole integration before you write a line of application code. Open the playground, paste a piece of real user text, and look at the twelve flags and the flagged value that come back. That is the exact payload your code will handle.

Then keep the API reference open beside you, create a cloud API key in the dashboard, and make one live POST /moderation request from your own terminal with the curl command above. Once you have seen the response for your own text, the rest is the branch logic you already wrote in this article.