CONTROL: Enterprise Content Moderation Pipeline

examples/30-control-moderation.halo policies |! people

A hardened moderation pipeline with multiple defense layers, audit trails, escalating human review, and budget-safe AI model selection. Shows how Halo handles compliance.

Open in the playground →

// ============================================================
// CONTROL: Enterprise Content Moderation Pipeline
// ============================================================
// A hardened moderation pipeline with multiple defense layers,
// audit trails, escalating human review, and budget-safe
// AI model selection. Shows how Halo handles compliance.
// ============================================================

// --- GUARDRAILS: Hard boundaries ---
policy PIIProtection   : Redact(SSN) & Redact(CreditCard) & Redact(HealthData)
policy OutputSafety    : Block(HarmfulContent) & Block(HateSpeech) & Block(SexualContent)
policy CostControl     : MaxSpend(2.00)

// --- TOOLS ---
tool ModerationAPI  : POST("api.openai.com/v1/moderations")
tool AuditLogger    : POST("api.example.com/audit", apiKey: Env.AUDIT_KEY)
tool EscalationQueue: POST("api.jira.com/rest/issue")

// --- MEMORY ---
tool PolicyRules    : VectorStore(Env.PINECONE_KEY, namespace: "content-policy-v3")
tool PastCases      : VectorStore(Env.PINECONE_KEY, namespace: "moderation-history")

// --- MINDS ---
// The triage agent is cheap — handles bulk content fast
agent TriageAgent [openai/gpt-4o-mini, temp: 0.0]
  system """
    You are a content triage specialist. Rapidly screen
    submissions for obvious policy violations. Classify as
    safe, borderline, or blocked. Optimize for throughput
    — most content should auto-clear in under 500ms.
  """
  requires CostControl

// The deep-review agent handles edge cases with precision
@timeout(60s)
agent ReviewAgent [google/gemini-flash, temp: 0.1]
  system """
    You are a nuanced content review agent. Handle edge cases
    that triage couldn't auto-resolve. Consider cultural
    context, satire, and intent. Report risk level with
    detailed reasoning for borderline decisions.
  """

// The policy expert knows regulations inside-out
agent PolicyExpert [anthropic/claude-sonnet-4, temp: 0.2]
  system """
    You are a regulatory compliance specialist. Verify content
    against platform policies and applicable regulations
    (GDPR, COPPA, DSA). Cite specific policy clauses in
    every decision. Never approve content you're uncertain about.
  """
  equips PolicyRules

// --- The Pipelines ---
// Every submission gets exactly one outcome: the branch (|? { ... })
// sends it down the first route whose condition holds.
@timeout(5m)
@retry(3, 10s)
pipeline ModerateContent(submission: ContentSubmission) -> Decision:
  submission
    // Layer 1: Quick triage scores every submission cheaply
    |> TriageAgent.screen(
        text: context.data.submission.body,
        user_flags: context.data.submission.user_flags
      ) as screen

    // Layer 2: Route it. Clearly fine content publishes, clearly toxic
    // content is removed, and everything in between gets a deep review.
    |? {
        (context.data.screen.toxicity_score < 0.3) => System.Publish(context.data.submission),
        (context.data.screen.toxicity_score > 0.9) => System.Notify("removed", reason: context.data.screen.reason),
        else => DeepReview(context.data.submission),
      } as outcome

    // Layer 3: Log every decision to the audit trail
    |> AuditLogger.record(
        event: "moderation_decision",
        screen: context.data.screen,
        outcome: context.data.outcome
      )
    |> System.Print(context.data.outcome)

// Ambiguous content: a careful model reviews it against policy.
@timeout(2m)
pipeline DeepReview(submission: ContentSubmission) -> Decision:
  submission
    |> ReviewAgent.evaluate(
        content: context.data.submission.body,
        policy: PolicyRules,
        history: PastCases
      ) as review

    // High risk goes to a human; the rest gets a final policy check
    |? {
        (context.data.review.risk_level == "high") => HumanReview(context.data.submission, context.data.review),
        else => PolicyExpert.final_review(content: context.data.submission.body, review: context.data.review),
      }

// High-risk content: a human moderator makes the call.
pipeline HumanReview(submission: ContentSubmission, review: Review) -> Decision:
  review
    |! yield(
        queue: EscalationQueue,
        context: context.data.review,
        timeout: 1h
      )
    ~> listen(EscalationQueue.Reaction) as human
    |? {
        (context.data.human.action == "approve") => PolicyExpert.final_review(
            content: context.data.submission.body,
            review: context.data.review,
            moderator: context.data.human
          ),
        else => System.Notify("removed by moderator", review: context.data.review),
      }