Use Jev as a CI/CD AI quality gate

Q: Why separate evidence from the CI gate decision?

An LLM or harness can score quality, tools, latency, and safety. That is evidence, not a merge policy. A separate typed Choice (pass / retry / review / block) plus Score and Noul lets code fail closed, retry flakiness, and hold for humans—without parsing "looks good enough" prose in the pipeline.

Q: How should CI enforce PASS / RETRY / REVIEW / BLOCK?

Auto-PASS only on high-confidence pass with strong evidence and needs_human false. High-confidence retry re-runs within a budget. Mid band or review: hold the job and notify a human. Low confidence, block, or safety fail: fail the pipeline and escalate.

Q: Can I bring my own API key?

Yes. Try the free demo above without signing in. To run it live on your own text, sign in with Google and connect a Flatkey or OpenRouter key (a quick test call runs when you save it). Manage or delete it anytime under Dashboard → BYOK (https://www.jev-llm.com/dashboard/). Keys are stored encrypted on the server, used only for runs you trigger, and never sent back to the browser. Need a Flatkey key? Get one for Jev 1.13 on Flatkey (https://flatkey.ai/models/typesafe~2Fjev-1.13).

Q: Is this the official TypeSafe Jev product page?

No. jev-llm.com is an independent research and practice site. Verify pricing, models and terms at the original TypeSafe sources.

Try it

Copy the code ↓

Data source
Decision schema
Model inference
Structured result

1 Data source CI evidence block · editable

2 Decision schema 3 typed questions

decisionchoicepassretryreviewblock
evidence_strengthscorehighmediumlow
needs_humannoulprobability 0–1

3 Model inference typesafe/jev-1.13

Waiting for a run.

4 Structured result

Run the demo to see the JSON answers, the highlighted pick and the final action.

Call Jev yourselfSame data and schema, straight to Flatkey. Set FLATKEY_API_KEY and run.

curl -sS https://router.flatkey.ai/api/alpha/decisions \
  -H "Authorization: Bearer $FLATKEY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "typesafe/jev-1.13",
  "state": "CI job: agent-eval-suite #4821 (PR #910)\nEvidence from LLM + harness (not a decision):\n- response_quality: 0.82\n- tool_success_rate: 0.94\n- p95_latency_s: 1.8\n- safety_check: pass\n- flaky_retries_so_far: 0\n\nPolicy:\n- PASS if quality≥0.80, tools≥0.90, latency≤2.0s, safety pass\n- RETRY on transient tool/latency flakes when quality still ok\n- REVIEW when metrics are borderline or conflict\n- BLOCK on safety fail or severe quality drop\n\nTask: typed CI gate — pass / retry / review / block (+ evidence strength, needs_human).\nNote: the LLM produced evidence; apply judgment separately.",
  "questions": {
    "decision": {
      "type": "choice",
      "instructions": "Decide the CI/CD gate for this AI test or eval run from the evidence only. Do not ask whether the run is \"good enough\" in free text—pick a typed action the pipeline can enforce.",
      "criteria": {
        "pass": "Evidence meets the policy bar (quality, tools, latency, safety); safe to proceed or merge",
        "retry": "Likely flaky or recoverable (timeouts, transient tool failures); re-run before human review",
        "review": "Ambiguous or borderline evidence; a human should inspect before merge",
        "block": "Clear policy miss—safety fail, severe quality drop, or hard SLA breach; fail the job"
      }
    },
    "evidence_strength": {
      "type": "score",
      "instructions": "How strong and consistent is the overall evidence for the gate you chose, ignoring the agent'\''s own confidence wording.",
      "criteria": [
        "high",
        "medium",
        "low"
      ]
    },
    "needs_human": {
      "type": "noul",
      "instructions": "A human should review this CI gate before the pipeline allows merge or production promote."
    }
  }
}'

// Node.js 18+ (built-in fetch). Run: FLATKEY_API_KEY=... node jev.mjs
const body = {
  "model": "typesafe/jev-1.13",
  "state": "CI job: agent-eval-suite #4821 (PR #910)\nEvidence from LLM + harness (not a decision):\n- response_quality: 0.82\n- tool_success_rate: 0.94\n- p95_latency_s: 1.8\n- safety_check: pass\n- flaky_retries_so_far: 0\n\nPolicy:\n- PASS if quality≥0.80, tools≥0.90, latency≤2.0s, safety pass\n- RETRY on transient tool/latency flakes when quality still ok\n- REVIEW when metrics are borderline or conflict\n- BLOCK on safety fail or severe quality drop\n\nTask: typed CI gate — pass / retry / review / block (+ evidence strength, needs_human).\nNote: the LLM produced evidence; apply judgment separately.",
  "questions": {
    "decision": {
      "type": "choice",
      "instructions": "Decide the CI/CD gate for this AI test or eval run from the evidence only. Do not ask whether the run is \"good enough\" in free text—pick a typed action the pipeline can enforce.",
      "criteria": {
        "pass": "Evidence meets the policy bar (quality, tools, latency, safety); safe to proceed or merge",
        "retry": "Likely flaky or recoverable (timeouts, transient tool failures); re-run before human review",
        "review": "Ambiguous or borderline evidence; a human should inspect before merge",
        "block": "Clear policy miss—safety fail, severe quality drop, or hard SLA breach; fail the job"
      }
    },
    "evidence_strength": {
      "type": "score",
      "instructions": "How strong and consistent is the overall evidence for the gate you chose, ignoring the agent's own confidence wording.",
      "criteria": [
        "high",
        "medium",
        "low"
      ]
    },
    "needs_human": {
      "type": "noul",
      "instructions": "A human should review this CI gate before the pipeline allows merge or production promote."
    }
  }
};

const res = await fetch("https://router.flatkey.ai/api/alpha/decisions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.FLATKEY_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify(body),
});
if (!res.ok) throw new Error(`Jev request failed: ${res.status} ${await res.text()}`);
const { answers, usage } = await res.json();
console.log(JSON.stringify(answers, null, 2), usage);

# Python 3.8+ (pip install requests). Run: FLATKEY_API_KEY=... python jev.py
import json, os, requests

body = json.loads(r"""{
  "model": "typesafe/jev-1.13",
  "state": "CI job: agent-eval-suite #4821 (PR #910)\nEvidence from LLM + harness (not a decision):\n- response_quality: 0.82\n- tool_success_rate: 0.94\n- p95_latency_s: 1.8\n- safety_check: pass\n- flaky_retries_so_far: 0\n\nPolicy:\n- PASS if quality≥0.80, tools≥0.90, latency≤2.0s, safety pass\n- RETRY on transient tool/latency flakes when quality still ok\n- REVIEW when metrics are borderline or conflict\n- BLOCK on safety fail or severe quality drop\n\nTask: typed CI gate — pass / retry / review / block (+ evidence strength, needs_human).\nNote: the LLM produced evidence; apply judgment separately.",
  "questions": {
    "decision": {
      "type": "choice",
      "instructions": "Decide the CI/CD gate for this AI test or eval run from the evidence only. Do not ask whether the run is \"good enough\" in free text—pick a typed action the pipeline can enforce.",
      "criteria": {
        "pass": "Evidence meets the policy bar (quality, tools, latency, safety); safe to proceed or merge",
        "retry": "Likely flaky or recoverable (timeouts, transient tool failures); re-run before human review",
        "review": "Ambiguous or borderline evidence; a human should inspect before merge",
        "block": "Clear policy miss—safety fail, severe quality drop, or hard SLA breach; fail the job"
      }
    },
    "evidence_strength": {
      "type": "score",
      "instructions": "How strong and consistent is the overall evidence for the gate you chose, ignoring the agent's own confidence wording.",
      "criteria": [
        "high",
        "medium",
        "low"
      ]
    },
    "needs_human": {
      "type": "noul",
      "instructions": "A human should review this CI gate before the pipeline allows merge or production promote."
    }
  }
}""")

res = requests.post(
    "https://router.flatkey.ai/api/alpha/decisions",
    headers={"Authorization": f"Bearer {os.environ['FLATKEY_API_KEY']}"},
    json=body,
    timeout=60,
)
res.raise_for_status()
data = res.json()
print(json.dumps(data["answers"], indent=2), data.get("usage"))

No Flatkey key yet? Get one for Jev 1.13 →

About this use case

Use Jev as a CI/CD AI quality gate: incoming state, Jev decision, routed outcome

Who / when

Audience

CI/CD owners who need typed gates on AI-generated test or eval runs
QA and platform teams shipping LLM features behind merge policies
builders who want evidence in, decision out—without asking an LLM "is this good enough?"

Triggers

an AI eval or agent test run needs a merge-blocking gate, not a free-form summary
quality, tool success, latency, and safety scores should become PASS / RETRY / REVIEW / BLOCK
the LLM should produce evidence while a separate decision model applies judgment
auto-PASS only when confidence is high and needs_human is false

The workflow

01
Collect evidenceQuality, tools, latency, safetyInput
02
Score strengthhigh / medium / lowJev
03
Gate decisionpass / retry / review / blockJev
04
Enforce in CIMerge, retry, hold, or failYour code

Question shape

{
  "decision": {
    "type": "choice",
    "instructions": "Decide the CI/CD gate for this AI test or eval run from the evidence only. Do not ask whether the run is \"good enough\" in free text—pick a typed action the pipeline can enforce.",
    "criteria": {
      "pass": "Evidence meets the policy bar (quality, tools, latency, safety); safe to proceed or merge",
      "retry": "Likely flaky or recoverable (timeouts, transient tool failures); re-run before human review",
      "review": "Ambiguous or borderline evidence; a human should inspect before merge",
      "block": "Clear policy miss—safety fail, severe quality drop, or hard SLA breach; fail the job"
    }
  },
  "evidence_strength": {
    "type": "score",
    "instructions": "How strong and consistent is the overall evidence for the gate you chose, ignoring the agent's own confidence wording.",
    "criteria": [
      "high",
      "medium",
      "low"
    ]
  },
  "needs_human": {
    "type": "noul",
    "instructions": "A human should review this CI gate before the pipeline allows merge or production promote."
  }
}

Gate policy

≥ 0.90

Auto-PASS only when decision is pass, evidence_strength is high, and needs_human is false. Auto-retry when decision is retry at high confidence.

0.65–0.90

Hold for human review; treat as review even if the top choice is pass. Do not merge yet.

< 0.65

Block the job; escalate when decision is block, evidence_strength is low, or needs_human is true.

Failure modes

Pre-launch checks before you trust auto-routing:

Asking an LLM "is this test run good enough?" and parsing free-form prose in CI;
Letting the same model both invent metrics and decide the merge gate;
Auto-PASS on mid confidence because mean quality looks fine;
Skipping BLOCK on a safety fail because latency and tools looked green;
Retrying forever without escalating ambiguous REVIEW cases to a human;

Keep exact work in code: which jobs call the gate, PASS/RETRY/REVIEW/BLOCK thresholds, retry budgets, and how REVIEW blocks merge should be deterministic. Use Jev for the semantic judgment on evidence; the pipeline enforces the bands.

FAQ

Why separate evidence from the CI gate decision?

An LLM or harness can score quality, tools, latency, and safety. That is evidence, not a merge policy. A separate typed Choice (pass / retry / review / block) plus Score and Noul lets code fail closed, retry flakiness, and hold for humans—without parsing "looks good enough" prose in the pipeline.

How should CI enforce PASS / RETRY / REVIEW / BLOCK?

Auto-PASS only on high-confidence pass with strong evidence and needs_human false. High-confidence retry re-runs within a budget. Mid band or review: hold the job and notify a human. Low confidence, block, or safety fail: fail the pipeline and escalate.

Can I bring my own API key?

Yes. Try the free demo above without signing in. To run it live on your own text, sign in with Google and connect a Flatkey or OpenRouter key (a quick test call runs when you save it). Manage or delete it anytime under Dashboard → BYOK. Keys are stored encrypted on the server, used only for runs you trigger, and never sent back to the browser. Need a Flatkey key? Get one for Jev 1.13 on Flatkey.

Is this the official TypeSafe Jev product page?

No. jev-llm.com is an independent research and practice site. Verify pricing, models and terms at the original TypeSafe sources.

Sources

Seen on X

Instead of asking an LLM: ‘Is this test run good enough?’ Build a decision layer: Test evidence ↓ Decision model ↓ PASS / RETRY / HUMAN REVIEW / BLOCK ↓ CI/CD action… PASS = 0.91, RETRY = 0.06, REVIEW = 0.03… The LLM produces evidence. The decision model applies judgment. The pipeline enforces the policy.
— @im_pranavkakde View the post on X →