# CREDO — Scoring Rubric and Evaluation Criteria **CREDO: Christian Reformed Evaluation for Doctrinal Orthodoxy** Version 1.1 · July 2026 · License: CC BY 4.0 Maintained by Reformeer (Pty) Ltd · https://reformeer.org/benchmark This document specifies, in full, how CREDO scores an AI system's answers. Together with the task set (`/credo-task-set.json`: all 23 questions and their reference answers) and the raw results (`/benchmark-results.json`: every transcript, every judge critique, every score), it is everything needed to check or reproduce the published result. --- ## 1. What CREDO measures Theological reliability: how faithfully a system answers questions of the Christian faith when measured against a fixed, public confessional standard, the **Three Forms of Unity** (the Belgic Confession, the Heidelberg Catechism and the Canons of Dort) and the Reformed **Church Order**. CREDO does not grade against a neutral average of world religions. Reliability is only measurable relative to a stated standard; CREDO states its standard openly and publishes the reference answer for every question so readers who confess a different standard can re-grade the data against their own. ## 2. Task set 23 questions in two tiers: - **Tier A (7 questions):** the most-googled basic questions of the Christian faith (following The Gospel Coalition's 2025 AI Christian Benchmark), graded against a broadly-orthodox reference answer. - **Tier B (16 questions):** Reformed distinctives, from the extent of the atonement to the marks of the true church, graded against the Three Forms of Unity and the Church Order. Each question carries a reference answer: the confessionally correct answer the judge grades against. The full set, including reference answers, is published at `/credo-task-set.json`. ## 3. Prompting protocol - Every system receives the **raw question, verbatim**, in a single turn, exactly as an ordinary person would type it. No system prompt, no added context, no coaching. - General models run with **no retrieval and no tools**, at temperature 0, low reasoning effort, and a capped answer length, mirroring an everyday consumer query. - Reformeer, the one grounded system in the field, answers through its standard production pipeline: retrieval over its Reformed corpus, with source citations. - The v1.1 run was conducted in English in July 2026. ## 4. First judge: scoring bands Each answer is scored 0-100 by an independent AI judge (Gemini 3.5 Flash, temperature 0, reasoning recorded before the score, **blind to which system wrote the answer**). The judge grades doctrinal substance against the reference answer, not prose polish or length, and does not require exact article or question numbers. The bands, as given to the judge: | Band | Label | Criteria (verbatim from the judge instruction) | |---|---|---| | 85-100 | Reliable | States the correct answer clearly and accurately, no hedge that undermines it; a correct confessional/Scripture citation is a bonus but not required. | | 65-84 | Hedged | Substantially correct but hedged, missing the key distinctive, or softened with "many traditions believe...". | | 40-64 | All-sides | Partially correct or heavily "all-sides"; the correct answer appears only as one option among several presented as equally valid. | | 0-39 | Unreliable | Wrong, refuses, gives a non-answer, or frames the correct position as merely one opinion. | An answer that presents the Reformed/orthodox position alongside contrary views as coequal is scored in the 40-64 band. Empty answers score 0. ## 5. Second judge: cross-lab adjudication (v1.1) Reformeer is built by the maintainers of this benchmark. To remove the temptation of a friendly grade, v1.1 adds an adjudication pass that applies **only to Reformeer's answers**: - Every Reformeer answer was re-graded by a second frontier judge from a different lab (**Claude Fable 5, Anthropic**), instructed to grade adversarially: to find loose or mismatched citations, missing elements of the reference answer, circular argumentation, and imprecision. - For each question, the **lower** of the two judges' scores is published. - General models keep the neutral first judge's scores unchanged. Only the maintainers' own system is held to the stricter double standard. - Both scores and both critiques for every Reformeer answer are preserved in the raw results (`firstJudgeScore` and `adjudication` fields in `/benchmark-results.json`). In the v1.1 run the adjudicator revised 18 of Reformeer's 23 scores downward (range 88-100), moving Reformeer's published overall from 100 to 96. ## 6. Aggregation - Per-system tier scores are the arithmetic mean of that system's published per-question scores within the tier, rounded to the nearest integer. - The overall score is the mean across all 23 questions, rounded to the nearest integer. - No weighting, no dropped questions, no retries. Every question in the task set counts, and every transcript is published. ## 7. Known limitations - One run (July 2026), in English. Different sampling, higher reasoning-effort settings, or other languages could shift individual scores. - LLM judges, even banded and blind, carry noise. Treat single-digit gaps between systems as ties. - The standard is confessional by design (see §1). ## 8. How to cite > Reformeer (2026). CREDO v1.1: Christian Reformed Evaluation for Doctrinal > Orthodoxy. reformeer.org/benchmark Requests to include another system in the next release are welcome: support@reformeer.org.