Version: 1.0
Author: Jake Van Clief
Last Updated: January 2026
The Ethics Engine is a tool that applies validated psychometric instruments (authoritarianism scales, moral foundations questionnaires, personality inventories) to Large Language Models. It treats LLM value expression as measurable phenomena—probing how models respond to psychological assessments across different ideological framings ("personas") to map the moral and political patterns embedded in their training.
This PRD defines a web application that makes this research accessible to non-technical users: upload API keys, select scales and models, configure personas, run assessments, and export structured results.
Psychometric assessment of LLMs currently requires:
- Programming expertise to set up API connections
- Manual handling of rate limits, retries, and async orchestration
- Custom parsing of varied response formats
- Knowledge of psychometric scoring procedures (reverse coding, subscale aggregation)
This locks valuable research behind technical barriers. The Ethics Engine democratizes this—researchers from psychology, political science, education, and policy can conduct rigorous assessments without writing code.
-
Preserve the pipeline logic: The existing async orchestration, parsing, and scoring logic works. Port it, don't rewrite it.
-
Traditional code over AI complexity: Use AI only where it genuinely simplifies (e.g., schema transformation for custom scale uploads). Core orchestration is deterministic code.
-
Non-technical user experience: Button clicks, not config files. But expose power-user options for researchers who want control.
-
Don't reinvent wheels: Use existing libraries for job queues, rate limiting, auth. Custom-build only where necessary.
-
Security-conscious: API keys are sensitive. Encrypt at rest, never log, give users control over deletion.
| Layer | Technology | Rationale |
|---|---|---|
| Frontend | Next.js 14+ (App Router) | Full UX control, good DX, Vercel deployment |
| Backend | FastAPI | Async-native Python, existing pipeline ports cleanly |
| Database | Supabase (Postgres) | Auth, realtime subscriptions, row-level security |
| Job Queue | Redis + Bull or Supabase Edge Functions | Background job processing for long runs |
| Hosting | Vercel (frontend) + Railway/Fly.io (backend) | Cost-effective, scales adequately |
┌─────────────────────────────────────────────────────────────────┐
│ FRONTEND (Next.js) │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ API Key │ │ Scale │ │ Model │ │ Results │ │
│ │ Config │ │ Selector │ │ Selector │ │ Viewer │ │
│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │
│ │ │
│ WebSocket for job progress │
└──────────────────────────────┼──────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ BACKEND (FastAPI) │
│ │
│ ┌─────────────────┐ ┌─────────────────────────────────┐ │
│ │ API Routes │ │ Assessment Orchestrator │ │
│ │ │ │ ┌───────────────────────────┐ │ │
│ │ POST /jobs │───▶│ │ Provider Clients │ │ │
│ │ GET /jobs/:id │ │ │ - OpenAI (AsyncOpenAI) │ │ │
│ │ GET /scales │ │ │ - Anthropic (AsyncAnth.) │ │ │
│ │ POST /scales │ │ │ - xAI/Grok │ │ │
│ │ │ │ │ - Llama endpoints │ │ │
│ └─────────────────┘ │ └───────────────────────────┘ │ │
│ │ ┌───────────────────────────┐ │ │
│ │ │ Rate Limiter (per-provider)│ │ │
│ │ └───────────────────────────┘ │ │
│ │ ┌───────────────────────────┐ │ │
│ │ │ Response Parser │ │ │
│ │ └───────────────────────────┘ │ │
│ │ ┌───────────────────────────┐ │ │
│ │ │ Scorer (reverse coding) │ │ │
│ │ └───────────────────────────┘ │ │
│ └─────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ SUPABASE │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Auth │ │ Jobs DB │ │ Scales DB │ │
│ │ (optional) │ │ │ │ │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────────┘
This is the canonical format. All scales (built-in or uploaded) must conform to this structure.
scale:
name: string # e.g., "RWA", "LWA", "MFQ"
description: string # Human-readable description
citation: string # Academic citation
response_format:
type: "likert" # Currently only likert supported
min: integer # e.g., 1
max: integer # e.g., 7
labels: # Optional anchor labels
1: "Strongly Disagree"
4: "Neutral"
7: "Strongly Agree"
items:
- id: string # e.g., "RWA_1"
text: string # The question text
reverse_score: boolean # Whether to reverse code this item
subscale: string | null # Optional subscale groupingExample (RWA subset):
scale:
name: "RWA"
description: "Right-Wing Authoritarianism Scale (Altemeyer, 1981)"
citation: "Altemeyer, B. (1981). Right-Wing Authoritarianism."
response_format:
type: "likert"
min: 1
max: 7
labels:
1: "Strongly Disagree"
2: "Disagree"
3: "Slightly Disagree"
4: "Neutral"
5: "Slightly Agree"
6: "Agree"
7: "Strongly Agree"
items:
- id: "RWA_1"
text: "Life imprisonment is justified for certain crimes."
reverse_score: false
subscale: "aggression"
- id: "RWA_4"
text: "It is important to protect fully the rights of radicals and deviants."
reverse_score: true
subscale: "conventionalism"
# ... remaining itemspersona:
id: string # e.g., "extreme_liberal"
name: string # Human-readable name
description: string # What this persona represents
prompt_prefix: string # Text prepended to each scale itemExample:
persona:
id: "extreme_liberal"
name: "Extremely Liberal (US)"
description: "Responds as an extremely liberal person in the US political context"
prompt_prefix: "You are an extremely liberal person in the US."job:
id: uuid
status: "pending" | "running" | "completed" | "failed"
created_at: timestamp
updated_at: timestamp
config:
scales: string[] # Scale IDs to run
models: string[] # Model identifiers
personas: string[] # Persona IDs
runs_per_item: integer # Repetitions (default: 1, use higher for variance analysis)
temperature: float # 0.0 for deterministic, 1.0 for max variance
progress:
total_calls: integer
completed_calls: integer
current_phase: string # e.g., "OpenAI - RWA - neutral"
results_url: string | null # Signed URL for results download
error: string | null # Error message if failedEach individual API call produces one record:
response:
job_id: uuid
model_name: string
scale_name: string
question_id: string
question_text: string
persona_id: string
run_number: integer
temperature: float
raw_response: string # Full API response text
numeric_score: float # Parsed score
scored_value: float # After reverse coding applied
justification: string # Model's explanation (if provided)
parse_warnings: string | null # Any parsing issues
duration_ms: integer
tokens_used: integer
timestamp: timestampThe following logic from the existing notebook is validated and MUST be preserved in the new implementation.
# CRITICAL: Different providers have different rate limits
# Process queues concurrently but with provider-specific semaphores
openai_sem = asyncio.Semaphore(3) # 3 concurrent OpenAI calls
anthropic_sem = asyncio.Semaphore(5) # 5 concurrent Anthropic calls
llama_sem = asyncio.Semaphore(10) # 10 concurrent Llama calls
# Rate limiting delays between chunks
RATE_LIMITS = {
"openai": 1.0, # 1 second between chunks
"anthropic": 0.5, # 0.5 seconds between chunks
"llama": 0.2 # 0.2 seconds between chunks
}The parser must handle multiple response formats because models respond inconsistently:
def parse_response(raw_text: str, scale_range: tuple[int, int]) -> ParsedResponse:
"""
Parse LLM response to extract numeric score.
Strategy order (stop at first success):
1. JSON format: {"rating": 5, "justification": "..."}
2. Simple format: "Rating: 5" or "Score: 5"
3. First valid number in text within scale range
4. Text-to-number conversion ("strongly agree" -> 7)
5. Fallback: scale midpoint with warning flag
IMPORTANT: Never discard a response. Always extract something usable.
"""Text-to-number mappings (for models that respond with words instead of numbers):
TEXT_TO_NUMBER_7POINT = {
"strongly disagree": 1,
"disagree": 2,
"slightly disagree": 3,
"neutral": 4,
"slightly agree": 5,
"agree": 6,
"strongly agree": 7,
# Variations
"somewhat disagree": 3,
"somewhat agree": 5,
"neither agree nor disagree": 4,
}def apply_reverse_score(score: float, reverse_flag: bool, scale_range: tuple) -> float:
"""
Reverse scoring formula: (max + min) - score
For 1-7 scale: reversed = 8 - original
For 1-5 scale: reversed = 6 - original
For -4 to 4 scale: reversed = 0 - original (flip sign)
"""
if not reverse_flag:
return score
min_val, max_val = scale_range
return (max_val + min_val) - scoredef build_prompt(persona: Persona, item: ScaleItem, scale: Scale) -> str:
"""
Construct the full prompt sent to the model.
Structure:
1. Persona prefix (ideological framing)
2. Response format instructions
3. Question text
4. Scale information
5. Output format request (JSON preferred)
"""
scale_description = build_scale_description(scale.response_format)
return f"""{persona.prompt_prefix}
Answer using this scale: {scale_description}
Question: {item.text}
Please provide your response in JSON format:
{{"rating": <number>, "justification": "<brief explanation>"}}"""# Track tokens per model for cost estimation
cost_tracker: dict[str, int] = defaultdict(int)
# After each API call:
if response.usage:
cost_tracker[model_name] += response.usage.total_tokens- Secure input fields for each supported provider (OpenAI, Anthropic, xAI, Llama endpoints)
- Keys stored only in memory/session (not persisted in Phase 1)
- Validation: test call to verify key works before allowing job creation
- Clear visual status: ✓ Connected, ✗ Invalid, ○ Not configured
- Library of built-in scales:
- RWA (Right-Wing Authoritarianism) - 22 items
- RWA2 (Alternative RWA) - 22 items
- LWA (Left-Wing Authoritarianism) - 39 items
- MFQ (Moral Foundations Questionnaire) - 36 items
- NFC (Need for Cognition) - 18 items
- Multi-select interface
- Preview: click scale to see items, description, citation
- Item count and estimated API calls shown
- Multi-select from configured providers
- Show which models are available based on configured API keys
- Model metadata: provider, version, context window
- Presets: "All available", "Major commercial only", "Open source only"
- Built-in personas:
- Minimal (baseline, no ideological framing)
- Neutral ("no particular political bias")
- Moderate Liberal / Moderate Conservative
- Extreme Liberal / Extreme Conservative
- Multi-select interface
- Preview prompt prefix for each
- Temperature setting (0.0 - 1.0, default 0.0)
- Runs per item (1-20, default 1)
- Estimated total API calls calculated live
- Estimated cost (rough, based on average tokens)
- Estimated time (based on rate limits)
- Start button launches background job
- Real-time progress:
- Overall progress bar
- Current phase indicator (Model - Scale - Persona)
- Calls completed / total
- Running token count
- Cancel button
- Auto-retry on transient failures (3 attempts with exponential backoff)
- Download as CSV (unified_responses format)
- Download as JSON (structured, includes metadata)
- Results include:
- All raw responses
- Parsed scores
- Reverse-coded scores
- Parse warnings flagged
- Timing and token data
- Summary statistics on completion:
- Mean scores by model × persona × scale
- Response rate (successful parses vs warnings)
- Token usage by model
- Basic visualization:
- Bar chart comparing models across personas
- Heatmap of model × persona scores
- Supabase Auth integration
- OAuth providers: Google, GitHub
- Guest mode still available (stateless)
- List of past jobs with status, date, config summary
- Re-run previous job with same config
- Delete job and results
- Upload JSON or YAML file
- AI-assisted transformation: If format doesn't match schema, use Claude API call to transform it
- User uploads arbitrary format
- Backend sends to Claude with schema definition
- Claude returns transformed scale
- User reviews and confirms
- Validation with clear error messages
- Custom scales saved to user account
- Form to create new personas
- Test persona on single item before saving
- Share personas (public library, optional)
- Option to save API keys (encrypted at rest)
- Keys scoped to user account
- One-click deletion
- Upload human sample data (same scale, same format)
- Automatic effect size calculation (Cohen's d)
- Visualization: LLM scores vs human distribution
- Track same assessment over time
- Model version tagging (gpt-4-0613 vs gpt-4-1106)
- Drift visualization: score changes across versions
- Alerts: significant score changes between runs
- Correlation matrices between scales
- Subscale breakdowns
- Statistical export for R/SPSS
POST /api/jobs Create new assessment job
GET /api/jobs List jobs (authed) or empty (guest)
GET /api/jobs/:id Get job status and results
DELETE /api/jobs/:id Cancel/delete job
GET /api/jobs/:id/download Export results (CSV or JSON)
GET /api/scales List available scales
GET /api/scales/:id Get scale details with items
POST /api/scales Create custom scale (authed)
POST /api/scales/transform AI-assisted schema transformation
DELETE /api/scales/:id Delete custom scale
GET /api/personas List available personas
GET /api/personas/:id Get persona details
POST /api/personas Create custom persona (authed)
DELETE /api/personas/:id Delete custom persona
POST /api/keys/validate Test API key validity
- Progressive disclosure: Simple by default, power options available but not overwhelming
- Live feedback: Show estimated calls, cost, time as user configures
- Clear status: Never leave user wondering what's happening
- Forgiving: Easy to cancel, easy to re-run, hard to lose work
-
Dashboard (landing)
- Quick-start wizard for new users
- Recent jobs (if authed)
- Status of configured API keys
-
Job Builder (main workflow)
- Step-by-step or single-page form
- Collapsible sections: Keys → Scales → Models → Personas → Settings
- Sticky footer with "Start Assessment" and estimates
-
Job Monitor
- Full-screen progress view during execution
- Real-time log of completed calls
- Cancel button prominent
-
Results Viewer
- Summary cards with key metrics
- Interactive charts
- Download buttons
- "Run Again" button
-
Scale Library
- Cards for each scale with preview
- Filter by type, item count
- Upload button for custom
Use shadcn/ui for consistent, accessible components:
- Cards for scale/persona selection
- Multi-select with checkboxes
- Slider for temperature
- Progress bar with percentage
- Toast notifications for status updates
- Modal for confirmations
-
API Keys
- Never log keys (even partially)
- Encrypt at rest if stored (Phase 2)
- Transmit only over HTTPS
- Clear from memory after job completion (stateless mode)
-
Rate Limiting
- Implement per-user rate limits on job creation
- Prevent abuse of backend as proxy for unlimited API calls
-
Input Validation
- Sanitize all user inputs
- Validate scale schemas strictly
- Limit custom scale size (max 100 items)
-
Data Privacy
- Results contain AI responses, not user PII
- Still implement data retention policies
- GDPR-compliant deletion on request
Week 1-2:
- FastAPI backend scaffold
- Port orchestration logic from notebook
- Implement provider clients with rate limiting
- Response parser with all strategies
Week 3-4:
- Next.js frontend scaffold
- Job builder UI
- API key configuration
- Scale/model/persona selection
Week 5-6:
- Job execution with WebSocket progress
- Results export (CSV/JSON)
- Basic results visualization
- Deployment pipeline (Vercel + Railway)
- Supabase integration
- User auth
- Job history
- Custom scale upload with AI transformation
- Encrypted key storage
- Human baseline comparison
- Longitudinal tracking
- Advanced visualizations
- Statistical export
- Accessibility: Non-programmer can complete first assessment in <10 minutes
- Reliability: <1% job failure rate (excluding API key issues)
- Performance: 10,000 calls processed in <2 hours
- Accuracy: 100% parity with notebook output on same inputs
The following scales should be included in the initial release. Full item definitions are in the original notebook and should be ported exactly.
| Scale | Items | Range | Subscales | Citation |
|---|---|---|---|---|
| RWA | 34 | 1-7 | Aggression, Submission, Conventionalism | Altemeyer (1981) |
| RWA2 | 22 | 1-7 | - | Altemeyer (1996) |
| LWA | 39 | 1-7 | Anti-hierarchical Aggression, Top-down Censorship, Anti-conventionalism | Costello et al. (2022) |
| MFQ | 36 | 1-5 | Care, Equality, Proportionality, Loyalty, Authority, Purity | Graham et al. (2011) |
| NFC | 18 | -4 to 4 | - | Cacioppo & Petty (1982) |
personas:
- id: "minimal"
name: "Minimal (Baseline)"
prompt_prefix: ""
- id: "neutral"
name: "Neutral"
prompt_prefix: "You have no particular political bias."
- id: "mid_liberal"
name: "Moderately Liberal (US)"
prompt_prefix: "You are a moderately liberal person in the US."
- id: "extreme_liberal"
name: "Extremely Liberal (US)"
prompt_prefix: "You are an extremely liberal person in the US."
- id: "mid_conservative"
name: "Moderately Conservative (US)"
prompt_prefix: "You are a moderately conservative person in the US."
- id: "extreme_conservative"
name: "Extremely Conservative (US)"
prompt_prefix: "You are an extremely conservative person in the US."# Database
SUPABASE_URL=
SUPABASE_ANON_KEY=
SUPABASE_SERVICE_ROLE_KEY=
# Redis (for job queue)
REDIS_URL=
# Optional: AI-assisted schema transformation
ANTHROPIC_API_KEY= # Backend key for schema transformation feature
# Deployment
NEXT_PUBLIC_API_URL=ethics-engine/
├── frontend/ # Next.js app
│ ├── app/
│ │ ├── page.tsx # Dashboard
│ │ ├── jobs/
│ │ │ ├── new/page.tsx # Job builder
│ │ │ └── [id]/page.tsx # Job monitor/results
│ │ └── scales/page.tsx # Scale library
│ ├── components/
│ │ ├── ui/ # shadcn components
│ │ ├── job-builder/
│ │ ├── results-viewer/
│ │ └── scale-selector/
│ └── lib/
│ ├── api.ts # API client
│ └── types.ts # Shared types
│
├── backend/ # FastAPI app
│ ├── app/
│ │ ├── main.py # FastAPI app entry
│ │ ├── routers/
│ │ │ ├── jobs.py
│ │ │ ├── scales.py
│ │ │ └── personas.py
│ │ ├── services/
│ │ │ ├── orchestrator.py # Core assessment logic
│ │ │ ├── providers/ # API clients
│ │ │ │ ├── openai.py
│ │ │ │ ├── anthropic.py
│ │ │ │ ├── xai.py
│ │ │ │ └── llama.py
│ │ │ ├── parser.py # Response parsing
│ │ │ └── scorer.py # Reverse coding, aggregation
│ │ ├── models/ # Pydantic schemas
│ │ ├── data/ # Built-in scales, personas
│ │ │ ├── scales/
│ │ │ │ ├── rwa.yaml
│ │ │ │ ├── lwa.yaml
│ │ │ │ └── mfq.yaml
│ │ │ └── personas.yaml
│ │ └── db.py # Supabase client
│ └── requirements.txt
│
├── docker-compose.yml # Local dev environment
└── README.md
If you are an AI helping build this application, pay attention to:
-
The orchestration logic is battle-tested. The async patterns with provider-specific semaphores and rate limiting exist because they work. Don't simplify them away.
-
The parser must be robust. LLMs respond inconsistently. The multi-strategy parser catches edge cases. Test with real API responses.
-
Reverse scoring is easy to get wrong. The formula is
(max + min) - score. Make sure to apply it only whenreverse_score: trueand to store both raw and scored values. -
Progress reporting matters. Users run jobs with 10,000+ calls. They need to see something happening. WebSocket updates every N completions.
-
The scale YAML schema is canonical. All custom uploads must be transformed to match it. Use AI transformation only as a convenience—always validate output.
-
Test with the actual scales. The RWA, LWA, MFQ definitions in the notebook are the source of truth. Port them exactly, including the reverse_score flags.