Supply Chain Knowledge Benchmark

A comprehensive toolkit for evaluating Large Language Models (LLMs) on supply chain and operations management knowledge using questions extracted from university textbook test banks.

🎯 Purpose

This project provides an automated pipeline to:

1. Extract thousands of test questions from supply chain textbook test banks
2. Create standardized benchmark datasets from these questions
3. Evaluate any LLM (local or API-based) against this knowledge base
4. Generate detailed performance reports with accuracy metrics

📁 Project Structure

sc-knowledge-benchmark/
├── sc-question-extraction/     # Step 1: Extract questions from textbooks
│   ├── textbooks/              # Place .doc/.docx test banks here
│   ├── extract_questions.py    # Main extraction script
│   └── run_extraction_pipeline.py  # Complete pipeline runner
│
└── sc-llm-benchmark/           # Step 2: Benchmark LLMs
    ├── data/                   # Test questions go here
    ├── run_benchmark.py        # Main benchmarking script
    ├── results/                # Raw model outputs
    └── reports/                # Performance reports

🚀 Quick Start

Step 1: Extract Questions from Textbooks

First, extract test questions from your textbook test banks:

cd sc-question-extraction

# Install dependencies
pip install -r requirements.txt

# Add textbook test banks (.doc/.docx files) to textbooks/ directory
# Then run the extraction pipeline
python run_extraction_pipeline.py

This will generate:

• test_questions_for_models.json - Questions for testing (no answers)
• test_questions_for_models_answers.json - Answer key for grading

Step 2: Benchmark LLMs

Next, evaluate LLMs against the extracted questions:

cd ../sc-llm-benchmark

# Install dependencies
pip install -r requirements.txt

# Copy test files from extraction project
cp ../sc-question-extraction/test_questions_for_models*.json data/

# Configure API keys (if using API-based models)
cp .env.example .env
# Edit .env with your API keys

# Run benchmarks
python run_benchmark.py ollama/llama3.1:8b            # Local model via Ollama (recommended)
python run_benchmark.py ollama/tinyllama:latest       # Another Ollama example
python run_benchmark.py openai/gpt-4o                 # OpenAI API
python run_benchmark.py anthropic/claude-3-haiku      # Anthropic API

📊 Benchmark Results

After running a benchmark, you'll find:

• Results: results/{model}_results.json - Raw model responses
• Report: reports/{model}_report.md - Detailed accuracy report with:
  • Overall accuracy percentage
  • Breakdown of incorrect answers

🤖 Supported Model Providers

Local Models (via Ollama)

• Install Ollama and pull models locally
• Run: python run_benchmark.py ollama/model:tag
• Examples:
  • python run_benchmark.py ollama/llama3.1:8b
  • python run_benchmark.py ollama/tinyllama:latest
  • python run_benchmark.py ollama/mixtral:8x7b
  • Models with slashes: python run_benchmark.py ollama/chenzo/SCB_llama3_1_q8_3eps:latest

API-Based Models

Configure your API keys in .env file:

• OpenAI: openai/gpt-4o, openai/gpt-3.5-turbo
• Anthropic: anthropic/claude-3-opus, anthropic/claude-3-sonnet
• Azure OpenAI: azure/your-deployment-name
• OpenRouter: openrouter/meta-llama/llama-3-70b
• AWS Bedrock: bedrock/anthropic.claude-3-sonnet
• Groq: groq/mixtral-8x7b-32768
• Together AI: together/meta-llama/Llama-3-70b-chat-hf

📚 Textbooks

Testing for this project used test banks from major supply chain textbooks. The test banks are not shared on this repo. Replace with your own test banks.

🔧 Advanced Usage

Adding New Textbooks

1. Place .doc or .docx test bank files in sc-question-extraction/textbooks/
2. Run the extraction pipeline
3. Copy generated files to benchmarking project

Custom Question Filtering

Edit sc-question-extraction/filter_questions.py to customize which question types to include:

desired_types = ['multiple_choice', 'true_false']  # Modify as needed

Batch Benchmarking

Create a script to test multiple models:

#!/bin/bash
for model in ollama/tinyllama:latest ollama/gemma:2b ollama/phi3:mini; do
    python run_benchmark.py $model
done

📈 Example Results

Sample benchmark output:

Model: gpt-3.5-turbo
Total Questions: 2,561
Correct Answers: 2,187
Accuracy: 85.40%

🛠️ Troubleshooting

Extraction Issues

• Ensure .doc files have antiword installed: brew install antiword (macOS)
• Check that test banks follow standard Q&A formatting

Benchmark Timeouts

• Increase timeout in run_benchmark.py if models take longer to respond
• Some complex questions may cause smaller models to timeout

API Errors

• Verify API keys are correctly set in .env file
• Check API rate limits and quotas
• Ensure sufficient API credits

📝 License

MIT License - See individual project directories for details

---

Note: This project is for educational and research purposes. Ensure you have appropriate permissions to use any textbook materials.