A local LLM workflow for analyzing software requirements with validation-first output handling.

This project is not a generic chatbot or prompt demo. It explores how to make local LLM output safer and more useful for production-style software engineering workflows by combining:

• local LLM generation
• trusted context grounding
• trusted context schema validation
• deterministic normalization
• trusted-context enrichment
• JSON Schema validation
• context-driven semantic validation
• malformed JSON repair fallback
• positive and negative regression checks
• multi-context regression testing
• structured run reporting
• run-report schema validation
• repeatable CLI workflows
• local model comparison

Local LLMs can produce useful analysis, but they are unreliable when used directly.

Observed failure modes include:

• malformed JSON
• invented table names
• invented endpoint names
• unsupported database assumptions
• unsupported frontend assumptions
• schema drift
• missing required fields
• fake sample values
• empty but schema-valid sections
• weak traceability to known facts

This project treats model output as untrusted until it passes deterministic validation.

The model is used as a draft generator. Code decides whether the output is acceptable.

The project currently includes three requirement contexts:

contexts/payment-webhook-context.json
contexts/production-report-context.json
contexts/review-moderation-context.json

The contexts cover different requirement types:

payment-webhook: external API/webhook integration
production-report: report-oriented backend workflow
review-moderation: admin workflow/UI-oriented requirement

This helps prove that the validation workflow is not tied to one hardcoded requirement.

The demo workflow automatically discovers all files matching:

contexts/*-context.json

So adding another context does not require changing the demo script.

The system produces structured engineering output such as:

• facts used
• unknowns
• client questions
• backend tasks
• frontend tasks
• database considerations
• test cases
• hallucination checks

Current architecture:

trusted context JSON
→ trusted context schema validation
→ generated prompt
→ Ollama API
→ raw model JSON
→ malformed JSON repair fallback if needed
→ normalize
→ enrich from trusted context
→ schema validation
→ context-driven semantic validation v3
→ positive/negative regression tests
→ multi-context regression tests
→ structured run report
→ run-report schema validation
→ PASS/FAIL

The main development baseline is:

qwen3:4b

Running locally through Ollama on Windows 10.

A larger same-family comparison model has also been tested:

qwen3:8b

Repeated model-comparison runs showed that qwen3:8b produced cleaner JSON with fewer repairs, while qwen3:4b remains useful as a fast iteration model.

Test hardware:

CPU: Ryzen 7 5800X
GPU: RTX 2070 8GB
RAM: 16GB
OS: Windows 10

This project intentionally uses local models to test practical local LLM workflows rather than relying only on hosted APIs.

requirements-intelligence-assistant/
├── contexts/
│   ├── payment-webhook-context.json
│   ├── production-report-context.json
│   └── review-moderation-context.json
├── evals/
│   └── evaluation notes and model run summaries
├── model-outputs/
│   └── committed demo outputs from earlier workflow stages
├── prompts/
│   └── prompt templates and earlier prompt experiments
├── schemas/
│   ├── requirements-analysis.schema.json
│   ├── requirements-context.schema.json
│   └── run-report.schema.json
├── scripts/
│   ├── build_prompt_from_context.py
│   ├── enrich_model_output.py
│   ├── generate_with_ollama.py
│   ├── normalize_model_output.py
│   ├── run_demo_multi_context_workflow.py
│   ├── run_model_comparison.py
│   ├── run_multi_context_regression_tests.py
│   ├── run_ollama_requirements_workflow.py
│   ├── run_requirements_workflow.py
│   ├── run_validation_v3_negative_tests.py
│   ├── run_validation_v3_regression_tests.py
│   ├── semantic_validate_model_output.py
│   ├── semantic_validate_model_output_v2.py
│   ├── semantic_validate_model_output_v3.py
│   ├── validate_context.py
│   ├── validate_model_output.py
│   ├── validate_pipeline.py
│   ├── validate_pipeline_v2.py
│   ├── validate_pipeline_v3.py
│   └── validate_run_report.py
├── requirements.txt
└── README.md

Create and activate a virtual environment:

py -m venv .venv
.\.venv\Scripts\Activate.ps1

Install dependencies:

pip install -r requirements.txt

Required runtime tools:

Python 3.10+
Ollama
qwen3:4b

Optional comparison model:

qwen3:8b

Python dependency:

jsonschema>=4.0.0

Requirement-specific facts are stored in trusted context JSON.

Current context files:

contexts/payment-webhook-context.json
contexts/production-report-context.json
contexts/review-moderation-context.json

The trusted context is the semantic source of truth for:

• requirement name
• summary
• known facts
• required unknowns
• client questions
• backend tasks
• frontend tasks
• database considerations
• test cases
• hallucination checks

Trusted context files are validated against:

schemas/requirements-context.schema.json

Manual validation command:

python .\scripts\validate_context.py `
  .\contexts\payment-webhook-context.json `
  .\contexts\production-report-context.json `
  .\contexts\review-moderation-context.json

Expected result:

PASS: 3 trusted context file(s) match the schema.

This prevents the model from being the only source of truth.

The prompt can be generated from trusted context instead of being manually maintained.

Run:

python .\scripts\build_prompt_from_context.py `
  --context .\contexts\production-report-context.json `
  --output .\scratch\generated-prompt.txt

This keeps requirement-specific information in the context file and reduces duplication between prompt text and validation rules.

Run a single context through prompt generation, Ollama, normalization, enrichment, and validation:

python .\scripts\run_ollama_requirements_workflow.py `
  --model qwen3:4b `
  --context .\contexts\production-report-context.json `
  --generated-prompt-output .\scratch\context-only-generated-prompt.txt `
  --generated-output .\scratch\context-only-generated-output.json `
  --normalized-output .\scratch\context-only-normalized-output.json `
  --enriched-output .\scratch\context-only-enriched-output.json

Expected result:

OLLAMA WORKFLOW RESULT: PASS

Runtime files are written to scratch/, which is ignored by Git.

The processing workflow handles deterministic post-processing only:

model output
→ normalize
→ enrich with trusted context
→ processed output

Run:

python .\scripts\run_requirements_workflow.py `
  --input .\scratch\context-only-generated-output.json `
  --context .\contexts\production-report-context.json `
  --normalized-output .\scratch\context-only-normalized-output.json `
  --enriched-output .\scratch\context-only-enriched-output.json

Expected result:

WORKFLOW RESULT: PASS

This script intentionally stops before validation. Final validation is handled by validate_pipeline_v3.py or by the full Ollama workflow.

Schema validation checks whether the output has the expected JSON structure.

Run:

python .\scripts\validate_model_output.py .\scratch\context-only-enriched-output.json

This catches issues such as:

• missing required keys
• wrong field names
• missing object fields
• unexpected extra properties
• invalid JSON shape

Schema validation does not prove that the output is semantically correct. It only proves that the structure matches the schema.

Semantic validation v3 checks the model output against the supplied context JSON.

Run:

python .\scripts\semantic_validate_model_output_v3.py `
  .\scratch\context-only-enriched-output.json `
  .\contexts\production-report-context.json

It validates that the output covers required context items, including:

• facts used
• unknowns
• client questions
• backend tasks
• frontend tasks
• database considerations
• test cases
• hallucination checks

It also rejects risky unsupported content, including:

• invented database engines
• invented frontend frameworks
• fake sample values
• unsupported implementation estimates
• invalid task statuses
• blocked items without depends_on
• invalid hallucination check result values

Allowed task status values:

ready
blocked
optional

Allowed hallucination check result values:

pass
fail
warning
blocked

Pipeline v3 enforces the correct validation order:

schema validation
→ semantic validation v3

Run:

python .\scripts\validate_pipeline_v3.py `
  .\scratch\context-only-enriched-output.json `
  .\contexts\production-report-context.json

Expected result:

Running schema validation...
PASS: Model output matches the schema.
Running semantic validation v3...
SEMANTIC VALIDATION V3 RESULT: PASS
PIPELINE V3 RESULT: PASS

The negative test runner creates controlled bad outputs from a known-good enriched output.

Run:

python .\scripts\run_validation_v3_negative_tests.py `
  .\scratch\context-only-enriched-output.json `
  .\contexts\production-report-context.json

It verifies that the validator rejects:

• missing required facts
• invalid status values
• blocked items without depends_on
• invalid hallucination check result values
• invented database engines
• invented frontend frameworks
• unsupported implementation estimates
• fake sample values

Expected result:

NEGATIVE TEST RESULT: PASS

This proves the validator is not only passing the happy path. It also rejects known bad outputs.

The regression runner combines positive and negative validation checks into one command.

Run:

python .\scripts\run_validation_v3_regression_tests.py `
  .\scratch\context-only-enriched-output.json `
  .\contexts\production-report-context.json

Expected result:

REGRESSION TEST RESULT: PASS

This demonstrates:

known-good output passes
known-bad outputs fail

The multi-context regression runner verifies that the same validation system works across multiple requirement contexts.

Run:

python .\scripts\run_multi_context_regression_tests.py `
  .\scratch\payment-webhook-enriched-output.json `
  .\contexts\payment-webhook-context.json `
  .\scratch\production-report-enriched-output.json `
  .\contexts\production-report-context.json `
  .\scratch\review-moderation-enriched-output.json `
  .\contexts\review-moderation-context.json

Expected result:

MULTI-CONTEXT REGRESSION RESULT: PASS

This proves that context-driven semantic validation v3 is reusable across more than one requirement type.

The demo workflow regenerates outputs for all discovered requirement contexts before running regression tests.

By default, the demo runner automatically discovers all files matching:

contexts/*-context.json

This avoids relying on old files in scratch/ and allows new contexts to be picked up without changing the script.

Run:

python .\scripts\run_demo_multi_context_workflow.py --model qwen3:4b

This command performs:

all contexts matching contexts/*-context.json
→ validate trusted context files
→ generate prompt
→ call Ollama
→ repair malformed JSON if needed
→ normalize
→ enrich
→ validate with pipeline v3

all enriched outputs
→ multi-context regression tests
→ run-report.json
→ validate run-report.json

Expected result:

DEMO MULTI-CONTEXT WORKFLOW RESULT: PASS

The workflow includes a JSON repair fallback for malformed local model responses.

This matters because small local models can return invalid JSON even when prompted strictly. The repair fallback keeps generation failure handling explicit while still requiring the final output to pass schema validation, semantic validation v3, and regression tests.

Runtime files are written to:

scratch/demo-multi-context/

These files are ignored by Git.

The reproducible demo workflow writes a structured run report:

scratch/demo-multi-context/run-report.json

The report records:

• model name
• temperature
• trusted context validation result
• context discovery mode
• context count
• output directory
• context paths
• generated prompt paths
• generated output paths
• normalized output paths
• enriched output paths
• whether JSON repair was used
• failed generation path, if any
• per-step start and end timestamps
• per-step duration
• return codes
• multi-context regression result
• run-report validation result
• final workflow result

This makes the demo easier to inspect and helps prepare for future model comparisons.

The run report is also validated against:

schemas/run-report.schema.json

Manual validation command:

python .\scripts\validate_run_report.py .\scratch\demo-multi-context\run-report.json

Expected result:

PASS: Run report matches the schema.

The reproducible demo workflow runs this validation automatically before printing the final pass result.

The model comparison runner executes the reproducible demo workflow for one or more Ollama models and summarizes the results.

Run with the current local model:

python .\scripts\run_model_comparison.py --model qwen3:4b

The runner writes:

scratch/model-comparison/model-comparison-summary.json
scratch/model-comparison/<model-name>/run-report.json

It compares:

final result
context count
repair count
failed context count
multi-context regression result
run-report validation result
duration

Example multi-model usage:

python .\scripts\run_model_comparison.py `
  --model qwen3:4b `
  --model qwen3:8b

Runtime files are written to scratch/, which is ignored by Git.

The project includes eval notes that document the validation and model-comparison milestones.

Key evaluation areas include:

• local Qwen3 4B baseline behavior
• context-driven semantic validation v3
• negative validation tests
• multi-context regression
• reproducible demo workflow
• run-report validation
• trusted context schema validation
• qwen3:4b vs qwen3:8b model comparison

These eval notes help show how the workflow evolved from prompt experiments into a repeatable validation-first system.

The strongest workflow so far is:

trusted context
→ trusted context schema validation
→ prompt generation
→ local LLM generation
→ malformed JSON repair fallback
→ deterministic normalization
→ trusted-context enrichment
→ schema validation
→ context-driven semantic validation v3
→ positive and negative regression proof
→ multi-context regression proof
→ structured run report
→ run-report schema validation

This demonstrates a production-oriented approach:

Completed:

• local Qwen3 4B baseline evaluation
• controlled prompt experiments
• JSON Schema validation
• repair prompt experiments
• semantic validation v1 and v2
• validation pipeline v1 and v2
• deterministic normalization
• trusted-context enrichment
• Ollama API generation
• one-command local Ollama workflow
• context-generated prompt workflow
• context-only workflow
• trusted context schema validation
• context-driven semantic validation v3
• negative validation tests
• regression test runner
• third requirement context
• multi-context regression runner
• reproducible multi-context demo workflow
• malformed JSON repair fallback
• structured run report
• run report schema validation
• local model comparison runner
• qwen3:4b vs qwen3:8b comparison evaluation

Current main proof command:

python .\scripts\run_demo_multi_context_workflow.py --model qwen3:4b

The main local proof command is:

python .\scripts\run_demo_multi_context_workflow.py --model qwen3:4b

This command acts as a local CI-style check for the project.

It validates:

• trusted context files
• generated/enriched model outputs
• semantic grounding
• negative regression cases
• multi-context regression
• run-report.json

This project does not claim that a small local model can produce production-ready output by itself.

The main finding is the opposite:

Prompting alone is not enough.

Current limitations:

• only three requirement contexts have been implemented so far
• contexts are still hand-written structured JSON files
• semantic validation uses deterministic heuristics, not full human understanding
• the validator can catch defined hallucination patterns, not every possible bad answer
• JSON repair only handles malformed model output, not semantic correctness
• no UI or API layer yet
• no retrieval-augmented generation yet
• no GitHub Actions CI workflow yet
• no model comparison dashboard yet

Recommended next steps:

1. Add GitHub Actions CI for schema validation and deterministic regression tests.
2. Start deriving trusted context from less structured requirement input.
3. Add a lightweight API or UI around the workflow.
4. Move toward retrieval-augmented generation over real requirement documents, tickets, emails, or PDFs.
5. Add a small model comparison dashboard or summary view if more models are tested.

This project demonstrates practical LLM engineering skills:

• local model usage
• prompt testing
• schema-first design
• validation-first output handling
• deterministic normalization
• trusted-context grounding
• trusted context schema validation
• malformed JSON repair
• context-driven semantic validation
• hallucination rejection
• positive and negative regression testing
• multi-context validation
• structured run reporting
• run-report validation
• local model comparison
• production-style accept/reject workflows

The positioning is:

I can build practical LLM workflows that are grounded, validated, testable, and production-aware — not just call an LLM API and hope the answer is good.