This research project evaluates the accuracy of large language models (LLMs), specifically OpenAI’s ChatGPT and Google’s Gemini, in normalizing cybersecurity Indicators of Compromise (IOCs) into structured STIX 2.1 format. The repository includes reproducible scripts for STIX generation, validation, and content-level evaluation.

.
├── gpt_stix.py              # Uses ChatGPT to convert IOC list to STIX n times
├── gemini_stix.py           # Uses Gemini to convert IOC list to STIX n times
├── validate_format.py       # Validates output files against STIX 2.1 format
├── ip_search.py             # Checks IP presence and duplication across outputs
├── url_search.py            # Checks URL presence and duplication across outputs
├── hash_search.py           # Checks hash presence and duplication across outputs
├── input_data/
│   ├── ip_list.txt
│   ├── url_list.txt
│   └── hash_list.txt
├── stix_outputs_gpt_[IP/hash/URL]        # Sample output directories for GPT-generated STIX files
├── stix_outputs_gemini_[IP/hash/URL]/     # Sample output directory for Gemini-generated STIX files
└── README.md

Ensure the following packages are installed in your Python environment:

pip install openai google-genai stix2-validator pydantic

This script uses OpenAI's ChatGPT to convert a list of IOCs into STIX format multiple times.

User-defined fields to modify in the script:

OPENAI_API_KEY: Add your OpenAI API key input_file: Path to your IOC file (e.g., input_data/url_list.txt) output_dir: Directory to save generated files (e.g., stix_outputs_gpt_url) for i in range (1, [N]): Replace N with number of STIX files to generate

This script uses Google’s Gemini model to convert the input IOC list into STIX-formatted objects.

User-defined fields to modify in the script:

GOOGLE_API_KEY: Add your Gemini API key input_file: Path to the IOC list output_dir: Where to save results for i in range (1, [N]): Replace N with number of STIX files to generate

Validates whether each generated STIX file is properly formatted according to the STIX 2.1 schema.

User-defined fields to modify in the script:

stix_folder: Path to the directory containing the generated STIX JSON files (e.g., stix_outputs_gpt) summary_output_file: Path to save the validation summary output (e.g., stix_validation_summary.json)

Each script is specific to a different IOC type and checks the generated STIX files for:

Accurate inclusion of original values Unexpected or duplicate values User-defined fields to modify in each script:

input_file: Path to the original IOC list (e.g., input_data/ip_list.txt) stix_dir: Directory of output STIX files (e.g., stix_outputs_gpt_ip) output_file: Path to save the evaluation summary (e.g., summary_ips.json) Expected Output

Each STIX generation script will create stix_output_001.json, stix_output_002.json, etc.

The validation script will output a JSON file with a list of:

STIX file names Whether they passed validation Reasons for failure (if applicable) The search scripts will output JSON summaries containing:

Match percentage with original IOCs Presence of repeats or unexpected values

If you use this project for research or publication, please cite:

Peterson, N. (2025). Trust But Verify: Evaluating LLM Accuracy in Cybersecurity Data Normalization.