SplunkFlow

A Python-based intelligent log processing tool that automatically detects log types, extracts structured fields, sends data to Splunk, and generates custom dashboards based on detected log patterns.

This tool revolutionizes log analysis by combining pattern recognition with automated Splunk integration. It intelligently identifies various log formats (firewall, DNS, Apache, syslog, JSON) and creates tailored visualizations without manual configuration.

• 🔍 Intelligent Log Detection: Automatically identifies log types using regex pattern matching
• 📊 Multi-Format Support: Handles firewall, DNS, Apache, syslog, and JSON logs
• 🚀 Splunk Integration: Seamless connection and data ingestion to Splunk
• 📈 Auto-Dashboard Generation: Creates custom XML dashboards based on detected log type
• ⚡ High Performance: Efficient field extraction with detailed success metrics
• 🛡️ Robust Error Handling: Comprehensive error reporting and debugging information

splunk-ai-log-analyzer/
├── main.py                    # Main entry point and orchestration
├── log_processor_py.py        # Core log processing and pattern matching
├── splunk_connection_py.py    # Splunk API integration
├── .env                       # Configuration file 
├── generated_dashboard.xml    # Auto-generated dashboard (created after run)
├── README.md                  # This file
└── LICENSE                    # MIT License

• Pattern Detection: Uses regex patterns to identify log types
• Field Extraction: Extracts structured data from unstructured logs
• Supported Log Types:
  • Firewall (Custom): 2025-08-22 09:15:23 [ALLOW] TCP 192.168.1.45:3421 -> 203.0.113.15:80
  • DNS: client 192.168.1.55#12345: query: google.com IN A
  • Standard Firewall: action=allow src=192.168.1.10 dst=8.8.8.8
  • Apache: 192.168.1.101 - - [19/Jan/2025:10:30:01 +0000] "GET /index.html"
  • Syslog: Jan 19 10:15:32 localhost sshd[1024]: message
  • JSON: Any valid JSON formatted logs

• Authentication: Secure connection to Splunk instances
• Event Ingestion: Bulk upload of processed log events
• Dynamic Sourcetype: Automatically sets sourcetype based on detected log type

• Custom XML Generation: Creates tailored dashboards for each log type
• Log-Specific Visualizations:
  • Firewall: Top IPs, protocols, blocked traffic trends
  • Apache: URL analysis, status code distribution
  • DNS: Domain queries, query type breakdown
  • Syslog: Process analysis, message patterns

• Python 3.7+
• Splunk instance (local or remote)
• Access to Splunk with appropriate permissions

1. Clone the repository

   git clone https://github.com/yourusername/splunk-ai-log-analyzer.git
   cd splunk-ai-log-analyzer

2. Install dependencies

   pip install -r requirements.txt

3. Configure environment

   cp .env.example .env
   # Edit .env with your Splunk credentials and log file path

Update the .env file with your settings:

# Splunk Configuration
SPLUNK_HOST=localhost
SPLUNK_PORT=8089
SPLUNK_USERNAME=your_username
SPLUNK_PASSWORD=your_password

# Optional Settings
SPLUNK_SCHEME=https
SPLUNK_VERIFY_SSL=false
DEFAULT_INDEX=main
LOG_LEVEL=INFO

# Input/Output Paths
LOG_FILE=path/to/your/logfile.txt
OUTPUT_XML=generated_dashboard.xml

python main.py

The tool will:

1. 🔍 Analyze your log file and detect the format
2. 📊 Extract structured fields from raw logs
3. 🚀 Upload processed data to Splunk
4. 📈 Generate a custom dashboard XML file
5. ✅ Provide detailed success metrics

🚀 Starting Splunk AI Log Analyzer...
📁 Index: main
📄 Log file: C:\Users\vetha\Downloads\syslog_sample.txt
📖 Reading log file: C:\Users\vetha\Downloads\syslog_sample.txt
📊 Loaded 150 lines from log file
🔍 Detecting log type...
📊 Detection scores:
   syslog: 100.00%
   firewall_custom: 0.00%
   dns: 0.00%
   apache: 0.00%
   firewall: 0.00%
   json: 0.00%
✅ Detected log type: syslog
📋 Description: System logs with process information and messages
🔧 Extracting fields for syslog logs...
✅ Extracted 150 records from 150 non-empty lines (100.0% success rate)
🔌 Connecting to Splunk at localhost:8089
✅ Connected to Splunk successfully
📤 Sending 150 events to Splunk...
✅ Sent 150 events to 'main' with sourcetype=syslog
📄 Dashboard XML written to generated_dashboard.xml
🎉 Process completed successfully!
📊 Dashboard XML saved as: generated_dashboard.xml
🔍 You can now search your data in Splunk using: index=main sourcetype=syslog

After running the analyzer, you'll get a generated_dashboard.xml file. Here's how to import it into Splunk:

1. Login to Splunk Web

   • Open your browser and navigate to your Splunk instance
   • Login with your credentials

2. Navigate to Dashboards

   • Go to Settings > User Interface > Views
   • Or click Apps > Search & Reporting > Dashboards

3. Import Dashboard

   • Click "New Dashboard" or "Create New Dashboard"
   • Select "Dashboard Studio" or "Classic Dashboards"
   • Choose "Upload Dashboard" or "Import from XML"

4. Upload XML File

   • Browse and select your generated_dashboard.xml file
   • Click "Upload" or "Import"

5. Verify Dashboard

   • The dashboard will appear in your dashboards list
   • Click on it to view your auto-generated visualizations

1. Open the generated XML file in a text editor
2. Copy the entire XML content
3. In Splunk Web, go to Settings > User Interface > Views
4. Click "New View"
5. Paste the XML content in the source editor
6. Save the dashboard

# Copy to Splunk apps directory
cp generated_dashboard.xml $SPLUNK_HOME/etc/apps/search/local/data/ui/views/

# Restart Splunk to load the dashboard
$SPLUNK_HOME/bin/splunk restart

• Timeline View: Events over time
• Top Source IPs: Most active source addresses
• Top Destination IPs: Most targeted destinations
• Actions Distribution: Allow vs Block ratio (pie chart)
• Protocol Analysis: TCP, UDP, ICMP breakdown
• Top Destination Ports: Most accessed services
• Blocked Traffic Trends: Security incidents over time
• Raw Events Table: Latest 20 log entries

• Timeline View: Web requests over time
• Top Requested URLs: Most accessed pages
• HTTP Status Codes: Success, error, redirect distribution
• Raw Events Table: Recent access logs

• Timeline View: DNS queries over time
• Top Queried Domains: Most requested domains
• Query Types: A, AAAA, MX, etc. breakdown
• Raw Events Table: Recent DNS queries

• Timeline View: System events over time
• Top Processes: Most active system processes
• Frequent Messages: Common log messages
• Raw Events Table: Recent system logs

The generated dashboards are fully customizable. You can:

• Modify Time Ranges: Adjust search time windows
• Add New Panels: Create additional visualizations
• Change Chart Types: Switch between bar, line, pie charts
• Customize Colors: Apply your organization's color scheme
• Add Filters: Include dropdown filters for interactive analysis

Dashboard Not Showing Data:

• Verify data was successfully uploaded to Splunk
• Check that the index and sourcetype match your configuration
• Ensure sufficient data exists in the specified time range

XML Import Errors:

• Validate XML syntax in the generated file
• Check Splunk permissions for dashboard creation
• Verify compatible Splunk version (dashboard format)

Search Query Issues:

• Confirm field names match extracted data
• Test individual search queries in Splunk's Search app
• Check for case sensitivity in field values

Add new log types by extending the log_patterns dictionary in LogProcessor:

'custom_type': {
    'pattern': r'your_regex_pattern_here',
    'fields': ['field1', 'field2', 'field3'],
    'description': 'Description of your custom log type'
}

The dashboard generator creates type-specific visualizations. You can extend generate_dashboard_xml() for custom chart types.

• Detection Algorithm: Multi-pattern scoring system with confidence thresholds
• Performance: Processes 10,000+ log lines in seconds
• Memory Efficient: Streaming processing for large files
• Error Resilience: Continues processing even with malformed log entries

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

If you encounter any issues:

1. Check the Issues page
2. Create a new issue with:
   • Error message/logs
   • Sample log format
   • Environment details (Python version, OS, Splunk version)

This project is licensed under the MIT License - see the LICENSE file for details.

Q: What log formats are supported?
A: Currently supports firewall, DNS, Apache, syslog, and JSON formats. New patterns can be easily added.

Q: Can I run this on Windows/macOS/Linux?
A: Yes, the tool is cross-platform compatible.

Q: Does this work with Splunk Cloud?
A: Yes, just update the SPLUNK_HOST to your Splunk Cloud instance.

Q: Can I process multiple log files?
A: Currently supports single files. Batch processing can be added as an enhancement.

• Security Analysis: Automated firewall log monitoring
• Web Traffic Analysis: Apache log insights and anomaly detection
• Network Monitoring: DNS query pattern analysis
• System Administration: Centralized syslog analysis
• Compliance Reporting: Automated log processing for audits

• Real-time log streaming
• Machine learning anomaly detection
• Multiple file processing
• REST API interface
• Docker containerization
• Kubernetes deployment scripts