A fast, flexible, and memory-efficient command-line tool written in Rust for processing large CSV files. Inspired by xsv and built on Polars, it's designed for handling tens or hundreds of gigabytes of CSV data efficiently in workflows like log analysis and digital forensics.

• Pipeline-style command chaining: Chain multiple commands in a single line for fast and efficient data processing
• Flexible filtering and transformation: Perform operations like select, filter, sort, deduplicate, and timezone conversion
• YAML-based batch processing (Quilt): Automate complex workflows using YAML configuration files

To see available commands and options, run qsv without any arguments:

$ qsv -h

Here's an example of reading a CSV file, extracting rows that contain 4624 in the 'Event ID' column, and displaying the top 3 rows sorted by the 'Date and Time' column:

$ qsv load Security.csv - isin 'Event ID' 4624 - sort 'Date and Time' - head 3 - showtable

This command:

1. Loads Security.csv
2. Filters rows where Event ID is 4624
3. Sorts by Date and Time
4. Shows the first 3 rows as a table

qsv commands are composed of three types of steps:

• Initializer: Loads data (e.g., load)
• Chainable: Transforms or filters data (e.g., select, grep, sort, etc.)
• Finalizer: Outputs or summarizes data (e.g., show, showtable, headers, etc.)

Each step is separated by a hyphen (-):

$ qsv <INITIALIZER> <args> - <CHAINABLE> <args> - <FINALIZER> <args>

The - token (a single hyphen surrounded by spaces) is the command separator. A standalone - is never treated as data.

• To separate commands: qsv load file.csv - select col1 - head 5
• If you need - as an option value, use an attached form such as --separator=- or -s-.

A standalone - positional value, including stdin-style usage, is not currently supported.

Note: If no finalizer is explicitly specified, default builds automatically use showtable, making it easy to quickly view results:

$ qsv load data.csv - select col1,col2 - head 5
# Equivalent to:
$ qsv load data.csv - select col1,col2 - head 5 - showtable

Builds compiled without the optional table feature fall back to show instead, and the showtable command prints a rebuild hint.

Load one or more CSV or Parquet files.

Supported formats:

• CSV files (.csv, .tsv, .txt)
• Gzipped CSV files (.csv.gz)
• Parquet files (.parquet) - high performance, preserves data types

Environment Variables:

• QSV_CHUNK_SIZE: Default chunk size for CSV processing (overrides auto-detection, can be overridden by --chunk-size)
• QSV_MEMORY_LIMIT_MB: Memory limit for gzip decompression and streaming operations (default: 1024MB, range: 512-4096MB)

Example:

$ qsv load data.csv
$ qsv load data.csv.gz
$ qsv load data1.csv data2.csv data3.csv
$ qsv load "logs/*.tsv" -s $'\t'
$ qsv load "logs/*.tsv" --separator=$'\t'
$ qsv load data.csv --low-memory
$ qsv load data.csv --no-headers
$ qsv load data.csv --chunk-size 50000
$ qsv load cache.parquet                              # Load from parquet cache
$ qsv load cache1.parquet cache2.parquet              # Load multiple parquet files

Select columns by name, numeric index, or range notation.

Column Selection Formats:

• Individual columns: col1,col3 - Select specific columns by name
• Numeric indices: 1,3 - Select columns by position (1-based indexing)
• Range notation (hyphen): col1-col3 - Select range using hyphen
• Range notation (colon): col1:col3 - Select range using colon
• Numeric range: 2:4 - Select 2nd through 4th columns (e.g., col1, col2, col3)
• Quoted colon notation: "col:1":"col:3" - For column names containing colons
• Mixed formats: 1,col2,4:6 - Combine different selection methods

Disambiguation rule: If an exact column name matching col1-col3 exists, it is selected as-is. Range expansion only occurs when no exact match is found.

$ qsv load data.csv - select datetime                       # Select single column by name
$ qsv load data.csv - select col1,col3                      # Select specific columns by name
$ qsv load data.csv - select col1-col3                      # Select range using hyphen
$ qsv load data.csv - select col1:col3                      # Select range using colon
$ qsv load data.csv - select 1                              # Select 1st column (datetime)
$ qsv load data.csv - select 2:4                            # Select 2nd-4th columns (col1, col2, col3)
$ qsv load data.csv - select 2,4                            # Select 2nd and 4th columns (col1, col3)
$ qsv load data.csv - select "col:1":"col:3"                # For columns with colons in names
$ qsv load data.csv - select 1,datetime,3:5                 # Mixed selection methods

Filter rows where a column matches any of the given values.

$ qsv load data.csv - isin col1 1
$ qsv load data.csv - isin col1 1,4

Filter rows where a column contains a specific literal substring.

$ qsv load data.csv - contains str ba
$ qsv load data.csv - contains str BA -i
$ qsv load data.csv - contains str BA --ignorecase

Replace values in column(s) using a Regex pattern.

Warning: When --column is omitted, sed replaces across all columns. In log/DFIR data this can silently modify timestamps, EventIDs, file paths, and usernames. Always specify --column unless you intend a full-dataset replacement.

$ qsv load data.csv - sed foo foooooo                       # Replace 'foo' with 'foooooo' in all columns
$ qsv load data.csv - sed foo foooooo --column str          # Replace 'foo' with 'foooooo' in 'str' column only
$ qsv load data.csv - sed FOO foooooo -i                    # Case-insensitive replacement in all columns
$ qsv load data.csv - sed ".*o.*" foooooo --column str      # Regex replacement in specific column

Filter rows where any column matches a regex pattern.

Example:

$ qsv load data.csv - grep foo 
$ qsv load data.csv - grep "^FOO" -i                        # Case-insensitive search
$ qsv load data.csv - grep "^FOO" --ignore-case              # Long form case-insensitive
$ qsv load data.csv - grep "^FOO" -i -v                     # Case-insensitive inverted match
$ qsv load data.csv - grep "^FOO" --ignore-case --invert-match  # Long form inverted match
$ qsv load logs.csv - grep "FAILED" --column EventData
$ qsv load logs.csv - grep "192\\.168\\." --column src_ip,dst_ip

Displays the first N rows of the dataset.

$ qsv load data.csv - head 3
$ qsv load data.csv - head 10
$ qsv load data.csv - head -n 3
$ qsv load data.csv - head --number 10

Displays the last N rows of the dataset.

$ qsv load data.csv - tail 3
$ qsv load data.csv - tail 10
$ qsv load data.csv - tail -n 3
$ qsv load data.csv - tail --number 10

Sorts the dataset based on the specified column(s).

⚠️ Memory: This command materializes the full dataset into memory.

$ qsv load data.csv - sort str
$ qsv load data.csv - sort str -d
$ qsv load data.csv - sort str --desc
$ qsv load data.csv - sort col1,col2,col3 --desc

Count duplicate rows, grouping by all columns by default. Results are automatically sorted by count in descending order.

⚠️ Memory: This command materializes the full dataset into memory.

$ qsv load Security.csv - count EventID          # Count by one column
$ qsv load proxy.csv - count src_ip,dst_ip       # Count by multiple columns
$ qsv load data.csv - count                       # Count all unique rows (original behavior)
$ qsv load data.csv - count - sort col1          # Count and then sort by col1 instead

Filters unique rows, removing duplicates based on all columns.

⚠️ Memory: This command materializes the full dataset into memory.

$ qsv load data.csv - uniq

Changes the timezone of a datetime column.

Understanding --ambiguous option:

During Daylight Saving Time (DST) transitions in autumn, clocks "fall back" creating duplicate hours. For example, 2:30 AM occurs twice:

• First time: 2:30 AM DST (before transition)
• Second time: 2:30 AM Standard Time (after transition)

When encountering such ambiguous times:

• earliest: Uses the first occurrence (DST time)
• latest: Uses the second occurrence (Standard time)

Example:

$ qsv load data.csv - changetz datetime --from-tz UTC --to-tz Asia/Tokyo
# Output: 2023-01-01T09:00:00.123456+09:00 (ISO8601 with microsecond precision)

$ qsv load data.csv - changetz datetime --from-tz UTC --to-tz America/New_York --input-format "%Y/%m/%d %H:%M" --output-format "%Y-%m-%d %H:%M:%S"
# Custom output format

$ qsv load data.csv - changetz datetime --from-tz America/New_York --to-tz UTC --ambiguous latest
# Handle ambiguous DST times

# Automatic format detection (similar to Python dateutil.parser):
$ qsv load logs.csv - changetz timestamp --from-tz local --to-tz UTC
# Handles: "Jan 15, 2023 2:30 PM", "2023/01/15 14:30", "15-Jan-2023 14:30:00", etc.

# Fuzzy parsing with embedded text:
$ qsv load events.csv - changetz event_time --from-tz EST --to-tz UTC  
# Handles: "Meeting on January 15th, 2023 at 2:30 PM", "Call scheduled for Jan 15 2023"

TODO: Upgrade to 7-digit sub-second precision (100-nanosecond precision for Windows FILETIME compatibility) when chrono-tz library supports it. Current auto output uses microsecond precision.

Renames a specific column.

$ qsv load data.csv - renamecol current_name new_name

Converts data formats between JSON, YAML, and XML. Also supports formatting/prettifying data in the same format.

Supported conversions:

• Cross-format: json ↔ yaml, json ↔ xml, yaml ↔ xml
• Same-format (formatting): json → json, yaml → yaml, xml → xml

Features:

• Automatically handles malformed JSON with extra quotes
• Prettifies and formats data for better readability
• Preserves data structure during conversion

Example:

$ qsv load data.csv - convert json_col --from json --to yaml
$ qsv load data.csv - convert config --from yaml --to json
$ qsv load data.csv - convert data --from json --to xml
$ qsv load data.csv - convert messy_json --from json --to json  # Format/prettify JSON
$ qsv load data.csv - convert compact_yaml --from yaml --to yaml  # Format YAML

Aggregates data by time intervals, creating time-based summaries.

Features:

• Creates a time bucket column named timeline_{interval} (e.g., timeline_1h, timeline_30m)
• If no aggregation column is specified, only row counts are provided for each time bucket
• Supports various time interval formats: hours (1h), minutes (30m), seconds (5s), days (1d)

CLI and Quilt YAML mapping:

Example:

$ qsv load access.log - timeline timestamp --interval 1h
# Creates column: timeline_1h

$ qsv load metrics.csv - timeline time --interval 5m --avg cpu_usage
# Creates columns: timeline_5m, count, avg_cpu_usage

$ qsv load sales.csv - timeline date --interval 1d --sum amount
# Creates columns: timeline_1d, count, sum_amount

$ qsv load server.log - timeline timestamp --interval 30s --max response_time
# Creates columns: timeline_30s, count, max_response_time

Filters data based on time ranges, extracting records within specified time boundaries.

At least one of --start or --end must be specified. Both boundaries are inclusive ([start, end]). Supports various datetime formats including ISO8601, timestamps, and common log formats.

Example:

$ qsv load data.csv - timeslice timestamp --start "2023-01-01 00:00:00"
$ qsv load data.csv - timeslice timestamp --end "2023-12-31 23:59:59"
$ qsv load data.csv - timeslice timestamp --start "2023-06-01" --end "2023-06-30"
$ qsv load access.log - timeslice timestamp --start "2023-01-01T10:00:00"

Creates grouped aggregations over row and column keys.

Note: pivot currently performs a grouped aggregation (long-form output), not an Excel-style wide cross-tabulation. If you need wide output, use select + count or wait for the planned --wide flag. Alternatives that better describe this operation: groupby, aggregate, summarize.

At least one of --rows or --cols must be specified.

Example:

$ qsv load sales.csv - pivot --rows region --cols product --values sales_amount --agg sum
$ qsv load data.csv - pivot --rows category --cols year --values revenue --agg mean
$ qsv load logs.csv - pivot --rows date --cols error_type --values count --agg count
$ qsv load metrics.csv - pivot --rows department --values performance --agg median

Rounds datetime values to specified time units, creating a new rounded column while preserving the original.

Features:

• Rounds datetime values down to the nearest specified time unit boundary
• Useful for time-based grouping and analysis
• Supports both short (h, d) and long (hour, day) unit names
• Output format automatically adjusts to the specified unit (clean, minimal format)

Output formats by unit:

• year (y): 2023
• month (M): 2023-01
• day (d): 2023-01-01
• hour (h): 2023-01-01 12
• minute (m): 2023-01-01 12:34
• second (s): 2023-01-01 12:34:56

Example:

$ qsv load data.csv - timeround timestamp --unit d --output date_only
# Input:  2023-01-01 12:34:56
# Output: 2023-01-01

$ qsv load data.csv - timeround timestamp --unit h --output hour_rounded
# Input:  2023-01-01 12:34:56
# Output: 2023-01-01 12

$ qsv load logs.csv - timeround timestamp --unit m
# Rounds to minute boundary, writes to timestamp_rounded

$ qsv load metrics.csv - timeround created_at --unit year --output created_year
# Input:  2023-01-01 12:34:56
# Output: 2023

Finalizers are used to output or summarize the processed data. They are typically the last command in a chain.

Splits data into separate CSV files based on unique values in a specified column. Each unique value creates its own file.

The output directory will be created if it doesn't exist. Each file is named after the unique value in the partition column (with invalid filename characters replaced by underscores).

Example:

$ qsv load data.csv - partition category                    # Uses default ./partitions/ directory
$ qsv load data.csv - partition category ./partitions/      # Explicit directory
$ qsv load sales.csv - partition region ./by_region/
$ qsv load logs.csv - partition date ./daily_logs/
$ qsv load data.csv - select col1,col2 - partition col1 ./numeric_partitions/

Displays the column headers of the current dataset.

Example:

$ qsv load data.csv - headers
$ qsv load data.csv - headers -p
$ qsv load data.csv - headers --plain

Displays summary statistics for each column in the dataset (e.g., count, null_count, mean, std, min, max).

This command does not take any arguments or options.

Example:

$ qsv load data.csv - stats

Displays the Polars LazyFrame query plan. This is useful for debugging and understanding the operations being performed.

This command does not take any arguments or options.

Example:

$ qsv load data.csv - select col1 - showquery

Displays the resulting data as CSV to standard output. Header is included by default.

Example:

$ qsv load data.csv - head 5 - show
$ qsv load huge.csv - show --batch-size 2GB                 # Streaming mode for large files
$ qsv load data.csv - select col1,col2 - show --batch-size 512MB

Displays the resulting data in a formatted table to standard output. Shows table dimensions and intelligently truncates large datasets.

Features:

• Displays table size information (rows × columns) like Python Polars
• For datasets with 9+ rows: shows the first 8 rows and a truncation indicator (⋮)
• For datasets with 8 or fewer rows: shows all rows without truncation
• Automatically used as default finalizer when no explicit finalizer is specified

This command does not take any arguments or options. This command is controlled by the optional cargo feature table, which is enabled in the default build.

Tip for large files: Pipe through head N before showtable, or use show instead. showtable with implicit finalization collects all rows by default.

Example:

$ qsv load data.csv - select col1,col2 - head 3 - showtable
# Output includes: shape: (3, 2) followed by formatted table

$ qsv load large_data.csv - select col1,col2
# Automatically calls showtable if no finalizer specified

To build a smaller binary without table rendering support:

$ cargo build --release --no-default-features

In that build, showtable exits with a clear rebuild message and implicit finalization falls back to show.

Outputs the processing results to a CSV file.

Example:

$ qsv load data.csv - dump                                  # Saves to dump_<timestamp>.csv
$ qsv load data.csv - head 100 - dump -o results.csv
$ qsv load data.csv - head 100 - dump --output results.csv
$ qsv load data.csv - head 100 - dump -o results.csv -s ';'
$ qsv load huge.csv - dump -o output.csv --batch-size 2GB   # Streaming mode for large files

Saves the processing results as a Parquet cache file for fast reloading.

Features:

• Saves DataFrame as compressed Parquet format
• Preserves data types (unlike CSV)
• High-performance for large datasets
• Can be loaded back using the load command

Example:

$ qsv load data.csv - head 100 - dumpcache                 # Auto-named cache file
$ qsv load data.csv - select col1,col2 - dumpcache -o cache.parquet
$ qsv load data.csv - sort col1 - dumpcache --output processed_data

# Load from cache for fast access
$ qsv load cache.parquet - show

Quilt allows you to define complex data processing workflows in YAML configuration files. This is useful for automating repetitive tasks or creating reusable data processing pipelines.

The quilt command itself takes the path to a YAML configuration file. Input data sources and other parameters are typically defined within the YAML file.

$ qsv quilt <config_file_path.yaml> [options]

$ qsv quilt rules/my_workflow.yaml
$ qsv quilt rules/my_analysis.yaml -o custom_output.csv

The YAML configuration file (e.g., rules/my_workflow.yaml) defines the stages and steps. For example, the Sample YAML (rules/test.yaml) below defines a pipeline that:

1. Loads data (implicitly or explicitly via a load step in a process stage).
2. Performs selections and a join operation across different stages.
3. Displays the final result as a table.

Within a Quilt YAML file, stages can be of different types to orchestrate the flow.

sigma2quilt converts Zircolite JSON rules into a regular quilt YAML file. The generated quilt uses normal process, concat, and output stages, and each rule becomes a where step over ${input}.

• A single-rule JSON file defaults to quilt-<rule-title>.yaml
• Rule titles are converted to lowercase hyphen-joined filenames
• --separate writes one quilt file per rule
• Each conversion also writes one _mapping.json template for the whole input ruleset
• --annotate is mainly useful when multiple rules are kept in one generated quilt and you want the matched rows to retain per-rule metadata
• For rules_dir/ input, -o <dir> is required

Supported SQL in the generated where step:

1. =
2. LIKE ... ESCAPE '\'
3. NOT (...)
4. AND
5. OR

Field resolution is:

1. Explicit mapping via qsv quilt --mapping <file> or params.field_map
2. Exact CSV column match
3. Case-insensitive CSV column match
4. Otherwise warn and skip that condition

title: Sigma JSON Conversion: rules_windows_generic

stages:
  load_stage:
    type: process
    steps:
      load:
        path: ${input}

  detect_1_suspicious_high_integritylevel_conhost_legacy_option:
    type: process
    source: load_stage
    steps:
      where:
        sql: "SELECT * FROM logs WHERE Channel='Security' AND EventID=4688"
        annotate: true
        sigma_title: "Suspicious High IntegrityLevel Conhost Legacy Option"
        sigma_id: "3037d961-21e9-4732-b27a-637bcc7bf539"
        sigma_level: "informational"
        sigma_tags: "attack.defense-evasion,attack.t1202"

  output_stage:
    type: output
    source: detect_1_suspicious_high_integritylevel_conhost_legacy_option
    steps:
      dump:
        output: ${output}

Generated mapping template example:

{
  "Channel": "",
  "CommandLine": "",
  "EventID": ""
}

Recommended flow:

1. Run sigma2quilt to generate both the quilt YAML and _mapping.json
2. Fill in the CSV column names inside _mapping.json
3. Run quilt --mapping <generated_mapping.json>

Examples:

$ qsv sigma2quilt rules_windows_generic.json
$ qsv sigma2quilt rules_windows_generic.json -o custom.yaml
$ qsv sigma2quilt rules_dir/ -o generated_quilts/
$ qsv sigma2quilt rules_windows_generic.json --annotate
$ qsv sigma2quilt rules_windows_generic.json --separate -o generated_quilts/
$ qsv quilt quilt-rules_windows_generic.yaml --mapping quilt-rules_windows_generic_mapping.json --var input=events.csv --var output=alerts.csv

Timeline steps in Quilt use explicit aggregation keys:

stages:
  hourly_metrics:
    type: process
    steps:
      load:
        path: metrics.csv
      timeline:
        time_column: timestamp
        interval: 1h
        agg_type: avg
        agg_column: cpu_usage
      show:

qsv-rs supports streaming processing for huge files without loading them entirely into memory.

Not all commands stream. Before running a pipeline on a large file, check the memory behavior of each operation:

Warning: Running a materializing command on a multi-GB file may exhaust memory. Use head, timeslice, or isin to reduce the dataset first.

# Stream display huge files (1GB batches by default)
$ qsv load huge.csv - show

# Custom memory usage - 512MB batches
$ qsv load huge.csv - show --batch-size 512MB

# High-memory server - 2GB batches for maximum performance
$ qsv load huge.csv - show --batch-size 2GB

# Stream save large results to file with custom batch size
$ qsv load huge.csv - select important,columns - dump -o output.csv --batch-size 2GB

# Configure batch size for your system
--batch-size 512MB    # Low memory systems
--batch-size 1GB      # Default (balanced)
--batch-size 2GB      # High memory systems (2GB+)

# Configure gzip decompression memory (environment variable)
export QSV_MEMORY_LIMIT_MB=512   # Low memory systems
export QSV_MEMORY_LIMIT_MB=1024  # Default (1GB)
export QSV_MEMORY_LIMIT_MB=2048  # High memory systems (2GB+)

# Process large gzip files with different memory settings
$ QSV_MEMORY_LIMIT_MB=2048 qsv load huge.csv.gz - show
$ QSV_MEMORY_LIMIT_MB=512 qsv load huge.csv.gz - head 1000 - show  # Low memory

For repeated processing of large CSV files, convert to Parquet format for significantly faster loading.

Performance Benefits:

• Faster loading compared to CSV format
• Better compression (smaller file sizes)
• Preserves data types (no re-parsing needed)

# One-time conversion: CSV to Parquet cache
$ qsv load huge.csv - dumpcache -o huge.parquet

# Subsequent processing: Load from Parquet (much faster)
$ qsv load huge.parquet - select col1,col2 - show
$ qsv load huge.parquet - isin category "important" - dump -o result.csv

Download the latest release from GitHub Releases.

$ git clone https://github.com/sumeshi/qsv-rs.git
$ cd qsv-rs
$ cargo build --release

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

This project is licensed under the MIT License - see the LICENSE file for details.

Inspired by xsv.