This folder contains an ADBC driver for Databricks written in C#.

At this time pre-packaged drivers are not yet available. See Building for instructions on building from source.

The Databricks ADBC driver is a C# implementation built on the Apache Spark ADBC driver foundation. It adds Databricks-specific functionality including:

• OAuth Authentication: Support for personal access tokens and client credentials flow
• CloudFetch: High-performance result retrieval directly from cloud storage
• Enhanced Performance: Optimized batch sizes and polling intervals for Databricks SQL
• Comprehensive Metadata: Primary key, foreign key, and catalog support

using Apache.Arrow.Adbc.Drivers.Databricks;

// Create driver and database
var driver = new DatabricksDriver();
var database = driver.Open(new Dictionary<string, string>
{
    ["uri"] = "https://your-workspace.cloud.databricks.com/sql/1.0/warehouses/your-warehouse-id",
    ["adbc.spark.auth_type"] = "oauth",
    ["adbc.databricks.oauth.grant_type"] = "access_token",
    ["adbc.spark.oauth.access_token"] = "your-personal-access-token"
});

// Execute query
using var connection = database.Connect();
using var statement = connection.CreateStatement();
statement.SqlQuery = "SELECT * FROM main.default.my_table LIMIT 10";
using var reader = statement.ExecuteQuery();

// Process results
while (reader.ReadNextRecordBatch())
{
    // Process Arrow RecordBatch
}

The driver supports three flexible configuration approaches:

Pass properties directly when creating connections.

You can use either a full URI or separate host/path parameters:

// Option 1: Using full URI
var config = new Dictionary<string, string>
{
    ["uri"] = "https://workspace.databricks.com/sql/1.0/warehouses/...",
    ["adbc.spark.auth_type"] = "oauth",
    ["adbc.databricks.oauth.grant_type"] = "client_credentials",
    ["adbc.databricks.oauth.client_id"] = "your-client-id",
    ["adbc.databricks.oauth.client_secret"] = "your-client-secret"
};

// Option 2: Using separate host and path (from Spark driver)
var config = new Dictionary<string, string>
{
    ["adbc.spark.type"] = "http",
    ["adbc.spark.host"] = "workspace.databricks.com",
    ["adbc.spark.port"] = "443",
    ["adbc.spark.path"] = "/sql/1.0/warehouses/...",
    ["adbc.spark.auth_type"] = "oauth",
    ["adbc.databricks.oauth.grant_type"] = "client_credentials",
    ["adbc.databricks.oauth.client_id"] = "your-client-id",
    ["adbc.databricks.oauth.client_secret"] = "your-client-secret"
};

Create a JSON configuration file (all values must be strings).

You can use either a full URI or separate host/path parameters:

{
  "uri": "https://workspace.databricks.com/sql/1.0/warehouses/...",
  "adbc.spark.auth_type": "oauth",
  "adbc.databricks.oauth.grant_type": "client_credentials",
  "adbc.databricks.oauth.client_id": "your-client-id",
  "adbc.databricks.oauth.client_secret": "your-client-secret",
  "adbc.connection.catalog": "main",
  "adbc.connection.db_schema": "default"
}

Or using separate host and path (from Spark driver):

{
  "adbc.spark.type": "http",
  "adbc.spark.host": "workspace.databricks.com",
  "adbc.spark.port": "443",
  "adbc.spark.path": "/sql/1.0/warehouses/...",
  "adbc.spark.auth_type": "oauth",
  "adbc.databricks.oauth.grant_type": "client_credentials",
  "adbc.databricks.oauth.client_id": "your-client-id",
  "adbc.databricks.oauth.client_secret": "your-client-secret",
  "adbc.connection.catalog": "main",
  "adbc.connection.db_schema": "default"
}

Set the environment variable:

export DATABRICKS_CONFIG_FILE="/path/to/config.json"

Combine both methods with the adbc.databricks.driver_config_take_precedence property:

• true: Environment config overrides constructor properties
• false (default): Constructor properties override environment config

var config = new Dictionary<string, string>
{
    ["uri"] = "https://workspace.databricks.com/sql/1.0/warehouses/...",
    ["adbc.spark.auth_type"] = "oauth",
    ["adbc.databricks.oauth.grant_type"] = "access_token",
    ["adbc.spark.oauth.access_token"] = "your-personal-access-token"
};

var config = new Dictionary<string, string>
{
    ["uri"] = "https://workspace.databricks.com/sql/1.0/warehouses/...",
    ["adbc.spark.auth_type"] = "oauth",
    ["adbc.databricks.oauth.grant_type"] = "client_credentials",
    ["adbc.databricks.oauth.client_id"] = "your-client-id",
    ["adbc.databricks.oauth.client_secret"] = "your-client-secret",
    ["adbc.databricks.oauth.scope"] = "sql"  // Optional, defaults to "sql"
};

Note: OAuth U2M (User-to-Machine) authentication support is planned for a future release.

Authentication Properties:

Note: Basic username/password authentication is not supported.

Note: Either uri or the combination of adbc.spark.host + adbc.spark.path is required.

Performance Notes:

• Databricks default batch_size is 2000000 (vs Spark's 50000) - optimized for CloudFetch's 1024MB limit
• Databricks default polltime_ms is 100 (vs Spark's 500) - faster query status feedback

CloudFetch is Databricks' high-performance result retrieval system that downloads results directly from cloud storage. It's enabled by default and provides significant performance improvements for large result sets.

CloudFetch Properties:

Example Configuration:

var config = new Dictionary<string, string>
{
    // ... auth config ...
    ["adbc.databricks.cloudfetch.parallel_downloads"] = "5",
    ["adbc.databricks.cloudfetch.memory_buffer_size_mb"] = "500",
    ["adbc.databricks.cloudfetch.prefetch_count"] = "4"
};

The driver supports both Thrift/HiveServer2 protocol (default) and Statement Execution REST API for query execution.

Protocol Properties:

REST API Configuration:

Example - Using REST Protocol:

var config = new Dictionary<string, string>
{
    ["uri"] = "https://workspace.databricks.com/sql/1.0/warehouses/...",
    ["adbc.spark.auth_type"] = "oauth",
    ["adbc.databricks.oauth.grant_type"] = "access_token",
    ["adbc.spark.oauth.access_token"] = "your-token",
    ["adbc.databricks.protocol"] = "rest",
    ["adbc.databricks.warehouse_id"] = "your-warehouse-id",
    ["adbc.databricks.rest.result_disposition"] = "inline_or_external_links"
};

For debugging HTTP/HTTPS traffic, you can use mitmproxy with custom certificate configuration:

var config = new Dictionary<string, string>
{
    ["uri"] = "https://workspace.databricks.com/sql/1.0/warehouses/...",
    ["adbc.spark.auth_type"] = "oauth",
    ["adbc.databricks.oauth.grant_type"] = "access_token",
    ["adbc.spark.oauth.access_token"] = "your-token",

    // Configure to trust mitmproxy's certificate
    ["adbc.http_options.tls.trusted_certificate_path"] = "/path/to/mitmproxy-ca-cert.pem",
    // Or disable certificate validation (not recommended for production)
    ["adbc.http_options.tls.disable_server_certificate_validation"] = "true"
};

Steps:

1. Install and start mitmproxy: mitmproxy -p 8080
2. Export mitmproxy's CA certificate from ~/.mitmproxy/mitmproxy-ca-cert.pem
3. Configure your system to route traffic through the proxy (e.g., set HTTP_PROXY environment variable)
4. Use trusted_certificate_path to trust mitmproxy's certificate, or disable validation for testing

The driver automatically converts Databricks/Spark types to Apache Arrow types:

Data Type Conversion Mode:

• adbc.spark.data_type_conv = scalar (default): Converts DATE, DECIMAL, TIMESTAMP to native types
• adbc.spark.data_type_conv = none: No conversion, returns raw string representations

Properties prefixed with adbc.databricks.ssp_ are passed to the Databricks server as session configuration.

The behavior is controlled by adbc.databricks.apply_ssp_with_queries:

• false (default): Server-side properties are applied in the Thrift configuration when the session is opened
• true: Server-side properties are applied using SET queries during execution

Example:

var config = new Dictionary<string, string>
{
    ["adbc.databricks.ssp_use_cached_result"] = "true",
    ["adbc.databricks.apply_ssp_with_queries"] = "false"  // Apply during session open (default)
};

// With apply_ssp_with_queries = false:
// Properties are included in the Thrift session configuration

// With apply_ssp_with_queries = true:
// Results in: SET use_cached_result=true (executed as a query)

The driver includes built-in tracing support via OpenTelemetry.

Tracing Properties:

Enable File-Based Tracing:

Via environment variable:

export OTEL_TRACES_EXPORTER=adbcfile

Or via configuration:

var config = new Dictionary<string, string>
{
    ["adbc.traces.exporter"] = "adbcfile"
};

Trace File Locations:

• Windows: %LOCALAPPDATA%\Apache.Arrow.Adbc\Traces
• macOS: ~/Library/Application Support/Apache.Arrow.Adbc/Traces
• Linux: ~/.local/share/Apache.Arrow.Adbc/Traces

Files rotate automatically with a pattern: apache.arrow.adbc.drivers.databricks-<YYYY-MM-DD-HH-mm-ss-fff>-<process-id>.log

Default: 999 files maximum, 1024 KB each.

The driver includes a comprehensive test suite using xUnit framework.

Before running tests, you need to create a test configuration file.

Create a JSON file with your Databricks connection details. All values must be strings.

Option A: Using Personal Access Token (Recommended for local testing)

Create databricks-test-config.json:

{
  "uri": "https://your-workspace.cloud.databricks.com/sql/1.0/warehouses/your-warehouse-id",
  "adbc.spark.auth_type": "oauth",
  "adbc.databricks.oauth.grant_type": "access_token",
  "adbc.spark.oauth.access_token": "dapi1234567890abcdef",
  "adbc.connection.catalog": "main",
  "adbc.connection.db_schema": "default"
}

Option B: Using OAuth Client Credentials (M2M)

Create databricks-test-config.json:

{
  "uri": "https://your-workspace.cloud.databricks.com/sql/1.0/warehouses/your-warehouse-id",
  "adbc.spark.auth_type": "oauth",
  "adbc.databricks.oauth.grant_type": "client_credentials",
  "adbc.databricks.oauth.client_id": "your-client-id",
  "adbc.databricks.oauth.client_secret": "your-client-secret",
  "adbc.databricks.oauth.scope": "sql",
  "adbc.connection.catalog": "main",
  "adbc.connection.db_schema": "default"
}

Option C: Using Separate Host and Path (Alternative)

Create databricks-test-config.json:

{
  "adbc.spark.type": "http",
  "adbc.spark.host": "your-workspace.cloud.databricks.com",
  "adbc.spark.port": "443",
  "adbc.spark.path": "/sql/1.0/warehouses/your-warehouse-id",
  "adbc.spark.auth_type": "oauth",
  "adbc.databricks.oauth.grant_type": "access_token",
  "adbc.spark.oauth.access_token": "dapi1234567890abcdef",
  "adbc.connection.catalog": "main",
  "adbc.connection.db_schema": "default"
}

Configuration Notes:

• Save the file in a secure location (e.g., ~/.databricks/test-config.json)
• Never commit this file to version control - add it to .gitignore
• All property values must be strings (quoted), including numbers
• The catalog and db_schema specify which database the tests will use
• Ensure your user/service principal has access to the specified warehouse and catalog

Point the DATABRICKS_TEST_CONFIG_FILE environment variable to your config file:

On Windows (PowerShell):

$env:DATABRICKS_TEST_CONFIG_FILE="C:\Users\YourName\.databricks\test-config.json"

On Windows (Command Prompt):

set DATABRICKS_TEST_CONFIG_FILE=C:\Users\YourName\.databricks\test-config.json

On macOS/Linux:

export DATABRICKS_TEST_CONFIG_FILE="$HOME/.databricks/test-config.json"

Make it Permanent (macOS/Linux):

Add to your shell profile (~/.bashrc, ~/.zshrc, or ~/.bash_profile):

export DATABRICKS_TEST_CONFIG_FILE="$HOME/.databricks/test-config.json"

Then reload: source ~/.bashrc (or ~/.zshrc)

Verify Configuration:

# Check the variable is set
echo $DATABRICKS_TEST_CONFIG_FILE  # macOS/Linux
echo %DATABRICKS_TEST_CONFIG_FILE%  # Windows CMD
$env:DATABRICKS_TEST_CONFIG_FILE   # Windows PowerShell

# Verify the file exists and is readable
cat $DATABRICKS_TEST_CONFIG_FILE  # macOS/Linux
type %DATABRICKS_TEST_CONFIG_FILE%  # Windows CMD
Get-Content $env:DATABRICKS_TEST_CONFIG_FILE  # Windows PowerShell

Note: The test environment variable is DATABRICKS_TEST_CONFIG_FILE, which is different from the driver's runtime environment variable DATABRICKS_CONFIG_FILE. This separation allows you to use different configurations for testing vs. production use.

From the test project directory:

cd csharp/test
dotnet test

Run tests for a specific framework:

# Test on .NET 8.0
dotnet test -f net8.0

# Test on .NET Framework 4.7.2 (Windows only)
dotnet test -f net472

Run a specific test class:

dotnet test --filter "FullyQualifiedName~TracingDelegatingHandlerTest"

Run a specific test method:

dotnet test --filter "FullyQualifiedName~TracingDelegatingHandlerTest.TestTracePropagation"

Run tests matching a pattern:

# Run all tests with "CloudFetch" in the name
dotnet test --filter "FullyQualifiedName~CloudFetch"

Get detailed test output:

dotnet test --logger "console;verbosity=detailed"

1. Open Solution

   • Open csharp/Apache.Arrow.Adbc.Drivers.Databricks.sln in Visual Studio

2. Set Environment Variable

   • Right-click on the test project → Properties
   • Navigate to Debug → General → Open debug launch profiles UI
   • Add environment variable:
     • Name: DATABRICKS_TEST_CONFIG_FILE
     • Value: C:\path\to\your\config.json

1. Test Explorer

   • Open Test Explorer: Test → Test Explorer (or Ctrl+E, T)
   • Build the solution to discover all tests
   • Tests will appear grouped by namespace and class

2. Run/Debug Tests

   • Run a test: Right-click test → Run
   • Debug a test: Right-click test → Debug
   • Run all tests: Click "Run All" in Test Explorer
   • Run tests in a class: Right-click class → Run

3. Set Breakpoints

   • Open the test file or source code
   • Click in the left margin to set breakpoints
   • Debug the test - execution will pause at breakpoints
   • Use F10 (Step Over), F11 (Step Into), F5 (Continue)

• Filter tests: Use the search box in Test Explorer to filter by name
• Group by: Use the grouping dropdown to organize by Class, Namespace, or Project
• Run Failed Tests: After a test run, right-click failed tests → Run
• Live Unit Testing: Test → Live Unit Testing → Start (Enterprise edition)

Tests run automatically in CI:

• On every push to main
• On every pull request
• Multiple frameworks: .NET 8.0 (Linux) and .NET Framework 4.7.2 (Windows)

The C# driver includes a comprehensive benchmark suite for CloudFetch performance testing. For complete documentation including:

• How to run benchmarks locally and in CI
• Benchmark query details and characteristics
• Viewing results on GitHub Pages
• Understanding benchmark metrics and comparisons

See the Benchmark Documentation.

See CONTRIBUTING.md.

See CONTRIBUTING.md.