Overview

Relevant source files

• CONTRIBUTING.md
• README.md
• docs/SUMMARY.md
• docs/getting-started/architecture/write-patterns.md
• docs/getting-started/components/online-store.md
• docs/getting-started/components/overview.md
• docs/getting-started/concepts/data-ingestion.md
• docs/getting-started/quickstart.md
• docs/getting-started/third-party-integrations.md
• docs/how-to-guides/feast-on-kubernetes.md
• docs/how-to-guides/running-feast-in-production.md
• docs/how-to-guides/scaling-feast.md
• docs/how-to-guides/structuring-repos.md
• docs/project/maintainers.md
• docs/reference/codebase-structure.md
• docs/reference/feast-cli-commands.md
• docs/reference/feature-servers/offline-feature-server.md
• docs/reference/registries/README.md
• docs/reference/registries/hdfs.md
• docs/reference/registries/metadata.md
• docs/roadmap.md
• examples/operator-postgres-tls-demo/02-Install-feast.ipynb
• examples/operator-postgres-tls-demo/03-Uninstall.ipynb
• examples/quickstart/quickstart.ipynb
• infra/feast-operator/README.md
• infra/feast-operator/api/v1/featurestore_types.go
• infra/feast-operator/config/samples/kustomization.yaml
• infra/templates/README.md.jinja2
• protos/feast/core/Registry.proto
• sdk/python/feast/driver_test_data.py
• sdk/python/feast/errors.py
• sdk/python/feast/feature_store.py
• sdk/python/feast/infra/offline_stores/bigquery.py
• sdk/python/feast/infra/offline_stores/offline_store.py
• sdk/python/feast/infra/offline_stores/offline_utils.py
• sdk/python/feast/infra/offline_stores/redshift.py
• sdk/python/feast/infra/online_stores/online_store.py
• sdk/python/feast/infra/online_stores/sqlite.py
• sdk/python/feast/infra/passthrough_provider.py
• sdk/python/feast/infra/provider.py
• sdk/python/feast/infra/registry/contrib/hdfs/__init__.py
• sdk/python/feast/infra/registry/contrib/hdfs/hdfs_registry_store.py
• sdk/python/feast/infra/utils/aws_utils.py
• sdk/python/feast/repo_config.py
• sdk/python/feast/repo_operations.py
• sdk/python/feast/utils.py
• sdk/python/tests/foo_provider.py
• sdk/python/tests/unit/infra/registry/test_sql_registry.py

Purpose and Scope

This document provides a technical overview of the Feast feature store system, covering its core architecture, key components, and main workflows. It introduces the primary code entities and their interactions to help developers understand how Feast operates as a unified system for managing and serving machine learning features.

For detailed information about specific subsystems:

• System architecture details: See System Architecture
• Core concepts and abstractions: See Core Concepts
• Data integration specifics: See Data Integration
• Feature serving mechanisms: See Feature Serving
• Deployment strategies: See Deployment

Sources: README.md26-38 sdk/python/feast/feature_store.py100-109

What is Feast?

Feast (Feature Store) is an open-source feature store for machine learning that manages the lifecycle of features from definition to serving. At its core, Feast provides infrastructure to:

1. Define features using Python abstractions (Entity, FeatureView, DataSource)
2. Store metadata about features in a Registry
3. Retrieve historical features from an OfflineStore for training
4. Materialize features to an OnlineStore for low-latency inference
5. Serve features via the FeatureStore class or dedicated feature servers

The system is designed to be modular and extensible, supporting multiple storage backends, data sources, and deployment patterns through a plugin architecture.

Sources: README.md29-38 sdk/python/feast/feature_store.py100-115

Core Architecture

The following diagram illustrates the primary code entities in Feast and their relationships:

Sources: sdk/python/feast/feature_store.py100-203 sdk/python/feast/repo_config.py193-256 sdk/python/feast/infra/provider.py49-63 sdk/python/feast/infra/passthrough_provider.py58-130

FeatureStore: The Central Orchestrator

The FeatureStore class (sdk/python/feast/feature_store.py100-189) is the primary interface for all Feast operations. It acts as a facade that:

• Loads configuration from feature_store.yaml via RepoConfig
• Initializes the appropriate Registry implementation based on configuration
• Creates a Provider instance to orchestrate infrastructure operations
• Provides methods for feature registration, materialization, and retrieval

Key attributes:

• config: A RepoConfig instance containing all configuration
• repo_path: Path to the feature repository
• _registry: The registry implementation (file, SQL, or remote)
• _provider: The provider implementation (typically PassthroughProvider)

Sources: sdk/python/feast/feature_store.py100-203

Configuration: RepoConfig and feature_store.yaml

The RepoConfig class (sdk/python/feast/repo_config.py193-557) defines the structure of feature_store.yaml and handles:

• Project identification and metadata
• Registry configuration (type, path, caching)
• Offline store configuration (BigQuery, Snowflake, File, etc.)
• Online store configuration (Redis, DynamoDB, SQLite, etc.)
• Batch engine configuration (Local, Lambda, Snowflake, etc.)
• Authentication settings
• Provider selection

The configuration uses Pydantic models with validation to ensure correctness at load time. The feature_store.yaml file is typically located at the repository root and loaded automatically when instantiating a FeatureStore.

Sources: sdk/python/feast/repo_config.py193-557

Registry: Metadata Storage

The registry stores all metadata about feature definitions, including entities, feature views, data sources, and feature services. The abstract BaseRegistry interface defines the contract, with multiple implementations:

Registry Type	Class	Storage Backend
File	Registry	Local filesystem or cloud object storage (S3, GCS)
SQL	SqlRegistry	PostgreSQL, MySQL, or other SQL databases
Snowflake	SnowflakeRegistry	Snowflake table
Remote	RemoteRegistry	Remote registry server via gRPC

The registry provides methods like:

• apply_entity(), apply_feature_view() - Register objects
• get_entity(), get_feature_view() - Retrieve objects
• list_entities(), list_feature_views() - List all objects
• refresh() - Update cached registry state

For production deployments, the SQL registry is recommended for its transactional guarantees and concurrent-write safety.

Sources: sdk/python/feast/feature_store.py155-177 sdk/python/feast/repo_config.py39-44 docs/how-to-guides/running-feast-in-production.md34-37

Provider: Infrastructure Orchestration

The Provider interface (sdk/python/feast/infra/provider.py49-105) abstracts infrastructure operations. The primary implementation is PassthroughProvider (sdk/python/feast/infra/passthrough_provider.py58-130), which delegates to:

1. Offline Store - For historical feature retrieval and point-in-time joins
2. Online Store - For low-latency feature serving
3. Compute Engine - For materialization (moving data from offline to online store)

The provider pattern allows Feast to support cloud-specific optimizations (GCP, AWS, Azure) while maintaining a consistent interface.

Sources: sdk/python/feast/infra/provider.py49-105 sdk/python/feast/infra/passthrough_provider.py58-175

Feature Definition Abstractions

The following diagram shows the relationships between feature definition objects:

Sources: sdk/python/feast/entity.py sdk/python/feast/feature_view.py sdk/python/feast/on_demand_feature_view.py sdk/python/feast/data_source.py sdk/python/feast/feature_service.py

Entity

An Entity defines a join key used to retrieve features. It specifies:

• name: Unique identifier
• join_keys: List of column names used for joining (typically one, e.g., ["driver_id"])
• value_type: Data type of the join key
• description: Human-readable description

Entities are referenced by feature views to establish relationships between features and the objects they describe.

Sources: README.md145-147 docs/getting-started/quickstart.md145-147

FeatureView

A FeatureView (sdk/python/feast/feature_view.py) is the core abstraction for defining features. It includes:

• name: Unique identifier
• entities: List of entity references
• schema: List of Field objects defining feature names and types
• source: The DataSource containing raw data
• ttl: Time-to-live for features in the online store
• online: Boolean indicating whether to materialize to online store

Subclasses include:

• StreamFeatureView: For real-time streaming features
• BatchFeatureView: For features with time-window aggregations

Sources: README.md162-182 docs/getting-started/quickstart.md162-182

OnDemandFeatureView

An OnDemandFeatureView (sdk/python/feast/on_demand_feature_view.py) defines features computed on-the-fly during retrieval using:

• sources: Dictionary mapping names to FeatureView or RequestSource objects
• schema: Output feature schema
• transformation: Python function or Pandas UDF for computing features
• write_to_online_store: Boolean to materialize computed features

On-demand transformations enable feature engineering logic to be version-controlled and reused across training and serving.

Sources: README.md184-214 docs/getting-started/quickstart.md184-214

DataSource

DataSource (sdk/python/feast/data_source.py) is an abstract base class representing where raw feature data resides. Implementations include:

DataSource	Module	Description
FileSource	file.py	Parquet/CSV files (local or cloud storage)
BigQuerySource	bigquery_source.py	Google BigQuery tables
SnowflakeSource	snowflake_source.py	Snowflake tables
RedshiftSource	redshift_source.py	AWS Redshift tables
KafkaSource	kafka_source.py	Kafka topics (streaming)
KinesisSource	kinesis_source.py	AWS Kinesis streams
PushSource	push_source.py	Push API for real-time ingestion

Each data source specifies:

• timestamp_field: Column containing event timestamps
• created_timestamp_column: Optional column for data versioning
• field_mapping: Optional mapping to rename columns

Sources: sdk/python/feast/data_source.py README.md152-157

FeatureService

A FeatureService (sdk/python/feast/feature_service.py) groups features for a specific model or use case:

• name: Unique identifier
• features: List of feature references (e.g., "driver_hourly_stats:conv_rate")
• description: Purpose of the service
• tags: Key-value metadata

Feature services enable versioning of feature sets and facilitate model-centric feature tracking.

Sources: README.md216-231 docs/getting-started/quickstart.md216-231

Main Workflows

1. Feature Registration (feast apply)

The registration workflow validates and stores feature definitions in the registry:

Key Files:

• CLI entry point: feast apply command invokes apply_total() (sdk/python/feast/repo_operations.py399-416)
• Repository parsing: parse_repo() (sdk/python/feast/repo_operations.py114-220) discovers Python objects
• Application logic: FeatureStore.apply() (sdk/python/feast/feature_store.py821-1005) performs validation and registration
• Infrastructure updates: PassthroughProvider.update_infra() (sdk/python/feast/infra/passthrough_provider.py142-174) creates online store tables

The workflow includes:

1. Validation: Check schema consistency, entity references, data source compatibility
2. Inference: Infer missing schemas, timestamps, and entity columns from data sources
3. Registry Updates: Store metadata for all feature objects
4. Infrastructure Provisioning: Create necessary tables/indices in online stores

Sources: sdk/python/feast/repo_operations.py399-416 sdk/python/feast/feature_store.py821-1005 sdk/python/feast/infra/passthrough_provider.py142-174

2. Materialization (feast materialize)

Materialization copies features from the offline store to the online store for low-latency serving:

Key Components:

• Materialization orchestration: FeatureStore.materialize() (sdk/python/feast/feature_store.py1289-1376)
• Single view materialization: Provider.materialize_single_feature_view() (sdk/python/feast/infra/provider.py222-246)
• Compute engine: Loads data from offline store and writes to online store
  • Local: LocalComputeEngine (in-process, default)
  • Lambda: LambdaComputeEngine (AWS serverless)
  • Snowflake: SnowflakeComputeEngine (uses Snowflake compute)
• Data retrieval: OfflineStore.pull_latest_from_table_or_query() gets the most recent feature values
• Data writing: OnlineStore.online_write_batch() writes features indexed by entity keys

The materialization process:

1. Queries the offline store for features within the specified time range
2. Applies deduplication to keep only the latest values per entity
3. Converts data to protobuf format for storage
4. Writes to online store with entity keys as primary keys
5. Records materialization metadata in the registry

Sources: sdk/python/feast/feature_store.py1289-1376 sdk/python/feast/infra/passthrough_provider.py311-404 docs/getting-started/concepts/data-ingestion.md

3. Historical Feature Retrieval (get_historical_features)

Retrieves point-in-time correct features for model training:

Key Methods:

• Entry point: FeatureStore.get_historical_features() (sdk/python/feast/feature_store.py1738-1871)
• Offline store delegation: Each offline store implements get_historical_features():
  • BigQuery: sdk/python/feast/infra/offline_stores/bigquery.py235-340
  • Snowflake: Similar pattern
  • File: Uses Dask for local processing
• Point-in-time join: Ensures feature values are from before the prediction timestamp to prevent data leakage
• RetrievalJob: Lazy evaluation - query executes when .to_df() or .to_arrow() is called

The point-in-time join logic:

1. Uploads entity DataFrame with timestamps to a temporary table
2. Generates SQL that joins feature views using:
   • Entity key equality
   • Event timestamp <= entity timestamp (no future leakage)
   • ROW_NUMBER() to select the most recent feature value
3. Returns a RetrievalJob that can be executed to get results

Sources: sdk/python/feast/feature_store.py1738-1871 sdk/python/feast/infra/offline_stores/bigquery.py235-340 sdk/python/feast/infra/offline_stores/offline_utils.py168-350

4. Online Feature Retrieval (get_online_features)

Retrieves features from the online store for real-time inference:

Key Components:

• Entry point: FeatureStore.get_online_features() (sdk/python/feast/feature_store.py1469-1580)
• Online store implementations: Each online store implements get_online_features():
  • Redis: sdk/python/feast/infra/online_stores/redis.py
  • DynamoDB: sdk/python/feast/infra/online_stores/dynamodb.py
  • SQLite: sdk/python/feast/infra/online_stores/sqlite.py178-289
• Entity key serialization: serialize_entity_key() converts entity values to a consistent binary format
• OnlineResponse: Contains feature values, statuses, and metadata

The retrieval process:

1. Parses feature references to identify required feature views
2. Serializes entity values into binary keys using protobuf
3. Reads features from online store (usually a key-value lookup)
4. Deserializes feature values from protobuf
5. Applies on-demand transformations if specified
6. Returns results with feature statuses (PRESENT, NOT_FOUND, etc.)

Sources: sdk/python/feast/feature_store.py1469-1580 sdk/python/feast/infra/online_stores/sqlite.py178-289 sdk/python/feast/infra/passthrough_provider.py239-258

Repository Structure and Development

Repository Organization

The codebase follows a standard Python package structure:

feast/
├── sdk/python/feast/          # Core Python SDK
│   ├── feature_store.py       # Main FeatureStore class
│   ├── repo_config.py         # Configuration
│   ├── infra/                 # Infrastructure implementations
│   │   ├── provider.py        # Provider interface
│   │   ├── offline_stores/    # Offline store plugins
│   │   ├── online_stores/     # Online store plugins
│   │   ├── compute_engines/   # Compute engine plugins
│   │   └── registry/          # Registry implementations
│   ├── entity.py              # Entity definition
│   ├── feature_view.py        # FeatureView definition
│   └── data_source.py         # DataSource base class
├── protos/                    # Protobuf definitions
├── infra/                     # Infrastructure-related code
│   ├── feast-operator/        # Kubernetes operator
│   └── charts/                # Helm charts
├── docs/                      # Documentation
└── examples/                  # Example repositories

Sources: sdk/python/feast/

CLI and User Entry Points

The Feast CLI provides commands for all major workflows:

Command	Function	Implementation
feast init	Bootstrap new repository	sdk/python/feast/repo_operations.py448-503
feast apply	Register feature definitions	sdk/python/feast/repo_operations.py399-416
feast materialize	Load features to online store	CLI wrapper around FeatureStore.materialize()
feast materialize-incremental	Incremental materialization	CLI wrapper with automatic date range
feast ui	Launch Web UI	Starts React development server
feast serve	Launch feature server	Starts FastAPI server for HTTP/gRPC serving

CLI commands are implemented in sdk/python/feast/cli.py and delegate to functions in sdk/python/feast/repo_operations.py

Sources: docs/reference/feast-cli-commands.md1-16 sdk/python/feast/repo_operations.py399-503

Feature Repository Pattern

A typical Feast deployment uses a feature repository: a directory containing:

• feature_store.yaml: Configuration file
• *.py: Python files with feature definitions
• data/: Optional local data files
• .feastignore: Files to exclude from discovery

The parse_repo() function (sdk/python/feast/repo_operations.py114-220) scans the repository to discover feature objects:

1. Reads .feastignore to determine exclusion patterns
2. Finds all .py files in the repository
3. Imports each module and inspects attributes
4. Collects Entity, FeatureView, DataSource, and FeatureService instances
5. Returns a RepoContents object with all discovered objects

This pattern enables version control of feature definitions alongside model code.

Sources: sdk/python/feast/repo_operations.py114-220 docs/getting-started/quickstart.md76-103

Type System and Data Serialization

Feast uses a comprehensive type system to ensure consistency across different storage backends:

Key Files:

• ValueType enum: sdk/python/feast/value_type.py - Legacy enum for basic types
• FeastType hierarchy: sdk/python/feast/types.py - Modern type system supporting complex types
• Type conversions: sdk/python/feast/type_map.py - Bidirectional conversions between all type systems
• Protobuf definitions: protos/feast/types/Value.proto - Serialization format

Conversion Functions:

• python_type_to_feast_value_type(): Infers Feast types from Python types
• python_values_to_proto_values(): Serializes Python values to protobuf
• feast_value_type_to_python_type(): Deserializes protobuf to Python
• bq_to_feast_value_type(), pa_to_feast_value_type(): Database-specific conversions

The type system ensures:

1. Schema consistency: Feature schemas are validated at registration time
2. Safe serialization: Data is safely converted between Python and protobuf formats
3. Database compatibility: Feast types map correctly to backend-specific types
4. Complex type support: Arrays, maps, and nested structures are properly handled

Sources: sdk/python/feast/value_type.py sdk/python/feast/types.py sdk/python/feast/type_map.py protos/feast/types/Value.proto

Extensibility and Plugin System

Feast is designed to be extensible through a plugin architecture:

Custom Offline Stores

To add a custom offline store:

1. Extend OfflineStore (sdk/python/feast/infra/offline_stores/offline_store.py)
2. Implement required methods:
   • get_historical_features(): Point-in-time join logic
   • pull_latest_from_table_or_query(): Latest value retrieval for materialization
3. Create a config class extending FeastConfigBaseModel
4. Register the class name in OFFLINE_STORE_CLASS_FOR_TYPE or use the full class path in feature_store.yaml

Example offline stores: BigQuery (sdk/python/feast/infra/offline_stores/bigquery.py), Snowflake, Redshift, DuckDB, Spark.

Sources: sdk/python/feast/infra/offline_stores/offline_store.py docs/how-to-guides/customizing-feast/adding-a-new-offline-store.md

Custom Online Stores

To add a custom online store:

1. Extend OnlineStore (sdk/python/feast/infra/online_stores/online_store.py)
2. Implement required methods:
   • online_write_batch(): Write features
   • online_read(): Read features by entity keys
   • update(): Create/update tables
3. Create a config class with online store settings
4. Register in ONLINE_STORE_CLASS_FOR_TYPE or specify full class path

Example online stores: Redis, DynamoDB, SQLite (sdk/python/feast/infra/online_stores/sqlite.py), PostgreSQL, Cassandra.

Sources: sdk/python/feast/infra/online_stores/online_store.py sdk/python/feast/infra/online_stores/sqlite.py docs/how-to-guides/customizing-feast/adding-support-for-a-new-online-store.md

Custom Providers

The Provider interface (sdk/python/feast/infra/provider.py49-105) can be extended for cloud-specific optimizations. Most users can use PassthroughProvider, but custom providers enable:

• Cloud-native materialization jobs
• Specialized online/offline store integrations
• Custom infrastructure management

Sources: sdk/python/feast/infra/provider.py49-105 docs/how-to-guides/customizing-feast/creating-a-custom-provider.md

Custom Compute Engines

The ComputeEngine interface enables different materialization strategies:

• LocalComputeEngine: In-process (default)
• LambdaComputeEngine: AWS Lambda for serverless materialization
• SnowflakeComputeEngine: Uses Snowflake compute resources
• BytewaxComputeEngine: Kubernetes-based streaming

Custom compute engines can optimize for specific infrastructure requirements.

Sources: sdk/python/feast/infra/passthrough_provider.py92-129 sdk/python/feast/repo_config.py46-53

Summary

Feast provides a complete feature store system built around several key principles:

1. Centralized orchestration through the FeatureStore class
2. Pluggable infrastructure via the Provider pattern and storage interfaces
3. Declarative feature definitions using Python classes
4. Type safety through a comprehensive type system
5. Version control of features through Git-based repositories
6. Point-in-time correctness for training data
7. Low-latency serving for online inference

The system is designed to work with existing data infrastructure while providing a consistent interface for feature management across the ML lifecycle.

Sources: README.md29-38 sdk/python/feast/feature_store.py100-203 docs/getting-started/quickstart.md1-60