Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

Prelude

dd-trace-py now includes an OpenFeature provider implementation, enabling feature flag evaluation through the OpenFeature API.
This integration is under active design and development. Functionality and APIs are experimental and may change without notice.
For more information, see the Datadog documentation at https://docs.datadoghq.com/feature_flags/#overview

Upgrade Notes

• 32-bit linux is no longer supported. Please contact us if this blocks upgrading dd-trace-py.
• LLM Observability
  • Experiments spans now contain metadata from the dataset record.
  • Experiments spans' input, output, expected_output fields are now emitted as is so that if data in any of the columns are objects, they can be searchable in Datadog.
  • Experiments spans and children spans are now tagged with human readable names to allow better analysis of experiments data. New tags added are: dataset_name, project_name, project_id, experiment_name.
• aioredis
  • The aioredis integration has been removed.
• tornado
  • Updated minimum supported version to v6.1+.

Deprecation Notes

• tornado
  • Deprecated support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
  • Deprecates programmatic tracing configuration via the ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.
• LLM Observability
  • The ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.

New Features

• profiling
  • Add support for threading.BoundedSemaphore locking type profiling in Python. The implementation follows the same approach as threading.Semaphore, properly handling internal lock detection to prevent double-counting of the underlying threading.Lock object.
  • Add support for threading.Semaphore locking type profiling in Python. The Lock profiler now detects and marks "internal" Lock objects, i.e. those that are part of implementation of higher-level locking types. One example of such higher-level primitive is threading.Semaphore, which is implemented with threading.Condition, which itself uses threading.Lock internally. Marking internal lock as "internal" will prevent it from being sampled, ensuring that the high-level (e.g. Semaphore) sample is processed.
  • This adds support for Python 3.14 in the Continuous Profiler.
  • This adds the process_id tag to profiles. The value of this tag is the current process ID (PID).
  • The stack sampler supports async generators and asyncio.wait.
  • Shows fully qualified name of functions using codeobject.co_qualname in memory profiler and lock profiler flamegraphs for Python 3.11+. Stack profiler has already been using this. This aligns the user experience across different profile types.
  • This introduces tracking for the asyncio.as_completed util in the Profiler.
  • This introduces tracking for asyncio.wait in the Profiler. This makes it possible to track dependencies between Tasks/Coroutines that await/are awaited through asyncio.wait.
• AAP
  • attach Application and API Protection findings on API Gateway inferred spans to enable AppSec API Catalog coverage of lambda functions
  • This introduces proper support for API10 for redirected requests on urllib3
• anthropic
  • Adds support for the Anthropic Beta client API (client.beta.messages.create() and client.beta.messages.stream()). This feature requires Anthropic client version 0.37.0 or higher.
• aiokafka
  • Adds DSM instrumentation support.
  • Adds instrumentation support for aiokafka>=0.9.0. See the aiokafka<https://ddtrace.readthedocs.io/en/stable/integrations.html#aiokafka> documentation for more information.
• Added support for uWSGI with gevent when threads are also patched. The use of the keyword argument thread=False is no longer required when performing monkey-patching with gevent via gevent.monkey.patch_all.
• LLM Observability
  • Reasoning token counts are now captured from Google GenAI responses.
  • The OpenAI integration now captures prompt metadata (id, version, variables, and chat template) for reusable prompts when using the responses endpoint (available in OpenAI SDK >= 1.87.0).
  • Experiments can now be run multiple times by using the optional runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.
  • Non-root experiment spans are now tagged with experiment ID, run ID, and run iteration tags.
  • Adds additional tags to MCP client session and tool call spans to power LLM Observability MCP tool call features.
  • Reasoning token counts are now captured from OpenAI and OpenAI Agents responses.
  • openai
    • This introduces support for capturing server-side MCP tool calls invoked via the OpenAI Responses API as a separate span.
• langchain
  • Adds support for tracing RunnableLambda instances.
• mcp
  • Marks client mcp tool call spans as errors when the corresponding server tool call errored
• Crashtracker
  • This introduces a fallback to capture runtime stack frames when Python's _Py_DumpTracebackThreads function is not available.
• ASGI
  • Enable context propagation between websocket message spans.

Bug Fixes

• avro
  • Fixes an issue where Avro instrumentation does not return method results when DSM is enabled.
• crashtracker
  • Fixes missing env variables inheritance for receiver process.
• dynamic instrumentation
  • fix issue with line probes matching the wrong source file when multiple source files from different Python path entries share the same name.
  • uploading snapshots now retries on all HTTP error codes.
• exception replay
  • fixed the order in which frames are captured to ensure that the values of frames close to the point where the initial exception was thrown are always attached to the relevant spans.
  • fixed an infinite loop that could cause memory leaks when capturing exceptions, and improved overall speed and memory performance.
  • ensure exception information is captured when exceptions are raised by the GraphQL client library.
• Code Security
  • Fixes critical memory safety issue in IAST when used with forked worker processes (MCP servers with Gunicorn and Uvicorn). Workers previously crashed with segmentation faults due to stale PyObject pointers in native taint maps after fork.
• openai
  • Resolves an issue where instantiating an OpenAI client with a non-string API key resulted in parsing issues.
• tracing
  • Fixed a potential IndexError in partial flush when the finished span counter was out of sync with actual finished spans.
  • DD_TRACE_PARTIAL_FLUSH_MIN_SPANS values less than 1 now default to 1 with a warning.
  • CI Visibility: Ensure the http connection is correctly reset in all error scenarios.
• ray
  • This fix resolves an issue where Ray jobs that did not explicitly call ray.init() at the top of their scripts were not properly instrumented, resulting in incomplete traces. To ensure full tracing capabilities, use ddtrace-run when starting your Ray cluster: DD_PATCH_MODULES="ray:true,aiohttp:false,grpc:false,requests:false" ddtrace-run ray start --head.
• AAP
  • This fix resolves an issue where the appsec layer was not compatible anymore with the lambda/serverless version of the tracer.
• lib-injection
  • do not inject into the gsutil tool
• LLM Observability
  • Fixes an issue where LLMObs.export_span() would raise when LLMObs is disabled.
  • Resolves an issue where self was being annotated as an input parameter using LLM Observability function decorators.
  • This fix resolves an issue where LLMObs.annotation_context() properties (tags, prompt, and name) were not applied to subsequent LLM operations within the same context block. This occurred when multiple sequential operations (such as Langchain batch calls with structured outputs) were performed, causing only the first operation to receive the annotations.
  • This fix resolves an issue where evaluation-metric labels containing dots could be interpreted as nested objects by adding validation that rejects such labels and provides a clear error message instructing users to use alternative naming conventions.
  • Fixes an issue where the Google ADK integration would throw an AttributeError when trying to access the name or description attributes of a tool.
• opentelemetry
  • Fixed spans going unsampled when using opentelemetry.trace.get_current_span() or NonRecordingSpan. Spans are now kept and appear in the UI unless explicitly dropped by the Agent or sampling rules.
• profiling
  • This fix resolves a critical issue where the Lock Profiler generated release samples for non-sampled lock acquires, resulting in inflated or negative (when integer overflows) lock hold times (e.g., "3.24k days per minute", "-970 days per minute"). This affected virtually all customers using sampling rates < 100% (which should be the majority).
  • This fix prevents a use-after-free crash from the memory profiler on Python version 3.10 and 3.11. The previous attempt to fix this bug itself had a bug, which this fix addresses.
  • improve reliability when parsing an empty span.
  • Fixes a segmentation fault caused by accessing frame.f_locals while trying to retrieve class name...

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Deprecation Notes

• Tornado:
  • Deprecated support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
  • Deprecates programmatic tracing configuration via the ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.

Bug Fixes

• AAP: This fix resolves an issue where the appsec layer was not compatible anymore with the lambda/serverless version of the tracer.

• Code Security: Fixes critical memory safety issue in IAST when used with forked worker processes (MCP servers with Gunicorn and Uvicorn). Workers previously crashed with segmentation faults due to stale PyObject pointers in native taint maps after fork.

• Dynamic instrumentation: fix issue with line probes matching the wrong source file when multiple source files from different Python path entries share the same name.

• Exception replay: ensure exception information is captured when exceptions are raised by the GraphQL client library.

• Lib-injection: do not inject into the gsutil tool

• LLM Observability: Fixes an issue where the Google ADK integration would throw an AttributeError when trying to access the name or description attributes of a tool.

• Profiling:

  • This fix resolves a critical issue where the Lock Profiler generated release samples for non-sampled lock acquires, resulting in inflated or negative (when integer overflows) lock hold times (e.g., "3.24k days per minute", "-970 days per minute"). This affected virtually all customers using sampling rates < 100% (which should be the majority).
  • This fix prevents a use-after-free crash from the memory profiler on Python version 3.10 and 3.11. The previous attempt to fix this bug itself had a bug, which this fix addresses.
  • Fixes a segmentation fault caused by accessing frame.f_locals while trying to retrieve class name of a PyFrameObject.

• Tracing:

  • Fixed a potential IndexError in partial flush when the finished span counter was out of sync with actual finished spans.
  • DD_TRACE_PARTIAL_FLUSH_MIN_SPANS values less than 1 now default to 1 with a warning.

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.

dd-trace-py now includes an OpenFeature provider implementation, enabling feature flag evaluation through the OpenFeature API.

This integration is under active design and development. Functionality and APIs are experimental and may change without notice. For more information, see the Datadog documentation at https://docs.datadoghq.com/feature_flags/#overview

Breaking Changes

• Support for ddtrace with Python 3.8 is removed after being deprecated in the 3.0 release line. Use ddtrace 4.x with Python 3.9 or newer.
• 32-bit linux is no longer supported. Please contact us if this blocks upgrading dd-trace-py.
• mongoengine
  • Drops support for the ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.
• CI Visibility
  • Removed deprecated entry points for the pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.
• dynamic instrumentation
  • removed the deprecated DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.
• exception replay
  • removed the deprecated DD_EXCEPTION_DEBUGGING_ENABLED variable.
• tracing
  • Deprecated methods have been removed
    • Span.set_tag_str has been removed, use Span.set_tag instead.
    • Span.set_struct_tag has been removed.
    • Span.get_struct_tag has been removed.
    • Span._pprint has been removed
    • Span.finished setter was removed, please use Span.finish() method instead.
    • Tracer.on_start_span method has been removed.
    • Tracer.deregister_on_start_span method has been removed.
    • ddtrace.trace.Pin has been removed.
    • Span.finish_with_ancestors was removed with no replacement.
  • Some methods have had their type signatures changed
    • Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> None
    • Span.get_tag typing is now get_tag(key: str) -> Optional[str]
    • Span.set_tags typing is now set_tags(tags: dict[str, str]) -> None
    • Span.get_tags typing is now get_tags() -> dict[str, str]
    • Span.set_metric typing is now set_metric(key: str, value: int | float) -> None
    • Span.get_metric typing is now get_metric(key: str) -> Optional[int | float]
    • Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> None
    • Span.get_metrics typing is now get_metrics() -> dict[str, int | float]
  • Span.record_exception's timestamp and escaped parameters are removed
• LLM Observability
  • manual instrumentation methods, including LLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.
  • LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
    • LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)
    • LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)
• profiling
  • this updates echion (the Python stack sampler) to the latest version, which introduces an experimental faster memory copy function.
  • The V1 stack profiler is removed. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED is now removed.
• freezegun
  • The deprecated freezegun integration is now removed.
• opentracer
  • This change removes the deprecated opentracer package
• aioredis
  • The aioredis integration has been removed.
• google_generativeai
  • The google_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
    As an alternative, you can use the recommended google_genai library and corresponding integration instead.
• openai
  • Streamed chat/completions will no longer have token counts computed using the tiktoken library, and instead
    will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.
• django
  • This upgrades the default tracing behavior to enable minimal tracing mode by default (DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.
  • When DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.
• Other
  • This change removes the ddtrace.settings package. Environment variables should be used to adjust settings.
  • This change removes the deprecated non_active_span parameter to HttpPropagator.inject
  • This change removes the deprecated environment variable DEFAULT_RUNTIME_METRICS_INTERVAL.

Deprecation Notes

• Support for ddtrace with Python 3.9 is deprecated after Python 3.9 reached its end-of-life.
• Deprecates support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
• Deprecates programmatic tracing configuration via the ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.

New Features

• AAP
  • This introduces security response id for easy identification of blocking responses.
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling
  • Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility
  • This fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
  • This fix resolves performance issue affecting coverage collection for Python 3.12+
• data_streams
  • This fix resolves an issue where payload size statistics were not being sent to the backend for Data Streams Monitoring (DSM).
• core
  • This fix resolves an issue where forksafe locks used patched threading primitives from the profiling module, causing performance issues. The forksafe module now uses unpatched threading primitives (Lock, RLock, Event).
• LLM Observability
  • add support for HTTPS_PROXY.
  • Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
  • Corrected the description of the assessment argument in submit_evaluation().
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
• Error Tracking
  • Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
• dynamic instrumentation
  • fix issue with line probes matching the wrong source file when multiple source files from different Python path entries share the same name.
• profiling
  • This fix resolves an issue where importing the profiler module after an asyncio Event Loop had been started would make the Profiler blind to the existing Event Loop and its Tasks.
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and ...

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.

dd-trace-py now includes an OpenFeature provider implementation, enabling feature flag evaluation through the OpenFeature API.

This integration is under active design and development. Functionality and APIs are experimental and may change without notice. For more information, see the Datadog documentation at https://docs.datadoghq.com/feature_flags/#overview

Breaking Changes

• Support for ddtrace with Python 3.8 is removed after being deprecated in the 3.0 release line. Use ddtrace 4.x with Python 3.9 or newer.
• 32-bit linux is no longer supported. Please contact us if this blocks upgrading dd-trace-py.
• mongoengine
  • Drops support for the ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.
• CI Visibility
  • Removed deprecated entry points for the pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.
• dynamic instrumentation
  • removed the deprecated DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.
• exception replay
  • removed the deprecated DD_EXCEPTION_DEBUGGING_ENABLED variable.
• tracing
  • Deprecated methods have been removed
    • Span.set_tag_str has been removed, use Span.set_tag instead.
    • Span.set_struct_tag has been removed.
    • Span.get_struct_tag has been removed.
    • Span._pprint has been removed
    • Span.finished setter was removed, please use Span.finish() method instead.
    • Tracer.on_start_span method has been removed.
    • Tracer.deregister_on_start_span method has been removed.
    • ddtrace.trace.Pin has been removed.
    • Span.finish_with_ancestors was removed with no replacement.
  • Some methods have had their type signatures changed
    • Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> None
    • Span.get_tag typing is now get_tag(key: str) -> Optional[str]
    • Span.set_tags typing is now set_tags(tags: dict[str, str]) -> None
    • Span.get_tags typing is now get_tags() -> dict[str, str]
    • Span.set_metric typing is now set_metric(key: str, value: int | float) -> None
    • Span.get_metric typing is now get_metric(key: str) -> Optional[int | float]
    • Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> None
    • Span.get_metrics typing is now get_metrics() -> dict[str, int | float]
  • Span.record_exception's timestamp and escaped parameters are removed
• LLM Observability
  • manual instrumentation methods, including LLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.
  • LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
    • LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)
    • LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)
• profiling
  • this updates echion (the Python stack sampler) to the latest version, which introduces an experimental faster memory copy function.
  • The V1 stack profiler is removed. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED is now removed.
• freezegun
  • The deprecated freezegun integration is now removed.
• opentracer
  • This change removes the deprecated opentracer package
• aioredis
  • The aioredis integration has been removed.
• google_generativeai
  • The google_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
    As an alternative, you can use the recommended google_genai library and corresponding integration instead.
• openai
  • Streamed chat/completions will no longer have token counts computed using the tiktoken library, and instead
    will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.
• django
  • This upgrades the default tracing behavior to enable minimal tracing mode by default (DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.
  • When DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.
• Other
  • This change removes the ddtrace.settings package. Environment variables should be used to adjust settings.
  • This change removes the deprecated non_active_span parameter to HttpPropagator.inject
  • This change removes the deprecated environment variable DEFAULT_RUNTIME_METRICS_INTERVAL.

Deprecation Notes

• Support for ddtrace with Python 3.9 is deprecated after Python 3.9 reached its end-of-life.
• Deprecates support for Tornado versions older than v6.1. Use Tornado v6.1 or later.
• Deprecates programmatic tracing configuration via the ddtrace.contrib.tornado module. Configure tracing using environment variables and import ddtrace.auto instead.

New Features

• AAP
  • This introduces security response id for easy identification of blocking responses.
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling
  • Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility
  • This fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
  • This fix resolves performance issue affecting coverage collection for Python 3.12+
• data_streams
  • This fix resolves an issue where payload size statistics were not being sent to the backend for Data Streams Monitoring (DSM).
• core
  • This fix resolves an issue where forksafe locks used patched threading primitives from the profiling module, causing performance issues. The forksafe module now uses unpatched threading primitives (Lock, RLock, Event).
• LLM Observability
  • add support for HTTPS_PROXY.
  • Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
  • Corrected the description of the assessment argument in submit_evaluation().
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
• Error Tracking
  • Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
• dynamic instrumentation
  • fix issue with line probes matching the wrong source file when multiple source files from different Python path entries share the same name.
• profiling
  • This fix resolves an issue where importing the profiler module after an asyncio Event Loop had been started would make the Profiler blind to the existing Event Loop and its Tasks.
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and ...

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• profiling
  • This updates echion (the Python stack sampler) to the latest version, which introduces an experimental faster memory copy function.

Deprecation Notes

• tracing
  • Span.finished setter is deprecated, use Span.finish() method instead.
  • Span.finish_with_ancestors() is deprecated with no alternative.
• LLM Observability
  • The ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.

New Features

• AAP
  • This introduces security response id for easy identification of blocking responses.
• LLM Observability
  • Experiments can now be run multiple times by using the optional runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.
  • Non-root experiment spans are now tagged with experiment ID, run ID, and run iteration tags.
  • Adds additional tags to MCP client session and tool call spans to power LLM Observability MCP tool call features.
• mcp
  • Marks client mcp tool call spans as errors when the corresponding server tool call errored

Bug Fixes

• CI Visibility
  • This fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
• data_streams
  • This fix resolves an issue where payload size statistics were not being sent to the backend for Data Streams Monitoring (DSM).
• core
  • This fix resolves an issue where forksafe locks used patched threading primitives from the profiling module, causing performance issues. The forksafe module now uses unpatched threading primitives (Lock, RLock, Event).
• LLM Observability
  • Add support for HTTPS_PROXY.
• Error Tracking
  • Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
• profiling
  • This fix resolves an issue where importing the profiler module after an asyncio Event Loop had been started would make the Profiler blind to the existing Event Loop and its Tasks.
• other
  • Prevent a potential ResourceWarning in multiprocess scenarios.
  • Prevent startup failure when a temporary directory is not available.

Other Changes

• profiling
  • This removes the wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Upgrade Notes

• profiling
  • This updates echion (the Python stack sampler) to the latest version, which introduces an experimental faster memory copy function.

Deprecation Notes

• tracing
  • Span.finished setter is deprecated, use Span.finish() method instead.
  • Span.finish_with_ancestors() is deprecated with no alternative.
• LLM Observability
  • The ExperimentResult class' rows and summary_evaluations attributes are deprecated and will be removed in the next major release. ExperimentResult.rows/summary_evaluations attributes will only store the results of the first run iteration for multi-run experiments. Use the ExperimentResult.runs attribute instead to access experiment results and summary evaluations.

New Features

• AAP
  • This introduces security response id for easy identification of blocking responses.
• LLM Observability
  • Experiments can now be run multiple times by using the optional runs argument, to assess the true performance of an experiment in the face of the non determinism of LLMs. Use the new ExperimentResult class' runs attribute to access the results and summary evaluations by run iteration.
  • Non-root experiment spans are now tagged with experiment ID, run ID, and run iteration tags.
  • Adds additional tags to MCP client session and tool call spans to power LLM Observability MCP tool call features.
• mcp
  • Marks client mcp tool call spans as errors when the corresponding server tool call errored

Bug Fixes

• CI Visibility
  • This fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
• data_streams
  • This fix resolves an issue where payload size statistics were not being sent to the backend for Data Streams Monitoring (DSM).
• core
  • This fix resolves an issue where forksafe locks used patched threading primitives from the profiling module, causing performance issues. The forksafe module now uses unpatched threading primitives (Lock, RLock, Event).
• LLM Observability
  • Add support for HTTPS_PROXY.
• Error Tracking
  • Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
• profiling
  • This fix resolves an issue where importing the profiler module after an asyncio Event Loop had been started would make the Profiler blind to the existing Event Loop and its Tasks.
• other
  • Prevent a potential ResourceWarning in multiprocess scenarios.
  • Prevent startup failure when a temporary directory is not available.

Other Changes

• profiling
  • This removes the wrapt library dependency from the Lock Profiler implementation, improving performance and reducing overhead during lock instrumentation.

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.

dd-trace-py now includes an OpenFeature provider implementation, enabling feature flag evaluation through the OpenFeature API.

This integration is under active design and development. Functionality and APIs are experimental and may change without notice. For more information, see the Datadog documentation at https://docs.datadoghq.com/feature_flags/#overview

Breaking Changes

• Support for ddtrace with Python 3.8 is removed after being deprecated in the 3.0 release line. Use ddtrace 4.x with Python 3.9 or newer.
• 32-bit linux is no longer supported. Please contact us if this blocks upgrading dd-trace-py.
• mongoengine
  • Drops support for the ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.
• CI Visibility
  • Removed deprecated entry points for the pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.
• dynamic instrumentation
  • removed the deprecated DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.
• exception replay
  • removed the deprecated DD_EXCEPTION_DEBUGGING_ENABLED variable.
• tracing
  • Deprecated methods have been removed
    • Span.set_tag_str has been removed, use Span.set_tag instead.
    • Span.set_struct_tag has been removed.
    • Span.get_struct_tag has been removed.
    • Span._pprint has been removed
    • Span.finished setter was removed, please use Span.finish() method instead.
    • Tracer.on_start_span method has been removed.
    • Tracer.deregister_on_start_span method has been removed.
    • ddtrace.trace.Pin has been removed.
    • Span.finish_with_ancestors was removed with no replacement.
  • Some methods have had their type signatures changed
    • Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> None
    • Span.get_tag typing is now get_tag(key: str) -> Optional[str]
    • Span.set_tags typing is now set_tags(tags: dict[str, str]) -> None
    • Span.get_tags typing is now get_tags() -> dict[str, str]
    • Span.set_metric typing is now set_metric(key: str, value: int | float) -> None
    • Span.get_metric typing is now get_metric(key: str) -> Optional[int | float]
    • Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> None
    • Span.get_metrics typing is now get_metrics() -> dict[str, int | float]
  • Span.record_exception's timestamp and escaped parameters are removed
• LLM Observability
  • manual instrumentation methods, including LLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.
  • LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
    • LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)
    • LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)
• profiling
  • this updates echion (the Python stack sampler) to the latest version, which introduces an experimental faster memory copy function.
  • The V1 stack profiler is removed. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED is now removed.
• freezegun
  • The deprecated freezegun integration is now removed.
• opentracer
  • This change removes the deprecated opentracer package
• aioredis
  • The aioredis integration has been removed.
• google_generativeai
  • The google_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
    As an alternative, you can use the recommended google_genai library and corresponding integration instead.
• openai
  • Streamed chat/completions will no longer have token counts computed using the tiktoken library, and instead
    will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.
• django
  • This upgrades the default tracing behavior to enable minimal tracing mode by default (DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.
  • When DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.
• Other
  • This change removes the ddtrace.settings package. Environment variables should be used to adjust settings.
  • This change removes the deprecated non_active_span parameter to HttpPropagator.inject
  • This change removes the deprecated environment variable DEFAULT_RUNTIME_METRICS_INTERVAL.

Deprecation Notes

• Support for ddtrace with Python 3.9 is deprecated after Python 3.9 reached its end-of-life.

New Features

• AAP
  • This introduces security response id for easy identification of blocking responses.
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling
  • Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility
  • This fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
  • This fix resolves performance issue affecting coverage collection for Python 3.12+
• data_streams
  • This fix resolves an issue where payload size statistics were not being sent to the backend for Data Streams Monitoring (DSM).
• core
  • This fix resolves an issue where forksafe locks used patched threading primitives from the profiling module, causing performance issues. The forksafe module now uses unpatched threading primitives (Lock, RLock, Event).
• LLM Observability
  • add support for HTTPS_PROXY.
  • Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
  • Corrected the description of the assessment argument in submit_evaluation().
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
• Error Tracking
  • Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
• dynamic instrumentation
  • fix issue with line probes matching the wrong source file when multiple source files from different Python path entries share the same name.
• profiling
  • This fix resolves an issue where importing the profiler module after an asyncio Event Loop had been started would make the Profiler blind to the existing Event Loop and its Tasks.
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)
  • Upgrades echion to resolve an issue w...

Estimated end-of-life date, accurate to within three months: 05-2027
See the support level definitions for more information.

This is a major-version release that contains many backwards-incompatible changes to public APIs. To find which of these your code relies on, follow the "deprecation warnings" instructions here.

Breaking Changes

• Support for ddtrace with Python 3.8 is removed after being deprecated in the 3.0 release line. Use ddtrace 4.x with Python 3.9 or newer.
• mongoengine
  • Drops support for the ddtrace.Pin object with mongoengine. With this change, the ddtrace library no longer directly supports mongoengine. Mongoengine will be supported through the pymongo integration.
• CI Visibility
  • Removed deprecated entry points for the pytest_benchmark and pytest_bdd integrations. These plugins are now supported by the regular pytest integration.
• dynamic instrumentation
  • removed the deprecated DD_DYNAMIC_INSTRUMENTATION_UPLOAD_FLUSH_INTERVAL variable.
• exception replay
  • removed the deprecated DD_EXCEPTION_DEBUGGING_ENABLED variable.
• tracing
  • Deprecated methods have been removed
    • Span.set_tag_str has been removed, use Span.set_tag instead.
    • Span.set_struct_tag has been removed.
    • Span.get_struct_tag has been removed.
    • Span._pprint has been removed
    • Span.finished setter was removed, please use Span.finish() method instead.
    • Tracer.on_start_span method has been removed.
    • Tracer.deregister_on_start_span method has been removed.
    • ddtrace.trace.Pin has been removed.
    • Span.finish_with_ancestors was removed with no replacement.
  • Some methods have had their type signatures changed
    • Span.set_tag typing is now set_tag(key: str, value: Optional[str] = None) -> None
    • Span.get_tag typing is now get_tag(key: str) -> Optional[str]
    • Span.set_tags typing is now set_tags(tags: dict[str, str]) -> None
    • Span.get_tags typing is now get_tags() -> dict[str, str]
    • Span.set_metric typing is now set_metric(key: str, value: int | float) -> None
    • Span.get_metric typing is now get_metric(key: str) -> Optional[int | float]
    • Span.set_metrics typing is now set_metrics(metrics: Dict[str, int | float]) -> None
    • Span.get_metrics typing is now get_metrics() -> dict[str, int | float]
  • Span.record_exception's timestamp and escaped parameters are removed
• LLM Observability
  • manual instrumentation methods, including LLMObs.annotate(), LLMObs.export_span(), LLMObs.submit_evaluation(), LLMObs.inject_distributed_headers(), and LLMObs.activate_distributed_headers() now raise exceptions instead of logging. LLM Observability auto-instrumentation is not affected.
  • LLMObs.submit_evaluation_for() has been removed. Please use LLMObs.submit_evaluation() instead for submitting evaluations. To migrate:
    • LLMObs.submit_evaluation_for(...) users: rename to LLMObs.submit_evaluation(...)
    • LLMObs.submit_evaluation_for(...) users: rename the span_context argument to span, i.e. LLMObs.submit_evaluation(span_context={"span_id": ..., "trace_id": ...}, ...) to LLMObs.submit_evaluation(span={"span_id": ..., "trace_id": ...}, ...)
• profiling
  • this updates echion (the Python stack sampler) to the latest version, which introduces an experimental faster memory copy function.
  • The V1 stack profiler is removed. V2 has been enabled by default since v2.20.0. DD_PROFILING_STACK_V2_ENABLED is now removed.
• freezegun
  • The deprecated freezegun integration is now removed.
• opentracer
  • This change removes the deprecated opentracer package
• google_generativeai
  • The google_generativeai integration has been removed as the google_generativeai library has reached end-of-life.
    As an alternative, you can use the recommended google_genai library and corresponding integration instead.
• openai
  • Streamed chat/completions will no longer have token counts computed using the tiktoken library, and instead
    will default to having their token counts estimated if not explicitly provided in the OpenAI response object. To guarantee accurate streamed token metrics, set stream_options={"include_usage": True} in the OpenAI request.
• django
  • This upgrades the default tracing behavior to enable minimal tracing mode by default (DD_DJANGO_TRACING_MINIMAL now defaults to true). Django ORM, cache, and template instrumentation are disabled by default to eliminate duplicate span creation since library integrations for database drivers (psycopg, MySQLdb, sqlite3), cache clients (redis, memcached), template renderers (Jinja2), and other supported libraries continue to be traced. This reduces performance overhead by removing redundant Django-layer instrumentation. To restore all Django instrumentation, set DD_DJANGO_TRACING_MINIMAL=false, or enable individual features using DD_DJANGO_INSTRUMENT_DATABASES=true, DD_DJANGO_INSTRUMENT_CACHES=true, and DD_DJANGO_INSTRUMENT_TEMPLATES=true.
  • When DD_DJANGO_INSTRUMENT_DATABASES=true (default false), database instrumentation now merges Django-specific tags into database driver spans created by supported integrations (psycopg, sqlite3, MySQLdb, etc.) instead of creating duplicate Django database spans. If the database cursor is not already wrapped by a supported integration, Django wraps it and creates a span. This change reduces overhead and duplicate spans while preserving visibility into database operations.
• Other
  • This change removes the ddtrace.settings package. Environment variables should be used to adjust settings.
  • This change removes the deprecated non_active_span parameter to HttpPropagator.inject
  • This change removes the deprecated environment variable DEFAULT_RUNTIME_METRICS_INTERVAL.

Deprecation Notes

• Support for ddtrace with Python 3.9 is deprecated after Python 3.9 reached its end-of-life.

New Features

• AAP
  • This introduces security response id for easy identification of blocking responses.
  • API Security schema collection is now supported in AWS Lambda behind an Application Load Balancer or the Lambda Function URL service where the endpoint cannot be reliably known. API Security reuses the endpoint inferred by the trace resource renaming feature or recomputes it when it is not available to perform sampling instead.
  • AppSec instrumentation for downstream request is now enabled by default for urllib3 and requests. It does not require enabling APM instrumentation for urllib3 anymore.
• profiling
  • Add support for threading.RLock (reentrant lock) profiling. The Lock profiler now tracks both threading.Lock and threading.RLock usage, providing comprehensive lock contention visibility for Python applications.
• LLM Observability
  • Previous dataset versions can be optionally pulled by passing the version argument to LLMObs.pull_dataset
  • Datasets have new properties version and latest_version to provide information on the version of the dataset that is being worked with and the latest global version of the dataset, respectively

Bug Fixes

• CI Visibility
  • This fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.
  • This fix resolves performance issue affecting coverage collection for Python 3.12+
• data_streams
  • This fix resolves an issue where payload size statistics were not being sent to the backend for Data Streams Monitoring (DSM).
• core
  • This fix resolves an issue where forksafe locks used patched threading primitives from the profiling module, causing performance issues. The forksafe module now uses unpatched threading primitives (Lock, RLock, Event).
• LLM Observability
  • add support for HTTPS_PROXY.
  • Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.
  • Corrected the description of the assessment argument in submit_evaluation().
  • Resolves an issue where the langchain integration would incorrectly mark Azure OpenAI calls as duplicate llm operations even if the openai integration was enabled.
• Error Tracking
  • Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.
• profiling
  • This fix resolves an issue where importing the profiler module after an asyncio Event Loop had been started would make the Profiler blind to the existing Event Loop and its Tasks.
  • DD_PROFILING_API_TIMEOUT doesn't have any effect, and is marked to be removed in upcoming 4.0 release. New environment variable DD_PROFILING_API_TIMEOUT_MS is introduced to configure timeout for uploading profiles to the backend. The default value is 10000 ms (10 seconds)
  • Upgrades echion to resolve an issue where stack profiler can allocate a large amount of memory unnecessarily. Resolves another issue where the profiler can loop infinitely on Python 3.13.
  • This fix resolves an issue where AssertionError exceptions were silently suppressed in the _acquire method of the Lock profiler (note: this only occurs when assertions are enabled.)
• kafka
  • This fix resolves an issue where only the first message in a batch was dispatched to Data Streams Monitoring (DSM) when consuming multiple Kafka messages
• langchain
  • This fix resolves an issue where auto instrumented prompt templates incorrectly included a version field. The version field is now omitted unless explicitly set by the user.
    ...

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: this fix resolves an issue where repo tags would be fetched while unshallowing to extract commit metadata, causing performance issues for repos with a large number of tags.

• LLM Observability: fix support for HTTPS_PROXY.

• Error Tracking: Modifies the way exception events are stored such that the exception id is stored instead of the exception object, to prevent TypeErrors with custom exception objects.

Estimated end-of-life date, accurate to within three months: 08-2026
See the support level definitions for more information.

Bug Fixes

• CI Visibility: This fix addresses a performance issue where repository tags were fetched during the unshallow process to extract commit metadata, causing slowdowns in repositories with many tags.

• LLM Observability: Resolves an issue in the bedrock integration where invoking cohere rerank models would result in missing spans due to output formatting index errors.

• opentelemetry:

  • Fixed circular import when enabling multiple OpenTelemetry signals (metrics + logs) simultaneously.
  • Prevents OpenTelemetry OTLP exporter connections from being traced by ddtrace. ddtrace internal connections (gRPC and HTTP) are now excluded from tracing to prevent circular instrumentation.

• ray: This fix resolves an issue where the tracer raised an error when submitting Ray tasks without explicitly calling ray.init().

• tracer: This fix resolves an issue where an application instrumented by ddtrace could crash at start. Fix compatibility with zope.event==6.0