README.md

How to fix Workflows for Committers

The following is guidance on how to practically make changes that fix workflows.

Create a branch in https://github.com/apache/beam not your fork.

The reason to perform changes to a branch of the main repo instead of your fork is due to the challenge in replicating the environment within which Beam GitHub workflows execute. GitHub workflows allow you to execute against a branch of a repo.
Make changes in this branch you anticipate will fix the failing workflow.
Run the workflow designating your branch.

In the GitHub workflow interface, you can designate any branch of the repository to run the workflow against. Selecting your branch allows you to test the changes you made. The following screenshot shows an example of this feature.
Create a PR, pasting the link to your successful workflow run in the branch

When doing a PR, the checks will not run against your branch. Your reviewer may not know this so you'll want to mention this in your PR description, pasting the link to your successful run.
After PR merges, execute the workflow manually to validate your merged changes.

Running Workflows Manually

Most workflows will get kicked off automatically when you open a PR, push code, or on a schedule.

If you would like to manually trigger a job, you have 2 options:

Trigger Files: Test suites are configured to trigger whenever a particular trigger file is modified. Test workflows have pull_request paths, which include a trigger file. Whenever a trigger file is modified, the test suite will trigger on the pull request. Make any change to this file to trigger the job. The trigger file looks like the following: .github/trigger_files/<workflow_file_name.json>.
Committers only - Manual triggering: Any committer can start any job with a workflow_dispatch trigger defined (all jobs should have these). To do so, navigate to the Actions tab, click on your desired workflow in the left navigation bar, and then click Run Workflow.

Guidelines for Adding or Modifying Workflows

On top of normal Actions workflow steps, all new CI workflows (excluding release workflows or other workflow automation) should have the following:

Job name set via matrix (See below)
A set of specific triggers
An explicit checkout step
A set of GitHub token permissions
Concurrency Groups
Allowed Actions (See below)

Each of these is described in more detail below.

Job Name

Jobs typically set their name via matrix elements. See the following example:

jobs:
  beam_job_name:
    name: ${{ matrix.job_name }}
    strategy:
      matrix:
        job_name: [beam_job_name]
    if: |
      github.event_name == 'push' ||
      github.event_name == 'pull_request' ||
      (github.event_name == 'schedule' && github.repository == 'apache/beam') ||
      github.event_name == 'workflow_dispatch'
    steps:
      - uses: actions/checkout@v7
      - name: Setup repository
        uses: ./.github/actions/setup-action

And in case when the workflow already utilizes matrix do the following:

jobs:
  beam_job_with_matrix:
    name: ${{ matrix.job_name }} (${{ matrix.python_version }})
    runs-on: [self-hosted, ubuntu-24.04, main]
    timeout-minutes: 30
    strategy:
      fail-fast: false
      matrix:
        job_name: ["beam_job_with_matrix"]
        python_version: ['3.9','3.10','3.11','3.12']
    if: |
      github.event_name == 'push' ||
      github.event_name == 'pull_request' ||
      (github.event_name == 'schedule' && github.repository == 'apache/beam') ||
      github.event_name == 'workflow_dispatch'
    steps:
      - uses: actions/checkout@v7
      - name: Setup repository
        uses: ./.github/actions/setup-action

Notice how the matrix element of python_version is appended to the name.

Workflow triggers

GitHub allows workflows to define a set of triggers that dictate when a job should be run. For more info, see documentation here.

For the purposes of Beam, each CI workflow should define the following triggers:

A push trigger
A pull_request trigger
A scheduled trigger
A workflow_dispatch trigger

The push/pull_request triggers should only run when appropriate paths are modified. See https://github.com/apache/beam/blob/master/.github/workflows/beam_PreCommit_Go.yml#L4 for an example (you can copy and paste this into your workflow, you just need to change the paths).

Checkout step

We use setup-action after actions/checkout to ensure the PR merge commit (or head commit) is checked out and environment variables are configured. Please make sure that you checkout the code before using the composite action.

Token Permissions

You should explicitly define the GitHub Actions token scopes needed by your job (typically read permissions for most options). See

beam/.github/workflows/beam_PreCommit_Go.yml

Line 16 in 907c511

permissions:

for an example (you can copy and paste this into your workflow).

Concurrency Groups

Concurrency groups are a way of making sure that no more than one Actions run is running per job/group at any given time (previous ones can be cancelled). To reduce resource usage, you should define the following concurrency group:

concurrency:
  group: '${{ github.workflow }} @ ${{ github.event.pull_request.number || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.sender.login }}'
  cancel-in-progress: true

this defines the following groups:

Each workflow will represent a different set of groups
Within each workflow, each PR will represent a single group
Within each non-PR run for a workflow, each commit will represent a set of groups
Within each commit, each push event, schedule event, and manual run event will represent a set of groups

Allowed Actions

GitHub Actions under the trusted namespaces actions/*, github/*, and apache/* are automatically approved. Any third-party actions outside of these namespaces must be explicitly approved and added to the ASF-wide allowlist before use.

Additionally, third-party actions must be pinned to a specific Git commit SHA (e.g., zizmorcore/zizmor-action@6599ee8b7a49aef6a770f63d261d214911a7ce02) instead of a version tag or branch name to comply with ASF security policy.

For details on the allowlist and how to request new actions, see the ASF Infrastructure Actions Allowlist Check Documentation.

Testing new workflows or workflow updates

Testing New Workflows

New workflows are not added to the UI on the Actions tab until they are merged into the repo's main branch (master for Beam). To test new workflows, we recommend the following pattern:

Fork the Beam repo
Add your proposed workflow to the main branch of your fork.
Run the workflow in the Actions tab of your fork using the Run workflow button.

Note: most workflows use self-hosted runners with the main and ubuntu labels to execute (example). If you are testing on a fork, you likely will not have self-hosted runners set up. To work around this, you can start using hosted runners and then switch over when you're ready to create a PR. You can do this by changing runs-on: [self-hosted, ubuntu-24.04, main] (self-hosted, use in your PR) to runs-on: ubuntu-24.04 (GitHub hosted, use for local testing).

Note when using ubuntu-24.04 as the host, you might need to choose the Java version since some gradle tasks only work with a certain Java version. One example is below to use Java 11 when testing your workflow:

    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v3
        with:
          distribution: 'temurin'
          java-version: '11'

Testing Workflow Updates

If you need to make more changes to the workflow yaml file after it has been added to the repo, you can develop normally on a branch (using your fork or the main Beam repo if you are a committer). If you are a non-committer, this flow has several caveats mentioned below. To do this:

Make your change on a development branch.
Navigate to your workflow in the Actions tab. If your changes are on a fork, navigate to the fork's Actions tab instead. If you don't see the correct action, make sure that your fork's main branch is up to date with Beam's master branch.
Click run workflow. Before clicking submit, update to run on your branch.

Note: If you run a workflow from your fork of Beam, it will not have access to secrets stored in the Beam repository. This will cause some things like authenticating to external services to fail, which may cause your workflow to fail. If you run into this issue, you can either:

Ask for a committers help to add the workflow to a branch on the main apache/beam repo.
Upload secrets to your repo mirroring the secrets used in the main Beam repo.
Wait until the changes are merged into Beam to test (this should only be done rarely).

Additionally, as mentioned above your fork likely will not have self-hosted runners set up. To work around this, you can start using hosted runners and then switch over when you're ready to create a PR. You can do this by changing runs-on: [self-hosted, ubuntu-24.04, main] (self-hosted, use in your PR) to runs-on: ubuntu-24.04 (GitHub hosted, use for local testing).

Workflows

PreCommit Jobs

PreCommit Jobs run in a schedule and also get triggered in a PR if relevant sources has changed. To manually trigger certain PreCommit job, attach a file named "Trigger file" (listed below) under .github/trigger_files/ folder.

Workflow name	Matrix	Trigger file	Cron Status
PreCommit Community Metrics	N/A	`beam_PreCommit_CommunityMetrics.json`
PreCommit GHA	N/A	`beam_PreCommit_GHA.json`
PreCommit Go	N/A	`beam_PreCommit_Go.json`
PreCommit GoPortable	N/A	`beam_PreCommit_GoPortable.json`
PreCommit Java	N/A	`beam_PreCommit_Java.json`
PreCommit Java Dataflow Non-portable Worker	N/A	`beam_PreCommit_Java_Dataflow.json`
PreCommit Java Amazon Web Services2 IO Direct	N/A	`beam_PreCommit_Java_Amazon-Web-Services2_IO_Direct.json`
PreCommit Java Amqp IO Direct	N/A	`beam_PreCommit_Java_Amqp_IO_Direct.json`
PreCommit Java Azure IO Direct	N/A	`beam_PreCommit_Java_Azure_IO_Direct.json`
PreCommit Java Cassandra IO Direct	N/A	`beam_PreCommit_Java_Cassandra_IO_Direct.json`
PreCommit Java Cdap IO Direct	N/A	`beam_PreCommit_Java_Cdap_IO_Direct.json`
PreCommit Java Clickhouse IO Direct	N/A	`beam_PreCommit_Java_Clickhouse_IO_Direct.json`
PreCommit Java Csv IO Direct	N/A	`beam_PreCommit_Java_Csv_IO_Direct.json`
PreCommit Java Datadog IO Direct	N/A	`beam_PreCommit_Java_Datadog_IO_Direct.json`
PreCommit Java Delta IO Direct	N/A	`beam_PreCommit_Java_Delta_IO_Direct.json`
PreCommit Java Debezium IO Direct	N/A	`beam_PreCommit_Java_Debezium_IO_Direct.json`
PreCommit Java ElasticSearch IO Direct	N/A	`beam_PreCommit_Java_ElasticSearch_IO_Direct.json`
PreCommit Java Examples Dataflow	N/A	`beam_PreCommit_Java_Examples_Dataflow.json`
PreCommit Java Examples Dataflow Java11	N/A	`beam_PreCommit_Java_Examples_Dataflow_Java11.json`
PreCommit Java File-schema-transform IO Direct	N/A	`beam_PreCommit_Java_File-schema-transform_IO_Direct.json`
PreCommit Java Flink Versions	N/A	`beam_PreCommit_Java_Flink_Versions.json`
PreCommit Java GCP IO Direct	N/A	`beam_PreCommit_Java_GCP_IO_Direct.json`
PreCommit Java Google-ads IO Direct	N/A	`beam_PreCommit_Java_Google-ads_IO_Direct.json`	$.github\workflows\beam_PreCommit_Java_Google-ads_IO_Direct.yml$
PreCommit Java Hadoop IO Direct	N/A	`beam_PreCommit_Java_Hadoop_IO_Direct.json`
PreCommit Java HBase IO Direct	N/A	`beam_PreCommit_Java_HBase_IO_Direct.json`
PreCommit Java HCatalog IO Direct	N/A	`beam_PreCommit_Java_HCatalog_IO_Direct.json`
PreCommit Java Kafka IO Direct	N/A	`beam_PreCommit_Java_Kafka_IO_Direct.json`
PreCommit Java Kafka Streams Runner	N/A	`beam_PreCommit_Java_Kafka_Streams_Runner.json`
PreCommit Java InfluxDb IO Direct	N/A	`beam_PreCommit_Java_InfluxDb_IO_Direct.json`
PreCommit Java IOs Direct	N/A	`beam_PreCommit_Java_IOs_Direct.json`	N/A
PreCommit Java JDBC IO Direct	N/A	`beam_PreCommit_Java_JDBC_IO_Direct.json`
PreCommit Java Jms IO Direct	N/A	`beam_PreCommit_Java_Jms_IO_Direct.json`
PreCommit Java Kudu IO Direct	N/A	`beam_PreCommit_Java_Kudu_IO_Direct.json`
PreCommit Java MongoDb IO Direct	N/A	`beam_PreCommit_Java_MongoDb_IO_Direct.json`
PreCommit Java Mqtt IO Direct	N/A	`beam_PreCommit_Java_Mqtt_IO_Direct.json`
PreCommit Java Neo4j IO Direct	N/A	`beam_PreCommit_Java_Neo4j_IO_Direct.json`
PreCommit Java Parquet IO Direct	N/A	`beam_PreCommit_Java_Parquet_IO_Direct.json`
PreCommit Java Pulsar IO Direct	N/A	`beam_PreCommit_Java_Pulsar_IO_Direct.json`
PreCommit Java PVR Flink Batch	N/A	`beam_PreCommit_Java_PVR_Flink_Batch.json`
PreCommit Java PVR Flink Docker	N/A	`beam_PreCommit_Java_PVR_Flink_Docker.json`
PreCommit Java RabbitMq IO Direct	N/A	`beam_PreCommit_Java_RabbitMq_IO_Direct.json`
PreCommit Java Redis IO Direct	N/A	`beam_PreCommit_Java_Redis_IO_Direct.json`
PreCommit Java RequestResponse IO Direct	N/A	`beam_PreCommit_Java_RequestResponse_IO_Direct.json`	$.github\workflows\beam_RequestResponse_Java_RequestResponse_IO_Direct.yml$
PreCommit Java SingleStore IO Direct	N/A	`beam_PreCommit_Java_SingleStore_IO_Direct.json`
PreCommit Java Snowflake IO Direct	N/A	`beam_PreCommit_Java_Snowflake_IO_Direct.json`
PreCommit Java Solr IO Direct	N/A	`beam_PreCommit_Java_Solr_IO_Direct.json`
PreCommit Java Spark Versions	N/A	`beam_PreCommit_Java_Spark_Versions.json`
PreCommit Java Splunk IO Direct	N/A	`beam_PreCommit_Java_Splunk_IO_Direct.json`
PreCommit Java Thrift IO Direct	N/A	`beam_PreCommit_Java_Thrift_IO_Direct.json`
PreCommit Java Tika IO Direct	N/A	`beam_PreCommit_Java_Tika_IO_Direct.json`
PreCommit Kotlin Examples	N/A	`beam_PreCommit_Kotlin_Examples.json`
PreCommit Playground	N/A	N/A
PreCommit Portable Python	['3.9','3.11','3.12']	`beam_PreCommit_Portable_Python.json`
PreCommit Python	['3.9','3.10','3.11','3.12']	`beam_PreCommit_Python.json`
PreCommit Python Coverage	N/A	`beam_PreCommit_Python_Coverage.json`
PreCommit Python Dataframes	['3.9','3.10','3.11','3.12']	`beam_PreCommit_Python_Dataframes.json`
PreCommit Python Docker	['3.9','3.10','3.11','3.12']	`beam_PreCommit_PythonDocker.json`
PreCommit Python Docs	N/A	`beam_PreCommit_PythonDocs.json`
PreCommit Python Examples	['3.9','3.10','3.11','3.12']	`beam_PreCommit_Python_Examples.json`
PreCommit Python Formatter	N/A	`beam_PreCommit_PythonFormatter.json`
PreCommit Python Integration	['3.9','3.11','3.12']	`beam_PreCommit_Python_Integration.json`
PreCommit Python Lint	N/A	`beam_PreCommit_PythonLint.json`
PreCommit Python ML	['3.9','3.10','3.11','3.12']	`beam_PreCommit_Python_ML.json`
PreCommit Python PVR Flink	N/A	`beam_PreCommit_Python_PVR_Flink.json`
PreCommit Python Runners	['3.9','3.10','3.11','3.12']	`beam_PreCommit_Python_Runners.json`
PreCommit Python Transforms	['3.9','3.10','3.11','3.12']	`beam_PreCommit_Python_Transforms.json`
PreCommit RAT	N/A	N/A
PreCommit Spotless	N/A	`beam_PreCommit_Spotless.json`
PreCommit SQL	N/A	`beam_PreCommit_SQL.json`
PreCommit SQL Java17	N/A	`beam_PreCommit_SQL_Java17.json`
PreCommit Typescript	N/A	`beam_PreCommit_Typescript.json`
PreCommit Website	N/A	`beam_PreCommit_Website.json`
PreCommit Website Stage GCS	N/A	`beam_PreCommit_Website_Stage_GCS.json`
PreCommit Whitespace	N/A	`beam_PreCommit_Whitespace.json`
PreCommit Xlang Generated Transforms	N/A	`beam_PreCommit_Xlang_Generated_Transforms.json`
PreCommit YAML Xlang Direct	N/A	`beam_PreCommit_Yaml_Xlang_Direct.json`
PreCommit Flink Container	N/A	`beam_PreCommit_Flink_Container.json`
PreCommit GoPrism	N/A	`beam_PreCommit_GoPrism.json`
PreCommit Integration and Load Test Framework	N/A	`beam_PreCommit_ItFramework.json`
PreCommit Java PVR Prism Loopback	N/A	`beam_PreCommit_Java_PVR_Prism_Loopback.json`
PreCommit Java Solace IO Direct	N/A	`beam_PreCommit_Java_Solace_IO_Direct.json`
PreCommit Prism Python	['3.10', '3.13']	`beam_PreCommit_Prism_Python.json`
PreCommit Python Dill tests with dill deps installed	['3.12']	`beam_PreCommit_Python_Dill.json`

Additional PreCommit jobs running basic SDK unit test on a matrices of operating systems.

Workflow name	Cron Status
Go Tests
Java Tests
Python Tests
TypeScript Tests
Build Wheels

PostCommit Jobs

PostCommit Jobs run in a schedule against master branch and generally do not get triggered in a PR. To manually trigger certain PostCommit job, attach a file named "Trigger file" (listed below) under .github/trigger_files/ folder.

Workflow name	Matrix	Trigger file
PostCommit Go	N/A	`beam_PostCommit_Go.json`
PostCommit Go Dataflow ARM	N/A	`beam_PostCommit_Go_Dataflow_ARM.json`
PostCommit Go VR Flink	N/A	`beam_PostCommit_Go_VR_Flink.json`
PostCommit Go VR Spark	N/A	`beam_PostCommit_Go_VR_Spark.json`
PostCommit Java Avro Versions	N/A	`beam_PostCommit_Java_Avro_Versions.json`
PostCommit Java BigQueryEarlyRollout	N/A	`beam_PostCommit_Java_BigQueryEarlyRollout.json`
PostCommit Java Dataflow V1	N/A	`beam_PostCommit_Java_DataflowV1.json`
PostCommit Java Dataflow V2	N/A	`beam_PostCommit_Java_DataflowV2.json`
PostCommit Java Delta IO Dataflow	N/A	`beam_PostCommit_Java_Delta_IO_Dataflow.json`
PostCommit Java Examples Dataflow ARM	['8','11','17','21','25']	`beam_PostCommit_Java_Examples_Dataflow_ARM.json`
PostCommit Java Examples Dataflow	N/A	`beam_PostCommit_Java_Examples_Dataflow.json`
PostCommit Java Examples Dataflow Java	['8','17','21','25']	`beam_PostCommit_Java_Examples_Dataflow_Java.json`
PostCommit Java Examples Dataflow V2	N/A	`beam_PostCommit_Java_Examples_Dataflow_V2.json`
PostCommit Java Examples Dataflow V2 Java	['8','17','21','25']	`beam_PostCommit_Java_Examples_Dataflow_V2_Java.json`
PostCommit Java Examples Direct	N/A	`beam_PostCommit_Java_Examples_Direct.json`
PostCommit Java Examples Flink	N/A	`beam_PostCommit_Java_Examples_Flink.json`
PostCommit Java Examples Spark	N/A	`beam_PostCommit_Java_Examples_Spark.json`
PostCommit Java Hadoop Versions	N/A	`beam_PostCommit_Java_Hadoop_Versions.json`
PostCommit Java IO Performance Tests	N/A	`beam_PostCommit_Java_IO_Performance_Tests.json`
PostCommit Java InfluxDbIO Integration Test	N/A	`beam_PostCommit_Java_InfluxDbIO_IT.json`
PostCommit Java Jpms Dataflow	N/A	`beam_PostCommit_Java_Jpms_Dataflow.json`
PostCommit Java Jpms Dataflow Versions	N/A	`beam_PostCommit_Java_Jpms_Dataflow_Versions.json`
PostCommit Java Jpms Direct	N/A	`beam_PostCommit_Java_Jpms_Direct.json`
PostCommit Java Jpms Direct Versions	N/A	`beam_PostCommit_Java_Jpms_Direct_Versions.json`
PostCommit Java Jpms Flink Java11	N/A	`beam_PostCommit_Java_Jpms_Flink_Java11.json`
PostCommit Java Jpms Spark Java11	N/A	`beam_PostCommit_Java_Jpms_Spark_Java11.json`
PostCommit Java Nexmark Dataflow	N/A	`beam_PostCommit_Java_Nexmark_Dataflow.json`
PostCommit Java Nexmark Dataflow V2	N/A	`beam_PostCommit_Java_Nexmark_Dataflow_V2.json`
PostCommit Java Nexmark Dataflow V2 Java	['11','17']	`beam_PostCommit_Java_Nexmark_Dataflow_V2_Java.json`
PostCommit Java Nexmark Direct	N/A	`beam_PostCommit_Java_Nexmark_Direct.json`
PostCommit Java Nexmark Flink	N/A	`beam_PostCommit_Java_Nexmark_Flink.json`
PostCommit Java Nexmark Spark	N/A	`beam_PostCommit_Java_Nexmark_Spark.json`
PostCommit Java PVR Flink Batch	N/A	`beam_PostCommit_Java_PVR_Flink_Batch.json`
PostCommit Java PVR Flink Streaming	N/A	`beam_PostCommit_Java_PVR_Flink_Streaming.json`
PostCommit Java SingleStoreIO IT	N/A	`beam_PostCommit_Java_SingleStoreIO_IT.json`
PostCommit Java PVR Spark3 Streaming	N/A	`beam_PostCommit_Java_PVR_Spark3_Streaming.json`
PostCommit Java PVR Spark Batch	N/A	`beam_PostCommit_Java_PVR_Spark_Batch.json`
PostCommit Java PVR Spark4 Batch	N/A	`beam_PostCommit_Java_PVR_Spark4_Batch.json`
PostCommit Java PVR Spark4 Streaming	N/A	`beam_PostCommit_Java_PVR_Spark4_Streaming.json`
PostCommit Java PVR Spark4 StructuredStreaming	N/A	`beam_PostCommit_Java_PVR_Spark4_StructuredStreaming.json`
PostCommit Java Tpcds Dataflow	N/A	`beam_PostCommit_Java_Tpcds_Dataflow.json`
PostCommit Java Tpcds Flink	N/A	`beam_PostCommit_Java_Tpcds_Flink.json`
PostCommit Java Tpcds Spark	N/A	`beam_PostCommit_Java_Tpcds_Spark.json`
PostCommit Java ValidatesRunner Dataflow JavaVersions	['8','25']	`beam_PostCommit_Java_ValidatesRunner_Dataflow_JavaVersions.json`
PostCommit Java ValidatesRunner Dataflow Streaming	N/A	`beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.json`
PostCommit Java ValidatesRunner Dataflow V2 Streaming	N/A	`beam_PostCommit_Java_ValidatesRunner_Dataflow_V2_Streaming.json`
PostCommit Java ValidatesRunner Dataflow V2	N/A	`beam_PostCommit_Java_ValidatesRunner_Dataflow_V2.json`
PostCommit Java ValidatesRunner Dataflow	N/A	`beam_PostCommit_Java_ValidatesRunner_Dataflow.json`
PostCommit Java ValidatesRunner Direct JavaVersions	['8','25']	`beam_PostCommit_Java_ValidatesRunner_Direct_JavaVersions.json`
PostCommit Java ValidatesRunner Direct	N/A	`beam_PostCommit_Java_ValidatesRunner_Direct.json`
PostCommit Java ValidatesRunner Flink	N/A	`beam_PostCommit_Java_ValidatesRunner_Flink.json`
PostCommit Java ValidatesRunner Spark	N/A	`beam_PostCommit_Java_ValidatesRunner_Spark.json`
PostCommit Java ValidatesRunner SparkStructuredStreaming	N/A	`beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.json`
PostCommit Java ValidatesRunner Spark4	N/A	`beam_PostCommit_Java_ValidatesRunner_Spark4.json`
PostCommit Java ValidatesRunner Twister2	N/A	`beam_PostCommit_Java_ValidatesRunner_Twister2.json`
PostCommit Java ValidatesRunner ULR	N/A	`beam_PostCommit_Java_ValidatesRunner_ULR.json`
PostCommit Java	N/A	`beam_PostCommit_Java.json`
PostCommit Javadoc	N/A	`beam_PostCommit_Javadoc.json`
PostCommit PortableJar Flink	N/A	`beam_PostCommit_PortableJar_Flink.json`
PostCommit PortableJar Spark	N/A	`beam_PostCommit_PortableJar_Spark.json`
PostCommit Python	['3.9','3.10','3.11','3.12']	`beam_PostCommit_Python.json`
PostCommit Python Arm	['3.9','3.10','3.11','3.12']	`beam_PostCommit_Python_Arm.json`
PostCommit Python Dependency	N/A	`beam_PostCommit_Python_Dependency.json`
PostCommit Python Examples Dataflow	N/A	`beam_PostCommit_Python_Examples_Dataflow.json`
PostCommit Python Examples Direct	['3.9','3.10','3.11','3.12']	`beam_PostCommit_Python_Examples_Direct.json`
PostCommit Python Examples Flink	['3.9','3.11','3.12']	`beam_PostCommit_Python_Examples_Flink.json`
PostCommit Python Examples Spark	['3.9','3.11','3.12']	`beam_PostCommit_Python_Examples_Spark.json`
PostCommit Python MongoDBIO IT	N/A	`beam_PostCommit_Python_MongoDBIO_IT.json`
PostCommit Python Nexmark Direct	N/A	`beam_PostCommit_Python_Nexmark_Direct.json`
PostCommit Python ValidatesContainer Dataflow	['3.9','3.10','3.11','3.12']	`beam_PostCommit_Python_ValidatesContainer_Dataflow.json`
PostCommit Python ValidatesContainer Dataflow With RC	['3.9','3.10','3.11','3.12']	`beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.json`
PostCommit Python ValidatesRunner Dataflow	['3.9','3.11','3.12']	`beam_PostCommit_Python_ValidatesRunner_Dataflow.json`
PostCommit Python ValidatesRunner Flink	['3.9','3.11','3.12']	`beam_PostCommit_Python_ValidatesRunner_Flink.json`
PostCommit Python ValidatesRunner Spark	['3.9','3.10','3.11','3.12']	`beam_PostCommit_Python_ValidatesRunner_Spark.json`
PostCommit Python Xlang Gcp Dataflow	N/A	`beam_PostCommit_Python_Xlang_Gcp_Dataflow.json`
PostCommit Python Xlang Gcp Direct	N/A	`beam_PostCommit_Python_Xlang_Gcp_Direct.json`
PostCommit Python Xlang IO Dataflow	N/A	`beam_PostCommit_Python_Xlang_IO_Dataflow.json`
PostCommit Python Versions	N/A	`beam_PostCommit_Python_Versions.json`
PostCommit SQL	N/A	`beam_PostCommit_SQL.json`
PostCommit TransformService Direct	N/A	`beam_PostCommit_TransformService_Direct.json`
PostCommit Website Test	N/A	`beam_PostCommit_Website_Test.json`
PostCommit XVR GoUsingJava Dataflow	N/A	`beam_PostCommit_XVR_GoUsingJava_Dataflow.json`
PostCommit XVR Direct	N/A	`beam_PostCommit_XVR_Direct.json`
PostCommit XVR Flink	N/A	`beam_PostCommit_XVR_Flink.json`
PostCommit XVR JavaUsingPython Dataflow	N/A	`beam_PostCommit_XVR_JavaUsingPython_Dataflow.json`
PostCommit XVR PythonUsingJava Dataflow	N/A	`beam_PostCommit_XVR_PythonUsingJava_Dataflow.json`
PostCommit XVR PythonUsingJavaSQL Dataflow	N/A	`beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.json`
PostCommit XVR Spark3	N/A	`beam_PostCommit_XVR_Spark3.json`
PostCommit YAML Xlang Direct	N/A	`beam_PostCommit_Yaml_Xlang_Direct.json`
Python Validates Container Dataflow ARM	['3.9','3.10','3.11','3.12']	`beam_Python_ValidatesContainer_Dataflow_ARM.json`
PostCommit Java ValidatesRunner Dataflow Streaming TagEncodingV2	N/A	`beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.json`
PostCommit Java ValidatesRunner Dataflow Streaming Engine	N/A	`beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.json`
PostCommit Python Portable Flink	N/A	`beam_PostCommit_Python_Portable_Flink.json`
PostCommit Python Xlang IO Direct	N/A	`beam_PostCommit_Python_Xlang_IO_Direct.json`
PostCommit Python Xlang Messaging Direct	N/A	`beam_PostCommit_Python_Xlang_Messaging_Direct.json`

PerformanceTests and Benchmark Jobs

Workflow name	Matrix	Cron Status
CloudML Benchmarks Dataflow	N/A
Inference Python Benchmarks Dataflow	N/A
Java JMH	N/A
Performance Tests AvroIOIT HDFS	N/A
Performance Tests AvroIOIT	N/A
Performance Tests BigQueryIO Batch Java Avro	N/A
Performance Tests BigQueryIO Batch Java Json	N/A
Performance Tests BigQueryIO Streaming Java	N/A
Performance Tests BigQueryIO Read Python	N/A
Performance Tests BigQueryIO Write Python Batch	N/A
PerformanceTests Cdap	N/A
PerformanceTests Compressed TextIOIT HDFS	N/A
PerformanceTests Compressed TextIOIT	N/A
PerformanceTests HadoopFormat	N/A
PerformanceTests JDBC	N/A
PerformanceTests Kafka IO	N/A
PerformanceTests ManyFiles TextIOIT HDFS	N/A
PerformanceTests ManyFiles TextIOIT	N/A
PerformanceTests MongoDBIO IT	N/A
PerformanceTests ParquetIOIT HDFS	N/A
PerformanceTests ParquetIOIT	N/A
PerformanceTests PubsubIOIT Python Streaming	N/A
PerformanceTests SingleStoreIO	N/A
PerformanceTests SpannerIO Read 2GB Python	N/A
PerformanceTests SpannerIO Write 2GB Python Batch	N/A
PerformanceTests SparkReceiver IO	N/A
PerformanceTests SQLBigQueryIO Batch Java	N/A
PerformanceTests TextIOIT HDFS	N/A
PerformanceTests TextIOIT Python	N/A
PerformanceTests TextIOIT	N/A
PerformanceTests TFRecordIOIT HDFS	N/A
PerformanceTests TFRecordIOIT	N/A
PerformanceTests WordCountIT PythonVersions	['3.9','3.10','3.11','3.12']
PerformanceTests XmlIOIT HDFS	N/A
PerformanceTests XmlIOIT	N/A
PerformanceTests xlang KafkaIO Python	N/A
Python Cost Benchmarks Dataflow	N/A

LoadTests Jobs

Workflow name	Matrix	Cron Status
LoadTests Go CoGBK Dataflow Batch	N/A
LoadTests Go CoGBK Flink Batch	N/A
LoadTests Go Combine Dataflow Batch	N/A
LoadTests Go Combine Flink Batch	N/A
LoadTests Go GBK Dataflow Batch	N/A
LoadTests Go GBK Flink Batch	N/A
LoadTests Go ParDo Dataflow Batch	N/A
LoadTests Go ParDo Flink Batch	N/A
LoadTests Go SideInput Dataflow Batch	N/A
LoadTests Go SideInput Flink Batch	N/A
LoadTests Java CoGBK Dataflow Batch	N/A
LoadTests Java CoGBK Dataflow Streaming	N/A
LoadTests Java CoGBK Dataflow V2 Batch JavaVersions	['11','17']
LoadTests Java CoGBK Dataflow V2 Streaming JavaVersions	['11','17']
LoadTests Java CoGBK SparkStructuredStreaming Batch	N/A
LoadTests Java Combine Dataflow Batch	N/A
LoadTests Java Combine Dataflow Streaming	N/A
LoadTests Java Combine SparkStructuredStreaming Batch	N/A
LoadTests Java GBK Dataflow Batch	N/A
LoadTests Java GBK Dataflow Streaming	N/A
LoadTests Java GBK Dataflow V2 Batch	N/A
LoadTests Java GBK Dataflow V2 Batch Java17	N/A
LoadTests Java GBK Dataflow V2 Streaming	N/A
LoadTests Java GBK Dataflow V2 Streaming Java17	N/A
LoadTests Java GBK Smoke	N/A
LoadTests Java GBK SparkStructuredStreaming Batch	N/A
LoadTests Java ParDo Dataflow Batch	N/A
LoadTests Java ParDo Dataflow Streaming	N/A
LoadTests Java ParDo Dataflow V2 Batch JavaVersions	['11','17']
LoadTests Java ParDo Dataflow V2 Streaming JavaVersions	['11','17']
LoadTests Java ParDo SparkStructuredStreaming Batch	N/A
LoadTests Java Combine Smoke	N/A
LoadTests Python CoGBK Dataflow Batch	N/A
LoadTests Python CoGBK Dataflow Streaming	N/A
LoadTests Python CoGBK Flink Batch	N/A
LoadTests Python Combine Dataflow Batch	N/A
LoadTests Python Combine Dataflow Streaming	N/A
LoadTests Python Combine Flink Batch	N/A
LoadTests Python Combine Flink Streaming	N/A
LoadTests Python FnApiRunner Microbenchmark	N/A
LoadTests Python GBK Dataflow Batch	N/A
LoadTests Python GBK Dataflow Streaming	N/A
LoadTests Python GBK Flink Batch	N/A
LoadTests Python GBK reiterate Dataflow Batch	N/A
LoadTests Python GBK reiterate Dataflow Streaming	N/A
LoadTests Python ParDo Dataflow Batch	N/A
LoadTests Python ParDo Dataflow Streaming	N/A
LoadTests Python ParDo Flink Batch	N/A
LoadTests Python ParDo Flink Streaming	N/A
LoadTests Python SideInput Dataflow Batch	N/A
LoadTests Python Smoke	N/A
LoadTests Java PubsubIO	N/A
StressTests Java BigQueryIO	N/A
StressTests Java BigTableIO	N/A
StressTests Java KafkaIO	N/A
StressTests Java PubSubIO	N/A
StressTests Java SpannerIO	N/A

Other Jobs

Workflow name	Matrix	Cron Status
Cancel Stale Dataflow Jobs	N/A
Clean Up GCP Resources	N/A
Clean Up Prebuilt SDK Images	N/A
Cleanup Dataproc Resources	N/A
Community Metrics Prober	N/A
PostRelease Nightly Snapshot	N/A
Playground CI Nightly	["java", "go", "python"]
Publish Beam SDK Snapshots	N/A
Publish BeamMetrics	N/A
Publish Docker Snapshots	N/A
Publish Python VLLM Image	N/A
Publish Website	N/A
Release Nightly Snapshot	N/A
Release Nightly Snapshot Python	N/A
Rotate IO-Datastores Cluster Credentials	N/A
Rotate Metrics Cluster Credentials	N/A
Beam Metrics Report	N/A
GCP Security Log Analyzer	N/A
Infrastructure Policy Enforcer	N/A
Modify the GCP User Roles according to the infra/users.yml file	N/A
Service Account Keys Management	N/A
Upgrade GCP Libraries BOM	N/A