Create and manage batch operation jobs

This page describes how to create, view, list, cancel, and delete storage batch operations jobs. It also describes how to use Cloud Audit Logs with storage batch operations jobs.

Before you begin

To create and manage storage batch operations jobs, complete the steps in the following sections.

Configure Storage Intelligence

To create and manage storage batch operations jobs, configure Storage Intelligence on the bucket where you want to run the job.

Enable the storage batch operations API

Enable the storage batch operations API.

gcloud services enable storagebatchoperations.googleapis.com

Create a manifest

If you want to use a manifest for object selection, create a manifest file. Using a manifest is one of the ways you can select objects to process in a storage batch operations job.

Create a storage batch operations job

This section describes how to create a storage batch operations job.

To get the permissions that you need to create a storage batch operations job, ask your administrator to grant you the Storage Admin (roles/storage.admin) IAM role on the project. For more information about granting roles, see Manage access to projects, folders, and organizations.

You might also be able to get the required permissions through custom roles or other predefined roles.

Console

In the Google Cloud console, go to the Cloud Storage Buckets page.

Go to Buckets
In the list of buckets, click the name of the bucket that contains the objects on which you want to perform batch operations.

The Bucket details page opens, with the Objects tab selected.
Click Create batch operations.
In the Select operation pane, choose the operation type:
- Manage object holds: Select Temporary hold or Event-based hold. For more information, see object holds.
- Update object metadata: To add object metadata, do the following:
  - To add custom metadata, complete the following steps:
    1. In the Key field, enter a key name.
    2. In the Value field, enter a value for that key.
    3. Optional: Click + Add item to add more key-value pairs.
  - To update fixed-key metadata, complete the following steps:
    1. To expand the Update fixed-key metadata section, click the expander arrow.
    2. In the Select one or more metadata to update list, select metadata items to edit.
- Update/Rotate encryption key: To use or update the encryption key for objects, do the following:
  1. In the Select a Cloud KMS key list, select a customer-managed encryption key (CMEK).
  2. Optional: Select Switch project to pick a key from another project or select Enter key manually to fill details.
- Delete objects: To delete objects, do the following:
  1. Check whether Object Versioning is enabled.
  2. If Object Versioning is enabled, choose one of the following deletion options:
    - Select Delete all versions of the objects to remove both live and noncurrent versions.
    - Select Permanently delete live versions to remove only the live version.
    If Object Versioning is not enabled, any objects selected for deletion are permanently deleted.
  Warning: Accidental object deletion carries a risk of data loss. Enable soft delete to recover accidental object deletions.
Click Next.
In the Name operation & specify objects pane, do the following:
1. In the Name field, enter a name.
2. Optional: In the Description field, enter a description.
3. In the Specify objects section, define a criterion to process objects from the bucket. Choose one of the following options:
  - Select all objects: Includes all objects in the bucket.
  - Select objects by using prefix filters: To define the list of objects by using prefix filters, do the following:
    1. In the Enter Prefixes of the objects to be included field, enter a prefix.
    2. Optional: Click + Add prefix to specify additional prefixes.
  - Upload lists of objects using manifest CSV files: To use a manifest file for selecting objects, do the following:
    1. Upload your manifest CSV file to a bucket. This file must contain headers for Bucket name, Object key, and Generation number.
    2. In the Select manifest file mode list, choose one of the following options:
      - If you select Select a manifest file from Cloud Storage, click Browse in the Select a manifest file from Cloud Storage field. In the Select object dialog that appears, navigate to your manifest CSV file, then click Select.
      - If you select Select multiple manifest files using wildcard, enter the file path in the Enter manifest file location using wildcard field. For example, bucket-name/folder/manifest_*.
Click Create.

Command line

To define the list of objects for your batch operation job, you can choose one of the following source configurations:

Project as the source: Targets objects for a project using a Storage Insights dataset configuration. Instead of specifying individual buckets or prefixes, you can specify advanced filter parameters, such as --insights-dataset-config, --target-project, --bucket-filters, and --object-filters. For details see, Create a job using advanced filters.
Buckets as the source: Targets objects within specific buckets. You must specify one of the following flags:
- --bucket or --bucket-list to define the target buckets.
- A manifest CSV file (--manifest-location) or object prefixes (--included-object-prefixes) to define the target objects.

In the Google Cloud console, activate Cloud Shell.

Activate Cloud Shell

At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.
Use Google Cloud CLI version 516.0.0 or later.
To set the default project, run the gcloud config set project command:
```
gcloud config set project PROJECT_ID
```
Where PROJECT_ID is the ID of your project.
Optional: Run a dry run job. Before executing any job, we recommend that you run the job in dry run mode to verify the object selection criteria and check for any errors. The dry run doesn't modify any objects.

In your development environment, run the gcloud storage batch-operations jobs create command with the --dry-run flag:
```
gcloud storage batch-operations jobs create DRY_RUN_JOB_NAME \
{--bucket=BUCKET | --bucket-list=BUCKET_LIST} OBJECT_SELECTION_FLAG JOB_TYPE_FLAG \
--dry-run
```
Where:
- DRY_RUN_JOB_NAME is the name of the storage batch operations dry run job.
The other parameters are the same as for the actual job. For more information, see parameter descriptions.

To view the results of the dry run, see Get storage batch operations job details.
After a successful dry run, run the gcloud storage batch-operations jobs create command.
```
gcloud storage batch-operations jobs create JOB_NAME \
{--bucket=BUCKET | --bucket-list=BUCKET_LIST} OBJECT_SELECTION_FLAG JOB_TYPE_FLAG
```
Where the parameters are as follows:
- JOB_NAME is the name of the storage batch operations job.
  
  Note: When you create the actual job, specify a unique job name or delete the dry run job first. Reusing the dry run's job name causes the creation to fail with an already exists error.
- --bucket: BUCKET is the name of the bucket containing the objects you want to process.
- --bucket-list: BUCKET_LIST is a comma-separated list of one or more bucket names containing the objects you want to process. You can specify up to 1,000 buckets from any project, as long as each bucket is enrolled in a storage intelligence plan.
- OBJECT_SELECTION_FLAG is one of the following flags that you need to specify:
  - --included-object-prefixes: Specify one or more object prefixes. For example:
    - To match a single prefix, use: --included-object-prefixes='prefix1'.
    - To match multiple prefixes, use a comma-separated prefix list: --included-object-prefixes='prefix1,prefix2'.
    - To include all objects, use an empty prefix: --included-object-prefixes=''.
  - --manifest-location: Specify the manifest location. For example, gs://bucket_name/path/object_name.csv.
- JOB_TYPE_FLAG is one of the following flags that you need to specify, depending on the job type.
  - --delete-object: Delete one or more objects.
    - If Object Versioning is enabled for the bucket, current objects transition to a noncurrent state, and noncurrent objects are skipped.
    - If Object Versioning is disabled for the bucket, the delete operation permanently deletes objects and skips noncurrent objects.
  - --enable-permanent-object-deletion: Permanently delete objects. Use this flag along with the --delete-object flag to permanently delete both live and noncurrent objects in a bucket, regardless of the bucket's object versioning configuration.
  - --rewrite-object: Update the customer-managed encryption keys for one or more objects. You can also use this flag to change the object's storage class by specifying the storage-class key. Supported storage classes include STANDARD, NEARLINE, COLDLINE, and ARCHIVE. For example, --rewrite-object=storage-class=NEARLINE.
    
    Caution: If you change the storage class of an object in an Autoclass-enabled bucket, the operation has no effect. The operation reports success, but Cloud Storage resets the object's storage class to the default class defined by Autoclass.
    Note: Changing the storage class of an object involves rewriting its data. A rewrite is a Class A operation. These operations might incur additional charges depending on the age, location, and storage class of the data. For more information, see Cloud Storage pricing. If you enable Object Versioning for your bucket, Cloud Storage retains the original object as a noncurrent version. This version remains in the bucket until you explicitly delete it.
  - --set-object-acls-from-file: Patch object access control lists (ACLs). Provide a JSON or YAML file with grants to add or update for entities such as allUsers or allAuthenticatedUsers. For example: --set-object-acls-from-file=acl-updates.yaml or --set-object-acls-from-file=acl-updates.json.
    
    The structure of the YAML file for updates is as follows:
```
grants:
  - entity: allAuthenticatedUsers
    role: READER
  remove_entities:
  - allUsers
```
    The structure of the JSON file for updates is as follows:
```
{
"grants": [
  {
    "entity": "allAuthenticatedUsers",
    "role": "READER"
  }
],
"remove_entities": [
  "allUsers"
]
}
```
  - --put-object-event-based-hold: Enable event-based object holds.
  - --no-put-object-event-based-hold: Disable event-based object holds.
  - --put-object-temporary-hold: Enable temporary object holds.
  - --no-put-object-temporary-hold: Disable temporary object holds.
    
    The following example shows how to create a job to update the Content-Language metadata to en for all objects listed in manifest.csv.
```
gcloud storage batch-operations jobs create my-job \
--bucket=my-bucket \
--manifest-location=gs://my-bucket/manifest.csv \
--put-metadata=Content-Language=en
```
    The following example shows how to create a job targeting multiple buckets to update Content-Language to en-us:
```
gcloud storage batch-operations jobs create my-job \
--bucket-list=bucket1,bucket2 \
--included-object-prefixes='' \
--put-metadata=Content-Language=en-us
```
  - --put-metadata: Update object metadata. Specify the key-value pair for the object metadata you want to modify. You can specify one or more key-value pairs as a list. You can also set object retention configurations using the --put-metadata flag. To do so, specify the retention parameters using the Retain-Until and Retention-Mode fields. For example,
```
gcloud storage batch-operations jobs create my-job \
--bucket=my-bucket \
--manifest-location=gs://my-bucket/manifest.csv \
--put-metadata=Retain-Until=RETAIN_UNTIL_TIME,Retention-Mode=RETENTION_MODE
```
    Where:
    - RETAIN_UNTIL_TIME is the date and time, in RFC 3339 format, until which the object is retained. For example, 2025-10-09T10:30:00Z. To set the retention configuration on an object, you'll need to enable retention on the bucket which contains the object.
    - RETENTION_MODE is the retention mode, either Unlocked or Locked.
      
      When you send a request to update the RETENTION_MODE and RETAIN_UNTIL_TIME fields, consider the following:
      - To update the object retention configuration, you must provide non-empty values for both RETENTION_MODE and RETAIN_UNTIL_TIME fields; setting only one results in an INVALID_ARGUMENT error.
      - You can extend the RETAIN_UNTIL_TIME value for objects in both Unlocked or Locked modes.
      - The object retention must be in Unlocked mode if you want to do the following:
        
        Reduce the RETAIN_UNTIL_TIME value.
        
        Remove the retention configuration. To remove the configuration, you'll need to provide empty values for both RETENTION_MODE and RETAIN_UNTIL_TIME fields.
      - If you omit both RETENTION_MODE and RETAIN_UNTIL_TIME fields, the retention configuration remains unchanged.

--clear-all-object-custom-contexts: Delete all existing object contexts.

The following example shows how to create a job to clear all object contexts for objects listed in manifest.csv:
```
gcloud storage batch-operations jobs create my-job \
--bucket=my-bucket \
--manifest-location=gs://my-bucket/manifest.csv \
--clear-all-object-custom-contexts
```
Caution: Clearing all object contexts deletes all the existing contexts on the object. To preserve the previous object contexts, enable Cloud Logging. The logs contain contexts from before the update, which you can use to restore previous values.
--clear-object-custom-contexts: Remove contexts with specific keys. You can also update specific contexts along with removing keys by using both the --clear-object-custom-contexts flag and one of the following flags:
- --update-object-custom-contexts: Provide a map of key-value pairs.
  
  The following example shows how to create a job to remove the context with key temp-id and update or insert context with key project-id and cost-center for all objects listed in manifest.csv:
```
gcloud storage batch-operations jobs create my-job \
--bucket=my-bucket \
--manifest-location=gs://my-bucket/manifest.csv \
--clear-object-custom-contexts=temp-id \
--update-object-custom-contexts=project-id=project-A,cost-center=engineering
```
  Note: You can't specify the same key in both the --clear-object-custom-contexts flag and the --update-object-custom-contexts flag. Doing so results in an INVALID_ARGUMENT error.
- --update-object-custom-contexts-file: Provide the path to a JSON or YAML file with key-value pairs.
  
  The following example shows how to create a job to process objects defined in manifest.csv. The job does the following:
  - Removes all contexts with the temp-id key.
  - Updates existing contexts with the project-id and cost-center keys defined in the /tmp/context_updates.json file.
```
gcloud storage batch-operations jobs create my-job \
--bucket=my-bucket \
--manifest-location=gs://my-bucket/manifest.csv \
--clear-object-custom-contexts=temp-id \
--update-object-custom-contexts-file=/tmp/context_updates.json
```
  Where /tmp/context_updates.json contains the following object contexts:
```
{
"project-id": {"value": "project-A"},
"cost-center": {"value": "engineering"}
}
```

Create and manage batch operation jobs Stay organized with collections Save and categorize content based on your preferences.

Before you begin

Configure Storage Intelligence

Enable the storage batch operations API

Create a manifest

Create a storage batch operations job

Console

Command line

Client libraries

C++

PHP

JSON API

Get storage batch operations job details

Console

Command line

Client libraries

C++

PHP

JSON API

List bucket operations

Describe a bucket operation

List storage batch operations jobs

Console

Command line

Client libraries

C++

PHP

JSON API

Cancel a storage batch operations job

Console

Command line

Client libraries

C++

PHP

JSON API

Delete a storage batch operations job

Console

Command line

Client libraries

C++

PHP

JSON API

Create a storage batch operations job using Storage Insights datasets

Use advanced filters

Command line

JSON API

Use a manifest file

Create a manifest using Storage Insights datasets

Create a storage batch operations job using a manifest file

Console

Command line

Integration with VPC Service Controls

Use Cloud Audit Logs for storage batch operations jobs

Next steps

Create and manage batch operation jobs