Skip to content

ArrayQueryParameter seemingly turns empty list into NULL #2325

@nkoep

Description

@nkoep

Issue

It seems that empty lists are not properly serialized by the client libraries. The example below demonstrates the attempt to use an ArrayQueryParameter as optional filter list: if the list is empty, all items in the source table are supposed to be returned. To that end, we use a short-circuit condition that first checks if the provided array is empty via WHERE ARRAY_LENGTH(@country_codes) = 0 and only tries to match items via OR c IN UNNEST(@country_codes) when the first check evaluates to FALSE.

The observed behavior, however, implies that when an empty python list is passed, the parameter ultimately ends up being converted as NULL. To remedy this, it is necessary to wrap the first use of the parameter in a COALESCE(...) call which is very unintuitive.

Aside: this was initially observed by inspecting the job and noticing that the parameter is passed as this (notice the missing parameterValue object:

{"name":"country_codes","parameterType":{"type":"ARRAY","arrayType":{"type":"STRING"}}}

Environment details

  • OS type and version: MacOS 15.7.1
  • Python version: 3.11 (also tested on 3.14)
  • pip version: uv 0.9.2 :)
  • google-cloud-bigquery version: 3.29.0 (also tested on 3.30.0 and 3.38.0)

Steps to reproduce

  1. uv run --no-project --with "google-cloud-bigquery==3.29.0" --python 3.11 python mre.py

Code example

from google.cloud import bigquery

query_bare = """
SELECT
  c
FROM UNNEST(["ab", "cd"]) c
WHERE
  ARRAY_LENGTH(@country_codes) = 0 OR c IN UNNEST(@country_codes)
"""

query_coalesce = """
SELECT
  c
FROM UNNEST(["ab", "cd"]) c
WHERE
  ARRAY_LENGTH(COALESCE(@country_codes, [])) = 0 OR c IN UNNEST(@country_codes)
"""

query_exists = """
SELECT
  c
FROM UNNEST(["ab", "cd"]) c
WHERE
  EXISTS(SELECT 1 FROM UNNEST(@country_codes) ccs WHERE STARTS_WITH(c, ccs))
"""


def check_optional_filter(items: list[str]):
    client = bigquery.Client()
    for name, query in [("bare", query_bare), ("coalesce", query_coalesce), ("exists", query_exists)]:
        result = client.query(
            query,
            job_config=bigquery.QueryJobConfig(
                query_parameters=[
                    bigquery.ArrayQueryParameter("country_codes", "STRING", items)
                ]
            ),
        )
        rows = list(result.result())
        print(f"{name}: {rows}")


print("No filter -> return all")
check_optional_filter([])
print()

print("Filter ab -> ab")
check_optional_filter(["ab"])

Stack trace

No filter -> return all
bare: []
coalesce: [Row(('ab',), {'c': 0}), Row(('cd',), {'c': 0})]
exists: []

Filter ab -> ab
bare: [Row(('ab',), {'c': 0})]
coalesce: [Row(('ab',), {'c': 0})]
exists: [Row(('ab',), {'c': 0})]

Metadata

Metadata

Assignees

Labels

api: bigqueryIssues related to the googleapis/python-bigquery API.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions