-
Notifications
You must be signed in to change notification settings - Fork 321
Description
Issue
It seems that empty lists are not properly serialized by the client libraries. The example below demonstrates the attempt to use an ArrayQueryParameter as optional filter list: if the list is empty, all items in the source table are supposed to be returned. To that end, we use a short-circuit condition that first checks if the provided array is empty via WHERE ARRAY_LENGTH(@country_codes) = 0 and only tries to match items via OR c IN UNNEST(@country_codes) when the first check evaluates to FALSE.
The observed behavior, however, implies that when an empty python list is passed, the parameter ultimately ends up being converted as NULL. To remedy this, it is necessary to wrap the first use of the parameter in a COALESCE(...) call which is very unintuitive.
Aside: this was initially observed by inspecting the job and noticing that the parameter is passed as this (notice the missing parameterValue object:
{"name":"country_codes","parameterType":{"type":"ARRAY","arrayType":{"type":"STRING"}}}
Environment details
- OS type and version: MacOS 15.7.1
- Python version: 3.11 (also tested on 3.14)
- pip version: uv 0.9.2 :)
google-cloud-bigqueryversion: 3.29.0 (also tested on 3.30.0 and 3.38.0)
Steps to reproduce
uv run --no-project --with "google-cloud-bigquery==3.29.0" --python 3.11 python mre.py
Code example
from google.cloud import bigquery
query_bare = """
SELECT
c
FROM UNNEST(["ab", "cd"]) c
WHERE
ARRAY_LENGTH(@country_codes) = 0 OR c IN UNNEST(@country_codes)
"""
query_coalesce = """
SELECT
c
FROM UNNEST(["ab", "cd"]) c
WHERE
ARRAY_LENGTH(COALESCE(@country_codes, [])) = 0 OR c IN UNNEST(@country_codes)
"""
query_exists = """
SELECT
c
FROM UNNEST(["ab", "cd"]) c
WHERE
EXISTS(SELECT 1 FROM UNNEST(@country_codes) ccs WHERE STARTS_WITH(c, ccs))
"""
def check_optional_filter(items: list[str]):
client = bigquery.Client()
for name, query in [("bare", query_bare), ("coalesce", query_coalesce), ("exists", query_exists)]:
result = client.query(
query,
job_config=bigquery.QueryJobConfig(
query_parameters=[
bigquery.ArrayQueryParameter("country_codes", "STRING", items)
]
),
)
rows = list(result.result())
print(f"{name}: {rows}")
print("No filter -> return all")
check_optional_filter([])
print()
print("Filter ab -> ab")
check_optional_filter(["ab"])Stack trace
No filter -> return all
bare: []
coalesce: [Row(('ab',), {'c': 0}), Row(('cd',), {'c': 0})]
exists: []
Filter ab -> ab
bare: [Row(('ab',), {'c': 0})]
coalesce: [Row(('ab',), {'c': 0})]
exists: [Row(('ab',), {'c': 0})]