feat!: reading JSON data as a custom arrow extension type #1458
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
7f8d18c to
5dcbdc0
Compare
5dcbdc0 to
94ef33b
Compare
Collaborator
|
Getting some test failures in presubmit: Also, could we make sure we add a |
94ef33b to
5a2baa6
Compare
65770f3 to
a2edcbf
Compare
tswast
approved these changes
Mar 10, 2025
Collaborator
tswast
left a comment
There was a problem hiding this comment.
Thanks! One suggestion, but otherwise looks good.
bigframes/dtypes.py
Outdated
| GEO_DTYPE = gpd.array.GeometryDtype() | ||
| # JSON | ||
| JSON_DTYPE = db_dtypes.JSONDtype() | ||
| JSON_DTYPE = pd.ArrowDtype(db_dtypes.JSONArrowType()) |
Collaborator
There was a problem hiding this comment.
Should we switch to pd.ArrowDtype(pyarrow.json_(pyarrow.string())) if pyarrow.json_ is available?
Also, would be good to make sure we align with OBJ_REF_DTYPE by creating a JSON_ARROW_TYPE variable to use here and there.
Contributor
Author
There was a problem hiding this comment.
I will follow up with b/401055693. Thanks for reviewing!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We initially implemented a local pandas extension (
db_dtypes.JSONType) for handling JSON data. Subsequently, the Arrow project introduced a native JSON data type in pyarrow after v19.0. We've opted to adopt this native type as our primary solution (see go/bf-json2 for internal design document). To ensure compatibility for users with older pyarrow versions, we've been using a custom Arrow extension as a fallback. This change transitions to using this custom Arrow extension as a stepping stone towards fully integrating the native pyarrow JSON type.Release-As: 1.40.0
Fixes internal issue 401054811🦕