Add pre-commit and Python >= 3.12 numpy dependencies to Poetry - #877
Add pre-commit and Python >= 3.12 numpy dependencies to Poetry#877rjurney wants to merge 6 commits into
Conversation
… problem with pandas/numpy version 2.5.1 being required by Python >= 3.10
…y/pyproject-numpy # Conflicts: # python/poetry.lock # python/pyproject.toml
There was a problem hiding this comment.
Pull request overview
Updates the Python package’s Poetry configuration to add development tooling and adjust Python-version-specific dependency resolution, aimed at addressing NumPy/Python compatibility issues.
Changes:
- Adds a new
[tool.poetry.dependencies]section including a PySpark requirement and conditional NumPy constraints by Python version. - Adds
pre-commitas a dev dependency. - Regenerates
python/poetry.lockto reflect the updated dependency graph.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| python/pyproject.toml | Introduces runtime dependencies (PySpark + conditional NumPy) and adds pre-commit to dev deps. |
| python/poetry.lock | Lockfile updated/regenerated to include new dependencies and resolution changes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #877 +/- ##
==========================================
+ Coverage 79.26% 79.58% +0.31%
==========================================
Files 81 81
Lines 4712 4682 -30
Branches 554 574 +20
==========================================
- Hits 3735 3726 -9
+ Misses 977 956 -21 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| poetry-plugin-export = ">=1.8" | ||
|
|
||
| [tool.poetry.dependencies] | ||
| pyspark = ">=3.5, <4.2" |
There was a problem hiding this comment.
why dont we allow spark 4.2?
There was a problem hiding this comment.
Why are we adding pyspark dependency while we are considered it as "provided"?
| poetry-plugin-export = ">=1.8" | ||
|
|
||
| [tool.poetry.dependencies] | ||
| pyspark = ">=3.5, <4.2" |
There was a problem hiding this comment.
Why are we adding pyspark dependency while we are considered it as "provided"?
|
|
||
| [tool.poetry.dependencies] | ||
| pyspark = ">=3.5, <4.2" | ||
| numpy = [ |
There was a problem hiding this comment.
Why are we adding numpy dependency?
Could you clarify please? We do not use pandas anyhow in the graphframes code.
I would prefer to achieve the same by just introducing a dependency group. |
What changes were proposed in this pull request?
Why are the changes needed?
I had numpy errors in Python 3.13/3.14 and they use a different version of numpy from pandas being a greater version and dropping Python <= 3.12 support. This fixes those errors with a conditional numpy dependency.
Also, pre-commit was not a dev dependency and it should be.
These changes are needed for my Pregel tutorial to work. Aside from this, it is now complete! I ran it end-to-end. See #578