Skip to content

Improve the InMemory Catalog Implementation #289

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Fokko merged 24 commits into from
Mar 13, 2024
Merged

Improve the InMemory Catalog Implementation #289

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
066e8c4
extract InMemoryCatalog out of test
kevinjqliu Jan 20, 2024
b1a99f7
generalize InMemoryCatalog
kevinjqliu Jan 20, 2024
32c449e
make write work
kevinjqliu Jan 21, 2024
3c6e06a
write to temporary location
kevinjqliu Jan 21, 2024
21b1e50
can override table location
kevinjqliu Jan 22, 2024
e2541ac
memory.py -> in_memory.py
kevinjqliu Jan 22, 2024
3013bdb
fix test_commit_table
kevinjqliu Jan 22, 2024
be3eb1c
rebase from main
kevinjqliu Jan 30, 2024
f80849a
revert fs changes
kevinjqliu Jan 30, 2024
faea973
fix tests
kevinjqliu Jan 30, 2024
4437a30
add docs and comments
kevinjqliu Jan 30, 2024
aabcde0
comma
kevinjqliu Jan 30, 2024
b91e1cf
comment
kevinjqliu Jan 30, 2024
2834bae
order
kevinjqliu Jan 30, 2024
341f5ba
fix test
kevinjqliu Jan 30, 2024
67c028a
add license
kevinjqliu Jan 30, 2024
96ba8de
`create_table` write metadata file
kevinjqliu Jan 31, 2024
c7f9053
Merge branch 'main' into kevinliu/test
kevinjqliu Mar 1, 2024
8a7b876
move InMemoryCatalog back to test_base
kevinjqliu Mar 1, 2024
10adb1c
remove unused references
kevinjqliu Mar 1, 2024
a466e5f
Update mkdocs/docs/configuration.md
kevinjqliu Mar 5, 2024
03ec82b
Update mkdocs/docs/configuration.md
kevinjqliu Mar 5, 2024
58b34ca
Update tests/catalog/test_base.py
kevinjqliu Mar 5, 2024
810c3c5
remove schema_id
kevinjqliu Mar 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
remove unused references
  • Loading branch information
@kevinjqliu
kevinjqliu committed Mar 1, 2024
commit 10adb1cb03d4fd2b6ec457b5d8689b24e353bd00
9 changes: 0 additions & 9 deletions mkdocs/docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -265,15 +265,6 @@ catalog:
The In-Memory catalog uses in-memory data-structures to store information.
This is useful for test, demo, and playground. Do not use in production as the data is not persisted.

While you can specify In-Memory catalog in the configuration file like this, it is not recommended since information is only persisted for the duration of the function call.

```yaml
catalog:
default:
type: in_memory
warehouse: /tmp/warehouse # default warehouse location
```

# Concurrency

PyIceberg uses multiple threads to parallelize operations. The number of workers can be configured by supplying a `max-workers` entry in the configuration file, or by setting the `PYICEBERG_MAX_WORKERS` environment variable. The default value depends on the system hardware and Python version. See [the Python documentation](https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor) for more details.
1 change: 0 additions & 1 deletion pyiceberg/catalog/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,6 @@ class CatalogType(Enum):
GLUE = "glue"
DYNAMODB = "dynamodb"
SQL = "sql"
IN_MEMORY = "in_memory"


def load_rest(name: str, conf: Properties) -> Catalog:
Expand Down