feat: add validate search field#1318
Closed
srini047 wants to merge 198 commits into
Closed
Conversation
* feat: add tensor type for ndarray * fix: fix mypy typing
* chore: add ruff and fix most of the errors Signed-off-by: Sami Jaghouar <sami.jaghouar@jina.ai> * chore: fix all of the rest of ruff problem Signed-off-by: Sami Jaghouar <sami.jaghouar@jina.ai> * chore: move from flake height to ruff in ci and pre commit Signed-off-by: Sami Jaghouar <sami.jaghouar@jina.ai> * chore: remove flake8 Signed-off-by: Sami Jaghouar <sami.jaghouar@jina.ai> * chore: remove line lenght ruff Signed-off-by: Sami Jaghouar <sami.jaghouar@jina.ai> Signed-off-by: Sami Jaghouar <sami.jaghouar@jina.ai>
* refactor: rename base node method Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * refactor: update docstrng Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* chore(ci): readd commint lint Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): add commit link to sucessful Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): fix ci Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): test disable commit lint comment Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): fix do ci only once Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): fix blsck Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): update on what trigger pr Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): apply black Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): add check mypy to success all Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): re add commit link comment Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* chore(ci): fix ci not running on dev branch Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore(ci): change branch name Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* refactor: change embedding and base node Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: expose any url to user Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * refactor: docarray type are relfected at the proto level Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: add test and fix dump to proto Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * refactor: move get nested class to document Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix mypy return type hint Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: add id as a type Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: adapt proto to the new id type Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix(proto): does not return mixin anymore type Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* feat(da): add get attribute mixin to da Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat(da): generate attribute on the fly at the da level Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat(da): add comment Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* refactor(v2): renamed variable and files Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * refactor(v2): renamd test Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* feat: add tensor type for ndarray * fix: fix mypy typing * feat: torch tensor type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: protobuf for pytorch type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * ci: install all extras in the ci Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: make nice looking * docs: update docarray/typing/tensor/torch_tensor.py Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> * refactor: code style Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: black and mypy Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: suppress mypy import error * ci: fix ci install Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com>
* feat: allow da bulk access to return da for document Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix mypy type pb Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: add link to the mypy issue Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: remove useless list type hint Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
* refactor(v2): renamed variable and files Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * refactor(v2): renamd test Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> feat: add read from proto Co-authored-by: samsja <55492238+samsja@users.noreply.github.com>
* feat: add tensor type for ndarray * fix: fix mypy typing * feat: torch tensor type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: protobuf for pytorch type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * ci: install all extras in the ci Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: make nice looking * docs: update docarray/typing/tensor/torch_tensor.py Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> * refactor: code style Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: black and mypy Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: suppress mypy import error * ci: fix ci install Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: add new type for image urls Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: add new type for image urls Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: add real existing url to test Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: test output of image buffer loading Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: add validation for image url Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: specify image axis permutation Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: add docstrings Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: make strings uniform Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: pass valid url as imageurl Signed-off-by: Johannes Messner <messnerjo@gmail.com> * chore: fix dependencies Signed-off-by: Johannes Messner <messnerjo@gmail.com> * chore(ci): add extras to mypy ci Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* feat: add tensor type for ndarray * fix: fix mypy typing * feat: torch tensor type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: protobuf for pytorch type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * ci: install all extras in the ci Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: make nice looking * docs: update docarray/typing/tensor/torch_tensor.py Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> * refactor: code style Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: black and mypy Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: suppress mypy import error * ci: fix ci install Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: add new type for image urls Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: add new type for image urls Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: add real existing url to test Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: test output of image buffer loading Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: add validation for image url Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: specify image axis permutation Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: add docstrings Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: make strings uniform Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: pass valid url as imageurl Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: add texturi type Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com>
* feat: allow da bulk access to return da for document Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix mypy type pb Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: add link to the mypy issue Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: remove useless list type hint Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: torch tensor type (#800) * feat: add tensor type for ndarray * fix: fix mypy typing * feat: torch tensor type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: protobuf for pytorch type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * ci: install all extras in the ci Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: make nice looking * docs: update docarray/typing/tensor/torch_tensor.py Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> * refactor: code style Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: black and mypy Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: suppress mypy import error * ci: fix ci install Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> * feat: add fastapi to dependency Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat(wip): add fake method to dump tensor to json Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat(wip): add fastapi test Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: add json dump for type and document Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: add json compatible with orjson Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * refactor: clean tests Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: remove duplicate Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: better json schema for tensor Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix fast api test Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * refactor: move to json test to integration Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: json laod from tensor type now working Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: add tensor to fastapi test Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: add new fastapi test Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: add more testing for text uri about json Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix default orson not returning Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: apply johannes suggestion on docstring Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: does not perform copy anymore on torch tensor unwrap Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: add johannes suggestion Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * fix: fix ruff line lenght Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix mypy pb Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <messnerjo@gmail.com>
* refactor: rename tensor type to ndarray type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: add tensor type for any kind of tensor Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: add more tests for proto Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <messnerjo@gmail.com>
* feat: parametrized torch tensor type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: fix typo in method name Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: make parametrized type implementation generic Signed-off-by: Johannes Messner <messnerjo@gmail.com> * refactor: make abstract tensor clearer Signed-off-by: Johannes Messner <messnerjo@gmail.com> * feat: parametrized ndarray type Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: add serializatin tests for param tensors Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: add costrings for tensor types Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: allow any kind of tensor shape Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: use view instead of reshape Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <messnerjo@gmail.com>
* chore: add documentation and ci Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: remove bad ci Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: add good ci Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: add labeler.yml Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: update readme to trigger the netifly Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: fix ci netifly Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: bump netfly python version Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: update API Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: update API Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: change doc netrlfy script Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * Revert "feat: change doc netrlfy script" This reverts commit ce59a1a. * fix: build cos Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: build cos Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix docs Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: add piblic api Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: try remove torch Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: reput torch Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix autodoc mock torch Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * fix: fix ci docs Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: update docs Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: update docs Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * feat: update docs Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* docs: rewrite the readme Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: meaking headings bigger Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: restructure headings Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: add link to discussions Signed-off-by: Johannes Messner <messnerjo@gmail.com> * docs: fix wording Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com>
Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* chore: update readme with instal intrsuction Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> * chore: update for release Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr> Signed-off-by: Sami Jaghouar <sami.jaghouar@hotmail.fr>
* refactor: doc index structure Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: import unused error Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: run black Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: remove noqa Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: doc_index to index Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: small changes Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: readme Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: add init file Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: move docindex outside integration tests Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: adjust config paths Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
* fix: remove tensor type from DocumentArray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix tensorflow test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: apply charllote suggestion Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> * feat: apply saba suggestion Co-authored-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Co-authored-by: Saba Sturua <45267439+jupyterjazz@users.noreply.github.com>
* refactor: bytes to bytes_ Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: missed fields Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
* feat: add from_files() Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add da classmethod from_files() Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: update docstring Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add example usage Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add get_paths, rm from_files Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add print to debug ci Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
* feat: add failing test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: shorten if else statememt Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: shorten if else statememt Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto and list Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix proto and dict Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add very complex test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: fix pure tensor stuff Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: fix pure tensor stuff Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix from protobuf Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix from protobuf tensorflow Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add more test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix mypy Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: add more test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: import ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
* feat: create documents from dict Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * fix: ignore type Signed-off-by: jupyterjazz <saba.sturua@jina.ai> * refactor: change fn names Signed-off-by: jupyterjazz <saba.sturua@jina.ai> --------- Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
Signed-off-by: Alex C-G <alexcg@outlook.com>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
* fix: support for torch and tf Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: allow arbitrary payloads, including tensors Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: mark tf tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: another attempt at fixing tf tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: remove parametrization of test Signed-off-by: Johannes Messner <messnerjo@gmail.com> * test: fix test Signed-off-by: Johannes Messner <messnerjo@gmail.com> * fix: add suggestion Signed-off-by: Johannes Messner <messnerjo@gmail.com> * ci: exlude tf tests from index tests Signed-off-by: Johannes Messner <messnerjo@gmail.com> --------- Signed-off-by: Johannes Messner <messnerjo@gmail.com>
…everse, sort, remove, pop (#1291) * feat: isort format fix Signed-off-by: agaraman0 <agaraman0@gmail.com> * refactor: comment fixes Signed-off-by: agaraman0 <agaraman0@gmail.com> * refactor: comment fixes Signed-off-by: agaraman0 <agaraman0@gmail.com> --------- Signed-off-by: agaraman0 <agaraman0@gmail.com>
* refactor: move streaming serialization into separate method Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: add binary io like protocol definition Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: ported push pull to JAC Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: protocol is not in 3.7 typing Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: make mypy happy Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: patch missing waterfall Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: jit import backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: implement cache in jinaai pull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble dependency to jina group Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: better division of concerns Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add concept of namespace Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: ignore missing hubble stubs Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: streaming protocol stubs Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: make more general buffered caching reader Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add tests for hubble pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add tests for file backend Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: remove hubble dependency from jina group This reverts commit b304421. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: implement push pull for local filesystem Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: test concurrent pushes and pulls in file protocol Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: resolve concurrent pushes and pulls correctly Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: rename text to textdoc Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: added some logging Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: s3 tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: s3 pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add smart open dependency Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add smart opens silly python bound Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: update hubble tests (failing) Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: fix delete return in hubble pushpull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * Revert "fix: add smart open dependency" This reverts commit cf78c6c. This reverts commit eb0e52b. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble and smart open dependencies Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: mypy fixes Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * ci: allow tests to see jina auth token Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add progress bars for streaming Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * style: blacken Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: buffer writes to s3 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: mypy no like sequence Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: make progress bar quieter when disabled Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: skip failing tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add tables when listing Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: add jina auth token to uncaped test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: mock s3 tests with minio container Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: silly error that cost me 2 hours of life Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: use tolerance ratio in file tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add caching to s3 pull Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: add log messages for unused parameters Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: take out unneeded buffering smart open already buffers Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: pick fastest protocol compression configuration for s3 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: bump tolerance ratio for s3 test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: reduce code duplication Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: put reader chunk size constant at top of file Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: reduce reader chunk size for memory tests Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: multipart uploads get stuck frequently lets just do big uploads for now... Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * docs: add docstrings to mixin and file backend Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * docs: add docstring for s3 and hubble backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * test: remove unused test Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: use literal in protocol Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: protocols dont need to be inherited Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add make mypy happy with the literals Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: literals not in 3.7 Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: move mixin out of init file Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: move cache path resolution to utils Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * feat: cache path is only evaluated once Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: loading backends makes more sense as debug log Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: add slow and internet marks Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: pin image tag Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: use abc instead of protocol for typing backends Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: revert - add hubble and smart open dependencies This reverts commit 1d1d2ee. Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: add hubble and aws dependencies Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: change all push pull mixin methods to class methods Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: misstyped class method self reference Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: rename pushpull to docstore and use more classmethods Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: separate remote backend implementations from mixin Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * fix: missed import refactor Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: change submodule name to store Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: remove list and delete from mixin Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: clear all the garbage in ci account Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * tests: skip test that is broken on ci Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> * refactor: standardize naming to jac Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> --------- Signed-off-by: Jackmin801 <56836461+Jackmin801@users.noreply.github.com> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com>
* refactor: rename document to doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename document to doc in da Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base doc in md files Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base base document ot base doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix(docs): fix docs building Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
* refactor: rename document to doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename document to doc in da Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base doc in md files Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * refactor: rename base base document ot base doc Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix(docs): fix docs building Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: ingore hubble test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add userguide install Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add awesome-pages Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add install Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: rename tutorials to how to Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * chore: add pre commit blacken docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * chore: add blacken docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: arr warning docarray version Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: repo url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add social Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add logo Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add first step emtpy page Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: add document docs Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add markdown documentation test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: remove content Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: fix ci Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
* wip Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: cleanup namespace utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add docstring test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
* wip Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: cleanup namespace utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add docstring test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video url docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix text url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix image url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fic audio url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: mesh 3d url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: mesh 3d url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove useless data Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix audio bytes Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: video and image bytes Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: move typing section Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
* fix: flatten schema of abstract index Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: catch exception when flatten schema Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: remove useless assignemnt Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: use Abstractensor as tensor doc_type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: add AbstractTensor to hnswlib Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: AbstractTensor as doc_type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: complete description about AbstracTensor Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com>
* feat: add utils for map to docs and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add utils for find and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video ndaray docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video find docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix map docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fileter docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix add reduce Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
Signed-off-by: Johannes Messner <messnerjo@gmail.com>
* fix: fix utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix map Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr>
* feat: __init__ of ElasticDocumentIndex Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add index func Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: get and del funcs Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: init and index creation Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: __init__ and _index Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: _get_items Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add _find Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add filter text and their batch version Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: store id and get nested doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: vector cannot be all zero Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: __getitem__ raise error Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more python types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: elastic index tests Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: comment scripts before ci setup Signed-off-by: AnneY <evangeline-lun@foxmail.com> * chore: add elasticsearch dependency to poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: elastic index ci setup Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add num_candidates to rumtime config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: let user pass index_settings Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: degrade to v7 and add query builder Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: remove elastic_transport Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: minor features Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: style fix Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: fix mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add chunk size to runtime config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: chunk size Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add chunk_size to funcs Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: rewrite elastic v7 query builder Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: db_type should be elastic types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: minor adjustment Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: rename elastic index files Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: remove comments Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: rename, batch operations, etc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: add test for persistency and col config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more field types and subclass Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more python types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: tf, tensor and more elastic field types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: elastic should be optional in toml Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> * refactor: rename class Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: change Dict to Mapping Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: add AbstractTensor Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: rename class and add tests Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai>
Signed-off-by: agaraman0 <agaraman0@gmail.com>
* chore: group extras and add instructions for pip installs Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: throw runtime error with install instructions for hnswlib Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for video imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for audio imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for 3d imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: add instructions for image imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: import only audiosegment from pydub Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: generalize audio and image imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for web imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for web imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for protobuf imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for lz4 imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: fastapi import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: revert changes in protobuf import Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for torch, without raising error Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for torch, with raising error Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add instructions for tensorflow Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: base doc io imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf in doc index abstract Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf in doc index abstract Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: tf import in doc index Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add getattr on module level Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: import torch for type checking Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add type checking Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: test cross backend Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add missing return statement Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: update error message Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: remove base document init Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add trimesh easy extra Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: pil immage importfix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * chore: add lz4 to mypy missing type hint section Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add instructions to doc index tutorial Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * chore: extra pandas and condense module where missing imports ignore Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: update poetry lock Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: missed imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: revert last commit This reverts commit 9aca06f. Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * revert "fix: missed imports" This reverts commit 353f029. Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: missed imports Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * wip Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: rename DocArrayProto to DocumentArrayProto (#1297) Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: docstring polish typing (#1299) * wip Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: cleanup namespace utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add docstring test Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video url docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix text url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix image url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fic audio url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: mesh 3d url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: mesh 3d url Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: remove useless data Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix docstring ndarray and torch tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fix audio url and audio ndarray Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video tensor Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix audio bytes Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: video and image bytes Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * docs: move typing section Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix for doc_string test Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: try short version in typing init getattr Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: shorter version in getattr Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: remove files (#1305) Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: flatten schema of abstract index (#1294) * fix: flatten schema of abstract index Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: _convert_dict_to_doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: catch exception when flatten schema Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: remove useless assignemnt Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: use Abstractensor as tensor doc_type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: add AbstractTensor to hnswlib Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: AbstractTensor as doc_type Signed-off-by: AnneY <evangeline-lun@foxmail.com> * docs: complete description about AbstracTensor Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> * fix: add type hint for lib Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: add import error to inits getattrs Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: add utils section (#1307) * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add utils for map to docs and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * feat: add utils for find and fix docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video ndaray docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix video find docstrng Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix map docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix fileter docstring Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix add reduce Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: fix docstring example of find_batched (#1308) Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * docs: fix map docstring (#1311) * fix: fix utils Signed-off-by: samsja <sami.jaghouar@hotmail.fr> * fix: fix map Signed-off-by: samsja <sami.jaghouar@hotmail.fr> --------- Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: elasticsearch document index (#1196) * feat: __init__ of ElasticDocumentIndex Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add index func Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: get and del funcs Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: init and index creation Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: __init__ and _index Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: _get_items Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add _find Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add filter text and their batch version Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: store id and get nested doc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: vector cannot be all zero Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: __getitem__ raise error Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more python types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: elastic index tests Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: comment scripts before ci setup Signed-off-by: AnneY <evangeline-lun@foxmail.com> * chore: add elasticsearch dependency to poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: elastic index ci setup Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add num_candidates to rumtime config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: let user pass index_settings Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: degrade to v7 and add query builder Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: remove elastic_transport Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: minor features Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: style fix Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: fix mypy Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add chunk size to runtime config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: chunk size Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: add chunk_size to funcs Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: rewrite elastic v7 query builder Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: db_type should be elastic types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: minor adjustment Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: rename elastic index files Signed-off-by: AnneY <evangeline-lun@foxmail.com> * refactor: remove comments Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: rename, batch operations, etc Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: add test for persistency and col config Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more field types and subclass Signed-off-by: AnneY <evangeline-lun@foxmail.com> * feat: support more python types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: tf, tensor and more elastic field types Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: elastic should be optional in toml Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> * refactor: rename class Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: change Dict to Mapping Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: add AbstractTensor Signed-off-by: AnneY <evangeline-lun@foxmail.com> * test: rename class and add tests Signed-off-by: AnneY <evangeline-lun@foxmail.com> * fix: poetry Signed-off-by: AnneY <evangeline-lun@foxmail.com> --------- Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Charlotte Gerhaher <charlotte.gerhaher@jina.ai> * fix: add case for elastic search Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * refactor: map_docs_batch to map_docs_batched (#1312) Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: clean up Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * feat: torch backend basic operation tests (#1306) Signed-off-by: agaraman0 <agaraman0@gmail.com> Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: ci add --fix-missing to apt-get Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: revert "fix: ci add --fix-missing to apt-get" Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: ci apt-get update Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply samis suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> * fix: apply samis suggestions from code review Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> --------- Signed-off-by: anna-charlotte <charlotte.gerhaher@jina.ai> Signed-off-by: samsja <sami.jaghouar@hotmail.fr> Signed-off-by: AnneY <evangeline-lun@foxmail.com> Signed-off-by: Anne Yang <evangeline-lun@foxmail.com> Signed-off-by: Johannes Messner <messnerjo@gmail.com> Signed-off-by: agaraman0 <agaraman0@gmail.com> Co-authored-by: samsja <55492238+samsja@users.noreply.github.com> Co-authored-by: Anne Yang <evangeline-lun@foxmail.com> Co-authored-by: Johannes Messner <44071807+JohannesMessner@users.noreply.github.com> Co-authored-by: Aman Agarwal <agaraman0@gmail.com>
Contributor
Author
|
Please ignore this PR due to the improper target branch selected. Correct PR at #1319 Sorry for the inconvenience😔 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
11 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Goals: