Skip to content

Improve memory consumption when patching/removing multiple items with mongodb adapter #3626

@claustres

Description

@claustres

It appears that when using a patch/remove operation with a null ID to target multiple items, all matching items in the database are first retrieved with a find operation. As far as I know this is to conform to the expected FeathersJS behaviour of sending back patched/removed items to after hooks or realtime events.

However, when having a database with a large number of items, this behaviour might cause a too much large:

  1. memory consumption when the backend processes the operation,
  2. bandwidth consumption for realtime events.

We were required to mitigate this in our application on a specific service with a large number of items, this documentation might help to understand how we did that. However, it does not prevent the built-in behaviour of the adapter to load all items up-front, causing the possible memory issue to remain.

Not sure what will be the best way to tackle that at the adapter level as by the way MongoDB does not really require the items to be loaded first to perform such operations, but we need to stay at least as possible "compatible" with FeathersJS expected behaviour. Does a similar approach than our with a configurable limit from which the adapter will "simplify" result items be appropriate ? Any idea is welcome, we could try to help with a PR if you think it's worth it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions