Skip to content

Conversation

@ev-br
Copy link
Contributor

@ev-br ev-br commented Sep 25, 2025

Otherwise, a malloc failure in init_gesv(...) is not acted upon, and the python return value is silently wrong---inv(non_zero_array) returns an array of zeros.

On main, using the script under the fold,

  • limiting the memory to 30mb emits numpy._core._exceptions._ArrayMemoryError: Unable to allocate 191. MiB for an array with shape (5000, 5000) and data type float64 from somewhere else in the gufunc machinery;
  • limiting the memory to 300mb makes res be an array of all zeros, no error.

Note that I don't know what I'm doing with NPY_ALLOW_C_API_DEF etc ---here I simply parrot the macros from elsewhere in this source file, e.g. https://github.com/numpy/numpy/blob/main/numpy/linalg/umath_linalg.cpp#L1202
The same issue seems present in other linalg functions, so if this fix looks reasonable, I'll extend to other functions.

Details
import os
import psutil     # need to `pip install psutil`
import resource
import numpy as np


def set_limit(buffer):
     """
     Set a limit on process virtual memory space.
     """
     process = psutil.Process(os.getpid())
     memory_info = process.memory_info()
     used = memory_info.vms # current virtual memory space
     old_limit = resource.getrlimit(resource.RLIMIT_AS)
     new_limit = (used + buffer, old_limit[1]) # set new
     resource.setrlimit(resource.RLIMIT_AS, new_limit)


np.random.seed(123467)
arr = np.random.rand(25_000_000).reshape(5000, 5000)

set_limit(300 * 1024 * 1024) # ~300MB

res = np.linalg.inv(arr)
assert not (res == 0).all()

Otherwise, a malloc failure in `init_gesv(...)` is not acted upon,
and the python return value is silently wrong.
@seberg
Copy link
Member

seberg commented Sep 25, 2025

Using npy_gil_error might be nice, but not helpful without changing more in the file probably.

Thanks, this is right, but TBH, a bit half hearted. The same function is called in multiple places, but only one is fixed.
I would say: move the error setting into the init_gesv and also don't omit init_evd (maybe more?), then audit the actual usages if they set error again we can remove it.
(I.e. set error at the source, for "allocations" that might be a bit different but considering that we just repeat things otherwise...)

@ev-br
Copy link
Contributor Author

ev-br commented Sep 25, 2025

Thanks---yes, the initial attempt was exploratory, to double-check that the general direction makes sense. Now that you confirmed it (thanks!), the updated PR extends this check across umath_linalg.

The error message is not as nice as it is for other memory failures which are reported through errstate (cf #28782 (comment)), but it's better than a wrong answer anyway.

Copy link
Member

@seberg seberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ev-br! Awesome, this looks pretty now and a cleanup :).

NPY_ALLOW_C_API;
PyErr_NoMemory();
NPY_DISABLE_C_API;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's go with this, npy_gil_error, got enough other places.

@seberg seberg added the 09 - Backport-Candidate PRs tagged should be backported label Sep 25, 2025
@seberg
Copy link
Member

seberg commented Sep 25, 2025

Seems simple and enough and very safe to backport as well.

@seberg seberg merged commit 92ab40b into numpy:main Sep 25, 2025
77 of 78 checks passed
bwhitt7 pushed a commit to bwhitt7/numpy that referenced this pull request Sep 25, 2025
Otherwise, a malloc failure in `init_gesv(...)` etc. is not acted upon,
and the python return value is silently wrong.
bwhitt7 pushed a commit to bwhitt7/numpy that referenced this pull request Sep 25, 2025
Otherwise, a malloc failure in `init_gesv(...)` etc. is not acted upon,
and the python return value is silently wrong.
charris pushed a commit to charris/numpy that referenced this pull request Sep 30, 2025
Otherwise, a malloc failure in `init_gesv(...)` etc. is not acted upon,
and the python return value is silently wrong.
@charris charris removed the 09 - Backport-Candidate PRs tagged should be backported label Sep 30, 2025
charris added a commit that referenced this pull request Sep 30, 2025
BUG: linalg: emit a MemoryError on a malloc failure (#29811)
IndifferentArea pushed a commit to IndifferentArea/numpy that referenced this pull request Dec 7, 2025
Otherwise, a malloc failure in `init_gesv(...)` etc. is not acted upon,
and the python return value is silently wrong.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants