Fix PyTables 3.10+ incompatibility with pandas 2.0+ MultiIndex extension dtypes by Copilot · Pull Request #3163 · DeepLabCut/DeepLabCut

Copilot · 2025-12-26T10:03:08Z

Pandas 2.0+ uses StringDtype extension dtypes by default for MultiIndex string columns. PyTables 3.10+ cannot serialize these to HDF5, causing NotImplementedError: Saving a MultiIndex with an extension dtype is not supported when saving labeled data.

Changes

Added conversion helper (convert_multiindex_to_hdf_compatible()) that downgrades extension dtype levels to object dtype before HDF5 serialization
Updated 30+ to_hdf() callsites across the codebase to apply conversion:
- Core utilities (auxiliaryfunctions, conversioncode, auxfun_multianimal)
- Dataset management (trainingsetmanipulation, outlier_frames, stitch, tracklets)
- GUI tools (tracklet_toolbox)
- Post-processing (filtering, analyze_skeleton, triangulation)
- Both TensorFlow and PyTorch backends (evaluate, videos, analyze_images)
- Model zoo utilities
Updated PyTables requirement from unspecified to >=3.10.1 in setup.py and requirements.txt
Added test verifying round-trip HDF5 serialization with extension dtypes

Example

def convert_multiindex_to_hdf_compatible(df):
    """Convert MultiIndex extension dtypes to object dtype for HDF5 compatibility."""
    if isinstance(df.columns, pd.MultiIndex):
        new_levels = []
        for level in df.columns.levels:
            if hasattr(level.dtype, 'name') and level.dtype.name == 'string':
                new_levels.append(level.astype('object'))
            else:
                new_levels.append(level)
        df.columns = df.columns.set_levels(new_levels)
    # Same for index MultiIndex...
    return df

# Applied before all to_hdf() calls
df = convert_multiindex_to_hdf_compatible(df)
df.to_hdf(output_path, key="df_with_missing", mode="w")

Backward compatible with older pandas/PyTables versions. No API changes.

Original prompt

This section details on the original issue you should resolve

<issue_title>Pytables version incompatibility</issue_title>
<issue_description>### Is there an existing issue for this?

I have searched the existing issues

Operating System

Description: Ubuntu 24.04.3 LTS
Release: 24.04
Codename: noble

DeepLabCut version

3.0.0rc13

What engine are you using?

pytorch

DeepLabCut mode

single animal

Device type

GeForce RTX 4070

Bug description 🐛

Pytables 3.8.0 was installed as described in the installation documents. Attempting to save a layer of labelled points during frame labeling gives the error:

ImportError: Pandas requires version '3.10.1' or newer of 'tables' (version '3.8.0' currently installed).

This dependency conflict cannot be resolved. Upgrading pytables then leads to another error at the same point:

NotImplementedError: Saving a MultiIndex with an extension dtype is not supported.

Steps To Reproduce

I installed deeplabcut as described:

conda create -n DEEPLABCUT python=3.12
conda activate DEEPLABCUT
conda install -c conda-forge pytables==3.8.0
conda install pytorch cudatoolkit=11.3 -c pytorch
pip install --pre deeplabcut[gui]

python -c "import torch; print(torch.cuda.is_available())"
Returns True.

I created a project, extracted frames from a subset of videos using automatic k-means, with frame cropping from config. I then tried to label frames. When I try to save my layer, I get the error.

Relevant log output

With pytables 3.8.0

python -m deeplabcut
Loading DLC 3.0.0rc13...
Starting GUI...
QPixmap::scaled: Pixmap is a null pixmap
_pythonToCppCopy: Cannot copy-convert 0x653e45e97880 (NoneType) to C++.
_pythonToCppCopy: Cannot copy-convert 0x653e45e97880 (NoneType) to C++.
Traceback (most recent call last):
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari_deeplabcut/_widgets.py", line 656, in <lambda>
    lambda: _save_layers_dialog(
            ^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari_deeplabcut/_widgets.py", line 300, in _save_layers_dialog
    self.viewer.layers.save("", selected=True, plugin="napari-deeplabcut")
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/utils/_proxies.py", line 188, in __call__
    return self.create(self.__wrapped__(*args, **kwargs))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/components/layerlist.py", line 476, in save
    return save_layers(path, layers, plugin=plugin, _writer=_writer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/plugins/io.py", line 229, in save_layers
    _written, writer_name = _write_single_layer_with_plugins(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/plugins/io.py", line 442, in _write_single_layer_with_plugins
    written_paths, writer_name = _npe2.write_layers(
                                 ^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari/plugins/_npe2.py", line 111, in write_layers
    paths, writer = io_utils.write_get_writer(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/npe2/io_utils.py", line 115, in write_get_writer
    return _write(path, layer_data, plugin_name=plugin_name, return_writer=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/npe2/io_utils.py", line 318, in _write
    res = writer.exec(args=args)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/npe2/manifest/utils.py", line 61, in exec
    return self.get_callable(reg)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/napari_deeplabcut/_writer.py", line 79, in write_hdf
    df.to_hdf(path, key="keypoints", mode="w")
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/pandas/core/generic.py", line 2789, in to_hdf
    pytables.to_hdf(
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/pandas/io/pytables.py", line 313, in to_hdf
    with HDFStore(
         ^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/pandas/io/pytables.py", line 578, in __init__
    tables = import_optional_dependency("tables")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/josh/miniforge3/envs/deeplabcut/lib/python3.12/site-packages/pandas/compat/_optional.py", line 187, in import_optional_dependency
    raise ImportErr...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes DeepLabCut/DeepLabCut#3162

<!-- START COPILOT CODING AGENT TIPS -->
---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our [2 minute survey](https://gh.io/copilot-coding-agent-survey).

…ompatibility - core files updated Co-authored-by: MMathisLab <28102185+MMathisLab@users.noreply.github.com>

Co-authored-by: MMathisLab <28102185+MMathisLab@users.noreply.github.com>

C-Achard · 2026-02-13T13:41:25Z

@deruyter92 Is this fixed by #3184 ?

deruyter92 · 2026-02-13T14:06:27Z

@C-Achard, yes that is correct. I will close this one in favor of #3184

Initial plan

443e870

Copilot AI assigned Copilot and MMathisLab Dec 26, 2025

Copilot started work on behalf of MMathisLab December 26, 2025 10:03 View session

Copilot AI and others added 3 commits December 26, 2025 10:14

Add helper function to convert MultiIndex extension dtypes for HDF5 c…

f9872fa

…ompatibility - core files updated Co-authored-by: MMathisLab <28102185+MMathisLab@users.noreply.github.com>

Update all to_hdf calls and set PyTables version requirement to >=3.10.1

afd02bb

Co-authored-by: MMathisLab <28102185+MMathisLab@users.noreply.github.com>

Add test for MultiIndex extension dtype conversion to HDF5 compatibility

d56ae2e

Co-authored-by: MMathisLab <28102185+MMathisLab@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix Pytables version incompatibility issue~~ Fix PyTables 3.10+ incompatibility with pandas 2.0+ MultiIndex extension dtypes Dec 26, 2025

Copilot AI requested a review from MMathisLab December 26, 2025 10:23

Copilot finished work on behalf of MMathisLab December 26, 2025 10:23

deruyter92 closed this Feb 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix PyTables 3.10+ incompatibility with pandas 2.0+ MultiIndex extension dtypes#3163

Fix PyTables 3.10+ incompatibility with pandas 2.0+ MultiIndex extension dtypes#3163
Copilot wants to merge 4 commits intomainfrom
copilot/fix-pytables-version-incompatibility

Copilot AI commented Dec 26, 2025 •

edited

Loading

Uh oh!

C-Achard commented Feb 13, 2026

Uh oh!

deruyter92 commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

Copilot AI commented Dec 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Example

Operating System

DeepLabCut version

What engine are you using?

DeepLabCut mode

Device type

Bug description 🐛

Steps To Reproduce

Relevant log output

Uh oh!

C-Achard commented Feb 13, 2026

Uh oh!

deruyter92 commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Dec 26, 2025 •

edited

Loading