Andes and BDBV improvements - #983
Draft
anna-parker wants to merge 9 commits into
Draft
anna-parker wants to merge 9 commits into
anna-parker wants to merge 9 commits into
Conversation
anna-parker
commented
May 29, 2026
anna-parker
force-pushed
the
andes_improvements
branch
from
May 29, 2026 09:02
6d77db7 to
4aa7158
Compare
anna-parker
added a commit
to loculus-project/loculus
that referenced
this pull request
May 29, 2026
…s.*.privateSubstitutions correctly (#6538) required for pathoplexus/pathoplexus#982 ### Screenshot tested in pathoplexus/pathoplexus#983 <img width="814" height="362" alt="image" src="https://github.com/user-attachments/assets/7279b521-e465-4faf-8a01-df5d2d0e57ce" /> 🚀 Preview: Add `preview` label to enable
Merged
3 tasks
anna-parker
added a commit
to loculus-project/loculus
that referenced
this pull request
Jun 3, 2026
…ups (#6527) see [@theosanderson](https://github.com/theosanderson)'s report https://gist.github.com/theosanderson-agent/ded53e709683a52063233c174aa7c0c0 for why this would be good to have to andes virus Use NCBI release date as a segment-specific field in ingest when grouping but still have only one release date per segment-group (as this is used for calculating the earliestReleaseDate making it segment specific would be a massive overhaul - see #6390 for a starting attempt in this direction) ### BREAKING CHANGE This change is not breaking but will lead to users receiving a notification that they need to revoke and regroup some sequences - so I wanted to highlight this here. ### Changes in CCHF Based on claude's summary of me running the diff script. 1. individual → grouped Previously separate single-segment records are combined into groups: ``` ┌─────────────────────────────┬───────────────────────────┐ │ Removed (individual) │ Added (grouped) │ ├─────────────────────────────┼───────────────────────────┤ │ ON623080.1.M + ON142178.1.S │ ON623080.1.M/ON142178.1.S │ ├─────────────────────────────┼───────────────────────────┤ │ ON623086.1.L + ON142179.1.S │ ON623086.1.L/ON142179.1.S │ ├─────────────────────────────┼───────────────────────────┤ │ OQ935542.1.M + ON254096.1.S │ OQ935542.1.M/ON254096.1.S │ ├─────────────────────────────┼───────────────────────────┤ │ OQ935543.1.M + ON254100.1.S │ OQ935543.1.M/ON254100.1.S │ ├─────────────────────────────┼───────────────────────────┤ │ OQ935544.1.M + ON254097.1.S │ OQ935544.1.M/ON254097.1.S │ ├─────────────────────────────┼───────────────────────────┤ │ PV210235.1.L + PV168397.1.M │ PV210235.1.L/PV168397.1.M │ ├─────────────────────────────┼───────────────────────────┤ │ PV210236.1.L + PV168394.1.M │ PV210236.1.L/PV168394.1.M │ └─────────────────────────────┴───────────────────────────┘ ``` 2. grouped → split Previously grouped records (only L and M interestingly) are split into individual segment entries. These seem to be all from the same submitting group: `Stavropol State Research Anti-Plague Institute, Laboratory for diagnostics of viral infections` For example first two examples: https://pathoplexus.org/cchf/search?visibility_specimenCollectorSampleId=true&specimenCollectorSampleId=229-ARM-TI-2023 have the same specimenCollectorSampleId and metadata (only release date was used to split) so it actually does make sense to split them up: ``` ┌───────────────────────────┬─────────────────────────────┐ │ Removed (grouped) │ Added (individual) │ specimenCollectorSampleId ├───────────────────────────┼─────────────────────────────┤ │ PQ878918.1.L/PQ878909.1.M │ PQ878918.1.L + PQ878909.1.M │ 229-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PV105948.1.L/PV091003.1.M │ PV105948.1.L + PV091003.1.M │ 229-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PQ878919.1.L/PQ878910.1.M │ PQ878919.1.L + PQ878910.1.M │ 279-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PV105949.1.L/PV091004.1.M │ PV105949.1.L + PV091004.1.M │ 279-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PQ878920.1.L/PQ878911.1.M │ PQ878920.1.L + PQ878911.1.M │ 345-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PQ878922.1.L/PQ878913.1.M │ PQ878922.1.L + PQ878913.1.M │ 372-ARM-TI-2022 ├───────────────────────────┼─────────────────────────────┤ │ PQ878925.1.L/PQ878916.1.M │ PQ878925.1.L + PQ878916.1.M │ 423-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PQ878926.1.L/PQ878917.1.M │ PQ878926.1.L + PQ878917.1.M │ 521-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PV105950.1.L/PV091005.1.M │ PV105950.1.L + PV091005.1.M │ 345-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PV105951.1.L/PV091006.1.M │ PV105951.1.L + PV091006.1.M │ 423-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PV105952.1.L/PV091007.1.M │ PV105952.1.L + PV091007.1.M │ 521-ARM-TI-2023 ├───────────────────────────┼─────────────────────────────┤ │ PV105953.1.L/PV091008.1.M │ PV105953.1.L + PV091008.1.M │ 372-ARM-TI-2022 └───────────────────────────┴─────────────────────────────┘ ``` ### PR Checklist - [x] Test on andes virus to see if this improves grouping there as well: testing results on andes are good: pathoplexus/pathoplexus#983 - [x] test on staging that does not trigger a revision of all grouped sequences (should not happen as the hash value will be the same unless the release date was different and in these cases the grouping will change anyways: tested here: pathoplexus/loculus_deployments#743 (comment) 🚀 Preview: https://ingest-grouping-2.loculus.org
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#982, #930
Things to test:
nice only 29 single sequences failed preprocessing all because they had sequences that could be aligned
nice! this works but needs a small code change
Claude's summary:
Segment Grouping Changes
Three grouping improvement patterns:
- 3 EU564xxx pairs: EU564721.1.M + EU564715.1.S → EU564721.1.M/EU564715.1.S (and 2 more)
- 2 OR987854/855 pairs: OR987854.1.M + OR908892.1.S → OR987854.1.M/OR908892.1.S (and 1 more)
- PP003836/37 (were L+M only) → full L+M+S by adding OR908897/898.S
- PP151168-170, PP151172 (were L+M only) → full L+M+S by adding OR90888x.S
- PP151165/166/167, PP504849 (individual L segments) + separate M and S → combined L+M+S
- PQ373861.1.L + PQ356349.1.M + PQ181488.2.S → single combined entry
New Sequences
section — genuine new additions. (These are sequences where the M segment could not be aligned only assigned and with the alignmenttype ANY are now visible on PPX)
Removed Sequences — Restriction Check
All 20 truly removed sequences are RESTRICTED. The OPEN sequences appearing in the removed list (all the EU564xxx, ORxxx, PPxxx, PQxxx) are not actually lost — they are regrouped into combined entries in the added section.
🚀 Preview: Add
previewlabel to enable