Support user dictionary in ThaiTokenizer and ThaiTokenizerFactory (#16721) - #16722

Hi @rmuir and everyone — thank you for merging the Thai analysis improvements (#16717, #16718, #16720, #16722, #16727). Much appreciated.

I would like to ask about a possible backport to branch_10x.

These changes currently only exist on main (11.0.0, which requires Java 25). branch_10x still contains just the original 4 Thai files (ThaiAnalyzer, ThaiTokenizer, ThaiTokenizerFactory, package-info), so downstream consumers that cannot move to Java 25 yet get none of the new Thai functionality.

The concrete case is OpenSearch: it builds against Lucene 10.5.1 and its minimum JDK is still 21 (opensearch-project/OpenSearch#21174, the JDK 25 bump, is open and facing a Bouncy Castle FIPS objection). So OpenSearch cannot consume Lucene 11 yet, and consequently cannot use ThaiCharFilter/ThaiNormalizer, the curated stopwords, ThaiRepeatFilter, the ThaiTokenizer user dictionary, or the configurable buffer size. Thai search quality there is stuck on the 10.x Thai code.

I noticed that new features are backported to branch_10x when appropriate — e.g. #16680 ("Add type to compound word", a 10.6.0 New Feature) was backported on 2026-09-28 (0157353e). So I wanted to ask whether the Thai changes could be considered for a 10.x backport too.

I understand that backporting new features is a judgement call, so to keep the ask narrow and reviewable, the highest-value item is probably #16722 (user_dictionary on ThaiTokenizer/ThaiTokenizerFactory): self-contained, purely additive, no file-format or breaking public-API change. The rest are similarly additive and analysis-only:

• Add ThaiCharFilter, ThaiNormalizationFilter, and ThaiNormalizer (#16716) #16717 — new ThaiCharFilter / ThaiNormalizationFilter / ThaiNormalizer + ThaiAnalyzer wiring
• Curate Thai stopwords to prevent over-filtering of compound words (#5323, #14730) #16718 — curated Thai stopwords (data only)
• Add ThaiRepeatFilter to handle Thai Maiyamok (ๆ) reduplication (#16719) #16720 — new ThaiRepeatFilter / ThaiRepeatFilterFactory
• Allow configurable buffer size in SegmentingTokenizerBase and recognize whitespace as safe boundary in ThaiTokenizer (#10153) #16727 — configurable buffer size in SegmentingTokenizerBase + whitespace safe boundary in ThaiTokenizer (this one touches a shared base class, so possibly less suitable for a maintenance branch)

None of them change the index format, so they should be BWC-safe on 10.x.

I am happy to prepare the backport PR(s) against branch_10x myself if that helps — just say the word, or let me know if you would rather handle it as committers. And if the answer is simply "new features don't get backported", that is completely fine; I mainly want to know so I can advise OpenSearch users accurately.

Thanks!

@kamthorn I'll backport, no worries. easiest for me just to do it with git merge.

@kamthorn all done: https://github.com/apache/lucene/commits/branch_10x/

FYI this might be causing some test failures: #16736

@kaivalnp I'll take a look

I was hoping it would be something exciting, but was just the empty string of course.

b304eea

Hey, thanks for these improvements and thanks for merging them @rmuir !

The new components are a great addition, they can be used in custom analyzers and they were not available before.

I was wondering about the users impact of the set of changes made across those 5 PRs that have been backported. I believe besides providing new building blocks, they modify previously existing, built-in components (thai analyzer, thai tokenizer and stopwords). Is my understanding correct? Is this an ok change to make in a minor? I believe upon upgrade users will get different tokens with the same input, doesn't that require reindexing to ensure predictable results?

@javanna yes, I think it is ok. For example, this user dictionary functionality is just a new feature.

Users impacted by the other changes, well, if they are impacted by them, then they were getting problematic results before for said documents.

It isn't a database, its full text, so when we look at things like wrong segmentation, it has to be thought of from that perspective.

I understand about the new features, but it seemed to me that the changes have also modified the output of the thai analyzer, for instance by adding a char filter that was not there before. Is that to be considered a bugfix then because the analyzer was previously behaving incorrectly?

yeah, i mean if your output has changed, that means it wasn't working for you before, because e.g. words weren't getting segmented properly / won't match.

its not changes on the order of "oh we added a new stemmer and analyzer wasn't stemming before".

The changes are pretty well documented so you can understand what is happening here.

can't view full text analysis from a lens of "oh you changed the output, thats a break!!!". Not a database. We've got to be able to improve and fix bugs.

Agreed. Not arguing against this, just trying to understand the scope and reasoning. Thanks for explaining.

Perhaps one last question: how have we communicated this to users in the past with similar changes? We could add some wording to the changes entry about the need to reindex? Or is that not necessary? I worry that users may get silent issues otherwise. For majors we have a migrate guide but that's not the case for minors I believe.

its not gonna make anything worse. for example, let's take the tokenizer change.

for the tokenizer change, i made it try to "only break on sentences" because most CJK models are trained on whole sentences. So it was just a bias there. But for thai it doesn't make sense: the way the dictionary-based-break-iterator works, it doesn't make use of a whole sentence. So for thai this was just causing truncated analysis (like "words split in half").

Hence, if before you were getting words truncated due to that, its not going to make anything worse to no longer do it. if you want to take advantage of the improved analysis, you have to reindex in order to get the benefits. then you won't have such truncated tokens anymore.