An open, reproducible toolkit and teaching platform for semantic similarity ensembles (stacking).
Combine, optimize, evaluate and explain semantic similarity measures — and learn how stacking works, end to end.

What this is. This repository accompanies the peer-reviewed review paper "A comprehensive review of stacking methods for semantic similarity measurement" (Martinez-Gil, Machine Learning with Applications, 2022). The paper surveys and organizes ensemble/stacking strategies; this repository adds a small, dependency-light, fully reproducible toolkit and a set of tutorials so you can run, understand, and build on those ideas yourself.

Scientific honesty. Every number you see in the notebooks is computed by the code in this repo on real, citable datasets that you download from their original distributors. Nothing here is hard-coded or fabricated.

1. Why semantic similarity ensembles?
2. Results at a glance
3. Quickstart
4. What's inside
5. The six families of stacking
6. How this relates to existing tools
7. How do I… (recipes)
8. Datasets
9. Tutorials & docs
10. Reproducibility
11. Citing this work
12. Contributing
13. Contact

Semantic similarity — deciding how close in meaning two words, sentences, or documents are — is a foundational primitive in NLP. It powers information retrieval, question answering, paraphrase detection, knowledge-graph and ontology alignment, machine-translation evaluation, and clinical concept mapping.

The catch: no single similarity measure is best everywhere. A WordNet path measure shines on taxonomic noun pairs; a transformer embedding shines on sentences; a string measure shines on near-duplicate surface forms. Each captures a different slice of "meaning."

Why use an ensemble? Because different measures make different mistakes. When measures are individually decent but disagree (i.e., they are diverse), stacking them — combining their scores with a learned or evolved aggregator — recovers signal that no single measure has, and typically yields a more accurate, more robust similarity estimator. This is the central finding of the accompanying review.

This repository lets you see that happen on real data, in minutes, on a laptop.

Twelve base measures (6 lexical + 6 WordNet) and six ensembles, evaluated on four human-rated benchmarks. Supervised ensembles are scored on out-of-fold predictions (5-fold CV), so nothing is inflated by train/test leakage:

Read the MC-30 row as a feature, not a bug. The "best single measure" column is an oracle pick: the max over 12 base measures on the evaluation data itself, which is upwardly biased on small sets, while the ensembles are scored strictly out-of-fold. With 30 pairs, the 95% confidence interval around ρ = 0.818 spans roughly [0.64, 0.91] (Fisher z), so a −0.021 delta carries no signal. There is also no theorem that an ensemble must match its best member: the classical convex-loss bound only guarantees an average beats the average member, and it does not apply to rank metrics at all. The harness therefore reports a fold-honest best_single_cv baseline (measure chosen on training folds only) next to the oracle row, and summarize() prints confidence intervals plus a Williams-test p-value so small deltas are never over-read. On the largest, hardest benchmark (SimLex-999) stacking gives the biggest gain. Details: docs/02-why-ensembles.md and docs/04-evaluation-metrics.md.

Every number and figure above is regenerated by one command:

python benchmarks/reproduce.py     # → benchmarks/results/ (CSV·LaTeX·MD) + assets/

Full per-method tables: benchmarks/results/summary.md.

Statistical rigor is built in, not bolted on:

results = evaluation.benchmark(pairs, gold, with_ci=True)   # + 95% bootstrap CI columns
print(evaluation.summarize(results))
# Best single measure (oracle pick): wn_jcn (Spearman=0.818)  [95% CI 0.640, 0.911]
# Fold-honest single baseline      : best_single_cv (Spearman=...)
# Best ensemble (out-of-fold)      : linear_stacking (Spearman=0.797)  [95% CI ...]
# Delta vs. oracle single measure  : -0.021 Spearman
# Williams t-test (ens vs. oracle) : p=... (not significant at 0.05)

lo, hi = metrics.fisher_ci(0.818, n=30)                     # analytic interval, no resampling
delta, p = metrics.paired_bootstrap_test(pred_a, pred_b, gold)  # paired comparison

Why it works — measure diversity (lexical vs. WordNet blocks barely correlate):

git clone https://github.com/jorge-martinez-gil/similarity-ensemble.git
cd similarity-ensemble
pip install -r requirements.txt          # numpy, pandas, matplotlib (+ optional nltk)

Build an ensemble and benchmark it in ~10 lines (runs offline, no downloads):

import numpy as np
from similarity_ensemble import measures, evaluation, datasets

pairs = datasets.sample_pairs()                       # word pairs (offline sample)
gold  = datasets.load("mc30")["gold"].to_numpy()      # real human ratings (downloads once)
pairs = list(datasets.load("mc30")[["word1", "word2"]].itertuples(index=False, name=None))

# Compare every base measure AND every ensemble (CV-evaluated) in one call:
results = evaluation.benchmark(pairs, gold, measures=measures.LEXICAL_MEASURES)
print(results.round(3))
print(evaluation.summarize(results))

benchmark() returns a tidy table of Pearson / Spearman / Kendall / RMSE for each base measure and each ensemble (supervised ensembles are scored with cross-validation, so the numbers aren't inflated). Export it to a publication-ready LaTeX table with evaluation.to_latex_table(results).

Add knowledge-based (WordNet) measures (optional):

pip install nltk
python -m similarity_ensemble.datasets --nltk     # downloads WordNet corpora once

from similarity_ensemble import measures
wn_measures = {
    "path": lambda a, b: measures.wordnet_similarity(a, b, "path"),
    "wup":  lambda a, b: measures.wordnet_similarity(a, b, "wup"),
    "lin":  lambda a, b: measures.wordnet_similarity(a, b, "lin"),
}

similarity-ensemble/
├── similarity_ensemble/        ← the reference toolkit (NumPy + pandas core)
│   ├── measures.py             ← base measures: lexical + WordNet (path/wup/lch/res/lin/jcn)
│   │                              + word-embedding cosine (GloVe/word2vec text, NumPy-only loader)
│   ├── ensembles.py            ← algebraic, regression-stacking, evolutionary (GA) ensembles
│   ├── metrics.py              ← Pearson/Spearman/Kendall/RMSE + bootstrap & Fisher CIs
│   │                              + Williams/Steiger and paired-bootstrap significance tests
│   ├── datasets.py             ← downloaders: RG65, MC30, WordSim353, SimLex999, MEN, SimVerb3500
│   ├── evaluation.py           ← one-call CV benchmark harness + honest baseline + LaTeX export
│   └── viz.py                  ← publication-quality figures
├── benchmarks/                 ← reproduce.py + committed results (CSV · LaTeX · Markdown)
├── assets/                     ← the figures in this README (regenerated by reproduce.py)
├── docs/                       ← tutorials, taxonomy deep-dive, metrics guide, glossary, exercises
├── notebooks/                  ← executable Jupyter notebooks (Colab-ready, beginner → advanced)
├── lecture/                    ← lecture-ready material for courses
├── paper/                      ← JOSS paper draft (paper.md + paper.bib)
├── tests/                      ← offline test suite (pytest, runs in CI on 3 OSes × py3.9–3.13)
├── .github/workflows/          ← CI + PyPI trusted publishing
├── requirements.txt · pyproject.toml · CITATION.cff · .zenodo.json · RELEASING.md

The review organizes semantic-similarity stacking into six families. This repo implements transparent reference baselines for the families that run on a laptop, and documents all six (see docs/03-stacking-taxonomy.md).

To the best of our knowledge, no other maintained package benchmarks combinations of heterogeneous similarity measures under a single seeded, cross-validated protocol with uncertainty quantification. That evaluation gap is what this repository exists to close.

from similarity_ensemble import measures, ensembles
pairs = [("car","automobile"), ("gem","jewel"), ("noon","string")]
feat  = measures.build_feature_table(pairs, measures.LEXICAL_MEASURES)
X     = feat[measures.measure_columns(feat)].to_numpy()
scores = ensembles.MeanEnsemble().fit(X).predict(X)   # unsupervised

from similarity_ensemble import ensembles, metrics
ga = ensembles.GAWeightedEnsemble(metric=metrics.spearman, generations=80).fit(X, gold)
print(ga.weights)          # contribution of each measure
print(ga.history_[-1])     # best fitness reached

from similarity_ensemble import measures

vocab = {w for pair in pairs for w in pair}
vecs  = measures.load_word_vectors("glove.6B.300d.txt", vocab=vocab)  # loads MBs, not GBs
emb   = measures.embedding_measure(vecs)          # cosine in [0,1]; OOV-tolerant

my_set  = {**measures.LEXICAL_MEASURES, "emb_cos": emb}
results = evaluation.benchmark(pairs, gold, measures=my_set)

def my_measure(a, b): ...      # returns a float in [0, 1]
my_set = {**measures.LEXICAL_MEASURES, "mine": my_measure}
results = evaluation.benchmark(pairs, gold, measures=my_set)

from similarity_ensemble import viz
viz.measure_correlation_heatmap(feat)        # diversity among measures
viz.contribution_weights(ga.weights, cols)   # which measures drive the ensemble
viz.ga_convergence(ga.history_)              # optimization trajectory
viz.base_vs_ensemble_bars(results)           # did the ensemble actually help?

from similarity_ensemble import metrics
point, lo, hi = metrics.bootstrap_ci(pred, gold, metric=metrics.spearman)    # 95% CI (resampling)
lo, hi   = metrics.fisher_ci(point, n=len(gold))                             # 95% CI (analytic)
p        = metrics.steiger_williams_test(r_A, r_B, r_AB, n=len(gold))        # dependent correlations
delta, p = metrics.paired_bootstrap_test(pred_a, pred_b, gold)               # paired, nonparametric

from similarity_ensemble import evaluation
preds = evaluation.cross_val_best_single(X, gold, k=5, seed=0)  # out-of-fold predictions

The toolkit downloads the original, human-rated benchmarks from their distributors and caches them locally — it never ships transcribed scores.

python -m similarity_ensemble.datasets --list        # show the registry
python -m similarity_ensemble.datasets mc30          # download + cache one

More benchmarks (STS, SICK, MEN, SimVerb, multilingual and domain-specific sets) are catalogued with references in docs/05-datasets.md and are a great first contribution — see CONTRIBUTING.md.

Hands-on notebooks live in notebooks/ (beginner → advanced), and lecture-ready material for NLP / ML / IR / evolutionary-computation courses is in lecture/.

• Lightweight: the core needs only NumPy, pandas, matplotlib. The whole test suite runs offline.
• Deterministic: every stochastic component (GA, bootstrap, CV folds) takes a fixed seed.
• Traceable: datasets are fetched from cited sources and cached; results come only from code in this repo.
• One-command regeneration: every number and figure in this README comes out of benchmarks/reproduce.py — if you change the code, rerun it and the README's claims are re-verified.
• Continuously tested: GitHub Actions runs the suite on Linux, Windows and macOS across Python 3.9–3.13 on every push.

pip install -e ".[all]"
pytest -q                                    # offline test suite
python -m similarity_ensemble.datasets --nltk   # WordNet corpora (once)
python benchmarks/reproduce.py               # regenerate all results + figures

If this repository or the accompanying review is useful in your research, please cite the paper:

@article{martinez2022comprehensive,
  title   = {A comprehensive review of stacking methods for semantic similarity measurement},
  author  = {Martinez-Gil, Jorge},
  journal = {Machine Learning with Applications},
  volume  = {10},
  pages   = {100423},
  year    = {2022},
  publisher = {Elsevier},
  doi     = {10.1016/j.mlwa.2022.100423},
  url     = {https://www.sciencedirect.com/science/article/pii/S2666827022000986}
}

To cite the software itself (e.g., when you build on the toolkit), use the machine-readable CITATION.cff — GitHub's "Cite this repository" button renders it for you. Each tagged release is archived on Zenodo with its own DOI (see RELEASING.md), and a JOSS paper draft lives in paper/. Related work by the same author that this platform connects to:

• Automatic Design of Semantic Similarity Ensembles Using Grammatical Evolution (2023), arXiv:2307.00925.
• A Comparative Study of Ensemble Techniques Based on Genetic Programming: Semantic Similarity Assessment (IJSEKE, 2022), DOI:10.1142/S0218194022500772.

Contributions are very welcome — new measures, ensembles, dataset downloaders, docs, or notebooks. Please read CONTRIBUTING.md, especially the scientific integrity rules (no fabricated results, no hand-transcribed gold scores, report uncertainty, cite sources).

Jorge Martinez-Gil — Senior Researcher, Software Competence Center Hagenberg (SCCH), Austria 🌐 jorgemar.com · GitHub · Google Scholar

_{⭐ If this helps your research or teaching, consider starring the repo so other researchers can find it.}