A full-text search engine that tries to require as little thinking as possible.

Roughsearch is a lightweight full-text search engine built on DuckDB and BM25. It targets Japanese and English only. You can use it from the CLI or call it directly from Python.

So when you want to search a large pile of text files in a decent way, what do you do? Start by writing a Dockerfile? Spin up an Elasticsearch container? Install a morphological analysis plugin? Fire a huge number of API requests at it? Wait forever for indexing? No. Your life will end first.

Roughsearch gives up on flexibility completely. Fine-grained settings, grand scoring systems, and everything else are gone. This software has exactly one purpose: "search roughly." Run $ pip install roughsearch, and the environment is ready. Point a command at the target directory, and the indexing is done. That is all.

Roughsearch indexes loaded documents with the following pipeline:

1. It reads the text and runs morphological analysis with Sudachi.
2. It extracts major terms from the result, mainly nouns, verbs, and adjectives.
3. It indexes the normalized form of each extracted term and a romaji version of that term.

At search time, the original terms score higher and the transliterated alphabet forms score lower, producing weighted best-match results. All of this data is stored in a single .duckdb file, which makes the index highly portable.

$ pip install roughsearch

This software requires Python 3.11 or later.

When you want to add a simple full-text search engine to your own system.

import roughsearch

with roughsearch.Client("docs.duckdb", language="ja") as rs:
    rs.add("doc-001", title="いろはにほへと", body="あのイーハトーヴォのすきとおった風")
    rs.add("doc-002", title="ちりぬるを", body="夏でも底に冷たさをもつ青いそら")
    rs.reindex()

    results = rs.search("風")
    for hit in results.hits:
        print(hit.score, hit.title, hit.snippet)

When you just want to get it running.
The server exposes a REST API that any frontend can use for search.

$ roughsearch init docs.duckdb --language ja
$ roughsearch add docs.duckdb ./docs
$ roughsearch serve docs.duckdb --port 8080

add, serve, search, and dump normally use the default language saved by init. If needed, you can temporarily override it with --language.

When you want to connect to a running Roughsearch server and query it.

import roughsearch

rs = roughsearch.HttpClient("http://localhost:8080")
rs.add("doc-001", title="Sky", body="blue sky", reindex=True)
docs = rs.list_documents(limit=10)
rs.update_document("doc-001", title="Sky v2", body="clear blue sky", reindex=True)
results = rs.search("空")

$ pip install roughsearch

$ roughsearch init notes.duckdb --language ja
$ roughsearch add notes.duckdb ./notes --glob "*.md"
$ roughsearch search notes.duckdb "ニンジャ"

import roughsearch

with roughsearch.Client("notes.duckdb", language="ja") as rs:
    rs.add(
        "note-001",
        title="いろはにほへと",
        body="あのイーハトーヴォのすきとおった風",
        metadata={"tags": ["note", "japanese"], "source": "handbook"},
        source_uri="handbook/note-001.md",
    )
    rs.reindex()

    from roughsearch.search.query import SearchQuery, SearchFilters
    results = rs.search(
        SearchQuery(
            query="風",
            filters=SearchFilters(tags=["note"]),
            highlight=True,
            limit=10,
        )
    )

$ roughsearch serve docs.duckdb --port 8080 &

$ curl -s -X POST "http://localhost:8080/documents?reindex=true" \
  -H "Content-Type: application/json" \
  -d '{"id":"1","title":"いろはにほへと","body":"あのイーハトーヴォのすきとおった風"}'

$ curl -s -X POST http://localhost:8080/search \
  -H "Content-Type: application/json" \
  -d '{"query":"風","limit":5}' | python -m json.tool

$ curl -s http://localhost:8080/documents?limit=10 | python -m json.tool

import roughsearch

docs = [
    {"id": "1", "title": "いろはにほへと", "body": "あのイーハトーヴォのすきとおった風"},
    {"id": "2", "title": "ちりぬるを",  "body": "夏でも底に冷たさをもつ青いそら"},
]

with roughsearch.Client("bulk.duckdb") as rs:
    rs.add_documents(docs)
    rs.reindex()
    print(rs.search("風").total)

{
  "query": "風",
  "total": 1,
  "hits": [
    {
      "id": "doc-001",
      "score": 8.512,
      "title": "いろはにほへと",
      "snippet": "あのイーハトーヴォのすきとおった<mark>風</mark>",
      "body": "あのイーハトーヴォのすきとおった風",
      "language": "ja",
      "source_uri": null,
      "heading_path": null,
      "parent_id": null,
      "chunk_id": null,
      "metadata": {}
    }
  ]
}

• Reindexing is required after writes unless you opt in per request. Embedded Client.add() and Client.add_documents() still need reindex() by default. Over HTTP, POST /documents, POST /documents/bulk, and PUT /documents/{id} accept ?reindex=true if you want the write reflected in search immediately.
• Assume a single writer. DuckDB does not support concurrent writes. Run one server process and only one write operation at a time.
• It listens on localhost by default. If you need external access, place it behind a reverse proxy such as nginx.

MIT. See LICENSE for details.

powered by Sudachi: Apache License v2.0