lexsim
A dictionary-free, multilingual lexical similarity engine for Rust. Answers "are these the same?" and "is this relevant?" with one tokenizer. Powers memory retrieval and dedup inside handoff-mcp.
When working with AI coding agents across sessions, two questions keep coming up: "is this memory relevant to what I'm doing now?" and "have I already saved something like this?" lexsim answers both on top of a single tokenizer.
"Are these the same?" gets a Jaccard similarity score. "Is this relevant?" gets a BM25 ranking. A stable content hash handles change detection.
The defining choice is that it uses no morphological dictionary at all. For languages like Japanese and Chinese that lack whitespace boundaries, it slides a two-character window across the text to produce overlapping pairs. No dictionary means minimal dependencies and the same tokenizer works across every language it encounters. The flip side is that it cannot judge whether text in different languages means the same thing — translation-level similarity is out of scope.
Originally built as the memory engine inside Handoff, then extracted as a standalone crate. It is code shaped by real use, not a lab experiment.
Key Features
Challenges Addressed
• Need to search memories across sessions in both Japanese and English, but morphological dictionaries are too heavy
• Duplicate detection and relevance search need to agree on what a "term" is, or results feel inconsistent
• Embedding inside an MCP server means minimal dependencies and low latency are non-negotiable
Approach Taken
• Sliding two-character window for dictionary-free tokenization, with normalization for variant forms and mixed-language text
• Jaccard (duplicate detection) and BM25 (relevance ranking) built on the same token stream
• Only two dependencies: unicode-segmentation and unicode-normalization
Outcomes Achieved
• Running in production as the memory engine inside handoff-mcp
• Searches mixed Japanese–English memories without any dictionary
• Published on crates.io and GitHub as a standalone crate