lexsim v0.4 — Segmenting Japanese into Words, Still Without a Dictionary
A 2 KB trained model now splits Japanese at word boundaries. There is still no dictionary. Dropping the character-bigram approach improved search and near-duplicate detection alike.
Version 0.4 of lexsim, our lexical similarity engine, is out. How it handles Japanese has changed at the root.
No more sliding two characters at a time
Until now, lexsim treated languages without spaces by sliding a two-character window across the text. The Japanese for "memory feature" came apart into four pieces, two of which were fragments that mean nothing on their own. Avoiding a dictionary came at that cost.
In v0.4 the same phrase is cut into its two real words instead.
Search and duplicate detection are both built by counting these units. Now that the units are meaningful words, BM25 ranking, Jaccard near-duplicate detection, and keyword extraction all improved together.
There is still no dictionary
The obvious fix — ship a morphological dictionary — was not the one we took. A dictionary of several megabytes is heavy as a dependency for an MCP server, and avoiding exactly that is why lexsim exists.
Instead the crate carries a model that decides, for each boundary between two characters, whether a word ends there. It is an AdaBoost classifier over 42 features, trained on 1,276 diverse sentences, and the embedded binary is about 2 KB. Because it is not a dictionary, it will still take a guess at unknown words, proper nouns, and coinages.
- Merged inflections — conjugated verb forms collapse to a single base-form token
- Stopword filtering — particles, auxiliaries, and other terms that carry no search signal are dropped using a list of more than 180 entries
- Non-CJK is unchanged — English and similar scripts still use UAX#29 word boundaries
Keyword extraction, added
textrank_keywords— builds a co-occurrence graph within a single text and pulls out its important termsCorpus::cooccurrence_keywords— finds the terms that tend to appear alongside a query term across a corpusCorpus::normalized_tf— length-normalized term frequency, with stopwords excluded
A note on compatibility
Because Japanese tokens changed from character bigrams to words, scores do not match those from v0.3 and earlier. BM25 scores and Jaccard coefficients both shift. The quality is better, but if you compare against previously recorded numbers, you will need to re-establish your baseline.
lexsim powers memory search in Handoff. Handoff MCP picked up this version in v0.19.1, improving recall when searching Japanese memories.