autocomplete: tune autocorrect algorithm.
Tune autocorrect to produce fewer false positives. Before we used trigram similarity. Now we use Levenshtein edit distance with a dynamic typo threshold. Trigram similarity was able to correct large transpositions (e.g. `miku_hatsune` -> `hatsune_miku`), but it was bad at correcting small typos. Levenshtein is good at small typos, but can't correct large transpositions.
This commit is contained in:
@@ -100,7 +100,7 @@ class AutocompleteService
|
||||
end
|
||||
|
||||
def tag_autocorrect_matches(string)
|
||||
tags = Tag.nonempty.fuzzy_name_matches(string).order_similarity(string).limit(limit)
|
||||
tags = Tag.nonempty.autocorrect_matches(string).limit(limit)
|
||||
|
||||
tags.map do |tag|
|
||||
{ type: "tag", label: tag.pretty_name, value: tag.name, category: tag.category, post_count: tag.post_count }
|
||||
|
||||
@@ -234,16 +234,19 @@ class Tag < ApplicationRecord
|
||||
end
|
||||
|
||||
module SearchMethods
|
||||
def autocorrect_matches(name)
|
||||
tags = fuzzy_name_matches(name).order_similarity(name)
|
||||
end
|
||||
|
||||
# ref: https://www.postgresql.org/docs/current/static/pgtrgm.html#idm46428634524336
|
||||
def order_similarity(name)
|
||||
# trunc(3 * sim) reduces the similarity score from a range of 0.0 -> 1.0 to just 0, 1, or 2.
|
||||
# This groups tags first by approximate similarity, then by largest tags within groups of similar tags.
|
||||
order(Arel.sql("trunc(3 * similarity(name, #{connection.quote(name)})) DESC"), "post_count DESC", "name DESC")
|
||||
order(Arel.sql("levenshtein(left(name, 255), #{connection.quote(name)}), tags.post_count DESC, tags.name ASC"))
|
||||
end
|
||||
|
||||
# ref: https://www.postgresql.org/docs/current/static/pgtrgm.html#idm46428634524336
|
||||
def fuzzy_name_matches(name)
|
||||
where("tags.name % ?", name)
|
||||
max_distance = [name.size / 4, 3].max.floor.to_i
|
||||
where("tags.name % ?", name).where("levenshtein(left(name, 255), ?) < ?", name, max_distance)
|
||||
end
|
||||
|
||||
def name_matches(name)
|
||||
|
||||
Reference in New Issue
Block a user