Search
Try other settings
Try other settings replays an index's most searched queries against other search settings and scores each one by what people clicked. You see the lift on your own data before you change anything.
It never changes your results. Nothing changes until an owner or admin applies a setting, and Roll back puts the previous settings back in one click.
How it works
- In the dashboard, open Indexes, then Try other settings in the index's menu, and select Run.
- Headband takes up to 100 of the index's most searched queries from the last 30 days that people clicked results for.
- It runs each query with your current settings and with up to 5 other settings, then scores every list against the clicks.
- The table shows each setting's score, its lift over your current settings and how sure that lift is. Apply one, or keep what you have.
How the clicks become a score
Click feedback keeps a signal per query and result. A click adds 1, more when the result was further down the page. A conversion adds 5. The signal halves every 14 days.
For each query, the result with the strongest signal is the most relevant. A result scores 3 at half of that signal or more, 2 at a fifth or more, or with any conversion, and 1 below that, down to a signal of 0.5. Results nobody clicked score 0.
Each list is scored with nDCG@10, which rewards the relevant results near the top, and MRR@10, which looks at where the first one lands. These are the measures our own engine benchmarks use.
The lift comes with a 95% interval from a bootstrap over the queries, and a p-value from a paired permutation test. A setting is called better only when the interval is above zero and p is below 0.05.
What the score can and cannot see
A replay sends the query text only. Filters, sorts and tenant token rules from the original searches are not stored, so they are not replayed. Every setting, the current one included, is scored on the engine's own ranking, without the click feedback boost.
What it tries
Each setting is your current profile with one change, taken from what we measured on public data. Applying it keeps your synonyms, stop words and fields.
A setting that searches exactly like your current one is left out. So is one the engine keeps on the index, such as typo rules on Meilisearch, since trying it needs a copy of the index. The dashboard lists what was not tried and why. Trying another engine is not part of this yet.
Limits
With less data the run stops and says how much it found. A run that reaches its time limit scores the queries it finished.
What it never does
- It never changes your results, settings or documents. Only Apply does.
- Replayed searches never count as your searches: not on your bill, not against your plan's limits, and not in Analytics or click feedback.
- It never applies anything on its own. Apply and Roll back are owners and admins only.
- Apply refuses when the settings changed since the run, and Roll back refuses when they changed since Apply. Nothing is overwritten.
- It reads only the stored, redacted query text. A query whose text redaction changed is left out, since replaying it would search for something nobody searched.
Applying writes the same search profile as PUT /v1/indexes/:uid/profile, so every door to the index reads it.