yc × jev
1Try the search

Can Jev replace an LLM reranker?

Jev reached 110% of Claude Haiku’s ranking quality at 2.0× the speed and 16× lower cost.

Every grade so far comes from Claude Opus. The human check of that grader has not been done yet, so treat the quality numbers as provisional.

200 test queries·10,164 graded pairs·YC snapshot 2026-09-22·judge: Claude Opus

§2

The setup

Jev is not a search engine. It answers typed questions about a situation with a choice or a score, and a confidence. So the test puts it where a search stack already asks those questions: picking what the user wants, and scoring 100 retrieved candidates against the query.

Retrieval is held fixed. Every reranker scores the identical 100 companies, so any difference below is the reranker's alone.

  1. 1querya sentence someone wrote on Hacker News
  2. 2routepick an intent: competitors, products, jobs or open source
  3. 3filterhard filters from that intent, e.g. hiring only
  4. 4retrieve 100BM25 + embeddings, frozen once for every reranker
  5. 5rerankthe only step that changesNo rerankingBGE cross-encoderClaude Haiku 4.5Jev
  6. 6top 10what gets graded
The pipeline: query, route, filter, retrieve 100 candidates, rerank, top 10. Only the rerank step differs between the four rerankers.

Jev and Claude Haiku can each be asked two ways, and both got the same treatment: a pilot on 20 held-out dev queries, never reported as results, picked one formulation per model by nDCG@10, with latency breaking near-ties. Every number below uses the pick.

Jev
FormulationnDCG@10p50Per 1,000Requests per search
one call per candidate; state holds the query and one company card0.722.7 s$2.35100
one call per search; state holds the query, 100 questions each carry a cardchosen0.741.2 s$1.181
Claude Haiku 4.5
FormulationnDCG@10p50Per 1,000Requests per search
one prompt per search scoring all 100 cards0.643.3 s$17.111
10 prompts per search sent at once, 10 cards eachchosen0.671.4 s$18.3210
§3

Ranking quality

How good is the top 10 each reranker puts in front of the user, graded against the judge's labels.

Ranking quality per reranker, mean with 95% bootstrap interval over queries.
RerankernDCG@10Recall@10MRRQueries
No reranking
0.41
No reranking nDCG@10 0.41, 95% interval 0.37 to 0.45
0.40
No reranking Recall@10 0.40, 95% interval 0.35 to 0.45
0.72
No reranking MRR 0.72, 95% interval 0.67 to 0.77		198
BGE cross-encoder
0.47
BGE cross-encoder nDCG@10 0.47, 95% interval 0.43 to 0.50
0.48
BGE cross-encoder Recall@10 0.48, 95% interval 0.43 to 0.53
0.75
BGE cross-encoder MRR 0.75, 95% interval 0.70 to 0.80		198
Claude Haiku 4.5
0.73
Claude Haiku 4.5 nDCG@10 0.73, 95% interval 0.70 to 0.75
0.76
Claude Haiku 4.5 Recall@10 0.76, 95% interval 0.71 to 0.81
0.94
Claude Haiku 4.5 MRR 0.94, 95% interval 0.91 to 0.96		198
Jev
0.80
Jev nDCG@10 0.80, 95% interval 0.78 to 0.83
0.80
Jev Recall@10 0.80, 95% interval 0.75 to 0.84
0.94
Jev MRR 0.94, 95% interval 0.91 to 0.97		198
Bars run from 0 to the mean on a 0–1 scale. The black whisker is the 95% bootstrap interval over queries: where two whiskers overlap, the gap between those rerankers may be noise.
§4

Speed and cost

Jev's pitch is speed and price. Here is what each reranker costs in time and money for the quality it buys.

0.20.30.40.50.60.70.80.9100 ms1 s10 s100 smedian latency per search, log scale →nDCG@10 ↑No reranking 0.41 · adds no latencyBGE cross-encoder: nDCG@10 0.47 (0.43–0.50), p50 16.3 sBGE0.47 · 16.3 sClaude Haiku 4.5: nDCG@10 0.73 (0.70–0.75), p50 1.4 sHaiku0.73 · 1.4 sJev: nDCG@10 0.80 (0.78–0.83), p50 671 msJev0.80 · 671 ms
Up is better ranking, left is faster. Whiskers are 95% intervals on nDCG@10. Latency is the median per search of 100 candidates. A reranker that adds no step is drawn as a line: anything below it made results worse.
Latency and cost per reranker
Rerankerp50p99Per 1k searchesAt 1M a monthLatency measured as
No reranking——runs locallyyour hardwareno rerank step
BGE cross-encoder16.3 s25.4 sruns locallyyour hardwarewall clock
Claude Haiku 4.51.4 s4.8 s$18.23$18,231API time, via Claude Code
Jev671 ms2.1 s$1.17$1,173wall clock
§5

By intent

A reranker that wins on competitor searches and loses on job searches is a finding, so each intent gets its own panel.

Competitors

50 queries
nDCG@10 by reranker for Competitors queries
None0.52, 95% interval 0.46 to 0.58
BGE0.55, 95% interval 0.49 to 0.61
Haiku0.82, 95% interval 0.79 to 0.85
Jev0.90, 95% interval 0.88 to 0.92

Products

50 queries
nDCG@10 by reranker for Products queries
None0.25, 95% interval 0.18 to 0.32
BGE0.35, 95% interval 0.27 to 0.43
Haiku0.69, 95% interval 0.62 to 0.75
Jev0.83, 95% interval 0.79 to 0.87

Jobs

50 queries
nDCG@10 by reranker for Jobs queries
None0.50, 95% interval 0.45 to 0.55
BGE0.53, 95% interval 0.48 to 0.58
Haiku0.75, 95% interval 0.71 to 0.79
Jev0.72, 95% interval 0.66 to 0.78

Open source

50 queries
nDCG@10 by reranker for Open source queries
None0.38, 95% interval 0.29 to 0.47
BGE0.44, 95% interval 0.36 to 0.52
Haiku0.65, 95% interval 0.58 to 0.72
Jev0.76, 95% interval 0.68 to 0.82
nDCG@10 per intent, every panel on the same 0–1 scale, with 95% intervals.
§6

The judge-free check

Launch HN posts come with a correct answer: the company that wrote the post. Its rank needs no grader, which makes this the most trustworthy chart on the page.

No reranking

#1 in 13 of 50, top 10 in 27
median rank #6

BGE cross-encoder

#1 in 16 of 50, top 10 in 30
median rank #5

Claude Haiku 4.5

#1 in 20 of 50, top 10 in 34
median rank #2

Jev

#1 in 25 of 50, top 10 in 35
median rank #1

One dot per Launch HN query: where each reranker placed the company that wrote the post. Rank is on a log scale, so the left edge is the top result. No judge is involved in this chart.
§7

Routing

Before retrieval, the query is routed to one of four intents. Jev Choice and Claude Haiku got the same queries and the same list.

Claude Haiku 4.5

83%accurate
Claude Haiku routing confusion matrix: rows are actual intent, columns predicted
actual ↓ picked →Comp.ProductJobOSS
Comp.
23
23
1
3
Product
4
46
0
0
Job
0
0
50
0
OSS
0
3
1
46

200 queries · median 2.0 s · $0.39 per 1,000

Jev Choice

94%accurate
Jev routing confusion matrix: rows are actual intent, columns predicted
actual ↓ picked →Comp.ProductJobOSS
Comp.
39
8
2
1
Product
0
50
0
0
Job
0
0
50
0
OSS
0
2
0
48

200 queries · median 390 ms · $0.026 per 1,000

Rows are the intent the query came from; columns are what the router picked. The diagonal is correct.
§8

Does Jev’s confidence mean anything?

Jev returns a confidence with every answer. If it is honest, a product could act on it: skip the expensive step when Jev is sure, ask the user when it isn't.

000.250.250.50.50.750.7511perfectly calibratedConfidence 0.2–0.3: 1 queries, mean confidence 0.25, 0% correctConfidence 0.3–0.4: 9 queries, mean confidence 0.36, 78% correctConfidence 0.4–0.5: 10 queries, mean confidence 0.43, 60% correctConfidence 0.5–0.6: 12 queries, mean confidence 0.55, 83% correctConfidence 0.6–0.7: 9 queries, mean confidence 0.67, 89% correctConfidence 0.7–0.8: 21 queries, mean confidence 0.74, 95% correctConfidence 0.8–0.9: 24 queries, mean confidence 0.85, 100% correctConfidence 0.9–1.0: 114 queries, mean confidence 0.97, 98% correctJev's confidence →share correctqueries per bin

Each dot is a band of confidence. Its height is how often Jev's intent was right when it reported that much confidence. On the diagonal, a confidence of 0.8 means right four times in five.

Expected calibration error 0.101
weighted mean gap between confidence and accuracy; 0 is perfect

§9

What we expected, and what held

Written down before anything ran.

  1. H1

    Jev reaches at least 90% of Claude Haiku's nDCG@10 as a reranker.

    nDCG@10 Jev 0.80 vs haiku 0.73 (110%), n=198

    Supported
  2. H2

    Jev's p50 latency for 100 candidates is under 10% of Haiku's.

    p50 Jev 671 ms (wall) vs Haiku 1374 ms (api), 49%

    Rejected
  3. H3

    Jev matches Haiku within 3 points of accuracy on intent routing.

    accuracy Jev 94% vs Haiku 83%, n=200

    Rejected
  4. H4

    Jev beats the open-source cross-encoder on accuracy.

    nDCG@10 Jev 0.80 vs bge 0.47, n=198

    Supported
§10

Look at the rankings yourself

Pick a query and follow each company from retrieval order to each reranker's top 10. Heavier threads are better matches by the judge's grade.

200 queries

Competitors · source on news.ycombinator.com ↗

“We have an app and community to help people cut back on their compulsive (or just excessive) use of pornography and other digital activities like social media and gaming. Our community currently has 100k members.”

No reranking

hybrid retrieval order

BGE

bge-reranker-v2-m3 (ONNX)

Haiku

claude-haiku-4-5-20251001

Jev

jev (TypeSafe)

Judge's gradeexact matchpartial matchnot relevantnot gradedHover, focus or tap a company to follow it across columns.
Top 10 per reranker for the query: We have an app and community to help people cut back on their compulsive (or just excessive) use of pornography and other digital activities like social media and gaming. Our community currently has 100k members.
RankNo rerankingBGE cross-encoderClaude Haiku 4.5Jev
1Reframe (partial match)Relay Health (exact match)Relay Health (exact match)Atmana (exact match)
2Atmana (exact match)Atmana (exact match)Atmana (exact match)Relay Health (exact match)
3Relay Health (exact match)Twitch (not relevant)Sunflower (partial match)Sunflower (partial match)
4Twitch (not relevant)Reframe (partial match)Clearspace (exact match)Clearspace (exact match)
5Status (not relevant)Roforco (not relevant)Recover (partial match)Recover (partial match)
6Glimpse (not relevant)Reddit (not relevant)Pelago (partial match)Pelago (partial match)
7Plura (formerly Bloom Community) (not relevant)9gag (not relevant)Reframe (partial match)Reframe (partial match)
8Recurious (not relevant)Pocket Worlds (not relevant)Breathe Well-being (not relevant)GroupAhead (not relevant)
9Socialmoth (not relevant)Rhythmm (not relevant)Daybreak Health (partial match)Daybreak Health (partial match)
10GroupAhead (not relevant)Wakie (not relevant)Reddit (not relevant)Wakie (not relevant)
§11

How much to trust this

Every quality number above comes from one grader: Claude Opus, reading each query with 20 pooled companies at a time and grading them 0 (irrelevant), 1 (partial) or 2 (exact). It never saw which reranker found a company.

The hand-graded check of the judge has not been run yet.

A Claude judge may favour a Claude reranker. Read Claude Haiku's lead against the judge-free check in section 6: that chart uses no grader at all, so if Haiku's lead shrinks there, the judge is the likely reason.

Limitations

  • Queries come from Hacker News. They skew technical and toward developer tools, so the product and open-source intents are easier to fill than they would be on a general site.
  • Founder search is not tested. The YC directory mirror has no founder data, and Jev has no founder intent here.
  • Retrieval is held fixed. A reranker cannot recover a company retrieval missed, so every arm shares that ceiling.
  • Latency to a hosted API is mostly network. Claude Haiku ran through Claude Code, so its latency is API time reported by Claude Code, close to but not the same as calling the API directly. When a search sends several Haiku prompts at once, its latency is the slowest prompt's.
  • Agreement with the judge depends on where a card sat in the retrieval list. Correlation of each score with the judge's grade, for retrieval positions 1–25, 26–50, 51–75, 76–100: Jev 0.67, 0.60, 0.62, 0.61; Claude Haiku with all 100 cards in one prompt 0.62, 0.38, 0.39, 0.27; Claude Haiku with ten prompts of 10 cards 0.70, 0.61, 0.62, 0.52. The share of relevant cards barely moves across those bands (0.38, 0.43, 0.45, 0.43), so a falling correlation comes from the reranker, not the candidates. A 100-card prompt loses Haiku most of its agreement past the first 25 cards; ten cards per prompt, the formulation its pilot picked, keeps most of it.
  • The live search does not retrieve quite the way the benchmark did. Every number here used fp32 query embeddings; the deployed app uses the q8 weights, because the fp32 file is 133 MB and a serverless function has 250 MB for everything it needs. Across the 200 test queries the two agree on a median 96% of the retrieved 100 and 90% of the top 10, and they disagree on the first result for 8.5% of queries. In this page's own currency, q8 drops 13 of the 910 grade-2 companies out of the top 100 and costs nothing on the Launch HN check. Rerun it with bun run bench:embed-drift.
  • If the embedding model fails to load, the live search ranks with BM25 alone and says so under the results. Nothing on this page is measured that way, and it is not a free fallback: grade-2 companies reaching the top 100 falls from 910/910 to 709/910, and the Launch HN company is found in 33 of 50 queries instead of 37.
  • Costs use each provider's published list price on the run date.
§12

Reproduce it

The data snapshot, queries, labels and prompts are in the repository. Retrieval is frozen once, so every arm reranks the identical 100 candidates.

git clone https://github.com/PPRAMANIK62/yc-jev-bench && cd yc-jev-bench
bun install
cp .env.example .env        # add TYPESAFE_API_KEY for the Jev arm
bun bench/index.ts          # build the retrieval index from the frozen snapshot
bun run bench               # every arm and router, the Opus judge, then src/generated/results.json
bun run dev                 # open this report and the search at localhost:3000