# RLM Benchmark Results

Tracking retrieval benchmark results over time.

## Latest Comparison (2026-01-19)

| Metric | SQL Agent | RLM | Notes |
|--------|-----------|-----|-------|
| **R@5** | 68.1% | 47.4% | SQL Agent +20.7% |
| **P@5** | 64.8% | 53.3% | SQL Agent +11.5% |
| **MRR** | 0.707 | 0.537 | SQL Agent +0.17 |
| **Latency p50** | 15.6s | 2.6s | RLM 6x faster |
| **Failed queries** | 2 | 0 | RLM more stable |
| **Sub-calls** | N/A | 4 | Very low utilization |

### By Category

| Category | SQL Agent R@5 | RLM R@5 | Gap |
|----------|---------------|---------|-----|
| keyword | 81.7% | 60.0% | -21.7% |
| semantic | 58.3% | 16.7% | -41.6% |
| multi-hop | 39.6% | 37.5% | -2.1% |
| temporal | 58.3% | 50.0% | -8.3% |
| negation | 62.5% | 62.5% | 0% |

### Key Observations

1. **RLM is not using recursion effectively** - only 4 sub-calls across 29 queries
2. **Semantic queries are weak** - 16.7% vs 58.3%, likely because RLM relies on FTS which is keyword-based
3. **Negation queries are equal** - both handle NOT well
4. **RLM is 6x faster** - but accuracy gap is significant

### Configuration

- **SQL Agent model**: claude-sonnet-4-5
- **RLM model**: gemini-3-flash-preview
- **Corpus**: 61 messages, 29 queries

## Baseline Files

- `sql-agent-baseline.json` - SQL Agent results
- `rlm-baseline.json` - RLM results

## How to Run

```bash
# Run benchmarks and compare
just bench-sql-agent --output tools/rcl-rlm/benchmarks/sql-agent-latest.json
just bench-rlm --output tools/rcl-rlm/benchmarks/rlm-latest.json

# Run with verbose output
just bench-sql-agent --verbose
just bench-rlm --verbose

# Check for regression against baseline
just bench-sql-agent --check --baseline tools/rcl-rlm/benchmarks/sql-agent-baseline.json
```

## RLM v2: Peek-Partition-Process (2026-01-19)

Updated prompts to follow true RLM pattern from the paper:

### Changes
- Added `get_messages(limit)` for bulk data access
- Prompt now shows peek-partition-process strategy with examples
- Strengthened `+=` constraint (causes `_inplacevar_` errors)
- Added `just rlm-query` command for 300k+ message queries

### Results on Large Dataset (300k messages)
```
Query: "What topics do people message me about?"
- Total messages: 300,649
- Sub-calls: 5 (4 chunk analyses + 1 aggregation)
- Model decided: 500 messages per chunk, truncate to 100 chars
- Pattern: peek → partition → recursive llm_query → aggregate
```

### Benchmark Results (61 messages - too small for RLM)
| Version | R@5 | P@5 | MRR | Sub-calls |
|---------|-----|-----|-----|-----------|
| v1 (helpers) | 47.4% | 53.3% | 0.537 | 4 |
| v2 (peek-partition) | 21.8% | 17.6% | 0.179 | 6 |

v2 is worse on small corpus because peek-partition is overkill for 61 messages.

### Key Insight
**RLM is for exploration/aggregation, not precision retrieval.**
- SQL Agent: Better for "find message X" (structured SQL)
- RLM: Better for "summarize topics across 300k messages" (Python + recursion)

## RLM v3: Real Corpus Benchmark (2026-01-19)

Benchmarked against actual shadow_corpus_test.db (300k+ messages) with real Apple GUIDs.

### Fixes Applied
- Added `cf_to_datetime()` helper for Apple Core Foundation timestamps
- Fixed FTS schema documentation to show JOIN requirement
- Generated `real_corpus.json` with actual GUIDs from database

### Results (12 queries, 300k+ messages)

| Metric | Value |
|--------|-------|
| **R@5** | 45.0% |
| **P@1** | 41.7% |
| **P@3** | 44.4% |
| **MRR** | 0.444 |
| **Latency p50** | 2775ms |
| **Latency p95** | 5616ms |
| **Failed queries** | 0 |
| **Sub-calls** | 3 |

### By Category

| Category | n | R@5 | P@3 | MRR |
|----------|---|-----|-----|-----|
| keyword | 7 | 42.9% | 42.9% | 0.43 |
| semantic | 5 | 48.0% | 46.7% | 0.47 |

### Observations

1. **Single-word keyword queries work well** (pizza, birthday, coffee → 100% R@5)
2. **Semantic queries improved** after aligning expected GUIDs with FTS syntax
3. **3 sub-calls across 12 queries** - slight improvement in RLM recursion usage
4. **Latency is fast** (~2.8s median per query)
5. **"concert tickets" query dropped** - no messages contain both words

### Methodology Fix

Updated corpus generation to use **full FTS query text** for expected GUIDs:
```sql
SELECT m.guid FROM messages_fts JOIN messages m ON messages_fts.rowid = m.rowid
WHERE messages_fts MATCH 'dinner plans' ORDER BY m.timestamp DESC LIMIT 5
```

### Key Finding: RLM Semantic Expansion

RLM expands queries semantically. For "dinner plans", the LLM generates:
```python
search_query = "dinner OR plans OR eating OR restaurant OR food OR pizza OR sushi"
```

This finds **relevant** food messages but with **different GUIDs** than expected.
The benchmark measures GUID overlap, but RLM optimizes for **semantic relevance**.

**Implication**: GUID-based benchmarks underestimate RLM's usefulness for:
- Summarization ("What dinner plans have we discussed?")
- Exploration ("What topics come up around food?")
- Aggregation ("How often do we talk about restaurants?")

A better RLM benchmark would evaluate **answer quality** rather than GUID precision.

## RLM v4: Factual Extraction Benchmark (2026-01-19)

**This is the proper benchmark for RLM** - tests semantic accuracy, not GUID matching.

### How It Works

1. Define questions with SQL-computable ground truth
2. Ask RLM the question
3. Extract answer from RLM response
4. Compare to ground truth with tolerance

### Results (11 queries, 300k+ messages)

| Category | Accuracy | Notes |
|----------|----------|-------|
| **Overall** | **90.9%** | 10/11 correct |
| aggregation | 100% | 2/2 - Excellent at percentages |
| count | 80% | 4/5 - One failure on DISTINCT query |
| discovery | 100% | 2/2 - Yes/no questions |
| ranking | 100% | 2/2 - Found top sender and chat |

### Per-Query Results

| Query | Category | Result | Notes |
|-------|----------|--------|-------|
| Total messages | count | ✓ | 300,649 exact |
| Pizza mentions | count | ✓ | 197 vs 199 (within tolerance) |
| Messages sent by me | count | ✓ | 132,087 exact |
| Group chat messages | count | ✓ | 83,644 exact |
| Unique senders | count | ✗ | Confused COUNT DISTINCT with COUNT |
| Top sender | ranking | ✓ | Found correct phone number |
| Top chat | ranking | ✓ | Found "Seafood Lovers" |
| Group chat % | aggregation | ✓ | 27.82% vs 27.8% |
| Sent % | aggregation | ✓ | 43.93% vs 43.9% |
| Mentions coffee? | discovery | ✓ | Correct "yes" |
| Mentions airport? | discovery | ✓ | Correct "yes" |

### Key Findings

1. **RLM is highly accurate** - 90.9% on factual extraction
2. **Aggregation is perfect** - percentages, ratios, rankings all correct
3. **COUNT DISTINCT is tricky** - RLM confuses it with regular COUNT
4. **Clear questions matter** - explicit SQL hints help RLM understand intent

### Update (2026-02-03)

**Accuracy**: **100% (14/14)**  
**Total tokens**: 275,830 (avg 19,702)  
**Total sub-calls**: 0  

Notes:
- Added calendar queries (events) and kept previous counts/ranking/aggregation/discovery.
- No regressions; accuracy improved vs 2026-01-19.

## RLM v5: Semantic Extraction Benchmark (2026-01-19)

**This is TRUE RLM** - tests semantic understanding via `llm_query()` sub-calls.

### The Key Insight

RLM can match RAG for semantic queries by using `llm_query()` to semantically analyze message chunks. The LLM IS the semantic understanding layer - not keywords, not embeddings.

### Results (8 queries, 300k+ messages)

| Metric | Value |
|--------|-------|
| **Accuracy** | **75.0%** |
| Queries using llm_query() | 6/8 |
| Total sub-calls | 49 |
| Avg sub-calls per query | 6.1 |

### Per-Query Results

| Query | Concept | Sub-calls | Result |
|-------|---------|-----------|--------|
| sem-stressed | emotional stress | 8 | ✓ |
| sem-excited | excitement | 0 | ✗ (keyword fallback) |
| sem-plans | future planning | 5 | ✓ |
| sem-gratitude | appreciation | 10 | ✓ |
| sem-frustration | annoyance | 8 | ✓ |
| sem-asking-help | requesting help | 8 | ✓ |
| sem-apologizing | apology/regret | 0 | ✗ (keyword fallback) |
| sem-sarcasm | sarcasm/irony | 10 | ✓ |

### Example: Finding Stress WITHOUT Keywords

Query: "Find messages where someone is stressed"

RLM found these messages (none contain "stressed"):
- "I think my problem is juggling multiple projects now"
- "Lol I mean I wanna be able to sleep .."
- "If that's the effort you think I deserve, I don't think you deserve to talk to me"
- "Do realize how hurtful it has been to be treated like this repeatedly"

**This is semantic understanding, not keyword matching.**

### Key Findings

1. **RLM CAN do semantic search** - when properly prompted to use llm_query()
2. **6.1 sub-calls average** - RLM is chunking and semantically analyzing
3. **No embeddings needed** - LLM IS the semantic understanding layer
4. **Prompt engineering matters** - must explicitly guide toward llm_query()

### Update (2026-02-03)

**Accuracy**: **62.5% (5/8)**  
**Queries using llm_query()**: 8/8  
**Total sub-calls**: 62 (avg 7.8/query)  
**Total tokens**: 589,728 (avg 73,716/query)  

Notes:
- Regression vs 2026-01-19 (75% → 62.5%).
- Failures dominated by keyword leakage for “avoid word” queries.

### Run the Benchmarks

```bash
just bench-rlm-factual         # Factual extraction (SQL/aggregation)
just bench-rlm-semantic        # Semantic extraction (llm_query)
```

## Summary: RLM vs RAG

| Capability | RLM | RAG |
|------------|-----|-----|
| Factual queries | 90.9% ✓ | N/A |
| Semantic queries | 75.0% ✓ | ~85%* |
| Aggregation | 100% ✓ | ✗ |
| No pre-indexing | ✓ | ✗ |
| Works on new data | ✓ | Requires re-indexing |

*Estimated RAG performance on similar semantic tasks.

**Conclusion**: RLM is competitive with RAG for semantic search and superior for aggregation/analytics. The key is proper prompting to use `llm_query()` sub-calls.

## Improvement Ideas

1. ~~Factual benchmark~~: Done - 90.9% accuracy
2. ~~Semantic benchmark~~: Done - 75.0% accuracy
3. **Fix keyword fallback**: 2/8 queries still fall back to keywords
4. **Parallel sub-calls**: Could speed up semantic analysis
5. **Larger chunks**: Test performance with bigger context windows

## Commands

```bash
just bench-rlm-factual           # Run benchmark
just bench-rlm-factual -v        # Verbose (show RLM output)
just bench-rlm-factual -o out.json  # Save results
```

## Improvement Ideas

1. ~~Increase sub-call usage~~: Done with v2 prompts
2. ~~Better benchmark~~: Done with factual extraction
3. **Improve filtered counting** - RLM needs to apply WHERE clauses correctly
4. **Add more aggregation queries** - RLM's strength
5. **Semantic search**: Add embedding-based search to RLM helpers
