# Memory Vector System

**Status:** ✅ Supermonkey-Powered  
**Created:** 2026-03-02  
**Replaces:** Supermemory cloud embedding API

---

## Overview

Local semantic memory search using SQLite + Ollama embeddings. No cloud dependency, no API limits, works offline.

## Architecture

### File-Based Pipeline
```
Memory Files (markdown)
    ↓
memory_embedding_worker.py
    ↓
Ollama (nomic-embed-text) → 768-dim vectors
    ↓
SQLite + sqlite-vector extension
    ↓
Cosine similarity search
```

### Real-Time Session Pipeline (NEW)
```
Discord/Chat Messages
    ↓
OpenClaw Session Transcript (.jsonl)
    ↓
session_monitor.py (cron every 2 min)
    ↓
Count messages → At 15: summarize
    ↓
Ollama (nomic-embed-text)
    ↓
SQLite + sqlite-vector
```

**Innovation:** We read OpenClaw's own session transcripts to auto-capture conversations without manual tracking!

## Components

### Core Module
**File:** `memory_vector.py`
- `MemoryVectorDB` class — database wrapper
- `store_memory()` — save embedding
- `search_memories()` — semantic search
- `setup_memory_vectors()` — one-time init

### Worker Scripts
**File:** `tools/memory_embedding_worker.py`
- Daily/batch processing for memory.md files
- Processes section-by-section
- Called by cron at 3 AM

**File:** `tools/session_monitor.py` ⭐ NEW
- Reads OpenClaw session transcripts live
- Tracks message counts automatically
- Creates snapshots every 15 messages
- Cron: every 2 minutes
- Database: `session_tracking` table

**File:** `tools/session_snapshotter.py`
- Manual session capture (legacy)
- Use session_monitor.py for auto-tracking

**File:** `tools/search_memories.py`
- CLI tool for manual searches
- Interactive or one-shot mode

**File:** `tools/bulk_memory_loader.py`
- One-time historical import
- Processed 1,186 embeddings on first run

**File:** `scripts/memory-embeddings-cron.ps1`
- PowerShell wrapper for daily cron
- Checks Ollama availability

## Database Schema

### memory_embeddings
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| source_type | TEXT | "daily", "memory_md", "project", "session_snapshot", "auto_session" |
| source_path | TEXT | File path + section or timestamp |
| content_text | TEXT | First 500 chars (searchable preview) |
| embedding | BLOB | 768-dim Float32 vector (3,072 bytes) |
| created_at | TIMESTAMP | Auto-set |

### session_tracking ⭐ NEW
| Column | Type | Description |
|--------|------|-------------|
| session_id | TEXT | OpenClaw session UUID |
| channel_key | TEXT | discord:channel_id |
| transcript_path | TEXT | Path to .jsonl file |
| last_message_index | INTEGER | Last processed line |
| messages_since_snapshot | INTEGER | Counter since last embed |
| last_checkpoint_time | TIMESTAMP | Last check |
| is_active | BOOLEAN | Session still exists? |

## Cron Schedule

| Job | Schedule | What It Does |
|-----|----------|--------------|
| **Memory Embeddings Daily** | 3:00 AM | Process yesterday's memory file |
| **Session Monitor** ⭐ NEW | Every 2 min | Reads transcripts, auto-snapshots at 15 msgs |
| Session Snapshots (legacy) | Manual | Manual capture via script |

**How Session Monitor Works:**
1. Reads `.openclaw/agents/main/sessions/*.jsonl`
2. Tracks `last_message_index` per session
3. Counts new user messages
4. At 15 messages: summarize → embed → store
5. Updates checkpoint in `session_tracking` table

## Usage

### Search
```powershell
# Search by query
python tools/search_memories.py "home assistant automation"

# Interactive mode
python tools/search_memories.py --interactive
```

### Manual Snapshot
```powershell
python tools/session_snapshotter.py "Summary of important discussion"
```

### From Python
```python
from memory_vector import search_memories

# Generate query embedding with Ollama
# Then search
results = search_memories(query_embedding, k=5)
# Returns: [(source_path, content_text, distance), ...]
```

## Stats

| Metric | Value |
|--------|-------|
| Total embeddings | **1,623** |
| Daily notes | 818 |
| Project files | 332 |
| MEMORY.md | 33 |
| Manual session snapshots | 2 |
| **Auto session snapshots** ⭐ | **27** |
| Tracked sessions | 245 |
| Active sessions | 243 |
| Database size | ~5 MB |

**Live Stats Query:**
```powershell
python -c "import sqlite3; db=sqlite3.connect(r'C:\Users\admin\.openclaw\memory.db'); c=db.cursor(); c.execute('SELECT COUNT(*) FROM memory_embeddings'); print('Total:', c.fetchone()[0]); c.execute('SELECT COUNT(*) FROM memory_embeddings WHERE source_type=\'auto_session\''); print('Auto snapshots:', c.fetchone()[0]); c.execute('SELECT COUNT(*) FROM session_tracking WHERE is_active=1'); print('Active sessions:', c.fetchone()[0]); db.close()"
```

**Check Session Monitor Status:**
```powershell
# See last run
python tools/session_monitor.py

# Check if cron is running
openclaw cron list | findstr "Session Monitor"
```

## The Innovation ⭐

**Problem:** How to automatically capture live conversations without manual tracking?

**Solution:** Read OpenClaw's own session transcripts!

OpenClaw stores every session in `.openclaw/agents/main/sessions/[session-id].jsonl`. We discovered we can:

1. **Monitor these files live** — cron job every 2 minutes
2. **Track line position** — `last_message_index` checkpoint
3. **Count user messages** — parse JSONL for `role: "user"`
4. **Auto-snapshot at threshold** — 15 messages → summarize → embed

**Why this matters:**
- ✅ No manual message counting
- ✅ Survives session restarts
- ✅ Multi-channel aware (each channel = separate session file)
- ✅ No OpenClaw hooks required (we read their existing data)

**Credit:** Corey's genius idea 💡 *(Corey, 2026-03-03)*

---

## Comparison: Old vs New

| Feature | Supermemory | SQLite-Vector |
|---------|-------------|---------------|
| Cloud dependency | Required | None |
| API limits | Yes | No |
| Offline use | No | Yes |
| Embeddings stored | Cloud | Local |
| Search speed | Network latency | <100ms local |
| Reliability | Crashing | Stable |
| Cost | API-based | Free |

## Troubleshooting

### Ollama not running
```powershell
# Check status
Invoke-RestMethod -Uri "http://localhost:11434/api/tags"

# Start Ollama
ollama serve
```

### Missing model
```powershell
ollama pull nomic-embed-text
```

### Database locked
Close any GUI tools (DB Browser) before running scripts.

## Future Enhancements

- [ ] Keyword filtering alongside vector search
- [ ] Date range queries
- [ ] Source type filtering (e.g., only projects)
- [ ] Embedding quality scoring
- [ ] Auto-summarization improvements

---

**Status:** Fully operational  
**Last updated:** 2026-03-02