# 2026-03-09 - Daily Notes

## Projects Worked On

### ROM Library Organization
- **Goal**: Organize/categorize ~100K ROM files across multiple gaming systems
- **Library location**: R:\ drive (38.8 TB)
- **Quick scan completed**: 98,601 items, 1,701 GB total

**Breakdown by Manufacturer**:
- Computers (C64, Amiga, etc.): 47,327 items, 61.89 GB
- Arcade (MAME, Neo Geo): 12,951 items, 32.97 GB
- Atari: 12,399 items, 2.56 GB
- Nintendo: 12,017 items, 467.24 GB
- Sony (PSX, PS3, PSN): 3,106 items, 672.40 GB
- Sega: 2,747 items, 3.54 GB
- Microsoft: 1,661 items, 0.05 GB

**Top by count**: Commodore 64 (24,349), Atari (10,935), ArcadePC (8,750), MAME (8,651)
**Top by size**: PSN ISO Pack (672 GB), Nintendo 3DS (412 GB), TurboGrafx-CD (234 GB)

**Scripts created**:
- tools/rom-quick-scan.py - Quick inventory (completed)
- tools/rom-full-scan.py - Full scan with duplicate detection (running overnight)

**Output files**:
- rom-inventory/rom-inventory.json - Quick scan results
- rom-inventory/rom-full-*.json - Full scan results (pending)

**Phase 2**: Duplicate detection running overnight - MD5 hashing files under 50MB

## Issues Encountered
- Node gateway timeouts when running long scans (30s limit)
- Sub-agent reliability issues - spawning background processes more reliable
- Script escaping issues with f-strings in Python scripts

## Decisions Made
- Use folder counts for disc-based systems (PSX, PS2, etc.) instead of file counts
- Hash only files under 50MB for duplicate detection (speed vs coverage)
- Run full scan as background process, not sub-agent

## Overnight Scan Results (2026-03-10 00:02)

### Full Scan Completed
- **Total:** 107,069 items (6,103 GB / 6.1 TB)
- **Duplicates Found:** 4,790 duplicate sets (11,160 files)

**Worst Duplicate Offenders:**
- ArcadePC MegaPack: 100 copies of same binary
- M.A.M.E: 66 copies of big7.bin
- PopCap games: 48-36 copies of various .bin files

**Key Insight:** Many duplicates are internal game data files (not actual ROM duplicates) - .bin files used across multiple games in:
- ArcadePC MegaPack (100 copies of dialog data)
- Abandonware DOS (map tiles reused across levels)
- PopCap games (UI assets)

**Action Items:**
- Review duplicate report for actual ROM duplicates vs internal game data
- Consider excluding .bin files under 1MB from duplicate scanning (noise)
- Cross-reference with No-Intro DATs for missing ROMs

**Output File:** rom-inventory/rom-full-20260310-000245.json (1.6 MB)