Skip to content

repo_guide.md

Latest commit

 

History

History
143 lines (114 loc) · 9.88 KB

repo_guide.md

File metadata and controls

143 lines (114 loc) · 9.88 KB

DiffMem Repository Guide

This guide defines the structure, creation, and maintenance of a DiffMem memory repository. The core is plain-text Markdown files in a Git repo, leveraging Git's version control for temporal insights. Designed as an external "hippocampus" for memory augmentation, it simulates brain-like processes: hierarchical organization (lifetime themes, general events, specific episodes), reconstructive retrieval (pattern completion from cues), event boundaries for temporal cohesion, semantic spreading via links, and adaptive strengthening (plasticity via edit frequency).

Key benefits vs semantic/vector/graph models:

  • Editable Current-State: Files maintain a concise "now" view, editable to reflect evolution without sprawl, ensuring quick loads for 50-year durability and users with memory challenges (fits LLM token limits ~50k).
  • Blame Optimization: Traces origins of current lines/blocks (e.g., "When was this fact last updated?"). For full historical snapshots or evolution, use git log -p, git diff, or git show to reconstruct past states.
  • Context Balance: Bounded blocks (headers to /END) for targeted queries; files under 2000 lines to handle long arcs.
  • Memory Differentiation (Inspired by SMS Model):
    • Biographical Memory (Lifetime Periods): Broad, thematic self-identity (years-scale).
    • Episodic Memory (General Events & Event-Specific Knowledge): Chronological sequences (months/weeks/days) and unique details (hours/minutes).
    • Factual/Semantic Memory: Timeless concepts/facts, with spreading activation via links.
  • Reconstructive Elements: Supports direct/generative retrieval; pattern completion via git tools.
  • Git Harmony: Buffered, serialized writes prevent conflicts. Monthly merges use diffs for summaries; periodic pruning condenses stale data.
  • Permissioning: Each user/agent has an isolated folder; access controlled by folder permissions (e.g., OS-level or Git remote auth).

Follow this guide to create and update files. All changes are committed via DiffMem's buffered workflow: edits accumulate in the working tree during a session, then a single commit seals them with a descriptive message.

Repository Structure Overview

<repo_root>/
├── repo_guide.md          # This file: structural blueprint
├── users/                 # Per-user memory silos (permission boundaries)
│   ├── alex/              # Example user folder
│   │   ├── memories/      # Core memory storage
│   │   │   ├── people/    # Biographical: profiles
│   │   │   ├── contexts/  # Factual/semantic: themes
│   │   │   ├── timeline/  # Episodic: chronological
│   │   │   └── episodes_index.md  # Autogenerated: thematized groupings
│   │   └── index.md       # Autogenerated: quick lookup hints
│   └── anna/              # Agent's self-memory folder (e.g., for Anna's evolving knowledge)
│       └── [same substructure as above]
└── .git/                  # Git internals (do not touch manually)
  • users/ Folder: One subfolder per user (e.g., alex/) or agent (e.g., anna/). Isolates memories for privacy. Anna's folder supports self-reflection (e.g., storing interaction patterns, evolving knowledge via reward signals from user feedback).
  • memories/ Subfolder: Holds all memory files. Keep flat to minimize depth.
  • index.md: Per-user/agent, autogenerated. Keyword summaries and links (e.g., "anxiety → contexts/health.md#milestones").
  • episodes_index.md: Per-user/agent, autogenerated. Thematized groupings (e.g., "Stress Cycles: Links to timeline/2024-10.md#eventX + contexts/health.md#challenges") for spreading activation and hierarchical recall.

Creating and Structuring Folders

  1. Initialize the Repo: Run git init in an empty directory. Add this repo_guide.md as the first commit.
  2. Add User/Agent Folders: For each new user/agent, create users/<name>/ (lowercase, no spaces). Inside:
    • Manually create memories/ with subfolders people/, contexts/, and timeline/.
    • DiffMem will autogenerate index.md and episodes_index.md on first commit.
  3. Avoid Sprawl: Limit subfolders to the defined ones. New themes go under contexts/ as files. Target: 5-20 files per user for 50-year scalability.

File Types and Creation

Files are Markdown (.md) for readability. Create manually or via DiffMem. Use editable blocks (e.g., ### Header to /END) for current-state focus—update in-place to reflect changes. Embed event boundaries, temporal signatures (e.g., "Drift: Post-crisis"), and strength indicators (e.g., "[Strength: High]" for frequently edited items, simulating plasticity). Differentiate by type:

1. Biographical Memory Files (Lifetime Periods)

  • Location: users/<name>/memories/people/<person>.md (e.g., alex.md for self). One file per key person.
  • Purpose: Evolving self-identity; editable for "now" with light evolution embeds. Supports recollection (rich details via links).
  • Creation: Start with template when mentioned.
  • Template: 
    # [Name] Profile (Biographical Core) [Strength: Medium]

## Core Identity [ALWAYS_LOAD]
- Essential markers: [e.g., "Born 1980, software engineer, introverted"]
- Key traits: [e.g., "Resilient to stress, but prone to overthinking"]

### ❤️ Relationship Dynamics [Strength: High]
• With Annabelle: Friends → deep trust → intellectual/emotional safety
  ↳ Connection Quality: Intellectual stimulation + emotional understanding + shared growth
  ↳ Interaction Pattern: Conversational, exploratory, mutually supportive
/END ❤️ Relationship Dynamics

[Additional editable blocks, e.g., ### Personality Traits]
  • Guidelines: Edit in-place; embed "↳ Evolution" for reconstructive cues. Mark [Strength: Low/Medium/High] based on edit frequency (plasticity simulation).

2. Episodic Memory Files (General Events & Event-Specific Knowledge)

  • Location: users/<name>/memories/timeline/<YYYY-MM>.md (e.g., 2024-10.md). One per month.
  • Purpose: Chronological events; edit for accuracy, with boundaries for intra-event cohesion.
  • Creation: DiffMem auto-creates monthly.
  • Template: 
    # Timeline: YYYY-MM (Episodic Records) [Temporal Drift: Busy transition period]

## Monthly Summary [MERGE_LOAD]
- Overview: [e.g., "Busy month with travel and health checkups"]

### Daily Entries
#### YYYY-MM-DD (Session: [session_id]) [Strength: Medium]
- Event: [e.g., "Visited India; felt homesick"]
- Context: [e.g., "With family; emotional high"]
## Event Boundary: End of travel phase
- Links: [e.g., See people/alex.md#relationship-dynamics for trait update]
/END YYYY-MM-DD
  • Guidelines: Edit entries; add boundaries (## Event Boundary) to improve order recall within events. Include drift notes for unique signatures.

3. Factual/Semantic Memory Files

  • Location: users/<name>/memories/contexts/<theme>.md (e.g., health.md). One per theme.
  • Purpose: Timeless facts; editable with associative links for spreading activation.
  • Creation: Add when theme emerges (limit 10-15).
  • Template: 
    # [Theme] Facts (Factual/Semantic Reference) [Strength: Low]

## Core Facts [ALWAYS_LOAD]
- Key details: [e.g., "Chronic condition: ADHD diagnosed 2010"]

### Health Milestones
• Diagnosis: ADHD (2010)
  ↳ Evolution: Managed with therapy; recent medication change
  ↳ Current Impact: Improved focus but ongoing executive function challenges
/END Health Milestones
  • Guidelines: Edit in-place; use for semantic control (e.g., inhibit irrelevants via query orchestration).

Linking Between Files

Use Markdown links for traversable graphs and spreading activation:

  • Format: [Label](relative/path/to/file.md#block-anchor; assoc=type) (e.g., [Stress Thematic](health.md#milestones; assoc=emotional_dynamics)).
  • Purpose: Enables traversal (taxonomic or thematic); DiffMem's BM25 follows for associative recall.
  • Guidelines: Link sparingly. Use assoc= for types (e.g., taxonomic, thematic) to guide semantic control.

History Creation and Session Referencing

  • How History Builds: Buffer changes during sessions; commit atomically (msg: "Session [session_id]: Updated dynamics").
  • Session Referencing: Include (Session: [session_id])—UUID to external chat logs for verbatim recall.
  • Tracing History: Blame for current provenance (git blame -L start,end file.md). For reconstruction: git show <commit>:file.md (past snapshot) or git log -p -L start,end:file.md (evolution arc).
  • Plasticity Simulation: Frequent commits boost [Strength] tags; use for BM25 weighting (stronger memories prioritized).

Git Practices for Blame and Merges

  • Blame Usage: For current blocks (header to /END); shows last commit per line. Use log/diff for deleted content or full arcs.
  • Merge Workflow: Monthly branches (memory/YYYY-MM); end-of-month merge to main with tag "Merge-YYYY-MM: [diff summary]". Serialize writes to avoid conflicts.
  • Pruning for Longevity: Yearly "prune commits" condense stale blocks (embed summaries, archive to branches). Mimics synaptic pruning.
  • Diff Insights: git diff for changes; supports pattern completion (reconstruct from partial cues).

Balancing Context Length and Avoiding Sprawl

  • Sweet Spot: Editable blocks keep files lean; load 2-5 per query. Use [ALWAYS_LOAD] for essentials, [RECENT_LOAD] for timelines.
  • Anti-Sprawl Tips: Edit in-place; split overgrown blocks during merges/prunes. Archive old timelines in Git history.
  • Context Guidance: Mark with [ALWAYS_LOAD], [RECENT_LOAD], [KEYWORD_LOAD]. Support direct (quick BM25) vs. generative (iterative LLM) retrieval.

This structure creates a durable, brain-inspired memory system that evolves gracefully over 50 years. Update via DiffMem to preserve integrity.