Perry — Autonomous Personal AI Agent

Overview

Perry is a personal AI agent that runs continuously on a Raspberry Pi 4, reachable through Telegram. It’s a study in what an always-on, proactive agent looks like — one with persistent memory, a real tool ecosystem, and the autonomy to start conversations rather than only respond to them.

Architecture

flowchart TB
    TG["Telegram<br/>text · voice · images"]
    subgraph PI["Raspberry Pi 4 — 24/7"]
        AGENT["Agent runtime<br/>Gemini 2.5 Flash"]
        MEM[("Per-user memory<br/>isolated, cross-session")]
        BEAT["Heartbeat + scheduler<br/>proactive triggers"]
    end
    subgraph TOOLS["Tool Ecosystem (10+)"]
        GOOG["Calendar · Gmail · Drive"]
        NOTES["Notes knowledge base"]
        LIFE["Expenses · location · health"]
        WEB["Weather · web search"]
    end
    TG <--> AGENT
    AGENT <--> MEM
    AGENT <--> TOOLS
    BEAT --> AGENT
    BEAT -->|briefings · nudges · journal| TG

• Edge-hosted agent runtime — runs 24/7 on a Pi 4 on an agent framework paired with Gemini 2.5 Flash, with a Telegram bot as the conversational surface. No cloud server to keep alive.
• Multimodal I/O — handles text, transcribes voice notes, analyzes images, and generates images, all through the same chat interface.
• Persistent + isolated memory — session memory carries context across conversations, with fully isolated per-user data so an owner and approved users never share state.
• Tool ecosystem — integrations spanning Google Calendar / Gmail / Drive, a searchable notes knowledge base, weather, web search, expense tracking with auto-categorization, Wi-Fi-based location awareness, and health metrics (steps, sleep, heart rate) pulled from a smartwatch.

What Makes It Interesting

• Proactive behaviors — Perry initiates: a morning briefing (calendar + email triage + sleep), pre-meeting prep briefs, calendar nudges with travel notes when away, an evening recap, an auto-written daily journal, and a git-style weekly “life changelog.”
• Self-monitoring — a periodic heartbeat runs self-checks (long-absence alerts, upcoming-interview reminders, tool health).
• Data fusion — proactive messages blend calendar, email, expenses, location, and health into a single coherent picture rather than siloed notifications.

Highlights

• Always-on agent on edge hardware (Raspberry Pi 4)
• 10+ tool integrations behind one conversational interface
• Proactive, scheduled behaviors — the agent reaches out first
• Per-user data isolation with cross-session memory