Building stateful LLM applications

The discussion began by exploring the fundamental distinction between short-term and long-term memory in LLM applications. Short-term memory operates within single sessions using context windows and attention layers, while long-term memory persists across multiple interactions through external storage mechanisms like databases. This distinction is crucial for building applications that can maintain continuity and personalization across user interactions, from remembering preferences to tracking ongoing projects.

The conversation then delved into the MemGPT framework, which approaches LLMs as operating systems with sophisticated memory management. The system utilizes different types of memory categories: "out of context" memory (comprising archival and recall memory) and "in context" memory (including system prompts, core memory, and statistics). This architecture is complemented by tool memory functions that enable operations like message sending, conversation searching, and memory manipulation through commands such as archival_memory_insert and core_memory_append.

Finally, the discussion covered LangChain's approach to long-term memory, which organizes memory into three distinct categories: Semantic (for storing facts and learned information), Episodic (for experiences and past actions), and Procedural (for instructions and system prompts). This framework combines human and AI messages to create summaries and process information effectively, providing a comprehensive system for managing different types of information in AI applications.

Slides link: https://www.canva.com/design/DAGZP127uyI/RcxNuGx5waPaoimEm4AwNg/view?utm_content=DAGZP127uyI&utm_campaign=designshare&utm_medium=link2&utm_source=uniquelinks&utlId=h99893ab515