When Does Memory Help? A Cost-Aware: Tested & Reviewed

for Large Language Model (LLM) agents, long-term memory is often touted as a key feature that enables them to learn and adapt over time. However, the actual value of long-term memory in these agents is not well understood. In this article, we'll explore into a recent study that sheds light on the effectiveness of long-term memory in tool-using LLM agents, with a focus on its cost-aware evaluation.

What is Long-Term Memory in LLM Agents?

Long-term memory in LLM agents refers to the ability of the model to retain and recall information over an extended period. This is in contrast to short-term memory, which is typically limited to a single conversation or task. Long-term memory enables LLM agents to learn from their experiences and adapt to new situations, making them more effective in complex tasks.

Evaluating Long-Term Memory: The LoCoMo and LongMemEval Benchmarks

Traditionally, long-term memory in LLM agents is evaluated using conversational recall benchmarks such as LoCoMo and LongMemEval. These benchmarks measure a model's ability to answer questions based on its dialogue history. However, these benchmarks have a significant limitation: they do not account for the actual impact of long-term memory on the model's performance.

Introducing MERIT: A Cost-Aware Evaluation of Long-Term Memory

To address the limitations of traditional benchmarks, researchers have developed a new evaluation framework called MERIT (Memory Evaluation for Realistic Instrumented Tasks). MERIT provides a comprehensive assessment of long-term memory in LLM agents, taking into account the model's performance, cost, and energy consumption.

The MERIT Benchmark and using

The MERIT benchmark consists of episodic tool-use tasks in three domains: robotics, chemistry, and biology. These tasks are designed to simulate real-world scenarios where LLM agents need to use tools to complete tasks. The MERIT using is a software framework that enables researchers to easily implement and evaluate LLM agents on these tasks.

Key Features of the MERIT Benchmark

The MERIT benchmark has several key features that set it apart from traditional benchmarks:

• Episodic tool-use tasks: MERIT tasks are designed to simulate real-world scenarios where LLM agents need to use tools to complete tasks.
• Difficulty ladder: MERIT provides a difficulty ladder that gradually increases the complexity of the tasks, allowing researchers to evaluate the model's performance at different levels of difficulty.
• Controlled memory corruption: MERIT enables researchers to simulate memory corruption, which is a common issue in real-world systems.
• Full token and dollar metering: MERIT provides a detailed accounting of every memory operation, including the number of tokens and dollars used.

Experimental Results

The researchers conducted an extensive experiment using the MERIT benchmark and using. They evaluated the performance of three LLM models: GPT-4.1, Claude Haiku 4.5, and Claude Sonnet 5. The results showed that long-term memory significantly improves the performance of LLM agents, particularly in tasks that require the model to recall previously learned information.

Key Findings

The experiment revealed several key findings:

• Memory lifts dependent-task success: Long-term memory lifts dependent-task success from a leak-verified floor of 0.00 to 0.55-1.00.
• Embedding retrieval collapses unpredictably: Embedding retrieval collapses unpredictably, with a range of 0.30-0.95 across models.
• Agents act on a correctly retrieved value only 55% of the time: Agents act on a correctly retrieved value only 55% of the time, indicating a significant gap between the model's performance and its actual impact.
• Update-on-write stores remain at 0.70-1.00: Update-on-write stores, which are a structured fact store and LLM summarization, remain at 0.70-1.00, indicating a significant advantage over traditional embedding retrieval.
• The hybrid is worse than the fact store alone: The hybrid approach, which combines embedding retrieval and update-on-write stores, is worse than the fact store alone, indicating that the benefits of long-term memory are not always additive.

The MERIT benchmark and using provide a comprehensive evaluation framework for long-term memory in LLM agents. The experimental results demonstrate the significant impact of long-term memory on the performance of LLM agents, particularly in tasks that require the model to recall previously learned information. However, the results also highlight the limitations of traditional embedding retrieval and the importance of update-on-write stores in achieving high performance.