TamaBench — Small Models, Long Horizons

The project

TamaBench is a lightweight benchmark for small and local language-model agents. It places an agent in a persistent virtual-pet sandbox where actions have delayed consequences.

The pet is the simulation mechanism. The actual subject is long-horizon autonomy: observing state, planning, calling tools, managing resources, recovering from failure, and continuing after the model's last decision.

What the agent has to survive

Across three simulated days, the agent must:

• feed, clean, heal, and play with the pet;
• earn money through jobs before buying supplies;
• balance its own energy against the pet's needs;
• respond to random sickness events;
• produce valid structured actions;
• plan around time that keeps moving while it sleeps, works, or waits.

This creates pressure that ordinary question-answering benchmarks do not create. A response can be fluent and still fail because it chose the wrong priority, used invalid JSON, waited too long, or spent resources it could not replace.

The harness question

TamaBench compares more than one model against another. Its central comparison is:

snippet.txttext
EditCopy
1Raw model
2  vs
3Raw model + Wake Scheduler
4  vs
5Raw model + Harness V1

Harness V1 follows a DECIDE → CALCULATE → SCHEDULE loop. The model wakes when a care decision is needed, while routine economy actions can be handled deterministically. The intended test is whether better orchestration can reduce API calls, tokens, and compute while preserving or improving survival.

What it measures

The repository defines metrics for:

• survival across the full episode;
• average health and happiness;
• economic efficiency;
• first-pass and recovered schema compliance;
• planning, resource, truncation, and retry failures;
• latency, API calls per day, and reasoning or JSON tokens.

The runtime also records SQLite results, JSONL events, reasoning traces, and replay files. That makes the benchmark useful for investigating why an agent failed, not only whether it survived.

Why this belongs in the Lab

TamaBench is a good example of an evaluation that starts from a workflow rather than a model score. The controlled sandbox gives a small local model a concrete job, fixed rules, seeded events, and consequences that accumulate over time.

It still does not prove that an agent is safe for email, home automation, or production infrastructure. Those environments need their own tools, permissions, policies, and tests. TamaBench isolates reusable skills underneath them: observation, planning, structured tool use, recovery, and long-horizon control.

Run one episode

From the repository root, install the package, start a local model through Ollama, then run:

snippet.shbash
EditCopy
1python -m tamabench.cli run --agent harness_v1 --model <your-model> --episodes 1

The repository also includes a rule-based baseline that does not need a model. Keeping that baseline in the comparison makes it easier to separate model behavior from environment and harness behavior.

Current boundaries

TamaBench is an early research preview. Its README marks current result rows as TBD, warns that version 0.1.0 results are not cross-version comparable, and lists a multi-agent runner and result dashboard as future work.

This note credits Pichetpong Muangsiri, keeps the original repository as the source of truth, and does not turn the benchmark into a winner list.