Local LLM Inference Under Limited VRAM

An Exploratory Study of Mixture-of-Experts, Dense Models, and RAM Offloading on an RTX 3060

A Study of Local LLM Efficiency for MoE and Dense Models Under VRAM Constraints Using GPU–RAM Hybrid Offloading

Status: Preliminary Experimental Report Test Platform: NVIDIA RTX 3060 12 GB + DDR4 16 GB Dual-Channel Inference Backend: llama.cpp Primary Focus: Token generation throughput, context scaling, RAM offloading, agent behavior, and structured-output reliability

---

Abstract

Running large language models on your own machine is usually constrained by the amount of GPU VRAM, especially for models in the 20–30 billion parameter range. The common way to work around this limitation is to move part of the weights into system RAM for storage and computation — the so-called CPU/RAM offloading. However, Dense models often lose a great deal of performance when offloaded, because most of the model weights must be exercised for every generated token.

This experiment studies how much a Mixture-of-Experts (MoE) model can reduce the downside of RAM offloading, running on a machine with an NVIDIA RTX 3060 12 GB and dual-channel DDR4 16 GB — a system with clearly constrained memory.

The main models tested were Gemma 4 26B-A4B, Mistral Small 3.1 24B, and Qwen-AgentWorld 35B-A3B. We measured Time to First Token (TTFT), generation throughput in tokens/s, the ability to scale the context window, and behavior when the models are used for AI-agent style work.

The preliminary results show that Gemma 4 26B-A4B can hold roughly 12–19 tokens/s at a 16K context configuration, even after reducing the number of GPU-resident layers to 13. In contrast, the dense Mistral Small 3.1 24B at 4K context with 15 GPU layers managed only 3.35 tokens/s. The Qwen-AgentWorld 35B-A3B — with more total parameters but only 3B active — reached about 6.5–12.9 tokens/s generation, but with high TTFT and enough memory pressure that the system became unstable across several attempts.

These results support the initial hypothesis that total parameter count alone is insufficient to predict Local LLM inference speed. Active parameters, model architecture, expert routing, memory traffic, context implementation, and backend optimization all play a major role.

That said, this experiment is an exploratory experiment, not a controlled benchmark, because the models differ in architecture, generation, context configuration, and training purpose. The results should therefore not be interpreted as causal evidence that MoE is faster than Dense in all cases.

---

1. Introduction

One of the most important constraints on Local AI is VRAM.

A consumer GPU such as the RTX 3060 has 12 GB of VRAM, which is enough for small and medium models. Once a model exceeds that footprint, the inference runtime must choose between:

1. Lowering precision / quantization
2. Reducing the context window
3. Reducing the number of model layers kept on the GPU
4. Offloading model weights to system RAM
5. Using a GPU with more VRAM

Option 4 has the lowest financial cost, because system RAM is far cheaper per GB than VRAM — but its memory bandwidth is many times lower than GPU memory.

For a Dense Transformer this problem is especially severe, because almost all the weights are used for each generated token.

Mixture-of-Experts changes that equation.

An MoE model can have a large parameter capacity while activating only a subset of experts per token. For example, Gemma 4 26B-A4B has about 25.2B total parameters but only ~3.8B active per token, while Qwen-AgentWorld-35B-A3B has 35B total and ~3B active.

So the key question for Local AI is:

If a large expert pool can be kept in RAM and each token uses only part of it, can RAM act as an effective capacity tier for a Local LLM?

This experiment was started to explore exactly that question.

---

2. Research Questions

This experiment attempts to answer five main questions.

RQ1 — Can total parameters predict tokens/s?

Compare models with similar total parameters but different architectures, such as a Dense 24B vs an MoE 26B.

RQ2 — Do active parameters correlate with generation throughput better than total parameters?

Compare 35B-A3B vs 26B-A4B and Dense 24B.

RQ3 — Can MoE reduce the penalty of RAM offloading?

Observe how much speed drops when model layers must move from GPU to CPU/RAM.

RQ4 — Through what mechanism does the context window affect speed?

Consider the relationship between:

• KV cache
• Number of GPU-resident layers
• RAM usage
• TTFT
• Generation throughput

RQ5 — Is the fastest model the best fit for AI agents?

Besides throughput, also consider:

• Instruction following
• Planning
• Tool understanding
• JSON Schema generation
• Structured-output reliability

---

3. Background

3.1 Dense Transformer

In a Dense Transformer, the parameters in every layer are used continuously for every token.

Approximately:

snippet.txttext
EditCopy
1T_dense ∝ W_active ≈ W_total

When part of the weights live in system RAM, the CPU and its memory bandwidth become the bottleneck.

In a system where RAM bandwidth is far below GPU memory bandwidth, offloading can cut generation throughput severely.

---

3.2 Mixture-of-Experts

MoE adds several sets of expert networks to the model, but uses a router to select only some experts per token.

This creates a difference between P_total and P_active.

For example, Gemma 4 26B-A4B has about 25.2B parameters but ~3.8B active, using a routing scheme of 8 active experts out of 128 routed experts plus a shared expert, per the model card.

Qwen-AgentWorld-35B-A3B has 35B total / 3B active, 256 experts, selecting 8 routed experts plus a shared expert per MoE layer.

The system-level idea can be viewed as:

snippet.txttext
EditCopy
1VRAM
2├── Attention
3├── Router
4├── Shared components
5├── KV cache
6└── GPU-resident experts
7 
8System RAM
9└── Additional expert weights

llama.cpp currently offers --cpu-moe and --n-cpu-moe to keep all MoE weights — or only the expert weights of the first N layers — on the CPU side, allowing expert-specific offloading instead of whole-layer offloading.

---

4. Experimental Platform

Hardware

This system was not deliberately chosen as a high-end inference server. Instead, it is an environment well suited to studying the limits of RAM offloading, because both VRAM and RAM are scarce.

---

5. Models

5.1 Gemma 4 26B-A4B

Architecture:

• Mixture-of-Experts
• 25.2B total parameters
• 3.8B active parameters
• 30 layers
• 128 routed experts
• 8 active routed experts + shared expert
• Maximum context per the model card: 256K tokens

Quantization used: Q4

This model was the primary candidate for studying MoE offloading behavior.

---

5.2 Mistral Small 3.1 24B

Mistral Small 3.1 is a 24B model supporting up to 128K context per the model card, with a configuration of 40 transformer layers.

In this experiment it served as the Dense comparison.

Quantization: Q4

---

5.3 Qwen-AgentWorld-35B-A3B

Qwen-AgentWorld is an MoE language world model:

• 35B total parameters
• 3B active parameters
• 40 layers
• 256 experts
• 8 routed + 1 shared expert
• Maximum context of 262,144 tokens

The model was not designed as a general chatbot. It was trained to simulate the environment and next state of an AI agent, covering MCP, Search, Terminal, SWE, Android, Web, and OS.

Conversational-quality results should therefore not be compared directly against Gemma without context.

---

6. Experimental Tasks

The initial prompt suite covers several levels of work.

Task 1 — Simple Conversation

Can you help me?

Used to measure initial TTFT and conversational response.

Task 2 — Basic Coding

Can you write me simple Calculator for Python?

Used to look at:

• Coding generation
• Output length
• Error handling
• Throughput when producing longer answers

Task 3 — Agent Tool Understanding

If you are AI agent What possible tool you gonna use to read a file

Measures tool-use understanding.

Task 4 — Structured Tool Design

Design a schema for read_file

Measures:

• JSON Schema understanding
• Structural correctness
• Tool-interface reasoning

Task 5 — Agent Orchestration

Defines workers:

• Senior Engineer
• Junior Engineer
• Reporter

Then asks the model to design a workflow for building a website end to end.

This task measures high-level orchestration more than general knowledge.

---

7. Results

7.1 Gemma 4 — Initial Run

Configuration:

• Q4
• GPU layers: 18
• The context configuration of the first round was not clearly recorded in the current log.

Mean generation throughput: ≈ 21.48 tok/s

This result is the baseline that raised the question of why a 26B total model that is not fully VRAM-resident can still generate tokens as fast as much smaller models.

---

7.2 Gemma 4 — 8K Context

Context was raised to 8K.

To keep the KV cache in VRAM, we had to reduce:

snippet.txttext
EditCopy
1GPU layers: 18 → 16

Results:

Mean: 17.89 tok/s

Compared with the three prompts shared with the baseline, throughput dropped about 13%.

Even after increasing the context reservation and moving two GPU layers off-card, the system still answered comfortably in interactive time.

---

7.3 Gemma 4 — 16K Context

Raising to 16K, gpu_layer=14 could not load the required configuration, so it was reduced to:

snippet.txttext
EditCopy
1gpu_layer = 13

Results:

Mean: 15.07 tok/s

The conversation in this round accumulated continuously, so the later tasks carried more real conversation history than at the start. The throughput drop therefore cannot be attributed to context reservation alone.

The important point is that the 16K configuration kept generation throughput in the 12–19 tok/s range throughout the suite, and completed multi-agent planning with no failure.

---

7.4 Mistral Small 3.1 24B Dense

Configuration:

snippet.txttext
EditCopy
1Context = 4K
2gpu_layer = 15
3Quantization = Q4

Result from the first prompt:

This is clearly slower than Gemma, even though:

• Context is smaller
• Total parameters are similar
• The number of GPU layers is similar

However, this is not yet a controlled Dense-vs-MoE comparison, because Mistral and Gemma are different architecture families from different generations.

The result should be read as:

Under the tested configuration, the Dense 24B model had far lower RAM-offloaded inference throughput than Gemma 4 26B-A4B.

It should NOT be read as:

Every MoE model is 4–6× faster than every Dense model.

---

7.5 Qwen-AgentWorld 35B-A3B

Configuration:

snippet.txttext
EditCopy
1gpu_layer = 16
2KV cache = Q8
3Context = 2K

Context was limited to 2K for safety, because the system crashed three times while attempting to load or run the model before a working configuration was found.

The crash cause has not been instrumented well enough to conclude whether it was CPU OOM, GPU OOM, a driver failure, or something else.

Successful results:

Mean: 10.11 tok/s

Excluding the first, very short answer: ≈ 11.31 tok/s

The interesting finding is that a 35B MoE can still reach double-digit generation throughput on a 12 GB VRAM + 16 GB RAM system.

But TTFT is very high.

Qwen-AgentWorld was designed to use long chain-of-thought reasoning to simulate environment state, so there are architectural and training-objective reasons that could make its latency different from a typical chatbot.

Still, we cannot conclude reasoning is the whole cause, because conversation history grew every turn and the system memory pressure was also very high.

---

8. Cross-Model Comparison

This table is not an apples-to-apples benchmark, but it shows how real models behave on the same hardware configuration.

---

9. Key Observations

9.1 Parameter Count Alone Is Not a Good Predictor of Throughput

If total parameter count were the only factor:

snippet.txttext
EditCopy
124B Dense

should be faster than:

snippet.txttext
EditCopy
135B-A3B MoE

But the experimental results show the opposite.

Mistral 24B: 3.35 tok/s

Qwen 35B-A3B: 6.5-12.9 tok/s

Gemma 26B-A4B: 12-25 tok/s

So throughput correlates with something more complex than total parameter count.

---

9.2 Active Parameters Matter, but Do Not Explain Everything

If active parameters determined throughput directly, Qwen A3B should be faster than Gemma A4B.

But the real result is that Gemma is much faster.

We can write the relationship conceptually as:

snippet.txttext
EditCopy
1Throughput = f( P_active, P_total, B_memory, A_architecture, O_offload, K_backend, C_context )

where:

• P_active = active parameters
• P_total = total parameter capacity
• B_memory = effective memory bandwidth
• A_architecture = architecture
• O_offload = placement of weights
• K_backend = kernel/backend optimization
• C_context = context workload

---

9.3 MoE Appears More Tolerant of RAM Offloading

The Dense 24B dropped to 3.35 tok/s when part of the model had to work through RAM.

Gemma MoE stayed in the 12–19 tok/s range even at 16K with only 13 GPU layers.

This supports the hypothesis that expert sparsity can reduce the penalty of RAM offloading.

A plausible mechanism is that the weights in the expert pool do not all need to be exercised for every token.

However, the experiment cannot yet resolve exactly which expert tensors sit on CPU vs GPU at every point, so this mechanism should still be called a hypothesis supported by data, not proof.

---

9.4 Context Cost Is Not Only Attention Cost

The Gemma results show that as context grows:

snippet.txttext
EditCopy
1GPU layers: 18 → 16 → 13

because VRAM must be left for the KV cache.

So context adds cost in two ways:

Direct cost

KV cache size and attention workload increase.

Indirect cost

The VRAM used by the KV cache squeezes model weights off the GPU, pushing layers or experts into RAM.

On a Local LLM with limited VRAM, this indirect cost can matter more than the attention cost itself.

---

9.5 TTFT and Generation Throughput Must Be Measured Separately

Qwen shows this clearly:

snippet.txttext
EditCopy
1TTFT = 6–26 seconds
2Generation = ~6–13 tok/s

So the model can generate fairly quickly once it starts answering, but the user waits a long time before the first token.

For agent systems, the two metrics mean different things.

Interactive Assistant

TTFT matters a lot.

Background Agent / Verifier

Higher TTFT is acceptable if output quality is higher.

So the word "fast" should not be reduced to a single token/s value.

---

10. Qualitative Agent Observations

Gemma 4

Across the tested tasks, Gemma stood out on:

• Natural conversation
• Agent orchestration
• Role delegation
• Coding
• Workflow planning
• Maintaining conversational context

The multi-agent task showed the model can lay out a workflow:

snippet.txttext
EditCopy
1Requirements → Architecture → Implementation → Verification → Correction → Final delivery

in a sensible order.

However, when generating JSON Schema it produced at least one syntax error, such as a key without correct quotation marks.

So the semantic design can be good, but structured-output reliability still needs repeated testing.

---

Qwen-AgentWorld

In the experiment Qwen gave weaker general conversational answers than Gemma, but the schema it produced had more accurate JSON structure in the observed examples.

This is consistent with the model's purpose: trained to simulate agent environments and structured state transitions more than to converse.

However, the current sample size is too small to conclude that Qwen has higher schema reliability than Gemma.

Repeated trials are required first.

---

11. Current Interpretation

From the current data we can propose the working hypothesis:

For a Local LLM with limited VRAM, a low-active-parameter MoE can use system RAM as a model-capacity tier more effectively than a Dense architecture in some configurations.

This idea can be viewed as a memory hierarchy:

snippet.txttext
EditCopy
1Fast Tier
2GPU VRAM
3├── Attention
4├── Router
5├── Shared components
6├── KV cache
7└── Frequently used / GPU-resident experts
8 
9Capacity Tier
10System RAM
11└── Remaining model/expert weights
12 
13Storage Tier
14NVMe
15└── Model repository / loading

This approach differs from the traditional view that RAM offloading is a fallback that makes inference "too slow to use."

For sparse models, RAM may play a more architectural role than that.

---

12. Important Limitations

The current results have several limitations.

12.1 No repeated benchmarking

Many values come from a single run per prompt.

We therefore do not yet know:

• Variance
• Standard deviation
• Thermal variation
• Runtime scheduling noise

---

12.2 Context differs between models

Gemma was tested up to 16K.

Mistral was tested at 4K.

Qwen was tested at 2K.

So these numbers should not be used as a normalized model benchmark.

---

12.3 Architectures span different generations

Gemma is newer than Mistral Small 3.1.

So the differences may come from:

• Attention architecture
• Kernel optimization
• Quantization compatibility
• Vocabulary
• Layer design
• MoE
• Backend implementation

not from MoE alone.

---

12.4 GPU layer count does not mean equal offload volume

One layer of each model is not the same size.

So:

snippet.txttext
EditCopy
115 GPU layers

of Mistral cannot be compared directly with:

snippet.txttext
EditCopy
115 GPU layers

of Gemma.

---

12.5 KV cache configuration was not recorded consistently across all experiments

Qwen is clearly documented as using Q8 KV.

But the early Gemma/Mistral runs did not have complete KV-precision telemetry.

---

12.6 No power telemetry

We do not yet know:

• GPU watts
• CPU package power
• Wall power
• Tokens/Joule

---

12.7 The Qwen crash has not been root-caused

Qwen crashed the system three times before a working configuration.

There is no kernel log or telemetry to identify the cause.

So we should not conclude it was RAM OOM without evidence.

---

13. Future Work

This section is the most important if we want to move from an exploratory experiment to a credible benchmark.

---

Priority 1 — Gemma 4 12B Dense vs Gemma 4 26B-A4B MoE

This should be the most important next experiment.

Gemma 4 12B is a dense 11.95B model in the same Gemma 4 generation as 26B-A4B, which removes much of the architecture-generation confound.

Compare:

snippet.txttext
EditCopy
1Gemma 4 12B Dense vs Gemma 4 26B-A4B MoE

with:

• Same context
• Same Q4
• Same KV precision
• Same llama.cpp build
• Same sampling
• Same prompts

This would be much stronger evidence on Dense vs MoE than using Mistral.

---

Priority 2 — 7B / 14B Scaling Study

Add:

snippet.txttext
EditCopy
1Dense ~7B
2Dense ~14B
3MoE ~7–10B
4MoE ~14B

to build a scaling curve.

Measure Generation tok/s vs Total Parameters and Generation tok/s vs Active Parameters.

The goal is to see which variable explains performance better.

---

Priority 3 — Community MoE

Candidate:

Llama 3.2 4×3B MoE

This community model merges several Llama 3.2 3B models into roughly a 10B MoE, with Q4_K_M around 6.08 GB per the repository.

Interesting because it allows comparing:

snippet.txttext
EditCopy
1Native MoE vs Post-hoc / merged MoE

The question is whether expert specialization from native MoE training is required for efficiency.

---

Priority 4 — Bonsai 27B Compression

Bonsai 27B offers a different approach from MoE.

Instead of reducing active parameters, it reduces the number of bits used to represent dense weights.

It has 1-bit and ternary variants; some community conversions report sizes around:

snippet.txttext
EditCopy
1Binary ≈ 3.8 GB
2Ternary ≈ 7.2 GB

for the 27B model.

This helps answer the key question:

Between sparse compute and extreme compression, which suits consumer GPUs better?

Both should be measured on:

• token/s
• Schema accuracy
• Agent planning
• Coding
• Perplexity/quality proxy
• Failure rate

---

Priority 5 — Exact Schema Reliability: 10 Runs per Model

The schema task needs an exact reference.

Run each model 10 times:

snippet.txttext
EditCopy
1Gemma 12B       10 runs
2Gemma 26B-A4B   10 runs
3Llama MoE       10 runs
4Bonsai          10 runs

Check with a real parser.

Result: Schema Exact Match = X/10

Fail when:

• JSON does not parse
• Extra field added
• Field missing
• Wrong enum
• Wrong required
• Description mismatch
• Markdown wrapped around the JSON

This turns a qualitative impression into a quantitative reliability metric.

---

Priority 6 — Agent Planning Reliability

Create a fixed scenario:

snippet.txttext
EditCopy
1Agent
2├── Senior Engineer
3├── Junior Engineer
4└── Reporter

with constraints such as:

1. Senior must design the architecture first
2. Reporter must not write production code
3. Tests must come before deploy
4. Bugs must be sent back for fixing
5. Must retest after fixes

Run 10 rounds.

Report: Agent Planning Pass Rate = valid runs / 10

Instead of judging by the feeling that the output "looks smart."

---

Priority 7 — Long Context Scaling

Gemma should be tested further at:

snippet.txttext
EditCopy
116K 24K 32K 48K 64K 80K 100K

stepping up gradually.

llama.cpp supports offloading only the MoE expert weights via --n-cpu-moe, which should be compared with lowering --gpu-layers the classic way.

Test two strategies:

Whole-layer offload

snippet.txttext
EditCopy
1-ngl N

Expert-specific offload

snippet.txttext
EditCopy
1-ngl all
2-ncmoe N

The hypothesis is that the second approach keeps attention/shared computation on the GPU and may preserve token/s better.

---

Priority 8 — Reserved Context vs Filled Context

-c 100000 does not mean the model was tested with 100K of data.

Separate:

Reserved Test

Set context to 100K but keep the prompt short.

Measure memory allocation.

Filled Test

Fill with real prompt data: 25% 50% 75% 90% of context.

Measure:

• Prefill tok/s
• TTFT
• Generation tok/s
• Recall

---

Priority 9 — Long Context Recall / Needle Test

Place specific information near the start, e.g.:

snippet.txttext
EditCopy
1AETHER_TEST_KEY = 731904

Then fill in tens of thousands more tokens of content.

Ask near the end of the context:

snippet.txttext
EditCopy
1What is AETHER_TEST_KEY?

Do this at multiple positions: 10% 25% 50% 75% 90%

To separate:

context loads

from:

context actually works

---

Priority 10 — Hardware Upgrade Study

Repeat the same experiment on:

snippet.txttext
EditCopy
1RTX 3060 12GB + RAM 16GB Dual Channel
2vs
3Tesla P100 16GB + RAM 32GB

This isolates the effects of:

• +4 GB VRAM
• 2× RAM capacity
• Memory configuration
• GPU architecture
• HBM2 bandwidth

Then, if an RTX 3090 is available:

snippet.txttext
EditCopy
1RTX 3090 24 GB

the same models can be tested near/full GPU residency to establish an upper-bound performance.

---

Priority 11 — Memory Telemetry

Every run should record:

snippet.txttext
EditCopy
1VRAM used
2RAM used
3Swap used
4GPU utilization
5CPU utilization
6GPU power
7GPU temperature
8GPU clocks

plus:

snippet.txttext
EditCopy
1Model load time
2Prompt tokens
3Prompt processing tok/s
4TTFT
5Generated tokens
6Generation tok/s

---

Priority 12 — Energy Efficiency

Once power data is available:

snippet.txttext
EditCopy
1Tokens/Joule = Generated tokens / Energy consumed

or an easier-to-read metric:

snippet.txttext
EditCopy
1Wh per 1000 generated tokens

MoE may offer not only a throughput advantage but also an energy-efficiency advantage, which matters a lot for home servers.

---

Priority 13 — GLM-OCR as a Separate Specialized-Model Study

GLM-OCR should not be included in a conversational LLM benchmark.

It is a multimodal OCR model of only 0.9B parameters — about 0.4B visual encoder and 0.5B language decoder — designed specifically for document understanding.

Create a separate experiment:

snippet.txttext
EditCopy
1Generic VLM vs GLM-OCR 0.9B

Measuring:

• Page/s
• Character error
• Table extraction
• Formula extraction
• Markdown quality
• VRAM
• Power

This explores another idea:

Can specialization reduce compute requirements more than scaling model size?

---

14. Recommended Standard Benchmark Protocol

To make the next round comparable, lock in:

snippet.txttext
EditCopy
1Backend version:       Same commit
2Quantization:          Same class when possible
3Context:               Same
4KV precision:          Same
5Temperature:           0 for performance
6Seed:                  Fixed
7Prompt:                Identical
8Output budget:         Fixed
9Session:               Fresh
10Parallel requests:     1
11Swap:                  0
12Warm-up:               1 run
13Measured repetitions:  10

Per configuration, report:

snippet.txttext
EditCopy
1Median generation tok/s
2Median TTFT
3P10 / P90
4Peak RAM
5Peak VRAM
6Crash count
7Schema pass rate
8Agent pass rate

Median should be the headline number instead of the mean, because inference benchmarks can have outliers from OS scheduling and GPU warm-up.

---

15. Proposed Evaluation Matrix

---

16. What Can Be Claimed Now?

Based on the current results, the following can be reasonably stated:

Supported by the current experiment

1. Models with more total parameters do not necessarily have lower generation throughput.

2. Gemma 4 26B-A4B can run hybrid GPU/RAM inference at an interactive level on RTX 3060 12 GB + RAM 16 GB.

3. The tested Gemma configuration can scale context to 16K while keeping generation throughput in the tens of tokens/s.

4. Mistral Small 3.1 24B Dense was much slower than Gemma MoE in the tested configuration.

5. Qwen-AgentWorld 35B-A3B shows that a 35B-total model can produce tokens in the tens per second even on very constrained hardware, but at a high latency and stability cost.

6. Architecture clearly affects performance, and parameter count alone cannot explain all the results.

---

17. What Should Not Be Claimed Yet?

We should not yet say:

MoE is 5× faster than Dense in general.

or:

Active parameters determine tokens/s.

or:

RAM offloading has no effect on MoE.

or:

Gemma 4 is more efficient than Mistral because of MoE alone.

because there are confounding variables.

The safer statement is:

Under the tested hardware and inference configurations, sparse MoE models showed substantially better tolerance to GPU-to-RAM offloading than the tested 24B dense baseline. Further controlled experiments are required to isolate the contribution of sparsity from model architecture, quantization, context implementation, and backend optimization.

---

18. Conclusion

This experiment started from a simple question:

If VRAM is insufficient, will RAM offloading be too slow for Local AI?

The results show the answer depends heavily on model architecture.

The tested Dense 24B dropped to just 3.35 tokens/s in the hybrid configuration, while Gemma 4 26B-A4B held roughly 12–19 tokens/s at the 16K configuration, despite having more total parameters and model weights that could not fully fit in VRAM.

Qwen-AgentWorld 35B-A3B makes this relationship even clearer: a 35B model can produce output at tens of tokens/s on an RTX 3060, but total model size still matters through memory pressure, stability, and TTFT.

So the preliminary evidence from this experiment paints a picture of Local AI different from the idea that "VRAM must hold the entire model."

For sparse architectures, the system can be designed as a memory hierarchy:

snippet.txttext
EditCopy
1GPU = Compute / Hot Path Tier
2RAM = Model Capacity / Expert Tier
3NVMe = Model Storage Tier

If these results are confirmed by controlled experiments, this approach could let consumer and home-server hardware run models with far more parameter capacity than their VRAM, especially with sparse Mixture-of-Experts architectures.

The preliminary results do not show that RAM can replace VRAM.

They show something more interesting:

Model architecture can change how RAM and VRAM should be used together.

---

References

Gemma Team. Gemma 4 26B-A4B model card and technical report. Google, 2026. The model card specifies 25.2B total parameters, 3.8B active parameters, and up to 256K context.

Google / Hugging Face. Gemma 4 family overview. Gemma 4 12B is a dense model of about 11.95B parameters supporting long context.

Mistral AI. Mistral Small 3.1 24B model card and configuration. The model has 24B parameters, 40 text layers, and up to about 128K context.

Qwen Team. Qwen-AgentWorld-35B-A3B. A 35B total / 3B activated language world model trained for agentic environment simulation.

ggml-org / llama.cpp. llama.cpp supports CPU MoE and per-layer MoE CPU placement through --cpu-moe and --n-cpu-moe.

DavidAU. Llama-3.2-4X3B-MOE-Ultra-Instruct-10B community model. Community/merged MoE based on four Llama 3.2 3B models.

Prism ML. Bonsai 27B binary and ternary model family.

Z.ai. GLM-OCR Technical Report. 0.9B specialized multimodal document-understanding model.