- ~4GBRAM actually available for inference on an 8GB phone, after the OS reserves 3-4GB for itself
- 1.88GBsize of a 3B model after Q4_K_M quantization, down from roughly 6GB at full precision — a 68% reduction
- 13-18 tok/sspeed of Phi-4 Mini (3.8B) at Q4_K_M on an iPhone 17 Pro, the practical ceiling for usable on-device chat
The phone in your pocket almost certainly has less usable memory for AI than its spec sheet suggests, and that gap is the single most important thing to understand before picking a model to run on it. An 8GB phone doesn’t give an app 8GB to work with — the operating system reserves 3 to 4GB for itself, leaving roughly 4GB actually available for inference. From there, a rough formula holds: model file size times 1.5 approximately equals the actual RAM needed at runtime, with the extra covering the KV cache and activation memory a running model needs beyond its static weights.
What quantization actually buys you
Quantization is what makes any of this possible at all. Going from 16-bit floating point precision down to 4-bit integer quantization (commonly the Q4_0 or Q4_K_M formats from the llama.cpp ecosystem) roughly triples inference speed with minimal quality impact on most models, and shrinks file size dramatically in the process — a 3B parameter model that would be roughly 6GB at full precision comes down to about 1.88GB at Q4_K_M, a 68% reduction. That reduction is what separates a model that technically exists from a model that actually fits and runs acceptably on a phone with a few gigabytes of real headroom.
| Model | Parameters | Quantized size (Q4) | Fit on 8GB phone (~4GB usable) |
|---|---|---|---|
| Qwen 3 1.5B | 1.5B | ~1GB | Comfortable, fast |
| Gemma 3 4B / Qwen 3.5 4B | 4B | ~2.3-2.5GB | Fits with headroom for simple tasks |
| Phi-4 Mini | 3.8B | ~2.2GB | Fits, current sweet spot for usable speed and quality |
| 7B-class models | 7B | ~4-4.5GB | Tight to unfit on 8GB phones, better suited to 12GB+ |
Why Phi-4 Mini sits at the practical sweet spot
For most flagship phones with 8GB or more of RAM in 2026, Phi-4 Mini at Q4_K_M quantization has settled in as the largest model that runs at genuinely usable speed — around 13 to 18 tokens per second on an iPhone 17 Pro, comfortably faster than most people read. That places it above smaller 1.5B-3B options like Qwen 3 1.5B in output quality, while staying safely inside the memory budget an 8GB phone can actually deliver. Gemma 3 4B and Qwen 3.5 4B sit in similar territory and are genuinely good at simple tasks — quick questions, short summarization, basic drafting — without pushing into the memory pressure that larger models create on this class of hardware.
- 1. OS claims its share firstOn an 8GB phone, 3-4GB is reserved for the operating system before any app gets to use memory.
- 2. Quantization shrinks the model4-bit quantization (Q4_K_M) cuts a model’s size by roughly 68% versus full precision, with modest quality loss.
- 3. Runtime needs more than the file sizeActual RAM required is roughly 1.5x the quantized file size, covering KV cache and activations.
- 4. What’s left decides the model ceilingOn a typical 8GB phone, that math lands squarely on 3-4B models like Phi-4 Mini as the practical maximum.
Why 8GB of phone RAM realistically supports a 3-4B model, not more
Each step in this chain quietly eats into the number on the spec sheet — by the time all four are accounted for, the real ceiling is much lower than 8GB.
What it’s actually good for, and where it falls short
For everyday tasks like quick questions, summarization, drafting, and lightweight document analysis, a well-chosen quantized model on a modern phone is surprisingly capable compared to a cloud LLM, and it works entirely offline with no data leaving the device. It is not a replacement for a frontier cloud model on complex reasoning, long documents, or specialized knowledge tasks — the parameter count gap between a 4B on-device model and a frontier cloud model of hundreds of billions of parameters is real, and quantization narrows the memory footprint, not the underlying knowledge and reasoning capacity difference. Matching expectations to the actual task is the difference between a genuinely useful on-device assistant and a frustrating one.
Run a 3-4B model at Q4_K_M on an 8GB phone Sweet spot
Right for everyday questions, summarization, and drafting — fits comfortably in available memory at usable token speeds.
Attempt a 7B-class model on an 8GB phone Tight fit
Possible in some cases but leaves little memory headroom; a 12GB+ phone is a meaningfully safer target for this size.
Expect on-device models to match frontier cloud LLMs Wrong expectation
The parameter gap between a 4B on-device model and a hundreds-of-billions-parameter cloud model is real and not solved by quantization alone.
Common mistakes
Common mistake
Sizing a model choice against a phone’s advertised total RAM rather than the roughly half that’s actually available after the OS takes its share — this is the single most common reason a “should fit” model runs poorly or crashes.
Common mistake
Comparing quantized model file size directly to available RAM without accounting for the roughly 1.5x runtime overhead from KV cache and activation memory.
What worked
Users who picked models specifically in the 1.5B-4B range for 8GB phones, matched to realistic everyday tasks like summarization and quick questions rather than complex reasoning, got fast, reliable, fully offline performance without hitting memory limits.
Glossary
- Q4_K_M
- A mixed 4-bit quantization format from the llama.cpp ecosystem, commonly used to shrink model file size while preserving most of the original model’s quality.
- KV cache
- Memory used to store intermediate attention values during inference, which grows with context length and adds to a running model’s actual memory footprint beyond its static file size.
- Runtime memory overhead
- The additional RAM a model needs beyond its quantized file size while actively running, commonly estimated at roughly 1.5 times the file size.
- Tokens per second (tok/s)
- A measure of how fast a model generates output text, used to judge whether on-device inference speed feels usable to a human reader.
Key takeaways
- An 8GB phone has roughly 4GB of RAM actually available for AI inference once the operating system reserves its own 3-4GB.
- 4-bit quantization (Q4_K_M) reduces a model’s file size by roughly 68% versus full precision, with minimal quality impact on most models.
- Actual runtime memory needed is roughly 1.5 times a model’s quantized file size, covering KV cache and activation memory.
- Phi-4 Mini (3.8B) at Q4_K_M has become the practical sweet spot for 8GB phones, reaching 13-18 tokens per second on an iPhone 17 Pro.
- Gemma 3 4B and Qwen 3.5 4B are genuinely good at simple tasks like quick questions and basic drafting within the same memory budget.
- 7B-class models are a tight to unrealistic fit on 8GB phones and are better matched to devices with 12GB or more of RAM.
- On-device quantized models are well suited to everyday tasks like summarization and drafting, but the parameter gap versus frontier cloud models remains real and unsolved by quantization alone.
FAQs
How much RAM do I actually need to run an LLM on my phone?
Less matters than you’d think from the spec sheet: on an 8GB phone, only about 4GB is actually available after the OS takes its share, and a model needs roughly 1.5 times its quantized file size in runtime memory to run comfortably.
What’s the best LLM to run on an 8GB phone?
Phi-4 Mini (3.8B) at Q4_K_M quantization is currently the practical sweet spot for most flagship 8GB phones, reaching 13-18 tokens per second on an iPhone 17 Pro while staying comfortably within available memory.
What does 4-bit quantization actually do to a model?
It reduces the numerical precision of the model’s weights, shrinking file size by roughly 68% (a 3B model drops from about 6GB to 1.88GB at Q4_K_M) and roughly tripling inference speed, with only minimal quality loss for most models.
Can I run a 7B model on an 8GB phone?
It’s a tight fit at best — a 7B model at Q4 quantization needs roughly 4-4.5GB, which leaves little to no headroom on a phone with only about 4GB of usable RAM; a 12GB or higher device is a safer target for that size.
Why doesn’t my phone’s full 8GB of RAM go toward running the model?
Because the operating system itself reserves roughly 3-4GB for its own processes and background apps before any AI model gets a chance to use memory, leaving only about half the advertised total actually available.
Are on-device phone models as good as ChatGPT or other cloud LLMs?
Not for complex reasoning or specialized knowledge — the parameter count gap between a 3-4B on-device model and a hundreds-of-billions-parameter cloud model is real. For everyday tasks like quick questions and summarization, though, a well-chosen on-device model is surprisingly capable and works fully offline.
What token speed counts as “usable” on a phone?
Roughly 5 tokens per second or faster generally streams text faster than most people read; Phi-4 Mini’s 13-18 tokens per second on recent flagship hardware is comfortably above that threshold.
Does running an LLM on my phone drain the battery quickly?
It’s more power-intensive than typical app usage, but modern phone NPUs are specifically optimized for this kind of small-batch inference workload, making well-optimized quantized models considerably more battery-friendly than running the same task on a general-purpose CPU.
References
- PromptQuorum — “Best Mobile LLM 2026: Phi-4 Mini vs Gemma 3 vs SmolLM”
- Local AI Master — “Run an LLM on Your Phone (2026): Offline AI on Android and iPhone”
- Local AI Master — “Best Ollama Models for 8GB RAM 2026: 12 Tested Local Picks”
- DEV Community — “How to Run LLMs Locally on Your Android Phone in 2026”
Related reading on The Tech Trends: Local LLM Setup Guide, Small Language Models in 2027, NPU vs GPU vs CPU for AI Inference, and Privacy-First AI.
