How much memory do I need to run Llama 3.1 8B?

By our estimate, about 5.8 GB at Q4 with an 8K context. A Mac with 16 GB of memory or an NVIDIA card with 8 GB of VRAM runs it at Q4. An 8 GB Mac is too tight for it; a 3B model is the better choice there.

What do Q4, Q5, Q8 and FP16 mean?

They describe how many bits store each weight. FP16 is the original 16-bit model. Q8 uses 8 bits and is usually very close to the original. Q4 uses about 4 bits, needs a quarter of the memory of FP16, and loses a little quality. Q4 is the common default for running models at home.

Can I run a 70B model on my own computer?

Yes, with enough memory. A 70B model needs about 44.9 GB at Q4 by our estimate, so a Mac with 64 GB or more, or GPUs with 48 GB of VRAM between them. On a smaller GPU it can split across GPU and system RAM, but it becomes slow.

Why can a Mac run bigger models than many PCs?

Apple silicon Macs share one pool of memory between the CPU and GPU, so a Mac with 64 GB can give most of it to a model. On a PC the model runs fast only if it fits in the graphics card’s own VRAM, which is usually 8 to 24 GB.

No. It is an estimate using a simple formula shown on this page. Real use varies with the runtime, the exact file, the context length and what else is running. Treat a tight fit as a maybe and leave some headroom.

Which AI model can
my computer run?

Pick your machine and memory. See which open models fit, at which quantization, and how much memory each one needs. Everything runs in your browser.

How this works

For each model and quantization we estimate the memory it needs, then compare that with the memory your machine can give it. It is an estimate, not a benchmark.

Memory needed (GB) = parameters (billions) × bits per weight ÷ 8 × 1.2 + context allowance

Parameters × bits ÷ 8 is the size of the weights. An 8B model at 4 bits is about 4 GB.
× 1.2 covers runtime buffers and the few extra bits that quantized files carry.
Context allowance is room for the conversation (the KV cache). We use about 0.5 GB per 8K tokens for models under 5B, 1 GB for 5B to 15B, 2 GB for 15B to 40B and 2.5 GB above that. Real numbers vary by model design.

Example: Llama 3.1 8B at Q4 with 8K context is 8.03 × 4 ÷ 8 × 1.2 + 1 = about 5.8 GB.

What your machine can give a model

Mac with Apple silicon: memory is shared. By default macOS lets the GPU use roughly two thirds of it on machines up to 36 GB, and about three quarters above that. We use those shares.
NVIDIA GPU: a model runs fast when it fits in VRAM. We keep 0.5 GB back for the display. If it doesn't fit, runtimes like llama.cpp and Ollama can put some layers in system RAM (we count 60% of it), which works but is much slower.
CPU only: we count 70% of your RAM. Speed is limited by memory bandwidth, so small models (up to about 8B) are the comfortable choice.

Parameter counts come from each model's public model card. Speed is not estimated here: it depends heavily on your chip and memory bandwidth.

Ways to run them

All of these are free, run fully on your machine, and download the model once.

Ollama. The simplest start. Install it, then run one command to download and chat with a model. It also serves a local API that other apps can use. Mac, Windows and Linux.
llama.cpp. The open-source engine behind many local AI tools. It runs quantized GGUF files on CPUs and GPUs, including Apple silicon and NVIDIA, and can split a model between GPU and system RAM.
MLX. Apple’s machine learning framework for Apple silicon. With mlx-lm you can run and fine-tune models in MLX format, often in 4-bit or 8-bit.
LM Studio. A desktop app with a model browser and chat window. It runs GGUF models, and MLX models on Apple silicon.

We build apps like this

Soonbuilt makes Mac and Windows apps where the AI runs on the user’s own computer. We size the model to each machine, ship a normal installer, and sell it once instead of monthly.

Local AI app development See TallyReel Studio

Questions

How much memory do I need to run Llama 3.1 8B?: By our estimate, about 5.8 GB at Q4 with an 8K context. A Mac with 16 GB of memory or an NVIDIA card with 8 GB of VRAM runs it at Q4. An 8 GB Mac is too tight for it; a 3B model is the better choice there.
What do Q4, Q5, Q8 and FP16 mean?: They describe how many bits store each weight. FP16 is the original 16-bit model. Q8 uses 8 bits and is usually very close to the original. Q4 uses about 4 bits, needs a quarter of the memory of FP16, and loses a little quality. Q4 is the common default for running models at home.
Can I run a 70B model on my own computer?: Yes, with enough memory. A 70B model needs about 44.9 GB at Q4 by our estimate, so a Mac with 64 GB or more, or GPUs with 48 GB of VRAM between them. On a smaller GPU it can split across GPU and system RAM, but it becomes slow.
Why can a Mac run bigger models than many PCs?: Apple silicon Macs share one pool of memory between the CPU and GPU, so a Mac with 64 GB can give most of it to a model. On a PC the model runs fast only if it fits in the graphics card’s own VRAM, which is usually 8 to 24 GB.
Is this exact?: No. It is an estimate using a simple formula shown on this page. Real use varies with the runtime, the exact file, the context length and what else is running. Treat a tight fit as a maybe and leave some headroom.

More free tools

Want AI that runs
on your customer’s computer?

We build Mac and Windows apps with the model on the machine: sized to its memory, private by default, sold once. Tell us what you want to make.

Book a call Try the free idea search

Which AI model canmy computer run?

Models and memory needed