llamAmpere (v0.3): this fork runs Qwen3.8-27B at up to 245K context on one RTX 3090 / 3090 Ti with the model's own MTP head: 99 tok/s on clean agentic and coding fixtures at temperature 1 (1.46x stock llama.cpp, 1.28x v0.2), 93 tok/s at 100K KV depth, and 1.10x tuned vLLM single-stream at 32K. Start with QWEN_AMPERE.md and the write-up in docs/llamampere-v0.3/ARTICLE.md (v0.2: docs/llamampere-v0.2/ARTICLE.md). Successor of llama-cpp-qwen-ampere (v0.1). It also loads Agnes 3.0 Flash, including its MTP head, which upstream llama.cpp does not; see docs/agnes-3.0-flash.md. The rest of this README is upstream llama.cpp's.

Fastest configuration (v0.3, one RTX 3090 / 3090 Ti). Build from main as in QWEN_AMPERE.md, with the GGUF from jakeatx/Qwen3.8-27B-ATX-IQ4_XS-M-GGUF, then:

GGML_Q8_TURBO3_MMA_FUSED=1 ./build-sm86/bin/llama-server -m Qwen3.8-27B-ATX-4-XS.gguf \
  -c 245760 -b 4096 -ub 1024 -t 8 -tb 8 -ngl 99 -fa on -ctk q8_0 -ctv turbo3 \
  --parallel 1 --jinja --fit off \
  --cache-prompt --cache-ram 8192 --ctx-checkpoints 24 --checkpoint-min-step 10240 \
  --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0 \
  --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 \
  --spec-draft-vocab-map docs/mtp-vocab/atx_65536.txt

The speed comes from MTP-3 speculative decoding with exact p/q verification (on by default), --spec-draft-p-min 0, the 65,536-token draft vocabulary map, a q8_0 K / turbo3 V cache on the fused attention kernel, and a q8_0/q8_0 drafter cache. Lower -c if you need VRAM for something else; the decode flags stay the same. To avoid turbo3, -ctv q8_0 (~165K context) and -ctv q5_1 (~190K, needs a -DGGML_CUDA_FA_ALL_QUANTS=ON build, no fused kernel yet) are listed with their trade-offs under KV cache options.

A few options to get llama.cpp installed on your machine:

• Visit https://llama.app and follow the instructions
• Run with Docker - see our Docker documentation
• Download pre-built binaries from the releases page
• Build from source by cloning this repository - check out our build guide

Once installed:

# Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

# Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

VLM session with llama cli

Built-in web UI against llama serve

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

• Plain C/C++ implementation without any dependencies
• Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
• AVX, AVX2, AVX512 and AMX support for x86 architectures
• RVV, ZVFH, ZFH, ZICBOP and ZIHINTPAUSE support for RISC-V architectures
• 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization for faster inference and reduced memory use
• Custom CUDA kernels for running LLMs on NVIDIA GPUs (support for AMD GPUs via HIP and Moore Threads GPUs via MUSA)
• Vulkan and SYCL backend support
• CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity

The llama.cpp project is build on top of the ggml library.

• cli
• completion
• server
• GBNF grammars

• How to build
• Running on Docker
• Build on Android
• Multi-GPU usage
• Performance troubleshooting
• GGML tips & tricks
• XCFramework
• Completions
• Models

• Contributors can open PRs
• Collaborators will be invited based on contributions
• Maintainers can push to branches in the llama.cpp repo and merge PRs into the master branch
• Any help with managing issues, PRs and projects is very appreciated!
• Read the CONTRIBUTING.md for more information

• yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
• stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
• nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
• miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
• subprocess.h - Single-header process launching solution for C and C++ - Public domain