How to Run Hy4 Preview Locally: GGUF, vLLM and SGLang

01

Before you start

Tencent's Hy4 Preview is open-weight, but “run locally” needs a qualifier: it has 770B total parameters with 49B active per token. Practical deployment is aimed at multi-GPU infrastructure.

Start with the official tencent/Hy4-preview or tencent/Hy4-preview-FP8 checkpoints. Tencent licenses both under Apache 2.0 and publishes model-specific reasoning and tool-call parsers.

02

Option 1: vLLM

Tencent documents vLLM as an official serving path. Use the compatible version in the current repository, download the checkpoint and launch it with tensor parallelism sized for your GPU fleet. The resulting server exposes an OpenAI-compatible API.

03

Option 2: SGLang

Tencent's current SGLang recipe uses the FP8 checkpoint, Hy4-specific tool and reasoning parsers, EAGLE-style speculative decoding and the native MTP layer.

The official example uses tensor parallelism across eight devices. That proves eight-way serving is a supported example configuration; it is not a universal minimum-hardware claim.

04

FP8, context length and hardware sizing

The official FP8 checkpoint is the clearest starting point for reducing memory pressure. It remains extremely large, so consumer single-GPU claims should be treated cautiously unless they include a reproducible conversion and measured setup.

Tencent does not publish one universal minimum-VRAM figure. Precision, tensor parallelism, KV cache, speculative decoding, concurrency and requested context all matter. Attempting a 1M-token context can increase memory requirements sharply compared with short-context tests.

Tencent points to AngelSlim for compression and quantization. Any community conversion should be labelled community-produced rather than presented as an official Tencent checkpoint.

05

Community GGUF builds and current llama.cpp status

AngelSlim now publishes community GGUF conversions for Hy4 Preview. The maintained repository currently lists Q4_K_M at 435.20 GiB, UD-IQ1_M at 219.83 GiB and STQ1_0 at 213.66 GiB. Those file sizes are not minimum-RAM guarantees; runtime overhead, KV cache, context length and offload strategy add to the memory budget.

As of September 4, 2026, AngelSlim explicitly says the hyv4 architecture is not upstream in stock llama.cpp, so none of these builds should be presented as plug-and-play with an unmodified release. Apply the architecture patch linked by the maintainer; STQ1_0 requires an additional patch.

06

Practical deployment checklist

1. Choose the official full/FP8 runtime path or a clearly labelled community GGUF conversion.
2. Confirm current Transformers and runtime versions in Tencent's repository.
3. Select vLLM or SGLang and size tensor parallelism for the actual GPU fleet.
4. Start with a shorter context before testing extreme context lengths.
5. Verify the Hy4 tool and reasoning parsers.
6. For GGUF, confirm the current hyv4 architecture patch and any quant-specific patch before launch.
7. Benchmark latency, throughput and memory on the production workload.

Self-host when data control, dedicated capacity or custom infrastructure justify it. Hosted access is much simpler for experimentation with a model at this scale.