PrismML Releases Bonsai 27B: 1-bit and Ternary Builds of Qwen3.6-27B That Run on Laptops and Phones

0 0

PrismML just released Bonsai 27B. It is a low-bit representation of Qwen3.6-27B, not a new pretrain. The architecture is unchanged.

Two variants ship under Apache 2.0. Ternary Bonsai 27B uses {−1, 0, +1} weights at a true 1.71 bits per weight. Its ideal size is 5.9GB. 1-bit Bonsai 27B uses binary {−1, +1} weights at 1.125 bits per weight, for 3.9GB.

Both are multimodal. The split is ~24.8B language weights, a 0.46B vision tower, and 2.5B in embeddings and the LM head. The vision tower is held separately at 4-bit (HQQ). Context is 262K tokens, kept practical because ~75% of Qwen3.6-27B attention is linear.

That architecture shapes the compression method below.

How the Compression Works

Each weight is a code, with one shared FP16 scale per group of 128. The effective weight is w_i = s_g · t_i.

A ternary value carries log2(3) ≈ 1.585 bits. One FP16 scale per 128 weights adds 16/128, giving ≈1.71 bits per weight. That is a ~9.4× reduction against FP16. Binary costs 1 + 16/128 = 1.125 bits, a ~14.2× reduction.

The representation runs end to end across the matrix-heavy components. Those are embeddings, attention projections, MLP projections, and the LM head. Only a negligible tail of normalization and scale parameters stays higher precision.

Measured as a true average, the Qwen3.6-27B “4-bit” build (Q4_K_XL) is 5.2 bits per weight. The “2-bit” build (IQ2_XXS) is 2.8. Bonsai also departs from BitNet, which avoids collapse only by pretraining from scratch.

The obvious question is what compression costs in accuracy.

Performance

PrismML evaluated 15 benchmarks in thinking mode, using EvalScope with vLLM on H100 GPUs. Ternary Bonsai 27B retains 94.6% of the FP16 baseline, and 1-bit Bonsai 27B retains 89.5%.

Conventional sub-4-bit builds fail differently. IQ2_XXS falls to 57.5 on AIME26 and 56.4 on LiveCodeBench. It still scores 88.93 on MMLU-Redux, so short-form benchmarks mask the collapse. Gemma-4-31B Q2_K_XL repeats that pattern on a second base model.

Scores alone, however, do not explain the release. Memory does.

Memory is the Binding Constraint

Fitting a phone is stricter than storage numbers suggest. iOS limits a single app to roughly half of physical memory. A 12GB iPhone therefore exposes about 6GB.

The KV cache is the second budget. Only 16 of 64 layers carry a growing full-attention cache, so FP16 costs ≈64 KiB/token. A 262K window costs ≈17.2GB, and a 4-bit KV cache cuts that to ≈4.3GB.

Tolerance is measured. Against its own FP16-KV baseline, Ternary Bonsai shows 0.0011 nats of output forward-KL on MATH-500. Q4_K_XL shows 0.0146.

Peaks follow. At 100K tokens with an FP16 cache, 1-bit peaks at 11.6GB and ternary at 14.7GB. The derived Q4_K_XL row needs ≈25.6GB.

Once a model fits, throughput is the next question.

Throughput and DSpark Speculative Decoding

Generation is memory-bandwidth-bound, so fewer bytes per step means more tokens per second. Prefill is compute-bound and gains less.

PrismML also ships a DSpark drafter trained against the Bonsai 27B target. On an H100 at draft depth k=4, the binary target reaches accepted length τ=3.6. That is 143.8 tok/s, a 1.37× speedup. Verification is lossless, so output stays distribution-identical. On Apple Silicon the drafter is off by default at batch size 1.

Running It

Ternary 27B is the demo repo default. Start the server, or generate directly:

./scripts/start_llama_server.sh   # OpenAI-compatible API + chat/vision UI on :8080

./llama-cli -m ./Ternary-Bonsai-27B-gguf/Ternary-Bonsai-27B-Q2_0.gguf 
  --mmproj ./Ternary-Bonsai-27B-gguf/mmproj.gguf -c 0 
  -p "Explain KV cache growth."

mlx_lm.generate --model prism-ml/Ternary-Bonsai-27B-mlx-2bit 
  --prompt "Explain KV cache growth."

Tool calling uses the standard OpenAI-style tools array:

curl http://localhost:8080/v1/chat/completions 
  -H "Content-Type: application/json" 
  -d '{
    "messages": [{"role": "user", "content": "What is the weather in Lisbon?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
      }
    }]
  }'

The call returns in choices[0].message.tool_calls. Thinking mode is on by default; thinking_budget_tokens toggles it per request.

That maps onto four deployment patterns.

Use Cases

Laptop-local agents run the ternary build for full-repository code work over 262K tokens. Phone-local reasoning runs the 1-bit build; the whitepaper measures 672 tokens per 1% of iPhone battery. Privacy-sensitive and offline workflows keep prompts on-device by construction. Combined with the 4-bit KV cache, single-GPU serving fits 27B-class quality on a 24GB card.

Key Takeaways

• Bonsai 27B moves Qwen3.6-27B into binary or ternary weights, not a new pretrain.
• Ternary retains 94.6% of FP16 at 5.9GB; 1-bit retains 89.5% at 3.9GB.
• PrismML claims the 1-bit build is the first 27B-class model to fit a phone.
• Conventional sub-4-bit builds collapse selectively on AIME, LiveCodeBench, and agentic tasks.
• Everything ships under Apache 2.0, on llama.cpp (CUDA, Metal) and MLX.

Check out the Technical details, GitHub Repo and Model Weight. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us

The post PrismML Releases Bonsai 27B: 1-bit and Ternary Builds of Qwen3.6-27B That Run on Laptops and Phones appeared first on MarkTechPost.

​MarkTechPost

About Post Author

Manoj Balakrishnan

[email protected]

https://annapoornainfo.com

Happy

0 0 %

Sad

0 0 %

Excited

0 0 %

Sleepy

0 0 %

Angry

0 0 %

Surprise

0 0 %