Splash: why a model-specific engine generates responses faster, and what you give up in return

For the past two years, running language models locally has looked much the same regardless of the tool. You pick a general-purpose engine such as llama.cpp, Ollama or MLX, download a weights file and get to work. That interchangeability was the advantage: one program, hundreds of models.

In mid-September 2026, Inco AI released Splash, an engine that reverses that approach. Rather than support everything, it focuses on a few selected models. Each gets its own GPU kernels, memory plan and specially trained draft model, a smaller model that speeds up generation. According to Inco, this setup generates responses 1.7 to 5.3 times faster than competing engines, depending on the comparison.

What is Splash?

Splash is an open-source inference engine: software that uses an already trained model to generate responses. It runs only on Apple computers with an M3 chip (or newer) and macOS 26.4 (or later). Its code is available on GitHub under the Apache 2.0 licence. The engine starts a local server compatible with the OpenAI and Anthropic APIs, so you can connect tools such as Claude Code or Codex without rewriting your integrations. You can also select it as an experimental backend in LM Studio.

The first release supported two models: Qwen3.8-27B, a dense model in which all parameters are used for every token, and Qwen3.6-35B-A3B, a mixture of experts in which only about 3 billion of its 35 billion parameters are active at a time. The current version, 1.4.1 from 26 October, supports six model families, but the principle is unchanged: each gets its own optimisations, and you simply cannot run a model outside the list.

Why does generation wait on memory rather than computation?

A model builds its response one token at a time, with each token representing a word or part of one. To generate each token, the GPU has to read practically all the weights from memory. For 4-bit Qwen3.8-27B, those weights occupy about 15 GiB. The calculations themselves are relatively cheap. Moving gigabytes from memory to the cores takes the most time, leaving the GPU waiting for data for much of the process.

This is called memory-bandwidth-bound decoding. It gives us a simple rule: an engine generates faster if it produces more tokens per read of the weights or moves fewer bytes for each token. Speculative decoding follows the first route. Quantisation, which stores weights at lower precision, follows the second. Splash takes both routes, but tunes every component to a specific model.

Three things Splash tailors to each model

GPU kernels. These are small programs that run on the GPU to multiply matrices and perform attention calculations. A general-purpose engine needs kernels that work with matrices of any size, so it cannot make assumptions about the data in advance. Splash knows the exact tensor shapes, meaning the dimensions of the arrays of numbers that make up each supported model. It can therefore combine several operations into one kernel without writing intermediate results to memory.

Memory plan. At startup, the engine takes the memory limit recommended by Metal, subtracts the fixed cost of the weights and the draft model, and allocates the rest to the conversation cache. For Qwen3.8-27B, the weights take 15 GiB and the draft model 1.2 GiB. The engine therefore knows in advance how many concurrent requests it can handle and how much context will fit.

Draft model. This is what most clearly sets Splash apart from general-purpose engines.

A draft model trained for one target

With speculative decoding, a small, fast draft model proposes the next few tokens, and the large model checks them all in one pass. Because the large model has to read all its weights anyway, checking seven tokens costs little more than generating one. If the proposals are correct, you get several tokens for the cost of a single read. If not, the large model discards the incorrect part and you lose only a little time. We covered what this looks like in practice with MLX, and when it can backfire in long conversations, in our M4 Max inference benchmark.

The gain depends entirely on how often the draft model's proposals are accepted. A general-purpose draft model makes worse guesses because it has not learned the behaviour of the particular model it is helping. Splash uses DFlash 2, an extension of the DFlash method described by researchers at z-lab. The draft model for Qwen3.8-27B has five layers, occupies 1.2 GiB and reads the target model's internal states rather than just the final text. It proposes 7 tokens at each step. According to the model card, the large model verifies every token, so speculation is intended to affect speed, not the content of the response.

Speculation helps most when the GPU has spare capacity, meaning a single user. Measurements of DFlash 2 on an NVIDIA H200, cited by Ground Truth from the model card, show how quickly the gain shrinks as concurrent requests increase:

At 32 concurrent requests, the gain for ordinary chat all but disappears because the GPU has no idle time left to use.

What do the vendor's numbers show?

Inco published a comparison on a single machine: an M5 Pro with 48 GB of memory and a 16-core GPU. Splash was compared with oMLX, Lily and Ollama, plus uzu for the 27B model, on coding tasks from SPEED-Bench, with responses capped at 1,024 tokens. All the figures below come from the vendor, which chose both the competitors and the test conditions. There is no independent benchmark of Splash yet. The chart shows median generation speed for a single request with a short prompt and with a 32,000-token prompt.

The widely quoted 210 tokens per second refers to the mixture-of-experts model, which reads only a small fraction of its weights for each token. Dense Qwen3.8-27B reaches 74 tokens per second on the same hardware. According to the vendor, the lead over the fastest competitor on a short prompt is 1.7 times for the 35B model and 2 times for the 27B. The larger multipliers, from 2.5 to 5.3 times, come from a separate comparison against llama.cpp using identical GGUF files, where llama.cpp has no trained draft model.

The gap is not large everywhere. For long-prompt processing on the 27B model, the model card reports only a 1.2-fold lead over uzu. Cache reuse tells a more interesting story: when replaying an identical 32,000-token prompt, Splash returns the first token after 282 ms, compared with more than 2 seconds for oMLX. This is exactly the weakness that made the second turn of a conversation twelve times slower in our tests. The vendor is clear that the test replays an identical prompt, leaving out the new tokens added in a real conversation.

What are Splash's limitations?

An engine built for specific models takes extra work to prepare and maintain in exchange for higher speed. Before choosing it, consider these limitations:

• Short model list. Splash cannot load full-precision BF16, F16 or F32 models, while support for custom or fine-tuned versions is available “on request”.

• Version lock-in. According to the model card, a package built for Splash does not work in other engines. Version 1.3.1 changed the format, and older packages stopped loading.

• A new model needs a new draft model. When another model version comes out, someone must train a draft model for it and tune the kernels. Only then can it run in Splash.

• One platform. Apple silicon from M3 onwards and macOS 26.4 or later only. No Linux, Windows or NVIDIA GPUs.

• Smaller gains under load. When many people use the engine at once, speculation brings less benefit. Splash's strong results with concurrent requests owe much to memory management and request batching.

• No authentication by default. The server listens only on the Mac itself, but requires no key by default. Set SPLASH_API_KEY before making it available on your network.

None of these limitations rules Splash out. It is simply a different kind of tool: instead of a flexible workbench for testing models, you get a tuned setup for working with one of them.

What this means for a company that wants to run AI on its own hardware

A local model has two main advantages: the information it processes stays within your infrastructure, and costs do not rise with every request. For companies working with contracts, technical documentation or customer data, the first point is often decisive. Performance has been a weakness: no one wants to use a slow assistant. Splash shows that a single Mac can run a coding agent at a pace that does not frustrate users.

The hardware requirements are specific. The 4-bit version of the dense 27B model needs at least 36 GB of unified memory, and the vendor recommends 48 GB or more. Smaller GGUF variants run on Macs with 24 GB, at the cost of some quality.

In short, how should you choose?

• Splash makes sense when you know which model you want to use, it is supported, and your machine is a Mac with an M3 or newer chip.

• A general-purpose engine is better when you switch models frequently, use models fine-tuned on your own data, or need a Linux server with NVIDIA GPUs.

• Testing with your own prompts matters either way. A vendor benchmark based on coding tasks will not tell you how an engine handles long documents or ten people working at once.

The broader trend is clear. Until now, general-purpose engines had the advantage because new models appeared every week and nobody wanted to wait for optimisations. Now that a few open models have become standard in business applications, it pays to tailor the software to one of them. The cost is less freedom to choose.

At ZanReal, we design and maintain local AI environments for companies: from selecting a model and hardware, through measuring performance on real data, to integrating with the tools your team already uses. Considering a deployment like this? Get in touch. We can help you decide whether it makes sense for your business and where to start.