Enable speculative decoding
Howdy fine Ollama folks 👋 , Back this time last year llama.cpp added support for speculative decoding using a draft model parameter. https://github.com/ggerganov/llama.cpp/issues/2030 This can _massively_ speed up inference. I was wondering if there's any chance you could look at adding the option for llama.cpp's `--model-draft` parameter that enables this? --- It works by loading a smaller model (with the same tokeniser/family) in front of a larger model. I know with exllamav2 you can get 100%-200% speed increases (seriously!) and the best part is - with no loss of quality. For example, you might have: - Main model: Qwen 72b Q4_K_M - Draft model: Qwen 0.5b Q4_K_M The result would be the memory usage of main Qwen 72b model + the tiny 0.5b draft model but around 4x the tokens/s you'd see with just the main model loaded. I've been using exllamav2 instead of Ollama for this feature (and the 4bit K/V cache #5091) recently and the performance really is astounding. --- Parameters that can be passed to llama.cpp's server: 1. `--model-draft` (required) - the usage is the same as the existing `--model` used for loading normal models. 2. `--draft` (optional, but recommended to make available) 3. `--p-split` (optional, but recommended to make available) There is also the following, but I think the default is probably fine 99% of the time: - `--threads-draft` - `--threads-batch-draft` https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md?plain=1#L3
Posted by sammcj on Jul 19, 2024, 9:51 PM
Tags
Related issues
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
support (Sysadmin)
Support connecting to remote devcontainers (devcontainer on remote host)
Option to use Waffle menu or links displayed normally
I also attempted to package ROCM on musl.
But why is there no such pushback agains linux kernel?
OpenCode Support
Upcoming changes to the tado° API
Allow Mac users to use Apple native containerization
Support for the Containerization Framework on macOS
Enable speculative decoding
Howdy fine Ollama folks 👋 , Back this time last year llama.cpp added support for speculative decoding using a draft model parameter. https://github.com/ggerganov/llama.cpp/issues/2030 This can _massively_ speed up inference. I was wondering if there's any chance you could look at adding the option for llama.cpp's `--model-draft` parameter that enables this? --- It works by loading a smaller model (with the same tokeniser/family) in front of a larger model. I know with exllamav2 you can get 100%-200% speed increases (seriously!) and the best part is - with no loss of quality. For example, you might have: - Main model: Qwen 72b Q4_K_M - Draft model: Qwen 0.5b Q4_K_M The result would be the memory usage of main Qwen 72b model + the tiny 0.5b draft model but around 4x the tokens/s you'd see with just the main model loaded. I've been using exllamav2 instead of Ollama for this feature (and the 4bit K/V cache #5091) recently and the performance really is astounding. --- Parameters that can be passed to llama.cpp's server: 1. `--model-draft` (required) - the usage is the same as the existing `--model` used for loading normal models. 2. `--draft` (optional, but recommended to make available) 3. `--p-split` (optional, but recommended to make available) There is also the following, but I think the default is probably fine 99% of the time: - `--threads-draft` - `--threads-batch-draft` https://github.com/ggerganov/llama.cpp/blob/master/examples/server/README.md?plain=1#L3
Comments (0)
No comments yet. Start the discussion.