Add batched inference
- [x] Use `llama_decode` instead of deprecated `llama_eval` in `Llama` class - [ ] Implement batched inference support for `generate` and `create_completion` methods in `Llama` class - [ ] Add support for streaming / infinite completion
Posted by abetlen on Sep 30, 2023, 6:29 AM
Tags
Related issues
- some bike vendor told me that the biggest problem with regular to ebike conversion was that regular bike frames were not strong enough to...
- I think of this as the kiosk economy.
- What a goofy comment.
- There's lots of evidence for positive influences of Caffeine on the body (some negative as well), especially the brain.
- Compose support in WSLc
- [RFC] Roadmap to v2.0
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
support (Other)
some bike vendor told me that the biggest problem with regular to ebike conversion was that regular bike frames were not strong enough to...
support (Other)0 comments
I think of this as the kiosk economy.
support (Other)0 comments
What a goofy comment.
support (Other)0 comments
There's lots of evidence for positive influences of Caffeine on the body (some negative as well), especially the brain.
support (Other)0 comments
Compose support in WSLc
support (Other)0 comments
[RFC] Roadmap to v2.0
support (Other)0 comments
[Feature]: Add TurboQuant Support for KV Cache Quantization
support (Other)0 comments
Support for AGENTS.md and .agents/skills/, the community has been asking since August 2025
support (Other)0 comments
Add batched inference
abetlenSep 30, 2023, 6:29 AM
- [x] Use `llama_decode` instead of deprecated `llama_eval` in `Llama` class - [ ] Implement batched inference support for `generate` and `create_completion` methods in `Llama` class - [ ] Add support for streaming / infinite completion
Comments (0)
No comments yet. Start the discussion.