Llama.cpp now supports distributed inference across multiple machines.

Llama.cpp now supports distribution across multiple devices to boost speeds, this would be a great addition to Ollama https://github.com/ggerganov/llama.cpp/tree/master/examples/rpc https://www.reddit.com/r/LocalLLaMA/comments/1cyzi9e/llamacpp_now_supports_distributed_inference/

Posted by AncientMystic on May 26, 2024, 2:57 AM

Tags

• github (Software)

Related issues

• [Meta] Feedback on granular control over Copilot changes
• unexpected status 401 Unauthorized issue
• [MODEL] Measurable quality regression in Claude generation 5 (Fable 5 / Opus 5 / Sonnet 5): worse nonsense detection, ~2x verbosity, under-disclosed model fallb
• [CRITICAL] Subagent spawning and subagent pattern bugs trigger infinite recursion, infinite token usage, grossly inefficient token usage, and lost accumulated s
• Video stops playing after some minutes
• [MODEL] Claude verbose code comments by default — ignores instructions to stop.