[RFC] TorchStore - A Shared-Memory Tensor Store
## 🚀 Feature TorchStore is a key-value store that holds ATen tensors in shared memory so that they can be accessed across process boundaries without any expensive copy operations. ## Problem With the trajectory of ever-increasing model sizes the conventional handling of inference loads by multiple worker processes has started to become a memory bottleneck. Since each worker requires its own instance of the model (or models), the memory use proportionally increases with the number of workers. Considering that today’s multi-core systems can easily handle a large number of workers, this means a big portion of physical memory potentially goes wasted with redundant copies of the same model. Some users, as described [here](https://medium.com/ibm-data-ai/how-to-load-pytorch-models-340-times-faster-with-ray-8be751a6944c), have successfully leveraged Apache Arrow’s [Plasma Object Store](https://arrow.apache.org/docs/python/plasma.html) to circumvent the problem, but the overall user experience is less than ideal and a production-ready solution requires quite a bit of manual work. Similarly training tasks in various machine learning fields such as graph neural networks or knowledge distillation happen to use read-only models during training. The same problem as in inference exists here as well. Many redundant copies of the same model waste the precious memory resources of the host machine. The problem is even more amplified in the training case since the memory is typically alrea
Posted by cbalioglu on Sep 13, 2021, 3:37 PM
Tags
Related issues
- Living in a 200+ year old house, I've seen a lot of this first-hand through both planned and unplanned renovations and repairs.
- Is anything with the pebble time able to capture heart rate for historical data analysis?
- Datastar has evolved into a suite of tools [1].
- > real issue is that MSG is basically a public utility.
- Everything good is a slippery slope: every step in the right direction is bad if you overshoot.
- Status Update - May 2026
Raise real-world problems, find people facing the same issue, and discuss solutions.3,000 issues0 comments
pricing (EdTech)
Living in a 200+ year old house, I've seen a lot of this first-hand through both planned and unplanned renovations and repairs.
Is anything with the pebble time able to capture heart rate for historical data analysis?
Datastar has evolved into a suite of tools [1].
> real issue is that MSG is basically a public utility.
Everything good is a slippery slope: every step in the right direction is bad if you overshoot.
Status Update - May 2026
I think what you wrote makes sense, and you missed one critical aspect : shared access to excessively expensive capitalized facilities an...
Long-range anti-ship missiles of old are also obsolete, they and their launch problems are also too expensive for their vulnerability.
[RFC] TorchStore - A Shared-Memory Tensor Store
## 🚀 Feature TorchStore is a key-value store that holds ATen tensors in shared memory so that they can be accessed across process boundaries without any expensive copy operations. ## Problem With the trajectory of ever-increasing model sizes the conventional handling of inference loads by multiple worker processes has started to become a memory bottleneck. Since each worker requires its own instance of the model (or models), the memory use proportionally increases with the number of workers. Considering that today’s multi-core systems can easily handle a large number of workers, this means a big portion of physical memory potentially goes wasted with redundant copies of the same model. Some users, as described [here](https://medium.com/ibm-data-ai/how-to-load-pytorch-models-340-times-faster-with-ray-8be751a6944c), have successfully leveraged Apache Arrow’s [Plasma Object Store](https://arrow.apache.org/docs/python/plasma.html) to circumvent the problem, but the overall user experience is less than ideal and a production-ready solution requires quite a bit of manual work. Similarly training tasks in various machine learning fields such as graph neural networks or knowledge distillation happen to use read-only models during training. The same problem as in inference exists here as well. Many redundant copies of the same model waste the precious memory resources of the host machine. The problem is even more amplified in the training case since the memory is typically alrea
Comments (0)
No comments yet. Start the discussion.