DataLoader num_workers > 0 causes CPU memory from parent process to be replicated in all worker processes
# Editor note: There is a known workaround further down on this issue, which is to NOT use Python lists, but instead using something else, e.g., torch.tensor directly. See https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 . You can use a numpy array, but it only fixes the issue for the `fork` start method. See https://github.com/pytorch/pytorch/issues/13246#issuecomment-1364587359 for more details ## 🐛 Bug CPU memory will leak if the DataLoader `num_workers > 0`. ## To Reproduce Run the following snippet: ``` from torch.utils.data import Dataset, DataLoader from PIL import Image from torchvision import transforms import os class DataIter(Dataset): def __init__(self): path = "path/to/data" self.data = [] for cls in os.listdir(path): for img in os.listdir(os.path.join(path, cls)): self.data.append(os.path.join(path, cls, img)) def __len__(self): return len(self.data) def __getitem__(self, idx): with Image.open(self.data[idx]) as img: img = img.convert('RGB') return transforms.functional.to_tensor(img) train_data = DataIter() train_loader = DataLoader(train_data, batch_size=300, shuffle=True, drop_last=True, pin_memory=False, num_workers=18) for i, item in enumerate(train_loader): if i % 200 == 0: print(i) ``` ## Expected behavio
Posted by bfreskura on Oct 29, 2018, 1:23 PM
Tags
Related issues
- [BUG] Cowork (Windows): all Plan9 shares fail with "Plan9 mount failed: invalid argument" after Windows update KB5124008 (26200.9445) — uninstalling the KB fixe
- Feature Request: Auto-resume CLI session when usage limit resets
- Please deobfuscate the Python
- [Bug] MLX version of Gemma 4 26B fails to load: "Model type gemma4 not supported" (macOS)
- # 233 Developers Discussed Abandoning the Office Add-in Platform — An Open Letter on the Office.js Stability, Security & Trust Crisis
- [FEATURE] Allow users to configure which shell the Bash tool uses or inherit initial shell
Raise real-world problems, find people facing the same issue, and discuss solutions.4,000 issues0 comments
reliability (Software)
[BUG] Cowork (Windows): all Plan9 shares fail with "Plan9 mount failed: invalid argument" after Windows update KB5124008 (26200.9445) — uninstalling the KB fixe
Feature Request: Auto-resume CLI session when usage limit resets
Please deobfuscate the Python
[Bug] MLX version of Gemma 4 26B fails to load: "Model type gemma4 not supported" (macOS)
# 233 Developers Discussed Abandoning the Office Add-in Platform — An Open Letter on the Office.js Stability, Security & Trust Crisis
[FEATURE] Allow users to configure which shell the Bash tool uses or inherit initial shell
Admin Materials and Surfaces
AMD Support
DataLoader num_workers > 0 causes CPU memory from parent process to be replicated in all worker processes
# Editor note: There is a known workaround further down on this issue, which is to NOT use Python lists, but instead using something else, e.g., torch.tensor directly. See https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 . You can use a numpy array, but it only fixes the issue for the `fork` start method. See https://github.com/pytorch/pytorch/issues/13246#issuecomment-1364587359 for more details ## 🐛 Bug CPU memory will leak if the DataLoader `num_workers > 0`. ## To Reproduce Run the following snippet: ``` from torch.utils.data import Dataset, DataLoader from PIL import Image from torchvision import transforms import os class DataIter(Dataset): def __init__(self): path = "path/to/data" self.data = [] for cls in os.listdir(path): for img in os.listdir(os.path.join(path, cls)): self.data.append(os.path.join(path, cls, img)) def __len__(self): return len(self.data) def __getitem__(self, idx): with Image.open(self.data[idx]) as img: img = img.convert('RGB') return transforms.functional.to_tensor(img) train_data = DataIter() train_loader = DataLoader(train_data, batch_size=300, shuffle=True, drop_last=True, pin_memory=False, num_workers=18) for i, item in enumerate(train_loader): if i % 200 == 0: print(i) ``` ## Expected behavio
Comments (0)
No comments yet. Start the discussion.