[dagster_k8s] code location gRPC timeout
## TL;DR It has been observed for K8s deployments that the GRPC server stalls on health checks, causing grpc timeouts, due to the fact that sensors/schedules are pre-empting the grpcs thread from being run. ### What's the issue? Howdy, we've been using dagster for a while now. In our deployment, for the sake of simplicity, we have 2 codelocations deployed via helm. They are typically identical to each other. One is a staging codelocation to test changes that are about to go into the production code location. On our production codelocation, the following command: `dagster api grpc-health-check -p 3030` takes around **2 min** to complete, while with the same underlying image, it takes 3 secs on staging. Looking at my monitoring, the production codelocation doesn't seem stressed at all. Attached are some resource images to prove that point: <figure> <img width="2538" alt="image" src="https://github.com/user-attachments/assets/a94f1eb0-e2f2-4339-abdf-ac582e9ae1d0"> <figcaption>Cpu usage</figcaption> </figure> <figure> <img width="2538" alt="image" src="https://github.com/user-attachments/assets/d58c115c-c3de-401e-998e-58d70a106798"> <figcaption>Memory usage</figcaption> </figure> <figure> <img width="2547" alt="image" src="https://github.com/user-attachments/assets/d4c579d7-291f-452a-b90c-377061b4aee9"> <figcaption>Network Traffic summary</figcaption> </figure> _I have obfuscated certain pod names as they could be considered confidential_ When I look at the logs on my
Posted by abhinavDhulipala on Oct 7, 2024, 7:34 PM
Tags
Related issues
Raise real-world problems, find people facing the same issue, and discuss solutions.2,000 issues0 comments
github (Healthcare)
Add setting to disable the auto-resolve in 60 seconds for questions
Slower development continuing for forseeable future
[dagster_k8s] code location gRPC timeout
Health of Linkerd project
[dagster_k8s] code location gRPC timeout
## TL;DR It has been observed for K8s deployments that the GRPC server stalls on health checks, causing grpc timeouts, due to the fact that sensors/schedules are pre-empting the grpcs thread from being run. ### What's the issue? Howdy, we've been using dagster for a while now. In our deployment, for the sake of simplicity, we have 2 codelocations deployed via helm. They are typically identical to each other. One is a staging codelocation to test changes that are about to go into the production code location. On our production codelocation, the following command: `dagster api grpc-health-check -p 3030` takes around **2 min** to complete, while with the same underlying image, it takes 3 secs on staging. Looking at my monitoring, the production codelocation doesn't seem stressed at all. Attached are some resource images to prove that point: <figure> <img width="2538" alt="image" src="https://github.com/user-attachments/assets/a94f1eb0-e2f2-4339-abdf-ac582e9ae1d0"> <figcaption>Cpu usage</figcaption> </figure> <figure> <img width="2538" alt="image" src="https://github.com/user-attachments/assets/d58c115c-c3de-401e-998e-58d70a106798"> <figcaption>Memory usage</figcaption> </figure> <figure> <img width="2547" alt="image" src="https://github.com/user-attachments/assets/d4c579d7-291f-452a-b90c-377061b4aee9"> <figcaption>Network Traffic summary</figcaption> </figure> _I have obfuscated certain pod names as they could be considered confidential_ When I look at the logs on my
Comments (0)
No comments yet. Start the discussion.