OpenVLA freezing during manipulation task on RTX 4090 — reproducible crash

I'm running OpenVLA fine-tuned on our own pick-and-place dataset and it consistently crashes after ~12 minutes of continuous inference. GPU memory looks stable in nvidia-smi but the Python process just dies. Stack trace shows it's inside the vision encoder. Has anyone seen this? Running Ubuntu 22.04, CUDA 12.1, transformers 4.40.

Hi Marcus — this is a known issue with OpenVLA's vision encoder under sustained inference on Ampere-class GPUs. The root cause is typically a memory fragmentation buildup in the attention cache that the Python garbage collector doesn't reclaim aggressively enough.

Two things to try:

1. Add `torch.cuda.empty_cache()` and `gc.collect()` every 200 inference steps. This alone often resolves the 12-minute crash window.
2. Set `PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512` before launching — this changes the allocator strategy and significantly reduces fragmentation.

If you're on transformers 4.40, there's also a known issue in the `modeling_openvla.py` vision tower with gradient accumulation in eval mode — check if `model.eval()` is being called consistently or if there's a context manager leak.

Let us know if either of those helps. We can dig further into the stack trace if you want to share the full output.