Project Maya Adds Dual CUDA and ROCm Backend Support, Boosting Heterogeneous GPU Inference Speeds by up to 48%
CUDA+ROCm dual-back ends LLM inference for mixed-vendor GPUs (up to 48% faster)
Open-source inference project Project Maya has submitted a PR to introduce dual CUDA and ROCm backend support, enabling cross-vendor GPUs to collaboratively run LLM inference. Under this architecture, CUDA handles the main model execution, while an independently compiled HIP module leverages 96 GiB of UMA for GLM expert computation. Benchmarked on a platform equipped with a Ryzen AI Max+ 395 and an RTX 5090, decoding speeds improved by up to 48% compared to native CUDA and CPU configurations.
The project demonstrates a viable implementation pathway for collaborative LLM inference across multi-vendor GPUs, serving as a practical reference for joint acceleration combining consumer-grade unified memory and discrete graphics cards.
Source:Hacker News · AI · github.com