Serving LLMs on AMD MI300X: Best Practices
blog.vllm.ai
TL;DR: vLLM unlocks incredible performance on the AMD MI300X, achieving 1.5x higher throughput and 1.7x faster time-to-first-token (TTFT) than Text Generation Inference (TGI) for Llama 3.1 405B. It also achieves 1.8x higher throughput and 5.1x faster TTFT than TGI for Llama 3.1 70B. This guide explores 8 key vLLM settings to maximize efficiency, showing you how to leverage the power of open-source LLM inference on AMD. If you just want to see the optimal parameters, jump to the Quick Start Guide.

I landed on this article today about how to configure vLLM for AMD GPUs and it contained this specific snippet:

Meta recently announced they’re running 100% of their live Llama 3.1 405B model traffic on AMD MI300X GPUs […]

I thought that was an interesting piece of trivia and shows both how important it is for industrial partners to find alternatives to Nvidia and also how much AMD has improved its software suite to enable these use-cases.