vLLM

What it is

An open-source engine for serving large language models with high throughput and efficient memory use. Its PagedAttention approach helps reduce wasted GPU memory during inference.

Topics