GPU LLM Benchmarks

Koyeb provides these benchmarks for large language models running on Koyeb across multiple GPU types (NVIDIA A100, H100, and L40S) to showcase how performance scales with different workloads. The benchmarks capture latency, throughput, and token usage under controlled input/output sizes, so you can have clear comparisons between models and hardware.

Benchmarking purpose and methodology

By systematically measuring model behavior across a range of input and output sizes, we can identify scaling characteristics, throughput limits, and potential performance bottlenecks to help you make informed decisions about which models and GPU types are best suited for specific applications.

What we measure

Wall-clock time (s) – the real elapsed time of the benchmark run, useful for spotting scaling issues.

Token counts (Input tokens/Output tokens) – the number of input tokens, output tokens, and total tokens actually processed by the model.

Throughput (t/s) – the number of tokens processed per second, often broken down into input tokens, output tokens, and combined totals.

Avg Latency (s) – the time required to complete a single request from submission to completion.

How we measure

1. We define token shapes (e.g. 1024x1024) to represent target input and output lengths. These shapes specify the intended workload.

2. The benchmarking harness generates synthetic prompts of the requested shape and submits them to the deployed model endpoints.

3. Each request is wrapped in a standardized system prompt and instruction template. This means the actual input length (ie Input Tokens) will typically exceed the requested token shape due to added benchmarking overhead.

4. For every request, the benchmark records token counts, throughput, and latency.

Why this approach

By controlling the workload and measuring real responses, you can:

• Compare performance across different models and GPUs on equal terms.
• Observe how latency and throughput scale with increasing input and output sizes.
• Detect mismatches between expected and actual token usage.
• Provide reproducible data for performance tuning and capacity planning.

LLM inference benchmarks

The following charts outline performance of the designated LLM running on specific hardware:

Llama 3.1 8B Instruct

The following chart compares Llama 3.1 8B Instruct performance on different GPUs. Deploy Llama 3.1 8B instruct on Koyeb (opens in a new tab)

Deepseek R1 Distill Llama 8b

The following chart compares Deepseek-R1 Distill Llama 8b performance on different GPUs. Deploy Deepseek-R1 Llama 8b on Koyeb (opens in a new tab)