How I Hit 1700+ Tokens/Sec at the Velric × Foundery Hackathon
A technical breakdown of how our team secured 3rd place in the Agentic AI track — achieving 1700+ tokens/sec throughput across 16 concurrent requests with 11–13ms TTFT via FP8 quantization.
The Challenge: Agentic AI Under Latency Pressure
The Velric × The Foundery Hackathon Agentic AI track had a clear performance objective: build an agentic system that handles concurrent requests with minimal time-to-first-token (TTFT). Most teams optimized for capability. We optimized for throughput.
Our insight: in agentic systems, perceived responsiveness matters more than raw accuracy. A system that responds in 13ms and is 90% accurate beats one that takes 400ms and is 95% accurate — users abandon slow agents before they see the result.
FP8 Quantization: The Core Optimization
The key to our throughput was FP8 quantization — reducing model weights from 16-bit floats to 8-bit floats with minimal accuracy loss. This doubles the effective memory bandwidth and allows significantly more concurrent requests per GPU.
Combined with continuous batching (processing tokens from multiple requests in a single forward pass), we achieved 1700+ tokens/sec across 16 concurrent requests — roughly 3× the throughput of the baseline FP16 setup.
11–13ms TTFT: How We Got There
Time-to-first-token is dominated by the prefill phase — processing the input prompt through all transformer layers before generating the first output token. We reduced TTFT by: (1) aggressive prompt compression to minimize input length, (2) speculative decoding with a smaller draft model for common token sequences, and (3) keeping the model warm in GPU memory between requests using a persistent inference server rather than cold-loading per request.
Lessons for Production Agentic Systems
The hackathon taught me that throughput optimization is as important as model quality for agentic AI. The techniques we used — FP8 quantization, continuous batching, prompt compression, speculative decoding — are all applicable to production systems today. If you're building agentic AI and haven't profiled your inference pipeline, you're almost certainly leaving 2–3× performance on the table.
More to Discover
Tutorials
Mar 15, 2025
Building LLM Pipelines from Scratch: Transformers to Production
A practical walkthrough of building production-ready LLM pipelines — from transformer architecture fundamentals to deploying scalable inference on AWS SageMaker.
Tutorials
Jan 10, 2025
NLP Pipelines with spaCy and Scikit-learn: A Production Guide
How to build scalable NLP pipelines using spaCy and Scikit-learn that process 10K+ records reliably — lessons from reducing manual review workload by 45% at Oasis Infobyte.