General Tech

Show HN: TrnRT – An open-source LLM inference engine for AWS Trainium

TrnRT is an open-source LLM inference engine for AWS Trainium that serves GLM-5.3 flash on a single trn2.48xlarge instance, achieving 3.57M tokens per minute on SemiAnalysis's AgentX benchmarks. It uses NKI megakernels and keeps a prefix cache across device and host memory. The engine is open to contributions and can be tried on a trn2.48xlarge with Docker and an x86_64 host. The authors ran AgentX benchmarks with 32 concurrent agents and a 1M-token context, achieving a batch 12 cost of $0.167 per million tokens.

Read the full article at github.com →