Beating: Prime Intellect releases open-source inference service, processing nearly 1000000000000 Token

Article

Prime Intellect inference platform goes live: nearly 1000000000000 tokens processed internally every day

Beating AI news reports that Prime Intellect has released the open-source model inference service Prime Inference.

Users can call models directly on demand; if a task requires long-term stable operation, they can also lock in part of the inference capacity in advance. The API is compatible with OpenAI API, and the first publicly deployed model is GLM-5.3.

The system had previously been used internally at Prime Intellect. The company says it processes nearly 1000000000000 tokens every day for reinforcement learning, synthetic data, model evaluation, and Coding Agents. Since January this year, Prime Intellect has also been operating large-scale deployments for customers.

Prime Inference is optimized particularly for Agents' long-context workloads. It separates prompt processing and response generation across different GPU s, reducing p90-token inter-token latency by nearly 40%in testing. Prime Intellect also compresses the KV cache; with the same amount of VRAM, each decoder can cache tokens from 1090000 to 1630000, an increase of approximately 50%.

Prime Intellect previously provided training infrastructure including reinforcement learning, evaluation, and sandboxes. With the addition of inference services, it can connect model training and real-world deployment within the same platform, and data generated during actual operation can continue to be used for subsequent training.

Original link https://m.theblockbeats.info/flash/370242

Key points

Prime Intellect releases the open-source model inference service Prime Inference.
Users can call models on demand or lock in part of the inference capacity in advance.
The API is compatible with OpenAI API, and the first publicly deployed model is GLM-5.3。
The company says the system processes nearly 1000000000000 tokens every day for reinforcement learning, synthetic data, model evaluation, and Coding Agents.
In testing, the system reduced p90-token inter-token latency by nearly 40%。
With the same amount of VRAM, each decoder's cached tokens increased from 1090000 to 1630000, an increase of approximately 50%。
Prime Intellect says it has provided training infrastructure including reinforcement learning, evaluation, and sandboxes.

AI-assisted interpretation

The following is analysis, separate from reported facts. Verify important claims independently.

Prime Inference is a model inference service. Users can use models on demand or reserve capacity. It targets Agents' long-context tasks by assigning prompt processing and response generation to different GPU s, and compressing the KV cache.

Why it matters to readers

If performance in the public service is close to the test results, long-context Agent tasks may become more efficient; the platform also attempts to connect model training, deployment, and operational data.

It demonstrates AI integrated infrastructure for taking models from training to production use, as well as concepts such as latency, VRAM, and capacity reservation in inference services.

Risks and unknowns

The claim of nearly 1000000000000 tokens processed each day is an official disclosure and has not been independently verified.
Recent 40%of p90 token-to-token latency reduction and approximately 50%cache improvement came from testing; performance under public customer workloads has not been confirmed.
Public availability, pricing, and customer deployment status were not stated in the evidence.
GLM-5.3 Support for subsequent models beyond this was not specified.

Beating: Prime Intellect releases open-source inference service, processing nearly 1000000000000 Token

Article

AI-assisted interpretation

Why it matters to readers

Risks and unknowns

Related Developments

Related concepts

Keep following

Security Center