RunInfra

What is RunInfra?

RunInfra is a platform designed to provide transparency and control over AI model infrastructure. Unlike closed-source APIs that obscure the underlying model and infrastructure, RunInfra allows users to benchmark open models against their own latency, throughput, and cost targets. Users can export the stack or run it in their own cloud, ensuring complete ownership of their infrastructure.

The platform offers a range of features aimed at optimizing the deployment of AI models, including:

End-to-end encryption for enhanced security

Isolated GPU infrastructure to ensure performance

No training on user data, maintaining privacy

SOC 2 Type II compliance for security assurance

RunInfra Features

RunInfra offers a comprehensive solution for deploying and optimizing machine learning models with a focus on transparency and control. Users can benchmark open models against their own latency, throughput, and cost targets, ensuring that they achieve the best performance for their specific needs. The platform allows for the export of the entire stack or the option to run it in a user's own cloud environment, providing full ownership of the infrastructure. Additionally, RunInfra provides managed hosting as a convenient option, complete with a free deployment kit for easy exit.

Key features and capabilities of RunInfra include:

End-to-end encryption for data security.

Isolated GPU infrastructure to ensure optimal performance.

No training on user data, maintaining privacy and security.

Support for deploying pipelines as REST APIs with one-click deployment.

Benchmarking tools to measure latency, throughput, VRAM, and cost.

Optimization agents that apply compatible runtime settings and enhancements.

Why RunInfra?

RunInfra offers a unique value proposition by providing transparency and control over your machine learning models and infrastructure. Unlike closed-source APIs that obscure the underlying model and infrastructure, RunInfra allows users to see both, enabling them to benchmark open models against their own latency, throughput, and cost targets. This level of visibility ensures that users can make informed decisions about their deployments and optimize their performance effectively.

Additionally, RunInfra prioritizes data security and user autonomy. With end-to-end encryption and isolated GPU infrastructure, your inference data remains secure and is never used for training purposes. Users have the option to export their stack and run it in their own cloud, ensuring complete ownership of their data and models. The managed hosting option provides convenience, along with a free deployment kit as an exit strategy.

Transparent access to models and infrastructure

Benchmarking against custom latency and cost targets

End-to-end encryption and isolated infrastructure for data security

Option to export and run in your own cloud

Convenient managed hosting with a free deployment kit

How to Use RunInfra

To get started with RunInfra, you can build your first pipeline by simply typing what you want to run, such as 'a support copilot with Whisper and Qwen, tuned for specific tasks.' RunInfra will then check for compatible serving engines and GPU targets, benchmarking latency, throughput, VRAM, and cost to ensure optimal performance.

Once you have your pipeline set up, RunInfra applies compatible optimizations, including supported runtime settings, batching, quantization, KV cache, kernel, and routing optimizations. After applying these settings, you can review the evidence through a benchmark receipt that details the measured performance, cost, GPU fit, and reproduction notes. Finally, you can choose to deploy the measured configuration on RunInfra Cloud or export the runnable code, Docker, Kubernetes, and runbook artifacts.

Easy setup by typing your desired application.

Automatic benchmarking of performance metrics.

Support for various optimizations tailored to your model.

Options to deploy or export your configuration seamlessly.

Ready to see what RunInfra can do for you?[@portabletext/react] Unknown block type "span", specify a component for it in the `components.types` propand experience the benefits firsthand.