Sovereign AI at Scale: Orchestrating & Sandboxing Untrusted LLM Agents

As organizations rapidly transition from experimental LLMs to production-grade, multi-tenant AI agent architectures, open infrastructure faces critical, competing challenges regarding security and resource efficiency.

AI agents execute non-deterministic, untrusted code and pull unvetted third-party packages, making traditional shared-kernel Linux containers a massive security risk for host escape and data leakage. Meanwhile, GPUs are scarce and expensive; co-scheduling massive distributed training alongside dynamic, latency-sensitive inference workloads often leads to severe GPU fragmentation and low hardware utilization.

This talk presents a production-ready, open-source blueprint that solves these challenges by combining the latest OpenInfra advancements with cutting-edge scheduling research:

Sandboxing Untrusted AI Agents with Kata Containers: We explore how the latest Kata Containers release—with its Rust-based runtime promoted to the default implementation—provides hardware-level microVM isolation for AI workloads. We will demonstrate how Kata, combined with Confidential Containers, secures sensitive data and prevents host kernel escapes without sacrificing container-like deployment speeds.

Foundational GPU Orchestration with OpenStack: We discuss how OpenStack (utilizing Nova vGPU attachments, NUMA-aware placement, and Blazar for GPU instance reservations) acts as the robust, sovereign virtualization layer beneath Kubernetes.

Optimized Co-Scheduling: To maximize GPU utilization, we introduce a unified scheduling layer on Kubernetes (via Magnum or k0rdent) inspired by the recent Kant scheduling framework. We will detail how strategies like Enhanced Binpack and Backfill drastically reduce the GPU Node Fragmentation Ratio and optimize the GPU Allocation Ratio when running mixed training and inference workloads.

Attendees will walk away with a practical architectural guide to building a secure, highly efficient, and fully sovereign AI platform on open-source infrastructure.