As organisations rapidly transition from experimental large language models (LLMs) to production-grade, multi-tenant AI agent architectures, open infrastructure faces critical, competing challenges relating to security and resource efficiency.
AI agents execute non-deterministic, untrusted code and pull unvetted third-party packages, making traditional shared-kernel Linux containers a massive security risk in terms of host escape and data leakage. Meanwhile, GPUs are in short supply and expensive; co-scheduling large-scale distributed training alongside dynamic, latency-sensitive inference workloads often leads to severe GPU fragmentation and low hardware utilisation.
This talk presents a production-ready, open-source blueprint that addresses these challenges by combining the latest OpenInfra advancements with cutting-edge research into scheduling:
Sandboxing Untrusted AI Agents with Kata Containers: We explore how the latest release of Kata Containers—with its Rust-based runtime now the default implementation—provides hardware-level microVM isolation for AI workloads. We will demonstrate how Kata, combined with Confidential Containers, secures sensitive data and prevents host kernel escapes without compromising container-like deployment speeds.
Foundational GPU Orchestration with OpenStack: We discuss how OpenStack (using Nova vGPU attachments, NUMA-aware placement and Blazar for GPU instance reservations) acts as the robust, sovereign virtualisation layer underlying Kubernetes.
Optimised Co-Scheduling: To maximise GPU utilisation, we introduce a unified scheduling layer on Kubernetes (via Magnum or k0rdent) inspired by the recent Kant scheduling framework. We will explain in detail how strategies such as Enhanced Binpack and Backfill drastically reduce the GPU node fragmentation ratio and optimise the GPU allocation ratio when running mixed training and inference workloads.
Attendees will leave with a practical architectural guide to building a secure, highly efficient and fully sovereign AI platform on open-source infrastructure.
