Project Guide | LLM On-Prem Deployment Kit

Live Demo

• Open the public GitHub Pages demo
• Scope: credential-free, synthetic-data demo for deployment reviewers and evaluators.

A customer-owned deployment baseline for evaluating private, hybrid, or air-gapped LLM infrastructure. It is designed for a bounded readiness sprint and must be adapted and validated before production use.

---

System Overview

A private/hybrid LLM deployment kit for organizations that cannot send sensitive workloads to uncontrolled hosted endpoints.

Evaluation Path

• Start here: Read the reference architecture, then jump to compliance mappings and airgap runbooks.
• Local demo: Use dry-run infrastructure validation rather than deploying by default.
• Checks: Run make validate; targeted checks are make tf-validate, make helm-lint, and make shell-lint.

Local Validation Tiers

make verify    # repository surface + architecture blueprint; requires Python only
make validate  # Terraform, Helm, and ShellCheck validation; requires the infra toolchain

If Terraform, Helm, or ShellCheck are installed outside PATH, pass explicit paths, for example:

make TERRAFORM=/path/to/terraform HELM=/path/to/helm SHELLCHECK=/path/to/shellcheck validate

Architecture Notes

• Architecture guide summarizes the system scope, first files to inspect, runtime commands, and known boundaries.
• Quality notes lists the local checks, CI surface, and release expectations for this repository.
• Enterprise readiness notes outlines security, data, operations, integration, and handoff expectations.

Table of Contents

1. Why this exists
2. What you get
3. Reference Architecture
4. Quick Start
5. Compliance Mappings
6. Runbooks
7. Choosing a Cloud
8. Security Model
9. Extending
10. Related Projects
11. License

---

Why this exists

Shipping large language model (LLM) applications to regulated customers is not a model problem. It is an *infrastructure* problem.

By the time a customer has decided to buy, their deployment team is asking questions that have little to do with tokens per second:

• "Can we run this inside our existing VNet with no egress to the public internet?"
• "Where does the encryption key live and who rotates it?"
• "How do we mirror the container images into our private registry?"
• "What does your disaster recovery look like, and can you show us an actual runbook?"
• "Does this map to our SOC 2 and ISO 27001 controls?"

llm-onprem-deployment-kit packages infrastructure-as-code, a Helm baseline, runbooks, and control-mapping aids for a customer-owned readiness sprint. It shortens architecture review and pilot setup; it does not certify that a rendered template is a running, auditable production system.

Target audience:

• Enterprise infrastructure teams responsible for deploying third-party AI workloads in regulated environments (financial services, healthcare, public sector, defense, energy).
• Forward deployed engineering teams at AI-native vendors who need a consistent deployment story across dozens of customer environments.
• Security and compliance approvers who need to map a proposed deployment to existing control frameworks before granting change-management approval.

If the question is "how fast can I get a demo LLM running on my laptop," this is not the right repository. If the question is "what must we validate before a private LLM pilot can run inside the customer's account," this is the intended starting point.

---

What you get

• Terraform across three clouds - Reviewable Azure AKS, AWS EKS, and GCP GKE module starters with private-cluster, GPU-pool, private-endpoint, and optional customer-managed encryption controls.
• A guarded Helm baseline - llm-stack connects Traefik to API-key-protected vLLM, keeps Qdrant single-node until distributed consensus is explicitly designed, and optionally renders OpenTelemetry and External Secrets resources.
• Airgap runbooks and tooling - scripts/airgap-mirror.sh enumerates every container image the kit ships with their pinned digests and mirrors them into a customer-controlled registry. A step-by-step runbook walks through the procedure.
• Compliance mappings - Explicit control-by-control mapping for SOC 2 Type II and ISO 27001:2022 Annex A, plus an "airgap requirements" summary that most procurement teams can consume directly.
• Incident response playbook - Severity levels, paging thresholds, customer handoff pattern, and a diagnostic-bundle collection script that produces the artifacts support actually needs.
• Architecture Decision Records - Documented rationale for every load-bearing choice (vLLM vs TGI, Qdrant vs Weaviate, K8s vs ECS, secrets model, airgap image strategy), so downstream teams can challenge or extend decisions without reverse-engineering them.
• Validation surface - Terraform validation, Helm lint/render/schema checks, runtime-contract checks, and ShellCheck. Passing CI proves those repository checks only.

---

Reference Architecture

flowchart LR
    subgraph CustomerEdge[Customer Edge]
        IdP[Customer IdP<br/>OIDC / SAML]
        Access[Customer API Gateway<br/>Identity + Quotas + Rate Limits]
        OpsUser[Operator / SRE]
    end

    subgraph PrivateNetwork[Private Network / VNet]
        subgraph IngressTier[Ingress Tier]
            PLB[Private Load Balancer]
            TRAEFIK[Traefik Reverse Proxy<br/>TLS + Routing]
        end

        subgraph K8sCluster[Kubernetes Cluster - Private Control Plane]
            subgraph AppTier[Application Tier]
                VLLM[vLLM Inference<br/>GPU Node Pool]
                QDRANT[Qdrant Vector DB<br/>Single StatefulSet Pod]
            end
            subgraph PlatformTier[Platform Tier]
                OTEL[OpenTelemetry Collector]
                ESO[External Secrets Operator]
            end
        end

        subgraph DataPlane[Data and Secrets Plane]
            KMS[Cloud KMS /<br/>Key Vault]
            VAULT[HashiCorp Vault]
            REGISTRY[Private Container Registry]
            LOG[Loki + Prometheus]
        end
    end

    OpsUser -->|Private endpoint| Access
    IdP --> Access
    Access --> PLB
    PLB --> TRAEFIK
    TRAEFIK --> VLLM
    VLLM --> OTEL
    QDRANT -. metrics .-> OTEL
    TRAEFIK -. metrics .-> OTEL
    OTEL --> LOG
    ESO --> VAULT
    VAULT --> KMS
    VLLM -.->|Pull images| REGISTRY
    QDRANT -.->|Pull images| REGISTRY
    TRAEFIK -.->|Pull images| REGISTRY

Notable properties of this architecture:

• Private intent, customer verification. Terraform and service annotations request private surfaces, but the customer must verify routes, DNS, firewall policy, and provider-specific load-balancer behavior.
• Customer-owned keys. KMS / Key Vault / Cloud KMS options keep key administration in the customer account; the sprint records who can use and rotate each key.
• Narrow built-in auth. vLLM validates one customer-owned API key. End-user identity, tenant policy, quotas, and rate limiting remain customer gateway responsibilities.
• Observable components, not audit completeness. The chart configures metric scraping and collector exporters. Per-request application audit, immutable retention, alerts, and evidence collection require customer integration and acceptance testing.

For the long form, see `docs/architecture.md`.

---

Quick Start

The quick start assumes (a) a workstation with terraform, helm, kubectl, and the cloud CLI of your choice; (b) permissions to create a VPC/VNet, a Kubernetes cluster, and a KMS key in the target subscription/account/project; and (c) a private container registry that the cluster can pull from.

For a full airgapped install with image mirroring, see `docs/runbooks/initial-deploy.md`.

1. Pick a cloud and provision infrastructure

## Azure example
cd terraform/modules/azure-aks/examples/basic
terraform init
terraform plan  -out=tfplan -var-file=../../../../../examples/quickstart-azure/terraform.tfvars
terraform apply tfplan

Equivalent commands exist under terraform/modules/aws-eks/examples/basic and terraform/modules/gcp-gke/examples/basic.

2. Mirror container images (airgapped environments only)

export TARGET_REGISTRY=registry.customer.internal/llm-stack
scripts/airgap-mirror.sh --target "$TARGET_REGISTRY" --dry-run
scripts/airgap-mirror.sh --target "$TARGET_REGISTRY"

3. Create customer-owned pilot secrets

The baseline fails closed if its API-key or TLS secret is absent. For an evaluation cluster, create them directly; a customer pilot should use the approved secret manager and ExternalSecret path.

kubectl create namespace llm-stack
kubectl -n llm-stack create secret generic llm-stack-inference-api-key \
  --from-literal=api-key="$(openssl rand -hex 32)"
openssl req -x509 -newkey rsa:3072 -nodes -days 30 \
  -subj '/CN=llm.internal.example.com' \
  -keyout /tmp/llm-stack.key -out /tmp/llm-stack.crt
kubectl -n llm-stack create secret tls llm-stack-tls \
  --key /tmp/llm-stack.key --cert /tmp/llm-stack.crt
rm -f /tmp/llm-stack.key /tmp/llm-stack.crt

4. Install the llm-stack Helm chart

helm upgrade --install llm-stack ./helm/llm-stack \
  --namespace llm-stack \
  --values ./helm/llm-stack/values.yaml \
  --atomic --timeout 10m

values-airgap.yaml is an overlay, not a turnkey install. It requires mirrored images and model weights, an approved ClusterSecretStore, customer observability endpoints, TLS material, network sources, storage, and an acceptance plan.

5. Verify

make status
scripts/smoke-test.sh --namespace llm-stack --release llm-stack

The smoke test checks the routed vLLM health path, sends an API-key-protected inference request, and checks the single Qdrant endpoint. It does not prove identity integration, quotas, rate limiting, high availability, backup recovery, immutable audit, or an SLO.

---

Compliance Mappings

This kit is designed to produce evidence that maps directly to common enterprise control frameworks. Full mappings live under `docs/compliance/`.

At-a-glance control summary:

---

Runbooks

Each runbook is written to be executable by an on-call engineer with working kubectl/cloud CLI access and no prior context.

---