Implement supply chain security for AI/ML pipelines and model artifacts
Implementation Effort: High – Requires coordination across data science, platform, and security teams. User Impact: Low – Signing, checksum, and provenance controls for models and datasets are build-time actions for ML and platform teams; end users are not affected. Lifecycle Stage: Build
Overview
Apply software supply chain controls to AI and machine learning artifacts in three steps: (1) govern model and dataset registries, (2) verify integrity for models, datasets, and pipeline components, and (3) enforce provenance gates before promotion to production.
AI and machine learning workloads introduce supply chain risks that traditional DevSecOps controls do not address. Training pipelines consume datasets, pre-trained models, and specialized libraries from external sources — each of which can be poisoned, tampered with, or backdoored without detection if treated as trusted inputs.
Apply these controls in three stages. First, govern model and dataset registries by registering all models and governed datasets in approved registries, such as Azure Machine Learning Model Registry or an internal registry with versioning, access control, lineage, and audit logging. Second, verify integrity before use: if you package models as OCI artifacts — such as container images in Azure Container Registry — sign and verify them using Notation or Cosign, and for non-Azure registries apply equivalent signing and verification controls. Third, enforce provenance gates before deployment; for Azure Machine Learning registry artifacts, enforce integrity through checksums, lineage metadata, and CI/CD gates that verify approved provenance before promotion, and because Azure Machine Learning registry controls do not provide container-admission-control-like signature enforcement, those CI/CD gates are essential.
Generate or retain provenance records for training inputs, build steps, and model packaging outputs so you can prove where a deployed model came from and detect tampering before release.
ML pipeline components — custom training containers, feature engineering scripts, and evaluation harnesses — should be subject to the same controls as CI/CD pipeline components: pin dependencies to immutable references, scan container images for vulnerabilities, and enforce least-privilege access for pipeline service identities.
Avoid Python pickle and joblib formats for model serialization in production because they allow arbitrary code execution during deserialization, meaning a tampered model file can execute malicious code when loaded. For PyTorch models, use safetensors (the Hugging Face safe serialization format) as the primary recommendation. For cross-framework compatibility, use ONNX, and for TensorFlow models use the TensorFlow SavedModel format. All of these formats prevent code execution on load and support integrity verification.
When consuming pre-trained models from external hubs, verify model checksums match published values, review model cards for known issues, and test models in isolated environments before production deployment. Maintain cryptographic checksums (SHA-256) for all training datasets and generate SBOM or equivalent dependency inventories for model-serving packages and containers.
Implement automated behavioral validation that tests model outputs for expected behaviors and adversarial robustness — signature verification proves the model file is unmodified but does not guarantee the model behavior is safe. For agentic AI systems that execute code, call tools, or interact with external services, extend validation to include AI-specific red team exercises that probe for prompt injection, tool misuse, data exfiltration, and unintended autonomous actions. Microsoft's AI red teaming guidance and open-source safety tooling provide frameworks for systematic adversarial testing of agent workflows before production deployment.
Treating external models, datasets, and libraries as untrusted inputs applies the Assume breach principle, while signing, checksum, and provenance verification enforce Verify explicitly before any artifact is promoted to production.