Automating the Python Package Release Process

In modern Data Engineering and ML environments, consistently reaching out to the public internet (like PyPI) for every pipeline, test, or notebook spawn is not just a performance bottleneck—it introduces significant supply-chain risks. To address this, our engineering team deploys DevPi as an internal Python package proxy and caching index directly into our Kubernetes clusters.

This post breaks down our high-level architecture, the automated CI/CD lifecycle for custom packages, and the automated security janitor that actively keeps vulnerabilities out of our execution environments.

High-Level Architecture & Networking

At its core, our DevPi component provides a cluster-local index for all Python workloads. Instead of reaching external repositories directly, notebooks and pipeline jobs communicate with DevPi, which transparently caches necessary packages and serves locally hosted, custom libraries.

Architecture Highlights

• Service Layer: DevPi runs behind a Kubernetes ClusterIP service exposing ports 80 and 3141.
• Storage: Backed by an OBS/S3 volume (/devpi/files) for package artifacts and a PersistentVolumeClaim (/devpi/server) for fast metadata retrieval.
• Routing: Managed via an Istio Virtual Service, routing both ports to devpi.kubeflow.svc.cluster.local with a 120-second timeout, ensuring seamless in-cluster DNS resolution.

Automating the Release Process

Publishing internal code to DevPi is highly standardized using a comprehensive GitLab CI/CD release pipeline. We enforce strict quality and security checks before a package ever reaches the index:

• Linting: Fast, static code analysis and formatting checks using Ruff.
• Testing: Building the execution environment and running unit tests against package code via pytest.
• Static Security: Scanning third-party dependencies to generate a Software Bill of Materials (SBOM) to track supply chain compliance.
• Versioning: Automatic semantic versioning (MAJOR.MINOR.PATCH) calculation based on structured git commit messages.
• Build & Publish: Compiling binary wheels (.whl) and source distributions (.tar.gz) via Python’s build module, before pushing to DevPi using twine.

Active Security: The Trivy Janitor

Having an internal registry is excellent for speed, but hosting vulnerable code is dangerous. We built an automated, multi-tiered security flow centered around Trivy to actively scan and remediate our package cache.

1. Scheduled Scanning (The Hourly Cron)

A Kubernetes CronJob executes hourly to inspect the OBS-backed package cache. An init-container running Trivy scans the read-only mounted volume, reporting only CRITICAL severity vulnerabilities into a shared volume.

2. Event-Driven Scanning (Upload Triggers)

We do not wait for the hourly job if new packages arrive. We integrate Argo Events to monitor package uploads in real time:

• An EventSource exposes a webhook listening for OBS uploads on port 12000.
• A Sensor filters incoming events, reacting only to relevant artifacts ending in .whl, .tar.gz, or .zip.
• When a match is found, a one-off Kubernetes Job is triggered to immediately scan the new artifacts with Trivy.

3. Automated Remediation (The Janitor Script)

Scanning is only half the battle. Our custom Python janitor.py container takes over right after Trivy finishes. It parses the JSON output looking for vulnerable package names. If critical threats are found, the script automatically:

1. Applies a strict NetworkPolicy (isolate-notebooks) limiting ingress and egress traffic for the vulnerable environment (e.g., in the kubeflow namespace).
2. Updates the DevPi index configuration via the CLI to whitelist substitute the affected packages, preventing further pulls.

Developer Quickstart: Using DevPi

For developer environments and CI runners wanting to interact with the cache, environment variables easily point Python tooling to DevPi.

Configuration

import os

DEVPI_HOST = "devpi.kubeflow.svc.cluster.local"
DEVPI_PORT = "3141"

# Environment configuration for pip and twine
os.environ["PIP_INDEX_URL"] = f"http://{DEVPI_HOST}:{DEVPI_PORT}/root/pypi/+simple/"
os.environ["PIP_TRUSTED_HOST"] = DEVPI_HOST
os.environ["DEVPI_URL"] = f"http://{DEVPI_HOST}:{DEVPI_PORT}"
os.environ["DEVPI_INDEX_URL"] = f"http://{DEVPI_HOST}:{DEVPI_PORT}/root/dev/"
os.environ["TWINE_REPOSITORY_URL"] = f"http://{DEVPI_HOST}:{DEVPI_PORT}/root/dev/"

Installing Packages

Once configured, you can simply use standard pip with the formulated simple index endpoint:

import subprocess
import sys
import os

devpi_index_url = os.environ["DEVPI_INDEX_URL"]
trusted_host = os.getenv("PIP_TRUSTED_HOST", "devpi.kubeflow.svc.cluster.local")
simple_index_url = devpi_index_url.rstrip("/") + "/+simple/"

subprocess.run([
    sys.executable, "-m", "pip", "install",
    "--index-url", simple_index_url,
    "--trusted-host", trusted_host,
    "your-custom-package==0.1.0"
], check=True)

Building a Resilient Python Infrastructure

Bringing DevPi inside your Kubernetes boundary converts an unpredictable external dependency into a managed, high-speed utility. By pairing cluster-local package caching with an automated remediation loop driven by Trivy and Argo Events, engineering teams eliminate external bandwidth dependencies and active security risks simultaneously. This setup provides data science and engineering teams with the speed of local package management alongside the strict governance required for production enterprise platforms.