{"id":893,"date":"2026-11-05T13:57:15","date_gmt":"2026-11-05T13:57:15","guid":{"rendered":"https:\/\/datascientists.info\/?p=893"},"modified":"2026-11-05T13:57:15","modified_gmt":"2026-11-05T13:57:15","slug":"duckdb-kubernetes-data-warehousing-pgvector","status":"publish","type":"post","link":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/","title":{"rendered":"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \/ pgvector"},"content":{"rendered":"\n

Cloud data warehouses often present teams with a difficult trade-off: either accept unpredictable SaaS compute bills or take on the operational overhead of running heavy distributed query engines like Trino or Spark. Furthermore, modern analytical platforms increasingly require bridging structured analytical files with operational relational records and vector embeddings for retrieval-augmented generation (RAG) and semantic similarity search.<\/p>\n\n\n\n

By deploying DuckDB<\/a> <\/strong>inside Kubernetes,<\/strong> you can achieve sub-second analytical queries directly across raw object storage (via httpfs<\/code>) and transactional\/vector data residing in PostgreSQL with pgvector<\/a><\/code> (via the postgres_scanner<\/code> and vss<\/code> extensions)\u2014combining an in-process vectorized engine with cloud-native scalability, zero vendor lock-in, and native Kubeflow integration.<\/p>\n\n\n\n

Here is an architectural deep dive into how to build this unified analytical and semantic query layer on Kubernetes.<\/p>\n\n\n\n

Architectural Overview<\/h2>\n\n\n\n

The core design principle is stateless compute decoupled from durable object storage and operational databases<\/strong>. The DuckDB engine pods retain no persistent disk state; instead, they execute vectorized SQL queries federating directly across remote Parquet files on S3 and vector tables in PostgreSQL via connection pooling.<\/p>\n\n\n\n

Component<\/strong><\/td>Resource \/ Technology<\/strong><\/td>Description<\/strong><\/td><\/tr><\/thead>
Service Endpoint<\/strong><\/td>Kubernetes ClusterIP \/ Ingress Service<\/td>Exposes REST and Arrow Flight SQL endpoints (\/api<\/code>) alongside the interactive Web UI (\/<\/code>) to Kubeflow Notebooks, pipeline tasks, and data science workloads.<\/td><\/tr>
DuckDB Engine Pod<\/strong><\/td>FastAPI \/ Python + DuckDB Native<\/td>Stateless query runner executing vectorized SQL queries, federated joins, and vector distance operations over HTTP range requests and Postgres sockets.<\/td><\/tr>
DuckDB Web UI Pod<\/strong><\/td>DuckDB-UI \/ Web GUI Container<\/td>Interactive web console for visual SQL exploration, schema inspection, and ad-hoc federated query execution.<\/td><\/tr>
Storage Connection<\/strong><\/td>S3 Bucket via httpfs<\/code><\/td>Streams Parquet metadata, footers, and filtered row groups directly from object storage without full-file downloads.<\/td><\/tr>
Relational & Vector Source<\/strong><\/td>PostgreSQL + pgvector<\/code> via postgres_scanner<\/code><\/td>Attached relational database storing operational entities and high-dimensional embeddings for similarity searches.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

Key Components in Detail<\/h2>\n\n\n\n

Storage Connections: Object Storage & PostgreSQL \/ pgvector<\/h3>\n\n\n\n

S3 via httpfs<\/code><\/h4>\n\n\n\n

Traditional data processing pipelines download entire dataset dumps to a local volume before executing SQL queries. DuckDB circumvents this via its native httpfs<\/code> extension<\/strong>, which communicates with S3-compatible Object Storage Services (MinIO, Ceph, AWS S3).<\/p>\n\n\n\n

When DuckDB reads a Parquet file over S3:<\/p>\n\n\n\n

    \n
  1. It issues targeted HTTP byte-range requests to fetch the Parquet file footer<\/strong> (schema, column statistics, dictionary pages).<\/li>\n\n\n\n
  2. It pushes down query projections and filter predicates ($min\/max$ metadata checks) to determine which row groups are needed.<\/li>\n\n\n\n
  3. It fetches only the required byte ranges<\/strong> directly into memory.<\/li>\n<\/ol>\n\n\n\n
    \"\"<\/figure>\n\n\n\n

    PostgreSQL + pgvector<\/code> via postgres_scanner<\/code><\/h4>\n\n\n\n

    Using the postgres_scanner<\/code> extension, DuckDB connects directly to PostgreSQL using the binary protocol to read tables on the fly without staging them to disk. When paired with pgvector<\/code>, DuckDB can read embeddings (stored as vector<\/code> or array types) and execute hybrid joins\u2014such as enriching historical parquet transaction logs with live user profile vectors and product similarity scores.<\/p>\n\n\n

    \nINSTALL httpfs;\nLOAD httpfs;\nINSTALL postgres_scanner;\nLOAD postgres_scanner;\n\n-- Configure S3 credentials\nSET s3_endpoint = 's3_endpoint_path';\nSET s3_url_style = 'path';\nSET s3_use_ssl = true;\nSET s3_access_key_id = '${STORAGE_ACCESS_KEY}';\nSET s3_secret_access_key = '${STORAGE_SECRET_KEY}';\n\n-- Attach PostgreSQL database dynamically\nCALL postgres_attach('dbname=analytics user=postgres host=postgres-service.analytics.svc.cluster.local password=${POSTGRES_PASSWORD} port=5432', source_schema='public');\n\n-- Federated Query: Join massive Parquet lakehouse logs with pgvector customer embeddings\nSELECT \n    l.customer_id,\n    p.customer_segment,\n    p.embedding,\n    COUNT(DISTINCT l.transaction_id) AS total_orders,\n    ROUND(SUM(l.sale_amount), 2) AS total_revenue\nFROM read_parquet('s3:\/\/lakehouse-gold\/sales\/year=2026\/*.parquet') l\nJOIN analytics.customer_profiles p ON l.customer_id = p.id\nWHERE l.status = 'COMPLETED'\nGROUP BY l.customer_id, p.customer_segment, p.embedding\nORDER BY total_revenue DESC\nLIMIT 50;\n\n<\/pre><\/div>\n\n\n
    The Engine Pod: Python Native + FastAPI<\/h3>\n\n\n\n
    The engine pod acts as an analytical proxy. Built with FastAPI and native DuckDB bindings, it accepts SQL payloads or structured parameter blocks, runs them against the engine, and returns results in Apache Arrow format or JSON.<\/p>\n\n\n\n
    Using Apache Arrow Flight SQL<\/strong> or raw Arrow IPC buffers drastically minimizes serialization latency. Unlike traditional JDBC\/ODBC connections that incur heavy CPU overhead converting tabular records to JSON\/tuples, Arrow retains zero-copy columnar data structures from DuckDB straight to downstream clients.<\/p>\n\n\n\n
    A lightweight production wrapper (app.py<\/code>):<\/p>\n\n\n
    \nimport duckdb\nfrom fastapi import FastAPI, HTTPException\nfrom pydantic import BaseModel\nimport pyarrow as pa\nimport io\nimport os\n\napp = FastAPI(title="DuckDB K8s Federated Engine")\n\nclass QueryPayload(BaseModel):\n    query: str\n\ndef get_connection():\n    con = duckdb.connect(database=":memory:")\n    \n    # Load storage and database extensions\n    con.execute("INSTALL httpfs; LOAD httpfs;")\n    con.execute("INSTALL postgres_scanner; LOAD postgres_scanner;")\n    \n    # Object storage configuration\n    con.execute(f"SET s3_endpoint = '{os.getenv('S3_ENDPOINT')}';")\n    con.execute(f"SET s3_access_key_id = '{os.getenv('S3_ACCESS_KEY')}';")\n    con.execute(f"SET s3_secret_access_key = '{os.getenv('S3_SECRET_KEY')}';")\n    \n    # Engine limits &amp; concurrency\n    con.execute(f"SET threads = {os.getenv('DUCKDB_THREADS', '4')};")\n    con.execute(f"SET max_memory = '{os.getenv('DUCKDB_MAX_MEMORY', '8GB')}';")\n    \n    # Attach Postgres &amp; pgvector source\n    pg_conn = (\n        f"dbname={os.getenv('POSTGRES_DB', 'analytics')} "\n        f"user={os.getenv('POSTGRES_USER', 'postgres')} "\n        f"host={os.getenv('POSTGRES_HOST', 'postgres-service.analytics.svc.cluster.local')} "\n        f"password={os.getenv('POSTGRES_PASSWORD')} "\n        f"port={os.getenv('POSTGRES_PORT', '5432')}"\n    )\n    con.execute(f"CALL postgres_attach('{pg_conn}', source_schema='public');")\n    return con\n\n@app.post("\/api\/query")\ndef execute_query(payload: QueryPayload):\n    try:\n        con = get_connection()\n        arrow_table: pa.Table = con.execute(payload.query).arrow()\n        \n        # Serialize Arrow table to IPC stream\n        sink = io.BytesIO()\n        with pa.ipc.new_stream(sink, arrow_table.schema) as writer:\n            writer.write_table(arrow_table)\n            \n        con.close()\n        return sink.getvalue()\n    except Exception as e:\n        raise HTTPException(status_code=400, detail=str(e))\n\n<\/pre><\/div>\n\n\n
    Visual Exploration: DuckDB Web UI Pod<\/h3>\n\n\n\n
    For data scientists and analysts accustomed to browser-based SQL consoles, pairing the stateless engine with a dedicated UI container provides parity with modern cloud data warehouse interfaces.<\/p>\n\n\n\n
    The UI container connects to the query engine or mounts identical httpfs<\/code> and Postgres credentials, allowing users to:<\/p>\n\n\n\n
      \n
    • Inspect remote schemas and explore partitions across S3 paths.<\/li>\n\n\n\n
    • Inspect PostgreSQL catalog tables, view metadata, and check vector column dimensions directly.<\/li>\n\n\n\n
    • Interactively test filter predicates, joins, and aggregations before committing code to production DAGs.<\/li>\n\n\n\n
    • Download ad-hoc result sets directly as Parquet or CSV.<\/li>\n<\/ul>\n\n\n\n
      Kubernetes Service & Ingress Routing<\/h3>\n\n\n\n
      A standard Kubernetes Deployment<\/code> coupled with a unified Ingress<\/code> exposes both components under a single domain, isolating operational concerns:<\/p>\n\n\n
      \napiVersion: v1\nkind: Service\nmetadata:\n  name: duckdb-service\n  namespace: analytics\nspec:\n  type: ClusterIP\n  ports:\n    - name: http-engine\n      port: 8000\n      targetPort: 8000\n    - name: http-ui\n      port: 8080\n      targetPort: 8080\n  selector:\n    app: duckdb-warehouse\n---\napiVersion: networking.k8s.io\/v1\nkind: Ingress\nmetadata:\n  name: duckdb-ingress\n  namespace: analytics\n  annotations:\n    kubernetes.io\/ingress.class: nginx\nspec:\n  rules:\n  - host: warehouse.internal.data-do.de\n    http:\n      paths:\n      - path: \/api\n        pathType: Prefix\n        backend:\n          service:\n            name: duckdb-service\n            port:\n              number: 8000\n      - path: \/\n        pathType: Prefix\n        backend:\n          service:\n            name: duckdb-service\n            port:\n              number: 8080\n\n<\/pre><\/div>\n\n\n
      Operational Advantages<\/h2>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
        \n
      1. Near-Zero Idle Cost:<\/strong> Unlike multi-node warehouses that require always-on coordinator and worker nodes, DuckDB engine pods can be scaled down to zero or scaled dynamically using the Kubernetes Horizontal Pod Autoscaler (HPA) or KEDA based on HTTP request volume.<\/li>\n\n\n\n
      2. Predictable Memory Footprint:<\/strong> DuckDB adheres strictly to its configured max_memory<\/code> boundary, utilizing out-of-core disk spillover (backed by ephemeral K8s emptyDir<\/code> volumes) to prevent OOMKilled<\/code> termination during large joins or vector transformations.<\/li>\n\n\n\n
      3. True Hybrid Analytics (Lake + Relational + Vectors):<\/strong> By federating Parquet lakehouse storage with pgvector<\/code> inside PostgreSQL, analysts can evaluate vector embeddings alongside historical data without requiring separate vector synchronization pipelines or specialized graph databases.<\/li>\n\n\n\n
      4. Reproducibility with Kubeflow:<\/strong> Pipeline tasks executed in Kubeflow can call the internal [http:\/\/duckdb-service.analytics.svc.cluster.local:8000\/api\/query](http:\/\/duckdb-service.analytics.svc.cluster.local:8000\/api\/query)<\/code> endpoint, generating training feature tables directly from raw lakehouse files and live database state without custom driver configurations.<\/li>\n<\/ol>\n\n\n\n
        Running DuckDB on Kubernetes decouples data volume from compute cost, giving engineering teams a high-performance, compliant, and cost-efficient analytical foundation without the overhead of heavy distributed engines.<\/p>\n","protected":false},"excerpt":{"rendered":"
        Cloud data warehouses often present teams with a difficult trade-off: either accept unpredictable SaaS compute bills or take on the operational overhead of running heavy distributed query engines like Trino or Spark. Furthermore, modern analytical platforms increasingly require bridging structured analytical files with operational relational records and vector embeddings for retrieval-augmented generation (RAG) and semantic […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_ppma_block_editor_authors":"{\"authors\":[144,145],\"author_categories\":{\"144\":\"1\",\"145\":\"1\"},\"fallback_author_user\":\"1\",\"ppma_author_box_select\":\"\",\"selected_authors\":[{\"id\":144,\"display_name\":\"Marc Matt\",\"is_guest\":0,\"category_id\":\"1\"},{\"id\":145,\"display_name\":\"saidah\",\"is_guest\":0,\"category_id\":\"1\"}]}"},"categories":[2,3,125,4,6],"tags":[126,29,179],"ppma_author":[144,145],"class_list":["post-893","post","type-post","status-publish","format-standard","hentry","category-analytics-platform","category-big-data","category-data-engineering","category-data-lake","category-data-warehouse","tag-data-engineering","tag-data-science","tag-duckdb","author-marc","author-saidah"],"yoast_head":"\nServerless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \/ pgvector - DATA DO - \u30c7\u30fc\u30bf \u9053<\/title>\n<meta name=\"description\" content=\"Deploy a lean, stateless data warehouse on Kubernetes. Learn how to federate DuckDB queries across S3 Parquet lakes and PostgreSQL pgvector embeddings.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \/ pgvector - DATA DO - \u30c7\u30fc\u30bf \u9053\" \/>\n<meta property=\"og:description\" content=\"Deploy a lean, stateless data warehouse on Kubernetes. Learn how to federate DuckDB queries across S3 Parquet lakes and PostgreSQL pgvector embeddings.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/\" \/>\n<meta property=\"og:site_name\" content=\"DATA DO - \u30c7\u30fc\u30bf \u9053\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/DataScientists\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-05T13:57:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/datascientists.info\/wp-content\/uploads\/2026\/10\/image-1.png\" \/>\n<meta name=\"author\" content=\"Marc Matt, saidah\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Marc Matt\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/\"},\"author\":{\"name\":\"Marc Matt\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/#\\\/schema\\\/person\\\/723078870bf3135121086d46ebb12f19\"},\"headline\":\"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \\\/ pgvector\",\"datePublished\":\"2026-11-05T13:57:15+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/\"},\"wordCount\":827,\"publisher\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/datascientists.info\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/image-1.png\",\"keywords\":[\"Data Engineering\",\"Data Science\",\"DuckDB\"],\"articleSection\":[\"Analytics Platform\",\"Big Data\",\"Data Engineering\",\"Data Lake\",\"Data Warehouse\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/\",\"url\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/\",\"name\":\"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \\\/ pgvector - DATA DO - \u30c7\u30fc\u30bf \u9053\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/datascientists.info\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/image-1.png\",\"datePublished\":\"2026-11-05T13:57:15+00:00\",\"description\":\"Deploy a lean, stateless data warehouse on Kubernetes. Learn how to federate DuckDB queries across S3 Parquet lakes and PostgreSQL pgvector embeddings.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/#primaryimage\",\"url\":\"https:\\\/\\\/datascientists.info\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/image-1.png\",\"contentUrl\":\"https:\\\/\\\/datascientists.info\\\/wp-content\\\/uploads\\\/2026\\\/10\\\/image-1.png\",\"width\":843,\"height\":810},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/index.php\\\/2026\\\/11\\\/05\\\/duckdb-kubernetes-data-warehousing-pgvector\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/datascientists.info\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \\\/ pgvector\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/#website\",\"url\":\"https:\\\/\\\/datascientists.info\\\/\",\"name\":\"Data Scientists\",\"description\":\"Digging data, Big Data, Analysis, Data Mining\",\"publisher\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/datascientists.info\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/#organization\",\"name\":\"DATA DO - \u30c7\u30fc\u30bf \u9053\",\"url\":\"https:\\\/\\\/datascientists.info\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/datascientists.info\\\/wp-content\\\/uploads\\\/2026\\\/02\\\/Bildschirmfoto-vom-2026-02-02-08-13-21.png\",\"contentUrl\":\"https:\\\/\\\/datascientists.info\\\/wp-content\\\/uploads\\\/2026\\\/02\\\/Bildschirmfoto-vom-2026-02-02-08-13-21.png\",\"width\":250,\"height\":174,\"caption\":\"DATA DO - \u30c7\u30fc\u30bf \u9053\"},\"image\":{\"@id\":\"https:\\\/\\\/datascientists.info\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/DataScientists\\\/\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/datascientists.info\\\/#\\\/schema\\\/person\\\/723078870bf3135121086d46ebb12f19\",\"name\":\"Marc Matt\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/74f48ef754cf04f628f42ed117a3f2b42931feeb41a3cca2313b9714a7d4fdd2?s=96&d=mm&r=g53b84b5f47a2156ba8b047d71d6d05fc\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/74f48ef754cf04f628f42ed117a3f2b42931feeb41a3cca2313b9714a7d4fdd2?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/74f48ef754cf04f628f42ed117a3f2b42931feeb41a3cca2313b9714a7d4fdd2?s=96&d=mm&r=g\",\"caption\":\"Marc Matt\"},\"description\":\"Senior Data Architect with 15+ years of experience helping Hamburg's leading enterprises modernize their data infrastructure. I bridge the gap between legacy systems (SAP, Hadoop) and modern AI capabilities. I help clients: Migrate & Modernize: Transitioning on-premise data warehouses to Google Cloud\\\/AWS to reduce costs and increase agility. Implement GenAI: Building secure RAG (Retrieval-Augmented Generation) pipelines to unlock value from internal knowledge bases using LangChain and Vector DBs. Scale MLOps: Operationalizing machine learning models from PoC to production with Kubernetes and Airflow. Proven track record leading engineering teams.\",\"sameAs\":[\"https:\\\/\\\/data-do.de\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \/ pgvector - DATA DO - \u30c7\u30fc\u30bf \u9053","description":"Deploy a lean, stateless data warehouse on Kubernetes. Learn how to federate DuckDB queries across S3 Parquet lakes and PostgreSQL pgvector embeddings.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/","og_locale":"en_US","og_type":"article","og_title":"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \/ pgvector - DATA DO - \u30c7\u30fc\u30bf \u9053","og_description":"Deploy a lean, stateless data warehouse on Kubernetes. Learn how to federate DuckDB queries across S3 Parquet lakes and PostgreSQL pgvector embeddings.","og_url":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/","og_site_name":"DATA DO - \u30c7\u30fc\u30bf \u9053","article_publisher":"https:\/\/www.facebook.com\/DataScientists\/","article_published_time":"2026-11-05T13:57:15+00:00","og_image":[{"url":"https:\/\/datascientists.info\/wp-content\/uploads\/2026\/10\/image-1.png","type":"","width":"","height":""}],"author":"Marc Matt, saidah","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Marc Matt","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/#article","isPartOf":{"@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/"},"author":{"name":"Marc Matt","@id":"https:\/\/datascientists.info\/#\/schema\/person\/723078870bf3135121086d46ebb12f19"},"headline":"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \/ pgvector","datePublished":"2026-11-05T13:57:15+00:00","mainEntityOfPage":{"@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/"},"wordCount":827,"publisher":{"@id":"https:\/\/datascientists.info\/#organization"},"image":{"@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/#primaryimage"},"thumbnailUrl":"https:\/\/datascientists.info\/wp-content\/uploads\/2026\/10\/image-1.png","keywords":["Data Engineering","Data Science","DuckDB"],"articleSection":["Analytics Platform","Big Data","Data Engineering","Data Lake","Data Warehouse"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/","url":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/","name":"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \/ pgvector - DATA DO - \u30c7\u30fc\u30bf \u9053","isPartOf":{"@id":"https:\/\/datascientists.info\/#website"},"primaryImageOfPage":{"@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/#primaryimage"},"image":{"@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/#primaryimage"},"thumbnailUrl":"https:\/\/datascientists.info\/wp-content\/uploads\/2026\/10\/image-1.png","datePublished":"2026-11-05T13:57:15+00:00","description":"Deploy a lean, stateless data warehouse on Kubernetes. Learn how to federate DuckDB queries across S3 Parquet lakes and PostgreSQL pgvector embeddings.","breadcrumb":{"@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/#primaryimage","url":"https:\/\/datascientists.info\/wp-content\/uploads\/2026\/10\/image-1.png","contentUrl":"https:\/\/datascientists.info\/wp-content\/uploads\/2026\/10\/image-1.png","width":843,"height":810},{"@type":"BreadcrumbList","@id":"https:\/\/datascientists.info\/index.php\/2026\/11\/05\/duckdb-kubernetes-data-warehousing-pgvector\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/datascientists.info\/"},{"@type":"ListItem","position":2,"name":"Serverless Analytics & Vector Search on Kubernetes: Building a Lean Data Warehouse with DuckDB, Object Storage, and PostgreSQL \/ pgvector"}]},{"@type":"WebSite","@id":"https:\/\/datascientists.info\/#website","url":"https:\/\/datascientists.info\/","name":"Data Scientists","description":"Digging data, Big Data, Analysis, Data Mining","publisher":{"@id":"https:\/\/datascientists.info\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/datascientists.info\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/datascientists.info\/#organization","name":"DATA DO - \u30c7\u30fc\u30bf \u9053","url":"https:\/\/datascientists.info\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/datascientists.info\/#\/schema\/logo\/image\/","url":"https:\/\/datascientists.info\/wp-content\/uploads\/2026\/02\/Bildschirmfoto-vom-2026-02-02-08-13-21.png","contentUrl":"https:\/\/datascientists.info\/wp-content\/uploads\/2026\/02\/Bildschirmfoto-vom-2026-02-02-08-13-21.png","width":250,"height":174,"caption":"DATA DO - \u30c7\u30fc\u30bf \u9053"},"image":{"@id":"https:\/\/datascientists.info\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/DataScientists\/"]},{"@type":"Person","@id":"https:\/\/datascientists.info\/#\/schema\/person\/723078870bf3135121086d46ebb12f19","name":"Marc Matt","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/74f48ef754cf04f628f42ed117a3f2b42931feeb41a3cca2313b9714a7d4fdd2?s=96&d=mm&r=g53b84b5f47a2156ba8b047d71d6d05fc","url":"https:\/\/secure.gravatar.com\/avatar\/74f48ef754cf04f628f42ed117a3f2b42931feeb41a3cca2313b9714a7d4fdd2?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/74f48ef754cf04f628f42ed117a3f2b42931feeb41a3cca2313b9714a7d4fdd2?s=96&d=mm&r=g","caption":"Marc Matt"},"description":"Senior Data Architect with 15+ years of experience helping Hamburg's leading enterprises modernize their data infrastructure. I bridge the gap between legacy systems (SAP, Hadoop) and modern AI capabilities. I help clients: Migrate & Modernize: Transitioning on-premise data warehouses to Google Cloud\/AWS to reduce costs and increase agility. Implement GenAI: Building secure RAG (Retrieval-Augmented Generation) pipelines to unlock value from internal knowledge bases using LangChain and Vector DBs. Scale MLOps: Operationalizing machine learning models from PoC to production with Kubernetes and Airflow. Proven track record leading engineering teams.","sameAs":["https:\/\/data-do.de"]}]}},"authors":[{"term_id":144,"user_id":1,"is_guest":0,"slug":"marc","display_name":"Marc Matt","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/74f48ef754cf04f628f42ed117a3f2b42931feeb41a3cca2313b9714a7d4fdd2?s=96&d=mm&r=g","avatar_alt":"","avatar_decorative":"","author_category":"1","first_name":"Marc","last_name":"Matt","user_url":"https:\/\/data-do.de","job_title":"Senior Data Architect | GenAI & RAG Expert | GCP \/ AWS","description":"Senior Data Architect with 15+ years of experience helping Hamburg's leading enterprises modernize their data infrastructure. I bridge the gap between legacy systems (SAP, Hadoop) and modern AI capabilities.\r\n\r\nI help clients:\r\n\r\n \tMigrate & Modernize: Transitioning on-premise data warehouses to Google Cloud\/AWS to reduce costs and increase agility.\r\n\r\n\r\n \tImplement GenAI: Building secure RAG (Retrieval-Augmented Generation) pipelines to unlock value from internal knowledge bases using LangChain and Vector DBs.\r\n \tScale MLOps: Operationalizing machine learning models from PoC to production with Kubernetes and Airflow.\r\n\r\nProven track record leading engineering teams."},{"term_id":145,"user_id":2,"is_guest":0,"slug":"saidah","display_name":"saidah","avatar_url":"https:\/\/secure.gravatar.com\/avatar\/015737c94dd80772d772f2b24a55e96c868068f28684c8577d9492f3313e4dd3?s=96&d=mm&r=g","avatar_alt":"","avatar_decorative":"","author_category":"","first_name":"Saidah","last_name":"","user_url":"http:\/\/data-do.de","job_title":"","description":""}],"_links":{"self":[{"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/posts\/893","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/comments?post=893"}],"version-history":[{"count":3,"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/posts\/893\/revisions"}],"predecessor-version":[{"id":900,"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/posts\/893\/revisions\/900"}],"wp:attachment":[{"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/media?parent=893"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/categories?post=893"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/tags?post=893"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/datascientists.info\/index.php\/wp-json\/wp\/v2\/ppma_author?post=893"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}