DATA DO – データ 道

Category: Analytics Platform

  • Beyond Pure Vectors: Implementing High-Precision Hybrid Search Natively inside Postgres

    We observed a 500ms synchronization lag when separating our storage engine from an external search cluster, which causes broken user experiences when resolving exact alphanumeric queries. Dense embeddings capture abstract semantic meaning but fail at exact keyword matching, causing queries for precise serial numbers like X-992-B, legacy product SKUs, or industry-specific jargon like Kubernetes to…

  • From Notebook to Production: Building End-to-End ML Pipelines with Kubeflow, KServe, and Fractional GPU Sharing

    Transitioning a machine learning model from an experimental Jupyter Notebook to a highly available, auto-scaling production endpoint is rarely a linear path. In an enterprise environment, this migration introduces severe operational friction—not just in terms of rewriting code, but in handling infrastructure efficiency. With modern workloads demanding massive compute resources, assigning an entire enterprise-grade GPU…

  • Start Fresh, Don’t Lift and Shift: Scaling Analytics Platforms with dbt-core and PostgreSQL

    We observed that executing a “lift and shift” of legacy, sprawling SQL scripts onto an enterprise cloud data warehouse fails to resolve core structural data issues. It transitions architectural technical debt into a variable, unconstrained operational expense. Moving unoptimized queries onto infinite-compute cloud platforms masks underlying engineering deficiencies rather than fixing them. We reject this…

  • PostgreSQL Data Mesh: A Technical Guide to Schema Segmentation, Boundaries, and Governance

    We deploy PostgreSQL natively to execute a decentralized data mesh architecture, proving that multi-million dollar cloud platforms and proprietary vendor ecosystems are infrastructure bloat. By utilizing open-source database primitives, we eliminate dependencies on specific tech conglomerates and cloud provider pricing models. We enforce domain boundaries, query allocations, and data product contracts directly through the PostgreSQL…

  • Apache Nifi on Google Cloud Kubernetes Engine (GKE)

    Apache Nifi on Google Cloud Kubernetes Engine (GKE)

    Apache Nifi on GKE can be a good solution, if you want to have a low code solution for processing streaming data. If you set it up on GKE, a managed version of Kubernetes, you have a managed scalable environment and do not need to worry about handling the actual servers. Setup of the Apache…

  • Data Infrastructure in the Cloud

    Data Infrastructure in the Cloud

    Having your data infrastructure in the cloud has become a real option for a lot of companies, especially since the big cloud providers have a lot of managed services available for a modern data architecture aside from just a database management system.

  • Bringing machine learning models into production

    Developing and bringing machine learning models into production is a task with a lot of challenges. These include model and attribute selection, dealing with missing values, normalization and others. Finding a workflow that puts all the gears, from data preprocessing and analysis over building models and selecting the best performing one to serving the model…

  • AVRO schema generation with reusable fields

    Why use AVRO and AVRO Schema? There are several serialized file formats out there, so chosing the one most suited for your needs is crucial. This blog entry will not compare them, but it will just point out some advantages of AVRO and AVRO Schema for an Apache Hadoop ™ based system. Avro schema can…

  • Plumber: Getting R ready for production environments?

    R Project and Production Running R Project in production is a controversially discussed topic, as is everything concerning R vs Python. Lately there have been some additions to the R Project, that made me look into this again. Researching R and its usage in production environments I came across several packages / project, that can…

  • Analytics Platform: An Evolution from Data Lake

    Analytics Platform Having built a Data Lake for your company’s analytical needs, there soon will arise new use cases, that cannot be easily covered with the Data Lake architecture I covered in previous posts, like Apache HAWQ™: Building an easily accessable Data Lake. You will need to adapt or enhance your architecture to become more…