DATA DO – データ 道

Tag: LLM

  • Deterministic Control Over Non-Deterministic AI: Why Production Agents Demand PydanticAI and LangGraph

    In production services relying on raw string interpolation, unhandled execution faults stem directly from four output failure modes: dropped mandatory payload keys, string-to-float coercion failures, malformed JSON due to unescaped control characters, and hallucinated object keys. Parsing unstructured model outputs with standard string loaders guarantees runtime crashes downstream. Large language models (LLMs) operate as probabilistic…

  • Scaling Production RAG: High-Throughput Model Serving with NVIDIA Triton

    Building a Retrieval-Augmented Generation (RAG) prototype in a Jupyter notebook is straightforward. Scaling it to handle thousands of concurrent production users is a completely different engineering challenge. In previous posts, we focused heavily on optimizing the retrieval layer—from vector indexing strategies to unified Graph-RAG architectures in Postgres. However, once your data pipeline is optimized, the…

  • RAG Context Pruning for Efficiency and Cost Optimization

    After baseline production runs across our clients’ financial discovery pipelines, we observed an increase in Time-to-First-Token (TTFT) when retrieved context exceeded 2,500 tokens. Furthermore, the system’s retrieval accuracy score decayed when the target information was located in the middle 40% of the injected payload. We addressed this bottleneck by deploying an inline sentence-level extractive context…

  • Production Metric: 14.2% Semantic Decay

    After processing 2.8 million unstructured retail fragments, we observed that 14.2% of records passing traditional NOT NULL and regex constraints contained semantic noise specifically CAPTCHA text, “out of stock” redirects, and promotional modals that poisoned downstream RAG embeddings. We enforced a deterministic quality gate using PydanticAI and a sovereign vLLM cluster, which suppressed these failures…

  • Specialized Judges: Scaling RAG Evaluation with Prometheus-2 and PydanticAI

    Our production benchmarks utilize the Feedback Collection and Preference Collection datasets to establish the performance delta between generalist and specialized evaluators. We observed that Prometheus-2 (8x7B) achieves a Pearson correlation of $0.898$ with human-annotated ground truth, which is on par with GPT-4 ($0.882$) and significantly higher than previous iterations of small generalist models. By enforcing…

  • From Generalist to Specialist: Benchmarking the 25x Speedup of Fine-Tuned “Tiny Compilers”

    We measured a 96.7% reduction in inference latency by migrating our EDI logic from Llama 4 (70B) to a fine-tuned Llama 3.2 (1B) “Tiny Compiler.” In high-volume logistics testing, the generalist model averaged 2,800ms per transaction, while the specialized 1B model, quantized to 4-bit, stabilized at $92ms$ on consumer-grade hardware. We accept the 0.4% decay…

  • Part 4: The Human Interface — Enterprise RAG Deployment for 100+ Users

    1. Introduction: From Prototype to Enterprise Building a Retrieval-Augmented Generation (RAG) system that works on a laptop is a common starting point, but it is rarely enough for a corporate environment. Consequently, deploying it to handle 100+ concurrent employees each with unique access levels, real-time streaming requirements, and finite GPU resources represents an entirely different…

  • Part 2: The Multi-Step Retriever — Implementing Agentic Query Expansion

    1. Introduction: The Death of the “Simple Search” In Part 1, we defined the blueprint for a production-grade Agentic RAG system. We moved away from passive retrieval toward a “reasoning-first” architecture. But even the best reasoning engine fails if the data fed into it is garbage. When a business user asks, “What’s our policy on…

  • Modernizing Data Warehouses for AI: A 4-Step Roadmap

    It’s the same conversation in every boardroom and Slack channel: “How are we using LLMs? Where are our AI agents? When do we get our Copilot?” But for the teams in the trenches, the hype is hitting a wall of legacy infrastructure. The truth is that Modernizing Data Warehouses for AI is the invisible hurdle…

  • Designing Production-Grade GenAI Automation

    A dbt Ops Agent Case Study A small, well-instrumented workflow can turn dbt failures into reviewable Git changes by combining deterministic parsing, constrained LLM tooling, and VCS-native delivery — while preserving governance through traces, guardrails, and CI. This is a blueprint to build a first Production-Grade GenAI Agent. You can find the complete implementation and…