Yusuf Ganiyu on real-time ML, feature freshness, agentic AI, BigQuery, and Gemini. ClickHouse Cloud — Packt DataPro ClickHouse Sponsored by ClickHouse via AWS Marketplace Analytical Database Your queries shouldn't take seconds.They should take milliseconds. ClickHouse Cloud is the OLAP database engineered for data teams who process billions of rows and need answers now — not after a coffee break. ~1s query latency on10M+ row datasets 50–75% cost savings vs.Redshift & BigQuery 10x data compressionon raw ingestion Data engineers running real-time pipelines know the pain: you've got Kafka ingesting 100GB/day, dashboards that need sub-second refresh, and a warehouse bill that grows every quarter. ClickHouse was built to solve exactly that. Why data teams switch to ClickHouse: → Columnar storage with MergeTree engines — aggregations on billions of rows in seconds → Native connectors for Kafka, Python, Pandas, Grafana, Power BI, Tableau, and AWS SageMaker → Built-in vector search for AI/ML pipelines — no separate embedding store needed → Automatic scaling — handles variable workloads without manual provisioning → ISO27001 & HIPAA certified — compliance-ready for regulated data environments "ClickHouse delivers results in 1 second versus 6–7 seconds on PostgreSQL and MongoDB for the same data. It saves our team 4–5 hours per person, per week." — Verified AWS Marketplace Customer Review ClickHouse Cloud is available on AWS Marketplace — deploy in minutes, pay only for what you use, and start with $300 in free credits. No provisioning, no DBA overhead, no lock-in. Start Free — $300 in Credits → View ClickHouse Cloud on AWS Marketplace This is a message delivered in partnership with ClickHouse via AWS Marketplace,delivered via Packt DataPro.
(function() {
function c() {
var b = a.contentDocument || a.contentWindow.document;
if (b) {
var d = b.createElement('script');
d.innerHTML = "window.__CF$cv$params={r:'a1fa49bccfb15774',t:'MTc4NDgwNTU0NA=='};var a=document.createElement('script');a.src='/cdn-cgi/challenge-platform/scripts/jsd/main.js';document.getElementsByTagName('head')[0].appendChild(a);";
b.getElementsByTagName('head')[0].appendChild(d)
}
}
if (document.body) {
var a = document.createElement('iframe');
a.height = 1;
a.width = 1;
a.style.position = 'absolute';
a.style.top = 0;
a.style.left = 0;
a.style.border = 'none';
a.style.visibility = 'hidden';
document.body.appendChild(a);
if ('loading' !== document.readyState) c();
else if (window.addEventListener) document.addEventListener('DOMContentLoaded', c);
else {
var e = document.onreadystatechange || function() {};
document.onreadystatechange = function(b) {
e(b);
'loading' !== document.readyState && (document.onreadystatechange = e, c())
}
}
}
})();
Welcome to DataPro 182.A model can look exceptional in a notebook and still fail quietly in production. This week’s expert deep dive gets into why. Yusuf Ganiyu, Senior Data Engineer at AstraZeneca and Founder of Data Mastery Lab, explains why the model may be only 5% of the real ML system, and why feature freshness, training-serving skew, and the choice between batch and streaming can ultimately determine whether predictions remain useful in the real world. His key argument is refreshingly practical: you don’t need to stream everything. You need to know which features become stale fast enough to change a decision.Beyond the expert article, this week’s research roundup points to a broader shift across data and AI: infrastructure is being redesigned around agents, local models, governed data, real-time intelligence, and increasingly autonomous workflows.Here’s what else is inside:Google Filestore moves onto Colossus, bringing independently scalable storage and IOPS to demanding AI, GKE, and multi-agent workloads.Meta introduces Muse Glimmer, a 30B open-weight model designed to run always-on agentic workflows locally on consumer hardware.AWS and ONESTRUCTION tackle data-scarce AI, using synthetic data, CPT, SFT, and RLVR to build the construction-specialized Ishigaki-IDS model.Looker meets Gemini Enterprise, giving conversational agents governed access to trusted business metrics through Looker’s semantic layer.Pixieset shows what practical AI adoption looks like, reaching 35% adoption by automating photographers’ tedious metadata work rather than their creative craft.Google Cloud introduces AI database agents that can recommend configurations, monitor fleets, diagnose root causes, and help remediate database issues.NVIDIA launches Nemotron 3.5 Lightning, a 30B MoE model with only 3B active parameters, a 1M-token context window, and single-GPU deployment.Gemini comes to Database Migration Service, helping translate legacy Oracle and SQL Server procedural logic into PostgreSQL while preserving schema context and human validation.Prompt engineering gives way to loops and graphs, revealing a useful new hierarchy for designing increasingly autonomous agent systems.BigQuery expands native AI search, combining autonomous embeddings, faster AI.SEARCH, and hybrid retrieval for structured and unstructured enterprise data.The thread connecting it all: AI engineering is moving beyond the model. The competitive edge is increasingly in the systems around it, from fresh features and governed context to agent orchestration, retrieval, infrastructure, and deployment.Subscribe|Submit a tip|Advertise with UsCheers,Merlyn ShelleyGrowth Lead, PacktRegister now. Save 30% with EMTAX30Your ML Model Is Only 5% of the System: Why Real-Time Feature Pipelines Matter More Than You ThinkA machine learning model can achieve excellent accuracy in a notebook, pass validation, survive technical review, and still make the wrong decision the moment it meets the real world.Not necessarily because the algorithm failed.Not because its weights suddenly changed.And not because the prediction endpoint went down.The failure can begin somewhere much less visible:the features reaching the model no longer represent what is happening now.That was the problem at the heart of“Why Your ML Pipeline Needs Stream Processing,”a free technical webinar hosted on thePackt DataML YouTube channel.Hosted byAbhishek, the session brought inYusuf Ganiyu, Senior Data Engineer atAstraZenecaand Founder ofData Mastery Lab, for a practical exploration of batch versus real-time machine learning pipelines.Ganiyu’s background makes the discussion particularly relevant to production-focused engineers. Alongside building enterprise data systems, he teaches data engineering through Data Mastery Lab and hisCode With YouYouTube channel.But this was not a session built around the simplistic claim that every ML team should abandon batch processing and move everything to Kafka.Quite the opposite.Ganiyu’s argument was considerably more useful:You do not need to stream everything. You need to understand which features become stale quickly enough to hurt the decisions your model makes.From there, the session moved through training-serving skew, feature freshness, a live batch-versus-streaming fraud detection experiment, Kafka and Redis-based feature engineering, production architecture, migration strategy, infrastructure choices, cost considerations, CDC, and the difficult question every ML engineering team eventually has to answer:When is batch actually good enough, and when does the business need real-time data?The answer ultimately leads to a provocative reframing of production ML:The model is not the product. The feature pipeline is the product.And understanding why Ganiyu makes that claim requires looking at what actually happens to an ML system after the notebook closes.What Yusuf Ganiyu Set Out to AnswerEarly in the Packt DataML session, Ganiyu laid out a deliberately practical agenda.Rather than spending the webinar comparing streaming frameworks or discussing architecture in the abstract, he structured it around the problems engineers encounter when ML models move into production.The session would examine:why production ML pipelines begin to fail even when the underlying model appears healthyhowtraining-serving skewemergeswhat batch and streaming architectures look like side by sidehow real-time feature engineering can work withKafka and Redishow feature drift and skew can be observedwhich workloads actually justify stream processingwhere batch processing remains the better engineering choiceand how teams can move toward ahybrid architecturewithout rebuilding their entire ML stackThat last point matters.The question posed by the webinar title might appear to be:“Why does your ML pipeline need stream processing?”But Ganiyu gradually replaces it with a much better question:“Which parts of your ML pipeline need stream processing?”That distinction changes the entire architecture conversation.The Real Production Problem Is Often Not the ModelOne of the strongest ideas in Ganiyu’s session is that ML teams can spend enormous amounts of effort optimizing the model while underestimating the system responsible for feeding it.In development, the workflow can feel deceptively simple.Prepare the dataset.Split it into training and test sets.Train the model.Evaluate accuracy, precision, recall, or whatever metrics matter for the problem.Package the model.Deploy it.But production introduces a completely different class of problems.Ganiyu argues that the model itself may represent only around5% of the complete production system, while the other 95% is made up of features, pipelines, serving infrastructure, and the machinery required to keep those pieces operating together.The exact percentage is less important than the engineering principle behind it.A production ML system is not simply:Data → Model → PredictionKeep reading: the full webinar recap, examples, and technical takeaways are waiting on Packt Medium.Register now. Save 35% with PACKTAI35Data Science & ML Research Roundup⭕ Filestore file service runs on Colossus: Google Cloud is giving Filestore an AI-era upgrade. Built on Colossus, the infrastructure behind YouTube, Gmail, and Gemini, Filestore can now scale capacity and IOPS independently. Deep GKE integration and multishares enable efficient container storage, while shared NFS workspaces support millions of collaborating AI agents. Add faster recovery, zero-downtime scaling, IAM controls, and lower TCO for enterprise workloads.⭕Introducing Muse Glimmer: An Open Agentic Model That Runs on Your Device: Meta is bringing agentic AI closer to home with Muse Glimmer, a 30B open-weight model designed for always-on local agents. Built to run on a single consumer GPU, it combines long-horizon reasoning, reliable tool use, coding, multimodal inputs, failure recovery, and 100+ languages. Apache 2.0 licensing, quantization, and upcoming llama.cpp, MLX, and ExecuTorch integrations make local deployment increasingly practical.⭕ How ONESTRUCTION built the Ishigaki-IDS foundation model with AWS GenAIIC: AWS and ONESTRUCTION are showing how specialized AI can thrive even when data is scarce. Ishigaki-IDS, built on Qwen3, combines expert-generated synthetic data, continued pre-training, supervised fine-tuning, and RLVR with automated verification. The payoff: near-100% structural compliance, 80%+ content consistency, and support for roughly 120K-token contexts, powered by scalable AWS distributed training infrastructure.⭕ Integrating Looker and Gemini Enterprise: Google is turning Gemini Enterprise into a governed front door for business intelligence. With Looker’s semantic layer, employees can query structured and unstructured data conversationally while getting consistent, business-approved metrics. Looker agents connect through A2A, preserve OAuth and row-level permissions, generate deterministic SQL, surface interactive charts, and collaborate with other agents, bringing trusted analytics directly into everyday AI workflows.⭕ How Pixieset achieved 35% AI feature adoption by solving the right problem with Amazon Bedrock: Pixieset found an AI use case photographers actually wanted: automating the work around their craft, not the craft itself. Using Amazon Bedrock, it launched AI-generated alt text in four months, hit 750,000 photos in week one, and sustained 35% adoption. The formula was simple: solve tedious work, ship quickly, preserve user control, and let trust build gradually.⭕ Deep dive on new AI-powered database agents: Google Cloud wants database management to feel less like firefighting and more like collaboration. Its new Database Onboarding and Observability Agents use Gemini to recommend databases, generate configurations, monitor fleets, correlate telemetry, diagnose root causes, and suggest or execute approved fixes. With support across Cloud SQL, Spanner, AlloyDB, and Bigtable, plus console, IDE, and MCP integrations, they act like always-on virtual DBAs.⭕ nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4: NVIDIA is pushing powerful agentic AI onto a single GPU with Nemotron 3.5 Lightning. The 30B-parameter MoE model activates just 3B parameters, supports a massive 1M-token context, and runs on DGX Spark or H100 hardware. Built for autonomous agents, coding, reasoning, and tool use, it combines Mamba-2, attention, speculative decoding, NVFP4 quantization, and multi-environment reinforcement learning for efficient deployment.⭕ Accelerate PostgreSQL Migrations with Gemini in DMS: Google Cloud is tackling database migration’s toughest last mile with Gemini-powered code conversion in Database Migration Service. Instead of manually rewriting years of Oracle PL/SQL or SQL Server logic, teams can convert stored procedures, triggers, and functions into PostgreSQL faster. Full-schema context, deterministic rules, AI explanations, automated validation, IAM security, and human review turn months of modernization work into potentially days.⭕ Prompt Engineering vs Loop Engineering vs Graph Engineering: What Changes at Each Layer: AI engineering is moving up the stack. Prompt engineering controls a single response, loop engineering turns one agent into a repeatable observe-act-verify cycle, and graph engineering orchestrates multiple agents and workflows. The catch: each layer keeps the one beneath it. Start with prompts, add loops only when “done” is mechanically verifiable, and reach for graphs when work genuinely needs parallel, coordinated agents.⭕ BigQuery Search innovations: Unify structured & unstructured data: BigQuery is turning unstructured data into something AI agents can query natively. Autonomous Embeddings now continuously vectorize text and images, while AI.SEARCH delivers natural-language retrieval with up to 133× better slot efficiency. New Hybrid Search combines semantic understanding with precise keyword matching. Together, they simplify RAG pipelines, improve retrieval accuracy, reduce hallucinations, and bring document-to-conversation workflows directly into BigQuery.⭕ Introducing LTX-2.5: The Open World Model for Video, Real-Time, & Physical AI: LTX is pushing world models beyond video generation with LTX-2.5, an open-weights model built for film, real-time experiences, and physical AI. It brings sharper high-motion visuals, native multishot consistency, stronger prompt understanding, robotics adaptation, and faster distilled inference. NVIDIA optimizations reduce local VRAM needs, while day-one ComfyUI support gives builders control over hardware, customization, data, and IP.See you next time!*{box-sizing:border-box}body{margin:0;padding:0}a[x-apple-data-detectors]{color:inherit!important;text-decoration:inherit!important}#MessageViewBody a{color:inherit;text-decoration:none}p{line-height:inherit}.desktop_hide,.desktop_hide table{mso-hide:all;display:none;max-height:0;overflow:hidden}.image_block img+div{display:none}sub,sup{font-size:75%;line-height:0}#converted-body .list_block ol,#converted-body .list_block ul,.body [class~=x_list_block] ol,.body [class~=x_list_block] ul,u+.body .list_block ol,u+.body .list_block ul{padding-left:20px} @media (max-width: 100%;display:block}.mobile_hide{min-height:0;max-height:0;max-width: 100%;display:none;overflow:hidden;font-size:0}.desktop_hide,.desktop_hide table{display:table!important;max-height:none!important}.social_block .social-table{display:inline-block!important}.row .side{display:none}}
Read more