Featured Issues

DataPro is a weekly, expert-curated newsletter trusted by 120k+ global data professionals. Built by data practitioners, it blends first-hand industry experience with practical insights and peer-driven learning.Make sure to subscribe here so you never miss a key update in the data world. AbstractCloud-native analytics platforms provide organizations with flexibility, scalability, and the ability to dynamically allocate infrastructure resources. However, this flexibility introduces a significant challenge: determining how much memory a workload actually requires and translating that requirement into appropriate container and infrastructure capacity.Traditional infrastructure sizing approaches frequently focus on vCPU, total memory, or average utilization. These approaches can be insufficient for memory-intensive analytics workloads, where application behavior, concurrency, transient memory consumption, and container resource constraints can significantly influence performance and reliability.This article presents a memory-aware methodology for sizing cloud-native analytics platforms. The approach establishes a relationship between workload characteristics, application memory requirements, container resources, Kubernetes node capacity, and overall cluster sizing.The methodology emphasizes that memory should be treated as a first-class sizing dimension, rather than simply as a fixed ratio to CPU.1. IntroductionOrganizations increasingly deploy analytics workloads on cloud-native platforms to support large datasets, advanced analytics, artificial intelligence, and highly concurrent workloads.While cloud platforms provide elastic infrastructure, selecting the appropriate infrastructure configuration remains challenging.A common sizing approach is:                               Fig. (1) [1]Workload → vCPU → RAM → Cloud Instance However, cloud-native analytics introduces additional layers between the workload and infrastructure:Fig. (2) [2]Workload → Application → Container → Kubernetes → Node → ClusterEach layer has different resource requirements and constraints. An application may require a specific amount of memory to perform an operation successfully. The container running that application must provide sufficient memory, while the Kubernetes node must have enough allocatable memory to schedule the workload. The cluster must then provide sufficient capacity for concurrency, resiliency, and future growth.Consequently, sizing memory at only one layer can result in either under-provisioning or over-provisioning.2. Why Memory Requires Special AttentionMemory behaves differently from CPU in many analytics workloads.CPU utilization can fluctuate significantly without necessarily causing workload failure. Memory consumption, however, can reach a hard limit.When a workload exceeds its available memory, the result may include application failure, container termination, Kubernetes pod eviction, reduced concurrency, increased execution time, or additional infrastructure requirements.At the other extreme, allocating substantially more memory than the workload requires can result in low resource utilization, larger cloud instances, higher infrastructure costs, and inefficient capacity planning.The challenge is therefore to identify the appropriate memory envelope for the workload.3. Application Memory vs. Container MemoryOne of the most important concepts in cloud-native sizing is distinguishing between application memory and container memory.Application-level memory represents the amount of memory required by the application or workload to perform its processing. Container memory represents the memory available to the container in which that application executes.These values should not automatically be assumed to be identical.The container may require additional memory for runtime processes, application libraries, internal buffers, temporary allocations, operating-system interactions, initialization and shutdown, I/O processing, and other supporting processes.Therefore, a useful conceptual relationship is: Container Memory > Application Memory Requirement.The exact margin should be determined through platform documentation, workload testing, and observed resource utilization.4. The Memory Sizing HierarchyLevel 1 - Workload: Identify data volume, dataset size, user concurrency, session concurrency, workload type, processing complexity, expected growth, and performance requirements.Level 2 - Application: Determine application memory requirements, temporary memory requirements, peak memory behavior, and memory growth during processing.Level 3 - Container: Translate application requirements into memory request, memory limit, and required headroom.Level 4 - Kubernetes Node: Determine allocatable memory, CPU availability, pod capacity, scheduling requirements, and system overhead.Level 5 - Cluster: Determine number of nodes, high availability, workload concurrency, failure recovery, and growth requirements.This hierarchy helps prevent a common mistake: selecting a cloud instance before understanding the actual memory requirements of the workload.Fig. (3)[3]5. Average Memory Is Not EnoughA workload's average memory consumption can be misleading.Consider a hypothetical workload withMetricMemoryAverage100 GBP90 (90th Percentile)180 GBMaximum220 GB Sizing the environment for 100 GB would leave insufficient capacity for normal workload variation.A memory-aware methodology should therefore consider:Average + P90 + Maximum + Failure BehaviorThe objective is not necessarily to size every workload for its absolute theoretical maximum. Instead, the architect should understand the workload distribution and establish an appropriate level of operational headroom.6. Memory and ConcurrencyConcurrency is another major factor.Suppose an analytics workload requires20 GB per active sessionand the expected concurrency is:10 sessionsA simplistic calculation would be:20 GB x 10 sessions = 200 GBHowever, 200 GB should be considered a starting point, not the final infrastructure requirement.Additional capacity may be required for:Container overheadRuntime processesTemporary memoryPlatform servicesKubernetes overheadConcurrent workload variabilityHigh availabilityOperational headroomTherefore,| Per-session memory x concurrency is a workload estimate, not automatically a node-sizing recommendation7. Memory Is Not Always Proportional to Dataset SizeAnother common assumption is: Larger dataset = proportionally larger memory requirement.Analytics workloads are more complicated. Memory consumption can depend on data structures, algorithms, intermediate results, sorting, aggregations, joins, caching, parallel processing, number of concurrent users, and number of simultaneous operations.Two workloads processing datasets of the same size can therefore have substantially different memory requirements.Similarly, doubling the dataset size does not necessarily mean that infrastructure memory must simply double.This is why empirical workload characterization is important.8. Identifying the True BottleneckA memory-aware methodology should determine which resource is actually limiting workload performance.CPU-bound: The workload has sustained high CPU utilization and benefits from additional compute capacity.Memory-bound: Memory utilization approaches the container or node limit while CPU remains comparatively underutilized.I/O-bound: The workload spends significant time waiting for storage or data movement.Application-bound: Increasing infrastructure resources produces little improvement because the limiting factor is application behavior or algorithmic execution.Concurrency-bound: The workload performs adequately for individual users but degrades as concurrent sessions increase.The correct infrastructure solution depends on the bottleneck—not simply on the largest available resource.9. Why More CPU Does Not Necessarily Solve a Memory ProblemA common infrastructure response to slow analytics workloads is to increase vCPU. That approach can be ineffective when memory is the limiting resource.Consider a workload with moderate CPU utilization, high memory utilization, increasing execution time, and memory pressure during peak processing.Adding CPU may provide little improvement. Increasing available memory, changing workload execution characteristics, optimizing data processing, or reducing concurrency may provide substantially greater benefits.Therefore, sizing should begin with: What is limiting the workload? rather than: Which larger instance should we select?10. Memory HeadroomMemory headroom is necessary because workload consumption is rarely perfectly predictable.Headroom can accommodate workload variability, temporary memory spikes, application overhead, additional users, data growth, platform services, and operational events.However, excessive headroom can result in significant infrastructure waste.The objective should therefore be evidence-based headroom rather than arbitrary headroom.Benchmarking, production telemetry, and historical workload behavior can be used to establish appropriate margins.11. Memory-Aware Kubernetes SizingKubernetes distinguishes between the total resources available on a node and the resources that are allocatable to pods.Kubernetes defines Node Allocatable as the amount of compute resources available for pods. [1][2] For memory, Node Allocatable represents the memory capacity that Kubernetes makes available for pod workloads. The allocatable capacity can be affected by resources reserved for Kubernetes system components, operating system processes, and memory reserved to support eviction behavior.For memory-aware sizing, this distinction is important because the memory requirement derived from the application or workload should ultimately be evaluated against the memory capacity available pods, rather than simply against the physical memory of the underlying node.Conceptually:Node Capacity → Reserved Resources / Eviction Protection → Node Allocatable → Pod ResourcesKubernetes provides configuration mechanisms such as kubeReserved, and systemReserved for reserving resources for Kubernetes and operating-system system daemons. Eviction thresholds can also reserve memory to protect the node from memory pressure.For environments using the Kubernetes Memory Manager, reservedMemory can be used to specify reserved memory across NUMA nodes. The Memory Manager does not use this reserved memory for container workloads.The actual reservation values and enforcement behavior are configuration-dependent and should not be represented as a universal percentage of node memory.For example, Kubernetes provides a documented example with a 32 GiB node configured with:kubeReserved = 2 GiBsystemReserved = 1 GiBeviction threshold = 500 MiBUnder that configuration, Node Allocatable memory is 28.5 GiB Kubernetes states that the scheduler ensures the total memory requests across pods on that node does not exceed 28.5 GiBTherefore, a memory-aware sizing methodology should distinguish among:1. Physical Node Memory2. Node Allocatable Memory3. Memory Requested by the Workload4. Application/Container Memory RequirementThis distinction is particularly important when sizing memory-intensive analytics workloads because using physical node memory directly as application capacity can overstate the memory capacity available to pods.ConceptMeaningNode CapacityTotal memory reported for the Kubernetes nodeNode AllocatableMemory capacity Kubernetes makes available for podsPod Memory RequestMemory amount used by the scheduler when determining whether a pod can fitThis should not be interpreted as saying that the application can actually consume all Node Allocatable memory. It describes the scheduling capacity based on pod resource requests. Actual memory consumption is a separate consideration and can be affected by container limits, workload behavior, memory spikes, and node-level memory pressure. Kubernetes can evict pods when overall pod memory usage exceeds the enforced allocatable boundary.This distinction becomes particularly important for large-memory analytics workloads.A workload requiring hundreds of gigabytes of memory may require a specialized high-memory node even when the total cluster memory appears sufficient.12. Memory-Aware Workload TiersOrganizations can improve resource efficiency by creating workload tiers.TierWorkload CharacteristicsSizing ApproachSmallSmall datasets, interactive workloadsLower memory profileMediumModerate analyticsStandard memory profileLargeLarge datasets or complex processingHigh-memory profileExtra LargeHighly memory-intensive workloadsDedicated high-memory infrastructure13. A Practical Memory - Aware Sizing MethodologyStep 1 - Characterize the workload: Capture data volume, dataset size, user concurrency, session concurrency, workload type, processing complexity, and growth expectations.Step 2 - Establish the application memory requirement: Determine how much memory the application requires during representative execution.Step 3 - Determine container requirements: Translate application memory requirements into memory request, memory limit, and required headroom.Step 4 - Benchmark representative workloads: Measure execution time, average memory, P90 memory, maximum memory, CPU utilization, I/O, and failure behavior.Step 5 - Identify the bottleneck: Determine whether the workload is CPU-bound, memory-bound, I/O-bound, application-bound, or concurrency-bound.Step 6 - Validate scalability: Test changes in dataset size, user concurrency, memory, CPU, threads, and workload complexity.Step 7 - Translate workload requirements to infrastructure: Determine node memory, node count, cluster capacity, and availability requirements.Step 8 - Optimize cost versus performance: Select the infrastructure configuration that meets the business performance requirement with the lowest economically justified resource allocation.14. Cost OptimizationMemory-aware sizing ultimately supports a broader objective: cost optimization without sacrificing performance or reliability.Under-sizing can produce low cost but poor performance and workload failures. Over-sizing can produce high cost, low utilization, and wasted capacity. Right-sizing aims to provide required performance with sufficient capacity and optimized cost.The objective is therefore not simply to minimize infrastructure. It is to minimize unnecessary infrastructure while satisfying the workload's business requirements.15. A Better Cloud Sizing QuestionTraditional sizing asks:How much CPU and memory does this application need?A memory-aware methodology asks:What workload are we running?How does memory consumption change as workload size increases?How does memory scale with concurrency?What is the application's peak memory requirement?What container memory limit is required?How much memory is actually allocatable on the Kubernetes node?What resource is limiting performance?How much operational headroom is justified?What infrastructure configuration provides the required performance at the lowest reasonable cost?This represents a shift from static infrastructure sizing to dynamic workload-based sizing.16. ConclusionCloud-native analytics platforms require a different approach to infrastructure sizing.Memory can no longer be treated simply as a fixed companion to vCPU.Instead, memory must be evaluated across the complete execution hierarchy:Workload → Application → Container → Kubernetes → Node → ClusterA memory-aware sizing methodology provides organizations with a structured way to understand this relationship.The key principles are: measure workload memory rather than assuming it; separate application memory from container memory; consider peak and percentile utilization, not only averages; account for concurrency and workload growth; identify the actual performance bottleneck; consider Kubernetes allocatable capacity rather than physical node memory alone; use evidence-based headroom; and optimize infrastructure based on performance and business requirements.Ultimately, the goal of cloud-native sizing is not to provision the most infrastructure. It is to provision the right infrastructure for the workload.17. AcknowledgementGenerative AI Disclosure:[1] The image in fig. 1, was generated using Google Gemini to illustrate the conceptual workflow of cloud instance sizing allocation.[2] The image in fig. 2, was generated using Google Gemini to illustrate the multi-tier deployment topology from workload definition.[3] The image in fig. 3, was generated using Google Gemini to illustrate the five-level memory sizing workflow and cross-cutting architectural considerations across the cloud-native stack.18. References[1] Kubernetes, “Resource Management for Pods and Containers.”This documentation shows discussion of memory requests, memory limits, scheduling, and the fact that available pod resources are less than total node capacity.[2] Kubernetes, “Reserve Compute Resources for System Daemons.”Kubernetes explicitly explains that node resources need to account for system daemons and that Node Allocatable represents resources available for pods.[3] Kubernetes “Assign Memory Resources to Containers and Pods”Kubernetes states that Pod scheduling is based on requests and that a Pod is scheduled only when the node has enough available memory to satisfy the Pod's memory request.[4] Kubernetes “Control Memory Management Policies on a Node”This documentation specifically states that reserved memory is used to calculate the actual Node Allocatable memory available to Pods.See: Reserved memory configuration - reservedMemory[5] Y. Mao, Y. Fu, S. Gu, S. Vhaduri, L. Cheng, and Q. Liu, - “Resource Management Schemes for Cloud-Native Platforms with Computing Containers of Docker and Kubernetes,” 2020.[6] V. Medel, R. Tolosana-Calasanz, J. Á. Bañares, U. Arronategui, and O. F. Rana, “Characterising resource management performance in Kubernetes,” 2024.Author BioSomil Rastogi (Senior Member, IEEE) is a Principal Solutions Architect and Lead Product Engineer with extensive expertise in cloud capacity planning, hardware sizing automation, and enterprise system architecture. He earned a Post Graduate qualification in AI/ML and AI Agents for Business Applications from the McCombs School of Business, University of Texas at Austin. His current technical and research interests center on agentic AI frameworks, deterministic LLM guardrails, Retrieval-Augmented Generation (RAG), and data analytics tools for enterprise infrastructure optimization. In his present role, he leads the architectural design and full-lifecycle development of self-service sizing web applications across major public cloud providers.Mr. Rastogi serves as an active technical reviewer and contributor to scholarly computer science and software engineering.

Gemini 3.8 goes live, Toyota bets on physical AI, and causality takes center stage.AI Distilled #154: AI Is Moving From Prediction to InterventionWelcome to AI Distilled 154For decades, AI has been remarkably good at finding patterns. The harder question is:what happens when we act on them?That question sits at the heart of this week’s expert piece from Yousri El Fattah, Founder of Causal Computing. In “From Control Theory to Causal Inference,” Yousri traces a three-decade journey from stochastic control and learning systems to Bayesian networks and causal AI, showing why understanding cause and effect, interventions, and uncertainty becomes increasingly important as AI moves from recommending decisions to actually making them. And that transition is unfolding across this week’s AI landscape. Agents are no longer confined to chat windows. They are seeing, speaking, operating enterprise systems, coordinating supply chains, and increasingly interacting with the physical world.This week’s highlights:Gemini gets a face, voice, and eyes: Google’s Gemini 3.8 Live with Live Avatar brings speech-to-speech conversation, video avatars, tool calling, screen and camera understanding, and support for 97 languages to production enterprise agents.Warehouse AI moves from prediction to execution: Gartner maps the progression from optimisation and generative AI to semi-autonomous agents and physical robotics, as warehouses move beyond AI experiments into live operations.AWS tackles the “who said what?” problem: WhisperX on SageMaker AI adds speaker identification and precise word-level timestamps to speech-to-text, making large-scale audio more useful for analytics, captions, search, and compliance.Supply-chain agents start taking action: Multi-agent systems are moving beyond dashboards to rerouting freight, rebalancing inventory, and responding to disruptions, with Lenovo reporting 3x faster fulfilment decisions and 4x faster disruption responses.Databricks puts agents inside spreadsheets: Its acquisition of Row Zero brings Databricks Genie together with spreadsheets capable of handling 1M+ live rows, giving analysts a familiar interface for working with secure enterprise data using natural language.Toyota makes a $6.4B physical-AI bet: The automaker estimates its automation expansion could eventually involve 400,000 robots, while tackling the difficult problems of Sim2Real transfer, reliability, precision, and human-robot collaboration.Feather wants to build the “Android of robotics”: Rather than another closed humanoid, Feather is building a $30,000 modular robot platform that developers can customize and run with AI models from NVIDIA, Skild, Physical Intelligence, and others.The thread connecting all of this is increasingly clear: AI is moving from understanding the world to acting within it. And once models begin making consequential decisions, Yousri’s central question becomes much harder to ignore: Do we understand what will happen when we intervene? Let's get into it.Cheers,Merlyn Shelley,Growth Lead, Packt.Build Production-Ready AI Agents : From Workflow to Deployment📅 Saturday, September 26–Sunday, September 27 | 10:00 AM–2:00 PM EDTBuild production-ready AI agents withn8n, from real-world workflows and tool integrations to testing, security, governance, and deployment.Join Ashley Live →From Control Theory to Causal Inference: A Three‑Decade Journey That Led to My BookSome academic journeys begin with a clear plan. Mine began with a question that refused to leave me alone: How do we intervene in a system to achieve a desired effect? That question followed me across continents, across disciplines, and across decades — from aeronautical engineering in Cairo to stochastic adaptive control in Europe and Morocco, and eventually to causal inference and Bayesian networks in Southern California.Only in hindsight do I see the thread connecting it all. Every phase of my work — optimal control, learning automata, stochastic processes, modeling biological morphogenesis, market price formation, and communication flow control — was really an exploration of how systems behave, adapt, and respond when we act upon them. Causality was always present. I simply didn’t yet have the vocabulary for it.The First Half: Learning to Control What We Don’t Fully KnowMy early academic life was devoted to understanding how to control systems whose dynamics were uncertain or partially unknown. I focused on finite Markov chains and semi‑Markov processes, developing active, recursive learning mechanisms that unified control and estimation in real time.This work introduced several ideas that today echo in reinforcement learning:Randomized control policies to ensure continual explorationReward gradient formulas to quantify how policy changes affect long‑run performanceRecursive stochastic gradient algorithms that adapt policies safely toward optimalityAt the time, these were simply solutions to problems I found intellectually irresistible. Today, they resemble early versions of concepts that underpin modern adaptive systems.The figure below illustrates block diagrams from my book Learning Systems, depicting supervised learning of Bayes’ rule, a game between multiple learning automata, a flow‑control system regulating server activity, and a learning automaton.Designing Data Infrastructure for AI Agents & LLMs📅 Saturday, September 26 | 9:30 AM–1:30 PM EDTDesign production-ready data infrastructure for AI agents & LLMs, from safe write paths and state management to provenance, tracing, recovery, and production architecture.Join Sandipan LiveThe Turning Point: Discovering the Language of CausalityEverything changed in 1986.While teaching control systems and AI at California State University Long Beach, I attended a lecture by Judea Pearl on Bayesian networks. That single talk reframed my entire intellectual world. It connected the systems I had spent decades modeling with a new, powerful idea: causal structure.I realized that the question I had been chasing — how do we intervene? — was the beating heart of causal inference.From that moment on, my research, teaching, and industry work shifted toward causal modeling, Bayesian networks, and decision analysis. When Packt approached me in 2023 to write a book on causal inference with Bayesian networks, I knew it was time to synthesize everything I had learned across my eclectic academic journey.Three Years, Hundreds of Papers, and One Unifying VisionWriting Causal Inference with Bayesian Networks took three years of deep reading, coding, experimentation, and careful structuring. My goal was ambitious:Build a book that teaches causal inference not as abstract theory, but as a practical, computational discipline grounded in Bayesian networks, influence diagrams, and relational databases.Continue reading the full article on Packt’s Medium →What It Takes to Build Production-Ready LLM Applications📅 Saturday, October 3 | 9:30 AM–1:30 PM EDTBuild production-ready LLM applications with DSPy, moving beyond manual prompt tweaking to structured evaluation, automated prompt optimization, failure analysis, and experiment tracking with MLflowBuild with Serj & Brett →AI Pulse: This Week📛 Gemini 3.8 Live with Live Avatar is now generally available: Google’s Gemini 3.8 Live with Live Avatar is now generally available, bringing production-ready conversational video agents to enterprises. It combines synchronized video avatars with natural speech-to-speech dialogue, background tool calling, 97-language support, and live camera and screen understanding. Available through US and EU endpoints, it adds enterprise compliance, strict data governance, controlled custom avatars, and SynthID watermarking for transparency.📛 Gartner outlines four AI tiers in warehouse automation: Gartner says warehouse AI is moving from experiments to real-world deployment, driven by worker shortages, cheaper software, and more reliable automation. The four tiers span advanced optimisation, generative AI, semi-autonomous agents, and physical robotics. Together, they can forecast demand, dynamically assign work, generate instructions, and automate picking and packing, while keeping humans in control of critical decisions.📛 Speaker-labeled transcription with WhisperX on SageMaker AI: AWS is making speaker-aware transcription easier to deploy at scale with WhisperX on SageMaker AI. WhisperX improves standard speech-to-text with word-level timestamps, speaker identification, and faster transcription, enabling searchable meetings, accurate captions, compliance reviews, and contact-center analytics. AWS packages it in a GPU-ready container, supporting real-time endpoints for short clips and asynchronous processing for longer, high-volume audio.📛 Multi-agent AI systems are taking over supply chain execution: Supply chains are moving from AI that recommends to AI that acts. Multi-agent systems can use real-time data to reroute freight, rebalance inventory, allocate docks, and respond to disruptions automatically. Lenovo reports 3x faster fulfilment decisions, 4x faster disruption responses, and 30% better delivery accuracy. But deployments remain bounded by human approvals, financial limits, and strict operational guardrails.📛 Databricks buys Row Zero and is scouting for more startups to acquire: Databricks is bringing AI directly into the spreadsheet experience by acquiring Row Zero, a cloud spreadsheet capable of handling more than 1 million live rows. The plan is to combine Row Zero with Databricks Genie, letting analysts query secure enterprise data with natural language and familiar formulas, without moving it elsewhere. The acquisition also extends Databricks’ aggressive 2026 AI acquisition spree.📛 Toyota’s $6.4bn robotics estimate puts physical AI in focus: Toyota is putting physical AI at the center of its factory automation strategy, estimating it could eventually require 400,000 robots and $6.4 billion in annual spending from 2028. Its work spans autonomous logistics, parts-picking robots, humanoids, and reinforcement learning. Toyota is also tackling key challenges including Sim2Real gaps, reliability, precision, robot-learning data, and human-robot collaboration.📛 Meet Feather, the startup building the 'Android of robotics' for developers: Feather Robotics wants to become the “Android of robotics”, offering developers a modular humanoid platform instead of building one closed, general-purpose robot. Its customizable $30,000 hardware can run AI models from providers like NVIDIA, Skild, and Physical Intelligence. Backed by a $7.6 million pre-seed, Feather has already surpassed $1 million in revenue, with robots working in restaurants and science labs.Design a production-ready RAG architecture for enterprise GenAI applications📅 Saturday, October 31 | 9:30 AM–1:30 PM EDTDesign a production-ready RAG architecture for enterprise GenAI, covering ingestion, chunking, metadata enrichment, vector search, retrieval tuning, evaluation, governance, and cost control to build grounded, traceable, and reliable AI applications.Join Brian Live →📢 If your company is interested in reaching an audience of developers and, technical professionals, and decision makers, you may want toadvertise with us.If you have any comments or feedback, just reply back to this email.Thanks for reading and have a great day!*{box-sizing:border-box}body{margin:0;padding:0}a[x-apple-data-detectors]{color:inherit!important;text-decoration:inherit!important}#MessageViewBody a{color:inherit;text-decoration:none}p{line-height:inherit}.desktop_hide,.desktop_hide table{mso-hide:all;display:none;max-height:0;overflow:hidden}.image_block img+div{display:none}sub,sup{font-size:75%;line-height:0}#converted-body .list_block ol,#converted-body .list_block ul,.body [class~=x_list_block] ol,.body [class~=x_list_block] ul,u+.body .list_block ol,u+.body .list_block ul{padding-left:20px} @media (max-width: 100%;display:block}.mobile_hide{min-height:0;max-height:0;max-width: 100%;display:none;overflow:hidden;font-size:0}.desktop_hide,.desktop_hide table{display:table!important;max-height:none!important}.social_block .social-table{display:inline-block!important}}