Featured Issues

Open models, autonomous agents, and billions flowing into the infrastructure behind them.Query On-Prem Iceberg and Delta Tables Live From DatabricksImport a share into Unity Catalog and query on-prem data in place—no copying, no downsampling, results back in seconds. Watch the 7-minute walkthrough of MinIO AIStor Table Sharing connected live to Databricks.Watch the DemoSponsoredAI Distilled #151: AI is crossing an important thresholdWelcome to AI Distilled 151AI Is Becoming Autonomous. Your Product Metrics Haven’t Caught Up.AI is crossing an important threshold.OpenAI’s GPT-6 Astra is pushing further into autonomous computer use and professional workflows. AWS is putting agents inside the software development lifecycle and everyday email workflows. Google Cloud is rebuilding infrastructure around agent swarms, real-time AI, and orchestration. Meanwhile, billions are pouring into the foundations underneath it all, from NVIDIA’s $12.93 billion Hugging Face acquisition to Crusoe’s reported $30 billion valuation.The common thread is clear: AI is moving from generating outputs to taking actions.But as these systems become more autonomous, a harder question emerges: how do we know they are actually making products and experiences better?That is where our expert-led conversation begins.Lorenzo Toni, Lead Data Scientist — AI Product Analytics at Amazon, explains why the metrics AI teams traditionally trust can tell the wrong story. An AI feature can increase engagement, session duration, and interactions while quietly hurting conversion and user trust. Lorenzo explores how causal inference, better experimentation, and measuring downstream behaviour can reveal what conventional A/B tests miss.Because building more capable AI is only half the challenge. Knowing whether it creates real value is becoming just as important.This week’s highlightsNVIDIA to acquire Hugging Face for $12.93B — Can NVIDIA scale Hugging Face without compromising the openness that made it central to the AI ecosystem?Crusoe reportedly raises $3B at a $30B valuation — The former crypto miner’s valuation has tripled in 10 months as demand for AI compute explodes.OpenAI unveils GPT-6 Astra — OpenAI’s latest frontier model pushes AI further from answering prompts toward autonomously executing professional work.GPT-6 Astra vs Gemini vs Claude — As models become agents, which one is best positioned to reliably take action?AWS puts AI deeper into the development lifecycle with AgentCore — Agents generate engineering artifacts and automate security analysis while humans retain oversight.Google Cloud rethinks infrastructure for the agentic era — Agent swarms, stateless MCP, sandboxed workloads, and session-aware networking are changing the infrastructure underneath AI.Meta offers steep AI discounts in exchange for training data — A roughly 95% discount puts an unusually explicit price on the value of prompts and model outputs.AWS connects Amazon Quick with Outlook — AI moves beyond summarizing your inbox to drafting replies, scheduling meetings, and triggering workflows.Let's get into it.Cheers,Merlyn Shelley,Growth Lead, Packt.Live LLM Engineering Masterclass📅 Saturday, September 12 | 09:30 AM–1:30 PM EDTBuild reliable LLM systems withproduction evals, RAG, resilient agents, regression testing, observability and cost control.Save 45% with PRODLLM45Your AI Experiment Says the Feature Is Working. Your Users May Be Telling You OtherwiseAn AI feature can pass an experiment and still make the product worse.Engagement can rise. Session duration can increase. Users can interact with the feature more frequently. An A/B test can even produce statistically convincing results.And yet, a few weeks later, conversion starts falling.Users stop returning.Trust begins to erode.That is the measurement problem AI product teams increasingly need to confront.Traditional digital products generally give users a relatively deterministic experience. Change a button, ranking algorithm, checkout flow, or interface and teams can establish a reasonably clean treatment and control.Generative AI changes that assumption.The same feature can produce different answers for different users, and even different answers for the same user. A seemingly insignificant prompt adjustment can alter the length, tone, relevance, accuracy, and usefulness of an interaction.So what exactly should teams measure?And more importantly,how do you know whether an AI feature actually caused the outcome you are seeing?That question was at the centre of a conversation betweenNilesh Kowadkar, Relationship Lead at Packt, andLorenzo Toni, Lead Data Scientist — AI Product Analytics at Amazon, focused on experimentation and causal inference for AI products. Drawing on a generalized e-commerce case study, Lorenzo unpacked why apparently positive metrics can be misleading and why AI teams need to look beyond conventional A/B testing.Partner Spotlight | Building a startup? Pitch live to 20 investors in one day.Fundraising shouldn’t mean spending weeks cold-emailing VCs and waiting for replies.We’re partnering withVC Pitch Conferenceto give founders a more direct route to investors. OnSeptember 17, selected startups can join20 guaranteed 1:1 investor calls in a single day, with meetings matched bysector, stage, and geography.With290+ VCs and angel investorslooking across AI, fintech, B2B, and other sectors, it’s an opportunity to put your startup directly in front of investors actively looking for their next deal.Packt readers get an additional 10% off the early-bird ticket with codePACKT10.Pitch Your Startup to Investors →When More Engagement Is Actually a Warning SignConsider an e-commerce platform introducing an AI chatbot designed to help customers discover products.Instead of navigating through search results, users could ask questions, compare prices, describe what they wanted, and receive recommendations conversationally.A change was made to the prompt powering the chatbot.Initially, the numbers looked promising.Engagement increased. Users performed more actions. They also spent longer interacting with the chatbot.Under a conventional product analytics framework, those signals could easily be interpreted as evidence that the new experience was working.Then, roughly two weeks later, another pattern emerged.Chatbot interactions began declining.More importantly, overall website conversion started falling.The metric that initially appeared particularly encouraging,time spent with the chatbot, turned out to be negatively correlated with conversion.Users were not necessarily spending longer because the experience was valuable.Some were spending longer because the chatbot was taking too long to help them reach a decision.A ten-minute conversation might look like excellent engagement on a dashboard. To the customer, it might mean:I could have found this myself in thirty seconds.That distinction matters.Because when AI introduces friction instead of removing it, the damage may extend beyond a single failed interaction. Users learn from that experience. The next time they need something, they may bypass the chatbot entirely.The apparent engagement win can therefore become atrust problem.Dive into the full article on Packt’s Medium, with a visual infographic to bring the key ideas together.How Modern AI Systems Really Find Answers: Build GraphRAG Applications📅 Saturday, September 19 | 09:30 AM–1:30 PM EDTMove beyond vector search and build explainable AI withNeo4j, knowledge graphs, Cypher and LLM agents.Save 35% with MODERNAI35AI Pulse: This Week✅ NVIDIA to acquire Hugging Face for $12.93B: NVIDIA is buying Hugging Face for $12.93 billion, but the bigger story may be what it promises not to change. Hugging Face will remain hardware-neutral, multi-cloud, and open to competing models and accelerators, while NVIDIA pours resources into its infrastructure. With 18 million developers already on the platform, the deal raises a much bigger question: can this alliance keep open-source AI competitive as closed labs race ahead?✅ Crusoe reportedly raises $3B at a $30B valuation: Crusoe has reportedly raised $3 billion at a $30 billion valuation, tripling its valuation in just 10 months. But the bigger signal is where that money is flowing. Once a crypto miner, Crusoe is now building AI infrastructure for Meta, Microsoft, and OpenAI, and has landed a $13 billion deal with Jane Street. With IPO talks reportedly underway, Crusoe’s rapid rise shows just how valuable the infrastructure powering the AI boom has become.✅ OpenAI’s GPT-6 Astra: A new generation of intelligence. OpenAI has unveiled GPT-6 Astra, and the headline numbers are hard to ignore: 99.9% on ARC-AGI-3, 98% on FrontierMath Tier 4, and 100% on ExploitBench. But Astra’s bigger leap may be beyond benchmarks. OpenAI says it can navigate computers, browsers, code, research, and professional workflows with greater autonomy while staying within its authorized scope. Are we moving from AI that assists with work to AI we can actually delegate work to?✅ GPT-6 Astra vs Gemini vs Claude: Which AI agent is best? GPT-6 Astra has arrived, but does it actually put OpenAI ahead of Gemini and Claude? The new model pushes deeper into coding, reasoning, computer use, and autonomous workflows, intensifying a race that is increasingly about more than raw intelligence. As AI shifts from answering prompts to executing real work, the bigger question is which model can become the most capable and reliable agent.✅ AI-driven development lifecycle using Amazon Bedrock AgentCore: AI can write code, but can it take on the development lifecycle without losing human control? AWS is putting that idea into practice with Amazon Bedrock AgentCore, showing how agents can automatically turn SQL schemas into ER diagrams and scan code for vulnerabilities, CVEs, and policy violations. The bigger shift is toward AI handling routine engineering execution while humans stay in control of critical decisions.✅ What’s new in AI infrastructure this month: AI agents are getting more capable, but is today’s infrastructure actually ready to run them at scale? Google Cloud’s August updates tackle the bottlenecks emerging underneath agentic AI, from Filestore supporting agent swarms and sandboxed Ray clusters to always-on Cloud Run instances, stateless MCP, and session-aware load balancing. The bigger shift: AI infrastructure is being redesigned around agents, not traditional applications.✅ Meta is paying to peek at how you use their latest AI model: What if the cheapest AI model came with a hidden currency: your data? Meta is offering roughly 95% discounts on its new Muse Spark model to users willing to share prompts and outputs for future model training. The move turns training data into an explicit economic trade-off and raises a bigger question for enterprises: how much is your data actually worth when AI providers are willing to pay for it?✅ Integrating Outlook with Amazon Quick for AI-powered email automation: How much of your workday disappears into reading threads, chasing action items, and rebuilding context from email? AWS is bringing Amazon Quick into Outlook to automate that overhead, from summarizing conversations and drafting contextual replies to scheduling meetings and triggering multi-step workflows. The bigger idea is simple: what if your inbox stopped being a task list and started completing the tasks for you?Build Production-Ready AI Agents : From Workflow to Deployment📅 Saturday, September 26–Sunday, September 27 | 10:00 AM–2:00 PM EDTBuild production-ready AI agents withn8n, from real-world workflows and tool integrations to testing, security, governance, and deployment.Save 35% with PACKTAI35📢 If your company is interested in reaching an audience of developers and, technical professionals, and decision makers, you may want toadvertise with us.If you have any comments or feedback, just reply back to this email.Thanks for reading and have a great day!*{box-sizing:border-box}body{margin:0;padding:0}a[x-apple-data-detectors]{color:inherit!important;text-decoration:inherit!important}#MessageViewBody a{color:inherit;text-decoration:none}p{line-height:inherit}.desktop_hide,.desktop_hide table{mso-hide:all;display:none;max-height:0;overflow:hidden}.image_block img+div{display:none}sub,sup{font-size:75%;line-height:0}#converted-body .list_block ol,#converted-body .list_block ul,.body [class~=x_list_block] ol,.body [class~=x_list_block] ul,u+.body .list_block ol,u+.body .list_block ul{padding-left:20px} @media (max-width: 100%;display:block}.mobile_hide{min-height:0;max-height:0;max-width: 100%;display:none;overflow:hidden;font-size:0}.desktop_hide,.desktop_hide table{display:table!important;max-height:none!important}.social_block .social-table{display:inline-block!important}}

DataPro is a weekly, expert-curated newsletter trusted by 120k+ global data professionals. Built by data practitioners, it blends first-hand industry experience with practical insights and peer-driven learning.Make sure to subscribe here so you never miss a key update in the data world. IntroductionBuilding a Retrieval-Augmented Generation (RAG) system is only the first step; ensuring that it retrieves the right information and generates accurate, context-grounded responses is equally important. RAG evaluation helps identify weaknesses across both the retriever and generator by measuring retrieval relevance, ranking quality, answer relevance, and faithfulness.In this article, you’ll explore the two core areas of RAG evaluation—retrieval evaluation and response evaluation—and understand how the RAG TRIAD provides a holistic framework for assessing query, context, and response quality. You’ll also learn how key retrieval metrics such as precision, recall, F1 score, Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), and P@K can be used to evaluate and improve the effectiveness of RAG systems.Retrieval evaluation and response evaluationRetrieval evaluation and response evaluation are the two core components of the RAG system evaluation framework; let’s consider them in detail:Retrieval evaluation focuses on whether the model can retrieve accurate and relevant context during a query. It assesses whether the content found by the system from the knowledge base or external resources is closely related to the user’s query and can effectively support subsequent generation. Key metrics in retrieval evaluation include context relevance, retrieval precision, and recall. Retrieval evaluation often emphasizes assessing the quality of the embedding model (i.e., the core component of the retriever).Response evaluation, on the other hand, focuses on the quality of the answers generated by the model based on the retrieved content. Its goal is to ensure that the generated content is not only relevant to the user’s query but also faithful to the provided context information. Response evaluation typically looks at whether the generated answer is accurate, whether it contains “hallucinations”, and whether it is consistent with the context. Key metrics for response evaluation include answer relevance and faithfulness, which help teams better assess whether the generated answers meet expectations. Response evaluation often emphasizes assessing the level of the generative model (i.e., the core component of the generator).When an evaluation dataset is available, meaning some questions already have labeled standard answers and sources, for retrieval evaluation, more precise quantitative metrics such as precision and recall can be used; whereas for response evaluation, it is usually difficult to quantify answer quality with specific scores, and subjective factors play a relatively larger role.RAG TRIAD – holistic evaluationAt present, one evaluation approach is to assess the overall effectiveness of the Retrieval-Augmented Generation (RAG) process. This method is known as the RAG TRIAD framework.Figure 9.4: A flowchart showing relationships among query context and responseIn the RAG TRIAD framework, evaluation is focused on three core aspects: query, context, and response. These three aspects are measured by the following three key evaluation metrics:Contextual relevancy: An indicator for retrieval evaluation, mainly assessing whether the text blocks returned by the retriever are relevant to the user’s query and helpful for generating an ideal answer. This involves selecting hyperparameters such as block size, Top-K value, and embedding model. An efficient embedding model can ensure that the retrieved text blocks are semantically similar to the user’s query, while an appropriate combination of block size and Top-K value helps pick out the most critical information from the knowledge base.Faithfulness: A key metric in response evaluation, used to measure whether the generated answer is based on the provided context, i.e., whether the answer has a factual basis and does not contain “hallucinations.” This metric is often closely related to model selection. If the current model cannot generate accurate answers based on the retrieved context, it may be necessary to replace the model or fine-tune it.Answer relevancy: Also a response evaluation metric, it focuses on the degree to which the generated answer is relevant to the user’s query. As the model’s reasoning ability improves, this metric increasingly depends on the quality of the retrieval results rather than the capability of the model itself. If the score is low, it may be necessary to improve the retrieval process or optimize prompts to enhance the model’s ability to generate more relevant answers.The RAG TRIAD framework provides a reference-free evaluation system. Even without readymade evaluation datasets, it is still possible to evaluate the outputs of the RAG system from the above three aspects. Therefore, this evaluation system has a certain degree of generality and flexibility. Each metric corresponds to specific components or hyperparameter settings in the RAG process. If a RAG system scores highly in these three areas, it indicates that its components or hyperparameter settings are appropriate; otherwise, further adjustment and optimization are needed.This reference-free evaluation system saves the trouble of creating evaluation datasets, but at the same time, it also means that the evaluation metrics themselves have a certain degree of non-determinism. Therefore, the final performance of the RAG system should not only depend on these evaluation metrics but also require practical confirmation from business teams and end users.Retrieval evaluation metricsRetrieval evaluation is critical for RAG systems because it directly affects the quality of the context adopted by the generator, thereby indirectly impacting the relevance and accuracy of the final generated response.MTEB provides a comprehensive benchmarking platform for evaluating text embedding models, covering tasks such as text similarity, text classification, and retrieval. MTEB can effectively measure the performance of text embedding models across multiple tasks. Therefore, MTEB is regarded as a means to evaluate the embedding models used in the retrieval process and is suitable for local task evaluation.However, in practical RAG projects, retrieval evaluation is not limited to the direct assessment of the embedding model, but more broadly targets the entire retriever component, evaluating whether it can effectively identify and prioritize the most relevant documents. This process not only depends on the quality of the embedding model but is also influenced by factors such as chunking strategy and indexing strategy.Retrieval evaluation mainly uses metrics such as precision, recall, Mean Reciprocal Rank (MRR), and Mean Average Precision (MAP). Before introducing these metrics, it should be noted that the text chunks retrieved by the retriever are usually not singular but rather one or more (this is a configurable parameter). In addition, the context text chunks relevant to a specific query are often more than one, and all relevant text chunks should be listed in the evaluation dataset.PrecisionPrecision measures how many of the retrieved text blocks are relevant to the query, i.e., the proportion of relevant text blocks among all retrieved text blocks. Precision aims to answer the question: “Of all text blocks retrieved, how many are truly relevant?” The calculation formula for precision is as follows:             Number of relevant text chunks retrievedPrecision =           Total number of text chunks retrievedFigure 9.5: Formula for precisionSuppose a scientific literature information retrieval system returns 15 text blocks, of which 12 are directly related to the query, then the precision is 80%. High precision means that learners can obtain the needed information more quickly without being distracted by irrelevant text blocks.Tip from Lewis: “In the evaluation dataset, if the system can find the relevant ‘text chunks,’ it is considered a successful retrieval.However, depending on the task, we may also need to locate specific pages, paragraphs, sentences, or even the original document.”High precision also means that the retrieved content is more accurate, effectively reducing unnecessary information display. This metric is especially important in avoiding the presentation of irrelevant information, in fields such as medicine and law, high precision can effectively prevent the spread of misleading information.RecallRecall measures the comprehensiveness with which the system retrieves relevant text blocks, that is, the proportion of relevant text blocks retrieved by the system to all relevant text blocks in the database. Recall aims to answer the following question: Of all the relevant text blocks, how many did the system successfully retrieve?The formula for calculating recall is as follows:        Number of relevant text chunks retrievedRecall =         Total number of relevant text chunksFigure 9.6: The ratio of retrieved relevant documents to all relevant documentsAssume that in a legal document retrieval system, the database contains 50 documents relevant to the query topic, and the system successfully retrieves 40 of them. Then the recall is 80%. In scientific research, high recall can ensure that researchers do not miss key references.High recall is critical for avoiding the omission of crucial information. If recall is low, the model may generate incomplete answers or make mistakes due to missing key information. For example, in legal document retrieval, missing relevant information may affect the completeness of case analysis.F1 scoreIn a RAG system, improving precision often leads to a decrease in recall, and vice versa. Therefore, to achieve optimal retrieval performance, it is usually necessary to find a balance between precision and recall. This balance is typically quantified using the F1 score, which is the harmonic mean of precision and recall, used to find the optimal point suitable for the specific application requirements.The formula for calculating the F1 score is as follows:2 × (Precision × Recall)      Precision + RecallFigure 9.7: Equation for F1 score in showing the harmonic mean of precision and recall valuesMean reciprocal rankMean Reciprocal Rank (MRR) is an evaluation metric for the efficiency of retrieval systems, with a special focus on the rank of the first relevant text chunk. MRR helps us gauge whether a RAG system can quickly return the first relevant text chunk, and its value has a direct impact on user experience. The higher the MRR value, the faster the system is able to find the first suitable answer.The formula for calculating MRR is as follows:Figure 9.8: Mathematical formula for MRR showing the average inverse rank over Q queriesWhere Q represents the total number of queries. rankq denotes the position of the first relevant text chunk for q query q. The formula requires taking the reciprocal of the ranking of the first relevant text chunk, which means that MRR only considers the position of the first relevant text chunk returned for each query. With this calculation method, the higher the rank of the text chunk, the greater the reciprocal of rankq, and thus the greater the MRR.MRR is particularly important in question-answering or information retrieval systems, as it focuses on the ability to quickly obtain the first relevant answer. For example, in a customer service Q&A system, if a user poses a question and the system can select and respond with an answer within the top 3 results, the MRR value will be relatively high, indicating that the system has stronger speed and accuracy in responding to user needs.Mean average precisionMean Average Precision (MAP) is a precision metric evaluated across multiple queries. It not only considers the precision of the retrieval results but also emphasizes the importance of document ranking. MAP evaluates retrieval effectiveness by calculating the precision at various ranks for each query, ensuring that important relevant documents are positioned higher, thereby optimizing the user search experience.The calculation formula for MAP is as follows: 𝑀𝑀𝑀𝑀𝑀𝑀 𝑀 q Figure 9.9: MAP equals the mean of Average Precision over Q queriesHere, Q denotes the total number of queries. Average Precision is the mean precision calculated for each query, taking into account the order of relevant documents.MAP is very useful in systems where the quality of rankings is important, such as search engines, and is often used in product recommendation systems on e-commerce platforms.P@KP@K measures the precision of the top K retrieved results, ensuring that as many relevant pieces of information as possible are included in the first few results displayed.       Number of relevant documents in the top K search resultsP@K =  Figure 9.10: Mathematical formula for P@KHere, the numerator represents the number of relevant documents among the top K results, and the K in the denominator is a fixed return count.P@K is especially useful in scenarios where users are particularly concerned with the top few results. For example, in a news search system, P@K can ensure that the first few news items retrieved are highly relevant to the query topic, thereby improving reading efficiency.ConclusionEvaluating a RAG system requires looking beyond the final generated answer. Retrieval quality determines whether the model receives useful context, while response evaluation measures whether that context is transformed into relevant and faithful answers. Frameworks such as the RAG TRIAD help connect these two stages by assessing contextual relevancy, faithfulness, and answer relevancy.Metrics such as precision, recall, F1 score, MRR, MAP, and P@K provide practical ways to measure retrieval performance from different perspectives, including relevance, coverage, and ranking quality. By selecting metrics that align with your application requirements and combining quantitative evaluation with feedback from business teams and end users, you can identify weaknesses in the RAG pipeline and make more informed decisions about retrieval strategies, model selection, prompts, and system configuration.This article is an excerpt from RAG from First Principles. To explore RAG in greater depth—from its foundational concepts to the design, implementation, evaluation, and optimization of retrieval-augmented generation systems—read the full book on Packt:  Author BioJia Huang is a Lead Research Engineer at A*STAR (Agency for Science, Technology and Research), Singapore, where his work focuses on NLP, large language models, and applied AI engineering. With over twenty years of experience leading large-scale AI and data projects across government, finance, healthcare, and e-commerce, he brings an unusually practical lens to technically rigorous subjects. In recent years, his research has primarily focused on NLP pre-trained large models and FinTech applications. He is the author of six bestselling technical books, including Hands-on AI Agent Development for Large Model Applications selected as one of JD Best Books of 2024, and GPT: How Large Models Are Built, named CSDN's Most Influential IT Book of 2023. His online RAG engineering course has been completed by over 10,000 students.