Snyk’s annual Capture the Flag event, Fetch the Flag 2025, is back! Hosted by Snyk and John Hammond, the event will take place on February 27, 2025, from 9am ET to 9pm ET. Engage in 20+ real-world security challenges, enhance your professional development & earn CPE credits.

Hi ,

In today’sExpert Insight we bring you an excerpt from the book, Coding with ChatGPT and Other LLMs, which discusses using LLMs like ChatGPT to explain and document code, specifically explaining the Shapiro-Wilk normality test code.

News Highlights: Python in Visual Studio Code – February 2025 Release adds no-config debugging, test discovery cancellation, Native REPL, and AI code actions; CLI2 v4 introduces a simplified CLI, Sphinx integration, and 12-factor config; and Python docs expose risks in legacy web development with insecure CGI code.

My top 5 picks from today’s learning resources:

And, in From the Cutting Edge, we introduce dtaianomaly, an open-source Python library that standardises and simplifies time series anomaly detection for both researchers and industry, offering a scikit-learn-inspired API, extensibility for developing new models, and tools for preprocessing, benchmarking, and visual analysis.

Stay awesome!

Divya Anne Selvaraj

Editor-in-Chief

Sign Up|Advertise

In "dtaianomaly: A Python library for time series anomaly detection," Carpentier et al. introduce an open-source Python library for time series anomaly detection. The library aims to bridge the gap between cutting-edge research and practical applications in business and industry.

Time series anomaly detection (TSAD) involves identifying outliers or abnormal observations in time series data. Accurate anomaly detection is crucial in various fields, such as healthcare, finance, and IoT. However, many existing tools struggle to effectively integrate state-of-the-art research with practical industrial applications. dtaianomaly addresses this issue by simplifying anomaly detection for practitioners while providing a framework for researchers to extend the tool with new methods.

Key concepts in this domain include anomaly detection, which refers to the identification of unusual patterns in data; time series data, which consists of data points indexed by time; and APIs, which offer a standard interface for communication between software systems.

dtaianomaly is relevant to both researchers and industry practitioners. Researchers can use it to rapidly implement and test new algorithms, while businesses can apply its established tools for practical anomaly detection tasks, such as in sensor data or financial transactions. The library's user-friendly documentation and standardised API make it accessible to newcomers, while its extensibility makes it valuable for ongoing research.

dtaianomaly uses an object-oriented design similar to scikit-learn. Its core component, the BaseDetector, standardises the interface for all anomaly detection models, making it easy to integrate new methods. The library supports time series preprocessing, preparing data before anomaly detection, and offers an automated workflow for large-scale experimental validation. The LazyDataLoader ensures efficient handling of large datasets, while the Pipeline object allows for chaining preprocessing and detection steps. Evaluation is facilitated through both quantitative metrics (e.g., AUC, precision, recall) and qualitative visualisations.

The library also supports runtime and memory profiling, helping determine if a model is suitable for deployment on edge devices, which is particularly useful for resource-constrained industries. Its cross-platform compatibility and continuous testing ensure reliability.

You can learn more by reading the entire paper or accessing the library on GitHub.

Here’s an excerpt from “Chapter 3: Grounding Responses” in the book, Coding with ChatGPT and Other LLMs by Dr. Vincent Austin Hall.

Often, you’ll come across code that someone else (including an earlier version of you) has written and it’ll not make sense to you. You might have to go through each line and function and class, running it in your mind and testing each bit of code inan IDE.

Fortunately, there is now another

option: LLMs or chatbots. You can get one to be your personal code translator to clarify what is going on.

You can put some code in the chat window and ask the bot to explain what it does. You will generally also get detailed explanations when the chatbots giveyou code.

Statistical tests are useful for statisticians and data scientists, so, let’s do some. Jason Brownlee can help here [Brownlee_17Stats]. What is aShapiro-Wilk test?

Here is some code in Python for this, from 17 Statistical Hypothesis Tests in Python (Cheat Sheet):

# Example of the Shapiro-Wilk Normality Test
from scipy.stats import shapiro
data = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
stat, p = shapiro(data)
print('stat=%.3f, p=%.3f' % (stat, p))
if p > 0.05:
print('Probably Gaussian')
else:
print('Probably not Gaussian')

What does ChatGPT sayabout this?

Here isthe prompt:

Please explain this code to me, I don't understand it at all.

Here is ChatGPT’sresponse:

(I put it into a markdown interpreter to get it into a nice style, without the **around theheadings** [Dillinger].)

Certainly! The provided code is an example of how to use the Shapiro-Wilk Normality Test in Python, specifically with thescipy.statslibrary. This statistical test is used to determine whether a given sample follows a Gaussian (normal) distribution. Here’s a breakdown ofthe code:

# Importing the Shapiro-Wilk test from the scipy.stats module
from scipy.stats import shapiro
# Creating a sample dataset (in this case, a list of numbers)
data = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]
# Applying the Shapiro-Wilk test to the dataset
stat, p = shapiro(data)
# Printing the test statistic and the p-value
print('stat=%.3f, p=%.3f' % (stat, p))
# Checking the p-value and making a decision based on a significance level (e.g., 0.05)
if p > 0.05:
print('Probably Gaussian') # If p > 0.05, we fail to reject the null hypothesis, suggesting the data is probably Gaussian.
else:
print('Probably not Gaussian') # If p <= 0.05, we reject the null hypothesis, suggesting the data is probably not Gaussian.

Here’s astep-by-stepexplanation:

from scipy.stats import shapiro

This line imports theshapirofunction from thescipy.statsmodule.

data = [0.873, 2.817, 0.121, -0.945, -0.055, -1.436, 0.360, -1.478, -1.637, -1.869]

This line creates a list named data containing a set ofnumerical values.

stat, p = shapiro(data)

This line applies the Shapiro-Wilk test to the dataset, and the results are stored in two variables: stat (the test statistic) and p (the p-value).

print('stat=%.3f, p=%.3f' % (stat, p))

This line prints the test statistic and the p-value with threedecimal places.

if p > 0.05:
print('Probably Gaussian')
else:
print('Probably not Gaussian')

Based on a significance level (0.05is commonly used), this code prints whether the dataset is probably Gaussian or probably not Gaussian. Ifp > 0.05, we fail to reject the null hypothesis, suggesting the data is probably Gaussian. Ifp <= 0.05, we reject the null hypothesis, suggesting the data is probablynot Gaussian.

Here is my review: ChatGPT (not GPT-4, but definitely use that instead) tells us what the test is for, how to import the relevant library and where it comes from, how to create example data and what type it is, about the statistics returned, about printing, and what each statistics result means. That’sreally helpful!

What could it do better? Maybe it could suggest other tests that might be able to do this, when to not use the test, and whether it might be able to critique the code. We can askthe following:

ChatGPT, when should I not use thisstatistical test?

ChatGPT lists six points and details that the test gets better with larger sample sizes, but there is a limit. It might also find significance where there isn’t any, on very large sample sizes. ChatGPT also mentions non-Gaussian distributions and sensitivity to outliers and uses other tests and actions to confirm that the data is to benormally distributed.

There are far more details, which I won’t get into here, for brevity, but I’m sure if you ask the AI for it, it’ll give you goodinformation [ChatGPT].

We could also ask Gemini to critiquethe code:

Critique the code and give us some visualizations to help us understand theShapiro-Wilk test.

Let’s check how that compares with what Gemini says about the code...

Coding with ChatGPT and Other LLMs was published in November 2024. Packt library subscribers can continue reading the entire book for free.

And that’s a wrap.

We have an entire range of newsletters with focused content for tech pros. Subscribe to the ones you find the most usefulhere. The complete PythonPro archives can be foundhere.

If you have any suggestions or feedback, or would like us to find you a Python learning resource on a particular subject, just respond to this email!