Hi ,

Welcome to a brand new issue of PythonPro!

In today’sExpert Insight we bring you an excerpt from the recently published book, AI-Assisted Programming for Web and Machine Learning, which discusses how Copilot can assist in debugging and troubleshooting by adding error-handling features.

News Highlights: DBOS Transact launches with durable Python workflow recovery; Python in Excel now live for data analysis; Python 3.13 is coming October 2024 with new interpreter, JIT, and more; and Hackers use fake coding tests on GitHub to target Python developers.

Here are my top 5 picks from our learning resources today:

And, today’s Featured Study, introduces ComplexCodeEval, a benchmark designed to evaluate large code models (LCMs) in complex development environments.

Stay awesome!

Divya Anne Selvaraj

Editor-in-Chief

P.S.: This month’ssurvey is still live. Do take the opportunity to tell us what you think of PythonPro, request learning resources, and earn your one Packt Credit for this month.

Sign Up|Advertise

In ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code, Feng et al. introduce a new benchmark for assessing large code models (LCMs). The paper focuses on evaluating LCMs in real-world coding scenarios involving complex tasks and avoiding data leakage.

LCMs are AI models trained to handle coding tasks like code generation, completion, test case creation, and API recommendation. Existing benchmarks tend to evaluate LCMs on limited tasks, such as standalone code generation, without capturing the broader, more diverse challenges developers face. Additionally, they often overlook data leakage, where models are tested on data already seen during training, resulting in inflated performance scores.

ComplexCodeEval is a comprehensive benchmark designed to test LCMs on multiple coding tasks and scenarios, reflecting real-world programming challenges. It assesses how well LCMs perform in contexts that include dependencies on third-party libraries and the need to create test functions and recommend APIs.

This study is valuable for programmers and software engineers who use AI coding tools. ComplexCodeEval highlights which models perform best for tasks like generating Java code or recommending Python APIs, making it easier to select the right tools for complex programming tasks. It provides a realistic assessment of LCMs, avoiding inflated scores from limited or synthetic benchmarks.

For developers working on AI models, the study offers insights into how additional contextual information, such as dependencies and function histories, can significantly improve model performance.

To create ComplexCodeEval, the authors sourced Java and Python samples from GitHub repositories that relied on popular third-party libraries. Each sample was annotated with relevant metadata like API references, docstrings, and timestamps, simulating real-world coding tasks.

Ten LCMs, including StarCoder2, CodeLlama, DeepSeek-Coder, and GPT-3.5-Turbo, were tested on four tasks: code generation, code completion, API recommendation, and test case generation. CodeLlama-34B achieved the highest CodeBLEU score of 34.08 for Java code generation, and Python API recommendation saw an F1 score of 52.24.

The researchers tested the impact of adding context to the inputs provided to LCMs. Starting with basic function signatures and docstrings, they added more context (e.g., dependencies and library imports) and found that full context improved average CodeBLEU scores by 70.73% in Java and 31.90% in Python.

To assess data leakage, the team compared model performance on data created before and after the models’ knowledge cut-off dates. They found models performed better on leaked data, with average CodeBLEU scores increasing by 1.22 points in Java and 3.10 points in Python, demonstrating the importance of preventing data leakage in evaluations.

You can learn more by reading the entirepaper and accessing the ComplexCodeEvalGithub repository.

Here’s an excerpt from “Chapter 20: Increasing Efficiency with GitHub Copilot” in the book, AI-Assisted Programming for Web and Machine Learning by Christoffer Noring, Anjali Jain, Marina Fernandez, Ayşe Mutlu, and Ajit Jaokar, published in August 2024.

Debugging and troubleshooting

Being able to debug and troubleshoot code is an important skill to have. When you develop code, it seldom does what you need it to do the first time. You need

to be able to debug and troubleshoot code to figure out what’s wrong with it. The same applies if you have a reported bug; you need to be able to diagnose at what point it goes wrong. Copilot can help, and a good approach is using Copilot chat and asking it to add support for debugging and troubleshooting.

The following is a piece of sample code you’ve been given. It’s a REST API in the framework Flask for Python. It’s working code, but it might not be so easy to work with from a debugging and troubleshooting standpoint:

from flask import Flask, jsonify, request
import sqlite3
app = Flask(__name__)
# Create a connection to the database
conn = sqlite3.connect('products.db')
c = conn.cursor()
# Create a table for products if it doesn't exist
c.execute('''CREATE TABLE IF NOT EXISTS products
(id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT,
price REAL)''')
conn.commit()
@app.route('/products', methods=['GET'])
def get_products():
# Retrieve all products from the database
c.execute('SELECT * FROM products')
products = c.fetchall()
# Convert the products to a JSON response
response = []
for product in products:
response.append({
'id': product[0],
'name': product[1],
'price': product[2]
})
return jsonify(response)
@app.route('/products', methods=['POST'])
def add_product():
# Retrieve the product data from the request body
data = request.get_json()
name = data['name']
price = data['price']
# Insert the product into the database
c.execute('INSERT INTO products (name, price) VALUES (?, ?)', (name, price))
conn.commit()
# Return a success message
return jsonify({'message': 'Product added successfully'})
if __name__ == '__main__':
app.run(debug=True)

The above code works, but it doesn’t support debugging and troubleshooting; let’s see how we can add support for that by using Copilot Chat:

[Prompt]

Add support for troubleshooting and debugging

[End of prompt]

Copilot chat answers us with both text and code:

[Prompt response]

Sure, here’s an updated version of the Flask API that includes support for troubleshooting and debugging:

from flask import Flask, jsonify, request
import sqlite3
app = Flask(__name__)
# Create a connection to the database
conn = sqlite3.connect('products.db')
c = conn.cursor()
# Create a table for products if it doesn't exist
c.execute('''CREATE TABLE IF NOT EXISTS products
(id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT,
price REAL)''')
conn.commit()
@app.route('/products', methods=['GET'])
def get_products():
try:
# Retrieve all products from the database
c.execute('SELECT * FROM products')
products = c.fetchall()
# Convert the products to a JSON response
response = []
for product in products:
response.append({
'id': product[0],
'name': product[1],
'price': product[2]
})
return jsonify(response)
except Exception as e:
# Log the error and return an error message
app.logger.error(str(e))
return jsonify({'error': 'An error occurred while retrieving products'})
@app.route('/products', methods=['POST'])
def add_product():
try:
# Retrieve the product data from the request body
data = request.get_json()
name = data['name']
price = data['price']
# Insert the product into the database
c.execute('INSERT INTO products (name, price) VALUES (?, ?)', (name, price))
conn.commit()
# Return a success message
return jsonify({'message': 'Product added successfully'})
except Exception as e:
# Log the error and return an error message
app.logger.error(str(e))
return jsonify({'error': 'An error occurred while adding the product'})
if __name__ == '__main__':
app.run(debug=True)

The improvements are:

These changes will help us troubleshoot and debug the code.

[End of response]

Packt library subscribers can continue reading the entire book for free. You can buy AI-Assisted Programming for Web and Machine Learning,here.

Print discounts end in 5 days on the 22nd of September, 2024.

And that’s a wrap.

We have an entire range of newsletters with focused content for tech pros. Subscribe to the ones you find the most usefulhere. The complete PythonPro archives can be foundhere.

If you have any suggestions or feedback, or would like us to find you aPythonlearning resource on a particular subject, take thesurveyor just respond to this email!