Posts Tagged ‘Python’

I’ve created a Python script that converts WordPress export files to Blogger/Atom XML format. Here’s how to use it:

The script takes two command-line arguments:

• wordpress_export.xml: Path to your WordPress export XML file
• blogger_export.xml : Path where you want to save the converted Blogger/Atom XML file

To run the script:

python wordpress_to_blogger.py wordpress_export.xml blogger_export.xml

The script performs the following conversions:

• Converts WordPress posts to Atom feed entries
• Preserves post titles, content, publication dates, and authors
• Maintains categories as Atom categories
• Handles post status (published/draft)
• Preserves HTML content formatting
• Converts dates to ISO format required by Atom

The script uses Python’s built-in xml.etree.ElementTree module for XML processing and includes error handling to make it robust.
Some important notes:

• The script only converts posts (not pages or other content types)
• It preserves the HTML content of your posts
• It maintains the original publication dates
• It handles both published and draft posts
• The output is a valid Atom XML feed that Blogger can import

The file:

[python]#!/usr/bin/env python3
import xml.etree.ElementTree as ET
import sys
import argparse
from datetime import datetime
import re

def convert_wordpress_to_blogger(wordpress_file, output_file):
# Parse WordPress XML
tree = ET.parse(wordpress_file)
root = tree.getroot()

# Create Atom feed
atom = ET.Element(‘feed’, {
‘xmlns’: ‘http://www.w3.org/2005/Atom’,
‘xmlns:app’: ‘http://www.w3.org/2007/app’,
‘xmlns:thr’: ‘http://purl.org/syndication/thread/1.0’
})

# Add feed metadata
title = ET.SubElement(atom, ‘title’)
title.text = ‘Blog Posts’

updated = ET.SubElement(atom, ‘updated’)
updated.text = datetime.now().isoformat()

# Process each post
for item in root.findall(‘.//item’):
if item.find(‘wp:post_type’, {‘wp’: ‘http://wordpress.org/export/1.2/’}).text != ‘post’:
continue

entry = ET.SubElement(atom, ‘entry’)

# Title
title = ET.SubElement(entry, ‘title’)
title.text = item.find(‘title’).text

# Content
content = ET.SubElement(entry, ‘content’, {‘type’: ‘html’})
content.text = item.find(‘content:encoded’, {‘content’: ‘http://purl.org/rss/1.0/modules/content/’}).text

# Publication date
pub_date = item.find(‘pubDate’).text
published = ET.SubElement(entry, ‘published’)
published.text = datetime.strptime(pub_date, ‘%a, %d %b %Y %H:%M:%S %z’).isoformat()

# Author
author = ET.SubElement(entry, ‘author’)
name = ET.SubElement(author, ‘name’)
name.text = item.find(‘dc:creator’, {‘dc’: ‘http://purl.org/dc/elements/1.1/’}).text

# Categories
for category in item.findall(‘category’):
category_elem = ET.SubElement(entry, ‘category’, {‘term’: category.text})

# Status
status = item.find(‘wp:status’, {‘wp’: ‘http://wordpress.org/export/1.2/’}).text
if status == ‘publish’:
app_control = ET.SubElement(entry, ‘app:control’, {‘xmlns:app’: ‘http://www.w3.org/2007/app’})
app_draft = ET.SubElement(app_control, ‘app:draft’)
app_draft.text = ‘no’
else:
app_control = ET.SubElement(entry, ‘app:control’, {‘xmlns:app’: ‘http://www.w3.org/2007/app’})
app_draft = ET.SubElement(app_control, ‘app:draft’)
app_draft.text = ‘yes’

# Write the output file
tree = ET.ElementTree(atom)
tree.write(output_file, encoding=’utf-8′, xml_declaration=True)

def main():
parser = argparse.ArgumentParser(description=’Convert WordPress export to Blogger/Atom XML format’)
parser.add_argument(‘wordpress_file’, help=’Path to WordPress export XML file’)
parser.add_argument(‘output_file’, help=’Path to output Blogger/Atom XML file’)

args = parser.parse_args()

try:
convert_wordpress_to_blogger(args.wordpress_file, args.output_file)
print(f"Successfully converted {args.wordpress_file} to {args.output_file}")
except Exception as e:
print(f"Error: {str(e)}")
sys.exit(1)

if __name__ == ‘__main__’:
main()[/python]

Encoding is essential for handling text, binary data, and secure transmission across applications. Understanding advanced encoding techniques can help prevent data corruption and ensure smooth interoperability across systems. This post explores key encoding challenges and how Java/Kotlin, Node.js, and Python tackle them.

1️⃣ Handling Special Unicode Characters (Emoji, Accents, RTL Text)

Java/Kotlin

Java uses UTF-16 internally, but for external data (JSON, databases, APIs), explicit encoding is required:

String text = "🔧 Café مرحبا";
byte[] utf8Bytes = text.getBytes(StandardCharsets.UTF_8);
String decoded = new String(utf8Bytes, StandardCharsets.UTF_8);
System.out.println(decoded); // 🔧 Café مرحبا

✅ Tip: Always specify StandardCharsets.UTF_8 to avoid platform-dependent defaults.

Node.js

const text = "🔧 Café مرحبا";
const utf8Buffer = Buffer.from(text, 'utf8');
const decoded = utf8Buffer.toString('utf8');
console.log(decoded); // 🔧 Café مرحبا

✅ Tip: Using an incorrect encoding (e.g., latin1) may corrupt characters.

Python

text = "🔧 Café مرحبا"
utf8_bytes = text.encode("utf-8")
decoded = utf8_bytes.decode("utf-8")
print(decoded)  # 🔧 Café مرحبا

✅ Tip: Python 3 handles Unicode by default, but explicit encoding is always recommended.

2️⃣ Encoding Binary Data for Transmission (Base64, Hex, Binary Files)

Java/Kotlin

byte[] data = "Hello World".getBytes(StandardCharsets.UTF_8);
String base64Encoded = Base64.getEncoder().encodeToString(data);
byte[] decoded = Base64.getDecoder().decode(base64Encoded);
System.out.println(new String(decoded, StandardCharsets.UTF_8)); // Hello World

Node.js

const data = Buffer.from("Hello World", 'utf8');
const base64Encoded = data.toString('base64');
const decoded = Buffer.from(base64Encoded, 'base64').toString('utf8');
console.log(decoded); // Hello World

Python

import base64
data = "Hello World".encode("utf-8")
base64_encoded = base64.b64encode(data).decode("utf-8")
decoded = base64.b64decode(base64_encoded).decode("utf-8")
print(decoded)  # Hello World

✅ Tip: Base64 encoding increases data size (~33% overhead), which can be a concern for large files.

3️⃣ Charset Mismatches and Cross-Language Encoding Issues

A file encoded in ISO-8859-1 (Latin-1) may cause garbled text when read using UTF-8.

Java/Kotlin Solution:

byte[] bytes = Files.readAllBytes(Paths.get("file.txt"));
String text = new String(bytes, StandardCharsets.ISO_8859_1);

Node.js Solution:

const fs = require('fs');
const text = fs.readFileSync("file.txt", { encoding: "latin1" });

Python Solution:

with open("file.txt", "r", encoding="ISO-8859-1") as f:
    text = f.read()

✅ Tip: Always specify encoding explicitly when working with external files.

4️⃣ URL Encoding and Decoding

Java/Kotlin

String encoded = URLEncoder.encode("Hello World!", StandardCharsets.UTF_8);
String decoded = URLDecoder.decode(encoded, StandardCharsets.UTF_8);

Node.js

const encoded = encodeURIComponent("Hello World!");
const decoded = decodeURIComponent(encoded);

Python

from urllib.parse import quote, unquote
encoded = quote("Hello World!")
decoded = unquote(encoded)

✅ Tip: Use UTF-8 for URL encoding to prevent inconsistencies across different platforms.

Conclusion: Choosing the Right Approach

• Java/Kotlin: Strong type safety, but requires careful Charset management.
• Node.js: Web-friendly but depends heavily on Buffer conversions.
• Python: Simple and concise, though strict type conversions must be managed.

📌 Pro Tip: Always be explicit about encoding when handling external data (APIs, files, databases) to avoid corruption.

While Python dominates in model training, Java is becoming increasingly vital for deploying and serving AI/ML models in production. Its performance, stability, and enterprise integration capabilities make it a strong contender.

Java Example: Real-time Object Detection with DL4J and OpenCV

[java]
import …

public class ObjectDetection {

public static void main(String[] args) {
String modelPath = "yolov3.weights";
String configPath = "yolov3.cfg";
String imagePath = "image.jpg";
Net net = Dnn.readNet(modelPath, configPath);
Mat image = imread(imagePath);
Mat blob = Dnn.blobFromImage(image, 1 / 255.0, new Size(416, 416), new Scalar(0, 0, 0), true, false);

net.setInput(blob);

MatVector detections = net.forward(); // Inference

// Process detections (bounding boxes, classes, confidence)
// … (complex logic for object detection results)
// Draw bounding boxes on the image
// … (OpenCV drawing functions)
imwrite("detected_objects.jpg", image);
}
}

[/java]

Python Example: Similar Object Detection with OpenCV and YOLO

[python]

import numpy as np

net = cv2.dnn.readNet("yolov3.weights", "yolov3.cfg")
image = cv2.imread("image.jpg")
blob = cv2.dnn.blobFromImage(image, 1/255.0, (416, 416), swapRB=True, crop=False)
net.setInput(blob)
detections = net.forward()

# Process detections (bounding boxes, classes, confidence)
# … (simpler logic, NumPy arrays)
# Draw bounding boxes on the image
# … (OpenCV drawing functions)
cv2.imwrite("detected_objects.jpg", image)
[/python]

Comparison and Insights:

• Syntax and Readability: Python’s syntax is generally more concise and readable for data science and AI tasks. Java, while more verbose, offers strong typing and better performance for production deployments.
• Library Ecosystem: Python’s ecosystem (NumPy, OpenCV, TensorFlow, PyTorch) is more mature and developer-friendly for AI/ML development. Java, with libraries like DL4J, is catching up, but its strength lies in enterprise integration and performance.
• Performance: Java’s performance is often superior to Python’s, especially for real-time inference and high-throughput applications.
• Enterprise Integration: Java’s ability to seamlessly integrate with existing enterprise systems (databases, message queues, APIs) is a significant advantage.
• Deployment: Java’s deployment capabilities are more robust, making it suitable for mission-critical AI applications.

Key Takeaways:

• Python is excellent for rapid prototyping and model training.
• Java excels in deploying and serving AI/ML models in production environments, where performance and reliability are paramount.
• The choice between Java and Python depends on the specific use case and requirements.

Bill Engels and Chris Fonnesbeck, both brilliant software developers from PyMC Labs, delivered an insightful 90-minute tutorial at PyData Global 2024 titled “Making Gaussian Processes Useful.” Aimed at demystifying Gaussian processes (GPs) for practicing data scientists, their session bridged the gap between theoretical complexity and practical application. Using baseball analytics as a motivating example, Chris introduced Bayesian modeling and GPs, while Bill provided hands-on strategies for overcoming computational and identifiability challenges. This post explores their comprehensive approach, offering actionable insights for leveraging GPs in real-world scenarios.

Bayesian Inference and Probabilistic Programming

Chris kicked off the tutorial by grounding the audience in Bayesian inference, often implemented through probabilistic programming. He described it as writing software with partially random outputs, enabled by languages like PyMC that provide primitives for random variables. Unlike deterministic programming, probabilistic programming allows modeling distributions over variables, including functions via GPs. Chris explained that Bayesian inference involves specifying a joint probability model for data and parameters, using Bayes’ formula to derive the posterior distribution. This posterior reflects what we learn about unknown parameters after observing data, with the likelihood and priors as key components. The computational challenge lies in the normalizing constant, a multidimensional integral that probabilistic programming libraries handle numerically, freeing data scientists to focus on model specification.

Hierarchical Modeling with Baseball Data

To illustrate Bayesian modeling, Chris used the example of estimating home run probabilities for baseball players. He introduced a simple unpooled model where each player’s home run rate is modeled with a beta prior and a binomial likelihood, reflecting hits over plate appearances. Using PyMC, this model is straightforward to implement, with each line of code corresponding to a mathematical component. However, Chris highlighted its limitations: players with few at-bats yield highly uncertain estimates, leaning heavily on the flat prior. This led to the introduction of hierarchical modeling, or partial pooling, where individual home run rates are drawn from a population distribution with hyperparameters (mean and standard deviation). This approach shrinks extreme estimates, producing more realistic rates, as seen when comparing unpooled estimates (with outliers up to 80%) to pooled ones (clustered below 10%, aligning with real-world data like Barry Bonds’ 15% peak).

Gaussian Processes as a Hierarchical Extension

Chris transitioned to GPs, framing them as a generalization of hierarchical models for continuous predictors, such as player age affecting home run rates. Unlike categorical groups, GPs model relationships where similarity decreases with distance (e.g., younger players’ performance is more similar). A GP is a distribution over functions, parameterized by a mean function (often zero) and a covariance function, which defines how outputs covary based on input proximity. Chris emphasized two key properties of multivariate Gaussians—easy marginalization and conditioning—that make GPs computationally tractable despite their infinite dimensionality. By evaluating a covariance function at specific inputs, a GP yields a finite multivariate normal, enabling flexible, nonlinear modeling without explicitly parameterizing the function’s form.

Computational Challenges and the HSGP Approximation

One of the biggest hurdles with GPs is their computational cost, particularly for latent GPs used with non-Gaussian data like binomial home run counts. Chris explained that the posterior covariance function requires inverting a matrix, which scales cubically with the number of data points (e.g., thousands of players). This makes exact GPs infeasible for large datasets. To address this, he introduced the Hilbert Space Gaussian Process (HSGP) approximation, which reduces cubic compute time to linear by approximating the GP with a finite set of basis functions. These functions depend on the data, while coefficients rely on hyperparameters like length scale and amplitude. Chris demonstrated implementing an HSGP in PyMC to model age effects, specifying 100 basis functions and a boundary three times the data range, resulting in a model that ran in minutes rather than years.

Practical Debugging with GPs

Bill took over to provide practical tips for fitting GPs, emphasizing their sensitivity to priors and the need for debugging. He revisited the baseball example, modeling batting averages with a hierarchical model before introducing a GP to account for age effects. Bill showed that a standard hierarchical model treats players as exchangeable, pooling information equally across all players. A GP, however, allows local pooling, where players of similar ages inform each other more strongly. He introduced the exponentiated quadratic covariance function, which uses a length scale to define “closeness” in age and a scale parameter for effect size. Bill highlighted common pitfalls, such as small length scales reducing a GP to a standard hierarchical model or large length scales causing identifiability issues with intercepts, and provided solutions like informative priors (e.g., inverse gamma, log-normal) to constrain length scales to realistic ranges.

Advanced GP Modeling for Slugging Percentage

Bill concluded with a sophisticated model for slugging percentage, a metric reflecting hitting power, using 10 years of baseball data. The model included player, park, and season effects, with an HSGP to capture age effects. He initially used an exponentiated quadratic covariance function but encountered sampling issues (divergences), a common problem with GPs. Bill fixed this by switching to a Matern 5/2 covariance function, which assumes less smoothness and better suits real-world data, and adopting a centered parameterization for stronger age effects. These changes reduced divergences to near zero, producing a reliable model. The resulting age curve peaked at 26, aligning with baseball wisdom, and showed a decline for older players, demonstrating the GP’s ability to capture nonlinear trends.

Key Takeaways and Resources

Bill and Chris emphasized that GPs extend hierarchical models by enabling local pooling over continuous variables, but their computational and identifiability challenges require careful handling. Informative priors, appropriate covariance functions (e.g., Matern over exponential quadratic), and approximations like HSGP are critical for practical use. They encouraged using PyMC for its high-level interface and the Nutpie sampler for efficiency, while noting alternatives like GPFlow for specialized needs. Their GitHub repository, linked below, includes slides and notebooks for further exploration, making this tutorial a valuable resource for data scientists aiming to apply GPs effectively.

Links:

• Bill Engels’ LinkedIn
• Chris Fonnesbeck’s LinkedIn
• PyMC Labs
• PyMC
• GitHub Repository
• PyData Global 2024
• NumFOCUS

Introduction

Vincent Warmerdam delves into the illusions often encountered in predictive modeling, highlighting the cognitive traps and statistical misconceptions that lead to overconfidence in model performance.

The Seduction of Spurious Correlations

Models often perform well on training data by exploiting noise rather than genuine signal. Vincent emphasizes critical thinking and statistical rigor to avoid being misled by deceptively strong results.

Building Robust Models

Using robust cross-validation, considering domain knowledge, and testing against out-of-sample data are vital strategies to counteract the illusion of predictive prowess.

Conclusion

Data science is not just coding and modeling — it requires constant skepticism, critical evaluation, and humility. Vincent reminds us to stay vigilant against the comforting but dangerous mirage of false predictability.

Introduction

Thomas Vachon shares insights into scaling data-driven products, blending machine learning, engineering, and user-centric design to create impactful and intelligent applications.

Key Ingredients for Success

Building intelligent products requires aligning data pipelines, model training, deployment infrastructure, and feedback loops. Vachon stresses the importance of cross-functional collaboration between data scientists, software engineers, and product teams.

Real-World Lessons

From architectural best practices to team organization strategies, Vachon illustrates how to navigate the complexity of scaling data initiatives sustainably.

Conclusion

Intelligent data products demand not only technical excellence but also thoughtful design, scalability planning, and user empathy from day one.

Watch the video

Introduction

Thierry Cordier and Valentin Laurent introduce MAPIE, a Python library within scikit-learn-contrib, designed for uncertainty quantification in machine learning models.

MAPIE on GitHub

Managing Uncertainty in Machine Learning

In AI applications — from autonomous vehicles to medical diagnostics — understanding prediction uncertainty is crucial. MAPIE uses conformal prediction methods to generate prediction intervals with controlled confidence, ensuring safer and more interpretable AI systems.

Key Features

MAPIE supports regression, classification, time series forecasting, and complex tasks like multi-label classification and semantic segmentation. It integrates seamlessly with scikit-learn, TensorFlow, PyTorch, and custom models.

Real-World Use Cases

By generating calibrated prediction intervals, MAPIE enables selective classification, robust decision-making under uncertainty, and provides statistical guarantees critical for safety-critical AI systems.

Conclusion

MAPIE empowers data scientists to quantify uncertainty elegantly, bridging the gap between predictive power and real-world reliability.

Exploring Quarto Dashboard for Impactful and Visual Communication

Watch the video

Introduction

Christophe Dervieux introduces us to Quarto Dashboard, a powerful open-source scientific and technical publishing system. Designed to create impactful visual communication directly from Jupyter Notebooks, Quarto enables the seamless creation of interactive charts, dashboards, and dynamic narratives.

Building Visual Communication with Quarto

Quarto extends standard markdown with advanced features tailored for scientific writing. It offers support for multiple computation engines, allowing narratives and executable code to merge into various outputs: PDF, HTML pages, websites, books, and especially dashboards. The dashboard format enhances data communication by organizing visual metrics in an efficient and impactful layout.

Using Quarto, rendering a Jupyter notebook becomes simple: with just a command-line instruction (quarto render), users can output polished, shareable dashboards. Additional extensions, such as those available in VS Code, JupyterLab, and Positron IDEs, streamline this experience further.

Dashboard Features and Design

Dashboards in Quarto organize content using components like cards, rows, columns, sidebars, and tabs. Each element structures visual outputs like plots, tables, and value boxes, allowing maximum clarity. Customization is straightforward, leveraging YAML configuration and Bootstrap-based theming. Users can create multi-page navigation, interactivity through JavaScript libraries, and adapt layouts for specific audiences.

Recent updates even enable branding dashboards easily with SCSS themes, making Quarto ideal for both scientific and corporate environments.

Conclusion

Quarto revolutionizes technical communication by enabling scientists and analysts to produce professional-grade dashboards and publications effortlessly. Christophe’s session at PyData Paris 2023 showcased the simplicity, power, and flexibility Quarto brings to modern data storytelling.

At PyConUS2024, Reuven M. Lerner, an esteemed independent trainer and consultant from Lerner Consulting, presented an exhaustive tutorial titled “All About Decorators.” This session aimed to strip away the perceived complexity surrounding Python’s decorators, revealing their inherent power and versatility. Reuven’s approach was to guide attendees through the fundamental principles, practical applications, and advanced techniques of decorators, empowering developers to leverage this elegant feature for cleaner, more maintainable code. The tutorial offered a deep dive into what decorators are, their internal mechanics, how to construct them, and when to employ them effectively in various programming scenarios.

Functions as First-Class Citizens: The Foundation of Decorators

At the heart of Python’s decorator mechanism lies the concept of functions as first-class objects. Reuven Lerner began by elucidating this foundational principle, demonstrating how functions in Python are not merely blocks of code but entities that can be assigned to variables, passed as arguments to other functions, and returned as values from functions. This flexibility is pivotal, as it allows for the dynamic manipulation and extension of code behavior without altering the original function definition.

He illustrated this with simple examples, such as wrapping print statements with additional lines of text. Initially, this might involve manually calling a “wrapper” function that takes another function as an argument. This manual wrapping, while functional, quickly becomes cumbersome when applied repeatedly across numerous functions. Reuven showed how this initial approach, though verbose, laid the groundwork for understanding the more sophisticated decorator syntax. The ability to treat functions like any other data type in Python empowers developers to create highly modular and adaptable code structures, a cornerstone for building robust and scalable applications.

The Power of Closures: Functions Returning Functions

Building upon the concept of first-class functions, Reuven delved into the powerful notion of closures. A closure is a function that remembers the environment in which it was created, even after the outer function has finished executing. This is achieved when an inner function is defined within an outer function, and the outer function returns this inner function. The inner function retains access to the outer function’s local variables, forming a “closure” over that environment.

Lerner’s explanations made it clear that closures are a critical stepping stone to understanding how decorators work. The decorator pattern fundamentally relies on an outer function (the decorator) that takes a function as input, defines an inner “wrapper” function, and then returns this wrapper. This wrapper function “closes over” the original function and any variables from the decorator’s scope, allowing it to execute the original function while adding pre- or post-processing logic. This concept is essential for functions that need to maintain state or access context from their creation environment, paving the way for more sophisticated decorator implementations.

Implementing the Decorator Pattern Manually

Before introducing Python’s syntactic sugar for decorators, Reuven walked attendees through the manual implementation of the decorator pattern. This hands-on exercise was crucial for demystifying the @ syntax and showing precisely what happens under the hood. The manual approach involves explicitly defining a “decorator function” that accepts another function (the “decorated function”) as an argument. Inside the decorator function, a new “wrapper function” is defined. This wrapper function contains the additional logic to be executed before or after the decorated function, and it also calls the decorated function. Finally, the decorator function returns this wrapper.

The key step, as Reuven demonstrated, is then reassigning the original function’s name to the returned wrapper function. For instance, my_function = decorator(my_function). This reassignment effectively replaces the original my_function with the new, enhanced wrapper function, without changing how my_function is called elsewhere in the code. This explicit, step-by-step construction revealed the modularity and power of decorators, highlighting how they can seamlessly inject new behavior into existing functions while preserving their interfaces. Understanding this manual process is fundamental to debugging and truly mastering decorator usage.

Python’s Syntactic Sugar: The @ Operator

Once the manual mechanics of decorators were firmly established, Reuven introduced Python’s elegant and widely adopted @ syntax. This syntactic sugar simplifies the application of decorators significantly, making code more readable and concise. Instead of the explicit reassignment, my_function = decorator(my_function), the @ symbol allows developers to place the decorator name directly above the function definition:

@decorator
def my_function():
#### ...

Lerner emphasized that this @ notation is merely a convenience for the manual wrapping process discussed earlier. It performs the exact same operation of passing my_function to decorator and reassigning the result back to my_function. This clarity was vital, as many developers initially find the @ syntax magical. Reuven illustrated how this streamlined syntax enhances code readability, especially when multiple decorators are applied to a single function, or when creating custom decorators for specific tasks. The @ operator makes decorators a powerful and expressive tool in the Python developer’s toolkit, promoting a clean separation of concerns and encouraging reusable code patterns.

Practical Applications of Decorators

The tutorial progressed into a series of practical examples, showcasing the diverse utility of decorators in real-world scenarios. Reuven presented various use cases, from simple enhancements to more complex functionalities:

• “Shouter” Decorator: A classic example where a decorator modifies the output of a function, perhaps by converting it to uppercase or adding exclamation marks. This demonstrates how decorators can alter the result returned by a function.
• Timing Function Execution: A highly practical application involves using a decorator to measure the execution time of a function. This is invaluable for performance profiling and identifying bottlenecks in code. The decorator would record the start time, execute the function, record the end time, and then print the duration, all without cluttering the original function’s logic.
• Input and Output Validation: Decorators can be used to enforce constraints on function arguments or to validate the return value. For instance, a decorator could ensure that a function only receives positive integers or that its output adheres to a specific format. This promotes data integrity and reduces errors.
• Logging and Authentication: More advanced applications include decorators for logging function calls, handling authentication checks before a function executes, or implementing caching mechanisms to store and retrieve previously computed results.

Through these varied examples, Reuven underscored that decorators are not just an academic curiosity but a powerful tool for injecting cross-cutting concerns (like logging, timing, validation) into functions in a clean, non-intrusive manner. This approach adheres to the “separation of concerns” principle, making code more modular, readable, and easier to maintain.

Decorators with Arguments and Stacking Decorators

Reuven further expanded the attendees’ understanding by demonstrating how to create decorators that accept arguments. This adds another layer of flexibility, allowing decorators to be configured at the time of their application. To achieve this, an outer function is required that takes the decorator’s arguments and then returns the actual decorator function. This creates a triple-nested function structure, where the outermost function handles arguments, the middle function is the actual decorator that takes the decorated function, and the innermost function is the wrapper.

He also covered the concept of “stacking decorators,” where multiple decorators are applied to a single function. When decorators are stacked, they are executed from the bottom up (closest to the function definition) to the top (furthest from the function definition). Each decorator wraps the function that results from the application of the decorator below it. This allows for the sequential application of various functionalities to a single function, building up complex behaviors from smaller, modular units. Reuven carefully explained the order of execution and how the output of one decorator serves as the input for the next, providing a clear mental model for understanding chained decorator behavior.

Preserving Metadata with functools.wraps

A common side effect of using decorators is the loss of the decorated function’s original metadata, such as its name (__name__), docstring (__doc__), and module (__module__). When a decorator replaces the original function with its wrapper, the metadata of the wrapper function is what becomes visible. This can complicate debugging, introspection, and documentation.

Reuven introduced functools.wraps as the standard solution to this problem. functools.wraps is itself a decorator that can be applied to the wrapper function within your custom decorator. When used, it copies the relevant metadata from the original function to the wrapper function, effectively “wrapping” the metadata along with the code.

from functools import wraps

def my_decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
##### ... decorator logic ...
        return func(*args, **kwargs)
    return wrapper

This simple yet crucial addition ensures that decorated functions retain their original identity and documentation, making them behave more like their undecorated counterparts. Reuven stressed the importance of using functools.wraps in all custom decorators to avoid unexpected behavior and maintain code clarity, a best practice for any Python developer working with decorators.

Extending Decorator Concepts: Classes as Decorators and Decorating Classes

Towards the end of the tutorial, Reuven touched upon more advanced decorator patterns, including the use of classes as decorators and the application of decorators to classes themselves.

• Classes as Decorators: While functions are the most common way to define decorators, classes can also serve as decorators. This is achieved by implementing the __call__ method in the class, making instances of the class callable. The __init__ method typically takes the function to be decorated, and the __call__ method acts as the wrapper, executing the decorated function along with any additional logic. This approach can be useful for decorators that need to maintain complex state or have more intricate setup/teardown procedures.
• Decorating Classes: Decorators can also be applied to classes, similar to how they are applied to functions. When a class is decorated, the decorator receives the class object itself as an argument. The decorator can then modify the class, for example, by adding new methods, altering existing ones, or registering the class in some way. This is often used in frameworks for tasks like dependency injection, ORM mapping, or automatically adding mixins.

Reuven’s discussion of these more advanced scenarios demonstrated the full breadth of decorator applicability, showcasing how this powerful feature can be adapted to various architectural patterns and design needs within Python programming. This segment provided a glimpse into how decorators extend beyond simple function wrapping to influence the structure and behavior of entire classes, offering a flexible mechanism for meta-programming.

Links:

• Reuven M. Lerner on LinkedIn
• Reuven M. Lerner’s Professional Website
• LernerPython Website (Courses)
• Video URL

Hashtags: #Python #Decorators #PyConUS2024 #Programming #SoftwareDevelopment #Functions #Closures #PythonTricks #CodeQuality #ReuvenMLerner #LernerConsulting #LernerPython

Watch the video

Introduction

The team from INSEE presents Onyxia, an open-source, Kubernetes-based platform designed to offer flexible, collaborative, and powerful cloud environments for data scientists.

Rethinking Data Science Infrastructure

Traditional local development faces issues like configuration divergence, data duplication, and limited compute resources. Onyxia solves these by offering isolated namespaces, integrated object storage, and a seamless user interface that abstracts Kubernetes and S3 complexities.

Versatile Deployment

With a few clicks, users can launch preconfigured environments — including Jupyter notebooks, VS Code, Postgres, and MLflow — empowering fast innovation without heavy IT overhead. Organizations can extend Onyxia by adding custom services, ensuring future-proof, evolvable data labs.

Success Stories

Adopted across French universities and research labs, Onyxia enables students and professionals alike to work in secure, scalable, and fully-featured environments without managing infrastructure manually.

Conclusion

Onyxia democratizes access to powerful cloud tools for data scientists, streamlining collaboration and fostering innovation.