10 Data Visualization and Other Tricks

So far we have focused on using Python and Numpy for “crunching numbers”…but crunching numbers is not enough! It is necessary to present the results of our efforts so that others can learn from them. In this chapter we’ll take a look at a few ways to make data look good.

Figure 10.1: There are many different ways to visualize the same data.

Data visualization is actually an active area of academic research. There are always ways to present data that make it easier for a human see things more clearly. Below are some examples.

Figure 10.2: “Probably the best statistical graphic ever drawn, this map by Charles Joseph Minard portrays the losses suffered by Napoleon’s army in the Russian campaign of 1812. Beginning at the Polish-Russian border, the thick band shows the size of the army at each position. The path of Napoleon’s retreat from Moscow in the bitterly cold winter is depicted by the dark lower band, which is tied to temperature and time scales.” - Edward Tufte.

Figure 10.3: This is a funny variation of the famous “March on Russia” graphic, but looks at plots of famous movies.

Figure 10.4: This plot shows the scales of different sources of radiation, trying to point out that radiation is not an “ON/OFF” phenomena. This graph was produced by Randall Munroe in the wake of the Fukushima disaster and is discussed here.

Figure 10.5: These “circos” graphs were developed to show how genes interact with each other.

10.1 Introduction to Matplotlib

Other plotting packages

There are several other options for plotting in Python. Each package below generally tries to focus on making graphs look nicer by default since TBH Matplotlib’s default formatting is rather bland.

Matplotlib is the main package for plotting in Python. There are several other contenders which offer their own perspectives on plotting, but Matplotlib is really all we need.

Matplotlib is part of the Numpy/Scipy ecosystem of packages. It is designed to work with Numpy arrays.

We have seen several examples of Matplotlib graphs already in the course notes, but now we’ll take a deep dive into how we get the plots we want. Matplotlib also offers a gallery of sample plots, with code to get you started.

10.1.1 The Standard Charts

10.1.1.1 Scatter Plots

The scatter plot, or x-y plot, is probably the most useful and commonly used type of plot. It allows us humans to see trends and relationships instantly.

Let’s plot a quadratic curve: \(y = ax^2 + bx + c\).

import numpy as np
1import matplotlib.pyplot as plt

a, b, c = 2, 1, -20
x = np.linspace(-10, 10, 100)
y = a*x**2 + b*x + c

2fig, ax = plt.subplots()
3ax.scatter(x, y);

1: Matplotlib supports an implicit interface through pyplot (such as plt.plot(...)) and an explicit object-oriented interface (such as ax.plot(...)). Here we use pyplot to create the figure, then call methods on its objects. The older pylab interface is discouraged; see the Matplotlib quick start.
2: subplots() creates a Figure, the whole canvas, and an Axes, a plotting region on that canvas. Each ordinary two-dimensional Axes contains an x Axis and a y Axis, which manage scales, ticks, and tick labels. The returned variables refer to objects whose methods we can call to add data or change properties.
3: Here we tell the ax object to populate itself with a scatter plot of the x and y data. In a Jupyter cell, the optional trailing semicolon (;) suppresses the displayed representation of the returned object. It does not suppress print() output or warnings.

Figure and Axes Handles

The concept of a “handle” is common throughout programming. It is not an arcane inscrutable term…it means exactly what it sounds like: it allows you to hold and manipulate an object! It has exactly the same role for a computational object as for objects in the real world. Consider coffee:

Matplotlib has some features for dealing with these handles, like finding one that you’ve lost.

import matplotlib.pyplot as plt
x, y = [1, 2, 3, 4], [1, 4, 9, 16]
1fig, _ = plt.subplots(figsize=[2, 2])
2ax = fig.axes
3ax[0].plot(x, y)

1: Assume that we don’t have the ax handle here.
2: This will retrieve the axes handles which are contained within fig
3: plot draws connected lines by default, while scatter draws separate markers and allows a different size or colour for each point. plot(y) also allows x-data to be omitted, using range(len(y)) instead. Choose a line when the order of connected observations is meaningful; use a scatter plot to show individual x-y pairs.

import matplotlib.pyplot as plt
x, y = [1, 2, 3, 4], [1, 4, 9, 16]
1plt.subplots(figsize=[2, 2])
2fig = plt.gcf()
3ax = fig.axes
ax[0].plot(x, y)

1: Assume we don’t have any handles
2: plt.gcf() returns the current figure, creating one if none exists. The current figure can be an earlier figure that was made active again.
3: Then we can grab the Axes objects too

Customizing the formatting of Matplotlib plots can become quite verbose since there are so many options!. Let’s look at a few of the normal options:

a, b, c = 2, 1, -20
x = np.linspace(-10, 10, 21)
y = a*x**2 + b*x + c

fig, ax = plt.subplots()
1ax.scatter(
    x=x, 
    y=y, 
    s=np.abs(x*100) + 10, 
    c='r',
    marker='s', 
    alpha=0.5,
    edgecolors='k',
    linewidth=2,
2)
3ax.set_xlabel('Some X Values')
ax.set_ylabel('Some Y Values');

1: This code style is allowed by PEP8 and is quite helpful when a function has many arguments. This layout not only allows the coder to see each argument clearly, but it’s also easy to add and remove arguments by just inserting/deleting lines. Even the closing bracket is on its own line so that every argument can be easily adjusted in this way.
2: These are some common options accepted by scatter(), not its complete argument list. Additional properties depend on the kind of object the plotting method creates; consult the scatter documentation. Here s controls marker area in points squared, and linewidth controls the marker edge width in points.
3: Here is an example of applying some formatting to an Axes using the ax handle. There are a lot of things we can do here, which will also be explored below.

As mentioned above there are many more arguments that can be passed to matplotlib functions. In the following code block we use keyword-unpacking introduced in Chapter 7. We will put all our desired properties into a dict, then pass it to the function:

formatting_args = {
1    'linestyle': '--',
    'hatch': '/',
    'marker': 's', 
    'alpha': 0.5,
    'edgecolors': 'k',
    'linewidth': 2,
    'c': 'g',
}

a, b, c = 2, 1, -20
x = np.linspace(-10, 10, 21)
y = a*x**2 + b*x + c

fig, ax = plt.subplots()
2ax.scatter(x=x, y=y, s=np.abs(x*100) + 10, **formatting_args);

1: These properties apply to the collection of markers created by scatter: linestyle affects marker outlines and hatch adds a pattern inside them. They do not connect the points. Other plotting methods create different objects and need not accept the same keywords.
2: Here we pass all the values in the formatting_args dict using the ** notation which tells Python to assign each “key” to the corresponding keyword argument.

For the sake of completeness, let’s also review the positional argument unpacking as well:

formatting_args = {
    'marker': 's', 
    'alpha': 0.5,
    'edgecolors': 'k',
    'linewidth': 2,
}

a, b, c = 2, 1, -20
x = np.linspace(-10, 10, 21)
y = a*x**2 + b*x + c

1data_args = (x, y, np.abs(x*100) + 10)

fig, ax = plt.subplots()
2ax.scatter(*data_args, **formatting_args);

1: These need to be in the same order as they are expected by the scatter() method since the * notation indicates these are unpacked as “position arguments”.
2: This conventional order puts unpacked positional arguments before keyword arguments. Ordinary positional arguments must precede keyword arguments; Python also permits some calls with *args after explicit keywords, but this is usually harder to read.

10.1.1.2 Histograms and Bar Charts

Figure 10.7: It would be great if printers actually worked this easily.

A histogram groups numerical observations into bins and counts how many fall in each bin; the bin widths affect the appearance of the distribution. A bar chart instead compares separate categories. We talked briefly in the last chapter about how to use Numpy to generate data following different common distributions.

vals = np.random.normal(loc=1, scale=0.1, size=10000)

fig, ax = plt.subplots()
h = ax.hist(vals, bins=25, edgecolor='k', alpha=.5)

Another common use of bar charts is when the ‘x-axis’ is a category rather than a numerical value. Matplotlib lets us specify both an ‘x’ location for the bars,and individual labels:

1data = {
  'apples': 10,
  'pears': 3,
  'oranges': 7,
  'bananas': 9,
}

fig, ax = plt.subplots()
2ax.bar(x=data.keys(), height=data.values());

1: It is not necessary to store the data in a dict but it is obviously quite convenient.
2: Since the data are in a dict we can use keys() as the x-coord (i.e. category) and values() as the y-coord (i.e. height of the bars).

10.1.2 Formatting `ax` and `fig` objects

Getters and Setters

In programming there is a “design pattern” called getters and setters. This means functions which “get” data and functions which “set” data. It is common for these functions to start with the words get and set. In Table 10.1 we can see a selection of set methods on ax objects, but there are also a lot of get methods too.

In fact, when you write data to a dict in Python, like d[item] = value, behind the scenes Python actually runs a hidden function called __setitem__() with item and value as arguments like d.__setitem__(item, value). You can try this yourself! You can also retrieve data using d.__getitem__(item).

In the above sections we saw how to format the “data”, such as marker colors and sizes. We also often want/need to format the graphs themselves, like axis limits and background colors. The ax and fig objects have a lot of methods attached to them which allow us to manipulate the formatting. Of this huge list, the ones that start with set are for “setting” properties. Below is a list of methods on the ax objects. A similar selection of methods can be found on the fig object as well.

Table 10.1: Selected setter methods attached to an Axes object; available methods can vary by Matplotlib version

`set`	`set_adjustable`	`set_agg_filter`
`set_alpha`	`set_anchor`	`set_animated`
`set_aspect`	`set_autoscale_on`	`set_autoscalex_on`
`set_autoscaley_on`	`set_axes_locator`	`set_axis_off`
`set_axis_on`	`set_axisbelow`	`set_box_aspect`
`set_clip_box`	`set_clip_on`	`set_clip_path`
`set_facecolor`	`set_fc`	`set_figure`
`set_forward_navigation_events`	`set_frame_on`	`set_gid`
`set_in_layout`	`set_label`	`set_mouseover`
`set_navigate`	`set_navigate_mode`	`set_path_effects`
`set_picker`	`set_position`	`set_prop_cycle`
`set_rasterization_zorder`	`set_rasterized`	`set_sketch_params`
`set_snap`	`set_subplotspec`	`set_title`
`set_transform`	`set_url`	`set_visible`
`set_xbound`	`set_xlabel`	`set_xlim`
`set_xmargin`	`set_xscale`	`set_xticklabels`
`set_xticks`	`set_ybound`	`set_ylabel`
`set_ylim`	`set_ymargin`	`set_yscale`
`set_yticklabels`	`set_yticks`	`set_zorder`

Let’s see some of these in action:

data = {
  'apples': 10,
  'pears': 3,
  'oranges': 7,
  'bananas': 9,
}

fig, ax = plt.subplots()
ax.bar(x=data.keys(), height=data.values())

ax.set_facecolor('grey')
ax.set_yticks([0, 2, 4, 6, 8, 10])
ax.set_yticklabels(('a', 'b', 'c', 'd', 'e', 'f'))
ax.set_ylabel('Some nonsense values')
ax.set_xlabel('The fruit name')
ax.set_xmargin(0.1)

fig.set_facecolor('silver')
fig.set_edgecolor('black')
fig.set_linewidth(10)
fig.set_figwidth(4);

Note: This graph does not look very good. This is just a demonstration of how to go about making it look good.

10.1.3 Named Colors and Color Maps

Specifying colors is one of the main ways which we customize plots. Matplotlib has a large list of “named colors” which can be passed to any of the color-related argument like set_facecolor(). These colors are given below:

Figure 10.8: Table showing the officially recognized “named colors” in Matplotlib. More information can be found in the documentation.

Matplotlib also has a small subset of colors which work well together called the “tableau palette”. These are shown below:

Figure 10.9: The subset of colors known as the “tableau palette” which work well when plotting many datasets together.

And here is an example of using the “tableau palette” when combining 4 different data sets.

x1, y1 = np.random.rand(2, 1000)
x2, y2 = np.random.normal(1, 0.1, (2, 1000))
x3, y3 = np.random.weibull(a=2, size=(2, 1000))
x4, y4 = np.random.gamma(scale=1, shape=1, size=(2, 1000))

fig, ax = plt.subplots()
ax.scatter(x4, y4, marker='x', alpha=0.5, c='tab:orange', label='Gamma')
ax.scatter(x3, y3, marker='+', alpha=0.5, c='tab:green', label='Weibull')
ax.scatter(x2, y2, marker='s', alpha=0.25, c='tab:red', label='Normal')
ax.scatter(x1, y1, marker='.', alpha=0.15, c='tab:blue', label='Uniform')
ax.legend();

The Theory of Colormaps

In this talk, Nathaniel Smith, one of the main developers of numpy, explains how the team decided on a new default color map for Matplotlib.

Another way to use colors effectively is assigning each data point some different color based on some additional value other than just “x” and “y” locations.

x, y = np.random.gamma(scale=1, shape=1, size=(2, 1000))
color = (x**2 + y**2)**0.5

1fig, ax = plt.subplots(1, 2, figsize=(9, 4), layout='constrained')
2points0 = ax[0].scatter(x, y, c=color, vmin=0, vmax=color.max())
points1 = ax[1].scatter(x, y, c=color, cmap=plt.cm.turbo,
3                        vmin=0, vmax=color.max())
fig.colorbar(points0, ax=ax[0], label='Distance from origin')
fig.colorbar(points1, ax=ax[1], label='Distance from origin');

1: Here we create 2 subplots, which we’ll explore in more detail in the next section
2: We assign the array color to the c argument and Matplotlib will color each marker accordingly
3: We can override the default color map by passing a colormap to the cmap argument. Both panels use the same numerical limits; their colourbars show how colours map to distance. A sequential map such as the default viridis is usually easier to interpret for ordered magnitudes than a rainbow-like map such as turbo.

Use categorical colours for distinct groups, sequential colours for ordered magnitudes, and a diverging map for deviations around a meaningful centre such as zero. Labels and marker shapes help readers who cannot distinguish particular colours. There are quite a few colormaps provided by Matplotlib, as shown below:

Figure 10.10: Overview of available colormaps. See the Matplotlib documentation for more information and categorizations of the different maps.

A good tutorial on the subject of plotting with colors is given here

10.1.4 Multiple Plots

Matplotlib Gallery

The Matplotlib Gallery contains hundreds of examples to use as a starting point. You can browse this page until you find something that looks close to what you want.

It is pretty common to plot two things side-by-side, or even an N-by-N grid. Using the plt.subplots() function as our default approach allows this easily:

x1, y1 = np.random.rand(2, 1000)
x2, y2 = np.random.normal(1, 0.1, (2, 1000))

1fig, (ax1, ax2) = plt.subplots(1, 2)
2ax1.scatter(x1, y1, marker='.', c='tab:blue')
ax2.scatter(x2, y2, marker='.', c='tab:red');

1: With this 1-by-2 layout, subplots() returns a one-dimensional array of Axes objects, which we unpack into ax1 and ax2. A single subplot returns one Axes by default, while a grid with multiple rows and columns returns a two-dimensional array.
2: In this line and the next we add scatter plots to each of the axes we created.

In the above demo we created the exact number of axes when we “unpacked” the collection, but we can also just use the collection directly:

x1, y1 = np.random.rand(2, 1000)
x2, y2 = np.random.normal(1, 0.1, (2, 1000))
x3, y3 = np.random.weibull(a=2, size=(2, 1000))
x4, y4 = np.random.gamma(scale=1, shape=1, size=(2, 1000))

1fig, ax = plt.subplots(2, 2)
2ax[0][0].scatter(x1, y1, marker='.', c='tab:blue')
ax[0][1].scatter(x2, y2, marker='.', c='tab:red')
ax[1][0].scatter(x3, y3, marker='.', c='tab:green')
ax[1][1].scatter(x4, y4, marker='.', c='tab:orange')

1: Here we create a 2-by-2 grid of axes, and catch all of them in ax
2: To add data to each Axes we need to index into the ax collection according to the “row” and “column” location we want.

There is another version of the subplots() approach called subplot_mosaic() which is interesting since it gives us quite a lot of control over the layout. Instead of just a regular grid, we can do the following:

x1, y1 = np.random.rand(2, 1000)
x2, y2 = np.random.normal(1, 0.1, (2, 1000))
x3, y3 = np.random.weibull(a=2, size=(2, 1000))
x4, y4 = np.random.gamma(scale=1, shape=1, size=(2, 1000))

1fig, ax = plt.subplot_mosaic([['A', 'A', 'A', 'B'],
                              ['A', 'A', 'A', 'C'],
                              ['A', 'A', 'A', 'D']])

2ax['A'].scatter(x1, y1, marker='.', c='tab:blue')
ax['B'].scatter(x2, y2, marker='.', c='tab:red')
ax['C'].scatter(x3, y3, marker='.', c='tab:green')
ax['D'].scatter(x4, y4, marker='.', c='tab:orange')

1: We create a grid of where we want plots 'A' through 'D' to appear. Note that we can use any names we want like 'Panel A' or 'axis 1'.
2: We then use the names to index into the ax object to create each individual subplot.

10.1.5 Working with Images

We have already seen several examples of working with images. Images are a very common form of data since engineers use them for many things:

microscope images of surfaces or tiny features on a part or sample
aerial photographs of an area like a water treatment plant
maps with color coding for terrain type or elevation

Let’s take one more look at color images by reading in an image that is often used as a test case:

im = plt.imread('media/astronaut-eileen-collins.png')
print(im.shape)

fig, ax = plt.subplots(2, 2, layout='constrained')
ax[0][0].imshow(im)
ax[0][0].set_title('All Channels')
ax[0][1].imshow(im[..., 0], cmap=plt.cm.Reds)
ax[0][1].set_title('Red Channel')
ax[1][0].imshow(im[..., 1], cmap=plt.cm.Greens)
ax[1][0].set_title('Green Channel')
ax[1][1].imshow(im[..., 2], cmap=plt.cm.Blues)
ax[1][1].set_title('Blue Channel');

(512, 512, 4)

Number of channels

PNG stands for “Portable Network Graphics”. Depending on the file, imread can return a grayscale, RGB, or RGBA array. This image has four channels: red, green, blue, and alpha. Alpha controls opacity, with 0 fully transparent and 1 fully opaque in this floating-point array. Transparency is optional; not every PNG has an alpha channel. See the imread documentation.

Let’s look at this in greyscale. A grayscale image stores one intensity per pixel. The range depends on the data type and convention: for example, 0–255 for an unsigned 8-bit image or 0–1 for the floating-point conversion below. A two-dimensional scalar array can also represent elevation or region labels, but those values have their own units or meaning and need an appropriate colour scale.

1from skimage.color import rgb2gray

im = plt.imread('media/astronaut-eileen-collins.png')
2im2 = rgb2gray(im[..., :3])
fig, ax = plt.subplots(1, 2)
ax[0].imshow(im)
ax[1].imshow(im2, cmap=plt.cm.grey);

1: scikit-image is a separate image-processing package, imported as skimage. The distribution name used for installation need not match the import name; for example, scikit-learn is imported as sklearn. Hyphens are not allowed in identifiers in an ordinary Python import statement.
2: For this RGBA image, im[..., :3] selects the RGB channels and discards alpha; it does not composite transparent pixels onto a background. The ellipsis (...) fills in the preceding dimensions, as in NumPy indexing. For this three-dimensional image it is equivalent to im[:, :, :3]. rgb2gray combines the RGB channels using unequal weights to approximate perceived brightness.

Example 10.1 (Compute the average temperature in each month) Given the weather data from the UW Weather Station, compute the average temperature of each month in 2023 and communicate the result in a plot.

Solution

This example continues the weather-data example in chapter 04. We now use the recorded dates and times to resolve its warning about assuming fixed intervals. Our data is a six-column snapshot of the UW Weather Station’s 2023 archive, retrieved on 28 September 2026. It contains all 35,040 observations and their original temperature values, including invalid readings. Download the snapshot into a media folder beside your notebook to follow along; the book uses this local copy so rendering does not depend on a live download.

Source format and refreshing the snapshot

The original CSV has 22 named columns but 24 fields per data row. On inspection, the first unnamed field contains only NaN or -9999.90039, and the second is empty. Their meaning is undocumented here, so we do not interpret them as additional measurements or silently let pandas discard arbitrary fields. The following optional refresh code validates this observed format, then keeps the six named columns needed for this lesson. Unexpected extra values stop the refresh for inspection. Other named measurements remain available in the original archive.

The SHA-256 checksum of the source used for this edition is 57d3cf9b82d7b69e09b6dbb5f9b989e859be23921a44d8339db9f4fb807fce1e. A refreshed source may have a different checksum; review the data and rerun the checks and plots before replacing the teaching snapshot.

import csv
import hashlib
import io
from pathlib import Path
from urllib.request import urlopen

url = 'https://weather.uwaterloo.ca/download/Hobo_15minutedata_2023.csv'
with urlopen(url) as response:
    source = response.read()
print('Source SHA-256:', hashlib.sha256(source).hexdigest())
rows = list(csv.reader(io.StringIO(source.decode('utf-8'))))
header = [name.strip() for name in rows[0]]
columns = ['year', 'month', 'day', 'hour', 'minute', 'Temperature']
if len(header) != 22 or any(header.count(name) != 1 for name in columns):
    raise ValueError('Inspect changed column names before refreshing')
indices = [header.index(name) for name in columns]
selected = []
for row in rows[1:]:
    if (len(row) != 24 or row[23].strip() != ''
            or row[22].strip() not in {'NaN', '-9999.90039'}):
        raise ValueError('Inspect unexpected extra fields before refreshing')
    selected.append([row[i].strip() for i in indices])
Path('media').mkdir(exist_ok=True)
with open('media/weather-2023-temperature.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerow(columns)
    writer.writerows(selected)

First import the packages, load the snapshot, and construct timestamps from its named columns:

import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

columns = ['year', 'month', 'day', 'hour', 'minute', 'Temperature']
df = pd.read_csv('media/weather-2023-temperature.csv')
if list(df.columns) != columns:
    raise ValueError('Unexpected weather-data columns')
timestamps = pd.to_datetime(df[['year', 'month', 'day', 'hour', 'minute']])
if (timestamps.isna().any() or not timestamps.is_monotonic_increasing
        or timestamps.duplicated().any()):
    raise ValueError('Timestamps must be present, ordered, and unique')
start = pd.Timestamp('2023-01-01')
end = pd.Timestamp('2024-01-01')
if not ((timestamps >= start) & (timestamps < end)).all():
    raise ValueError('Expected observations from 2023')
df['elapsed days'] = (timestamps - start).dt.total_seconds() / (24 * 60 * 60)
print('First and last timestamps:', timestamps.iloc[0], timestamps.iloc[-1])
print('First elapsed time [days]:', df['elapsed days'].iloc[0])

First and last timestamps: 2023-01-01 00:00:00 2023-12-31 23:45:00
First elapsed time [days]: 0.0

Unlike a cumulative sum of row intervals, this calculation starts at zero for midnight on January 1 and preserves elapsed time if rows are missing. These timestamps use the station’s recorded clock; we have not assigned a time zone or performed daylight-saving conversions.

Now plot the raw temperature data to look for problems:

fig, ax = plt.subplots()
ax.plot(df['elapsed days'], df['Temperature'])
ax.set_xlabel('Time since January 1, 2023 [days]')
ax.set_ylabel('Recorded temperature [°C]');

The extreme negative value resembles a missing-data sentinel; the CSV alone does not establish whether it came from a sensor malfunction. For this Waterloo example, use one explicit screening rule for both the plot and the averages: keep finite temperatures strictly between −60 °C and 50 °C. These bounds reject implausible values, but do not certify every remaining measurement as accurate.

T = df['Temperature'].to_numpy()
valid = np.isfinite(T) & (T > -60) & (T < 50)
print(f'Excluded {np.count_nonzero(~valid)} of {len(T)} observations')

Excluded 168 of 35040 observations

We will compute the arithmetic mean of the retained observations in each month. This is a sample mean of valid recorded temperatures, not an estimate that fills missing intervals. To make the coverage visible, compare the number of valid readings with the expected number of quarter-hour observations in each calendar month:

summary_rows = []
for month_number in range(1, 13):
    in_month = df['month'].to_numpy() == month_number
    retained = in_month & valid
    count = np.count_nonzero(retained)
    month_start = pd.Timestamp(year=2023, month=month_number, day=1)
    expected = month_start.days_in_month * 24 * 4
    mean = T[retained].mean() if count else np.nan
    summary_rows.append({
        'month': month_number,
        'mean temperature [°C]': mean,
        'recorded': np.count_nonzero(in_month),
        'valid': count,
        'expected': expected,
        'coverage [%]': 100 * count / expected,
    })
monthly = pd.DataFrame(summary_rows)
print(monthly.round(2).to_string(index=False))

 month  mean temperature [°C]  recorded  valid  expected  coverage [%]
     1                  -2.02      2976   2976      2976        100.00
     2                  -2.65      2688   2688      2688        100.00
     3                  -0.50      2976   2970      2976         99.80
     4                   7.99      2880   2878      2880         99.93
     5                  13.12      2976   2976      2976        100.00
     6                  18.28      2880   2872      2880         99.72
     7                  20.52      2976   2854      2976         95.90
     8                  18.51      2976   2976      2976        100.00
     9                  16.88      2880   2877      2880         99.90
    10                  10.76      2976   2952      2976         99.19
    11                   2.47      2880   2880      2880        100.00
    12                   1.76      2976   2973      2976         99.90

Coverage below 100% can reflect missing rows or rejected readings. Here the source has regular 15-minute timestamps throughout 2023, so the coverage losses come from excluded temperatures. With irregular sampling, an unweighted mean overrepresents periods with more frequent observations; a time-weighted average would need an explicit treatment of gaps.

Finally, show the retained readings and each monthly mean together. A horizontal segment spans the calendar month it summarizes, and the table above reports how much data contributed:

fig, ax = plt.subplots(figsize=(9, 4), layout='constrained')
ax.scatter(df.loc[valid, 'elapsed days'], T[valid], s=1,
           color='tab:blue', alpha=0.35, label='Valid 15-minute readings')
for i, row in monthly.iterrows():
    month_number = int(row['month'])
    left = pd.Timestamp(year=2023, month=month_number, day=1)
    right = left + pd.offsets.MonthBegin(1)
    left_day = (left - start).total_seconds() / (24 * 60 * 60)
    right_day = (right - start).total_seconds() / (24 * 60 * 60)
    ax.hlines(row['mean temperature [°C]'], left_day, right_day,
              colors='black', linewidth=2,
              label='Monthly mean of valid readings' if i == 0 else None)
ax.set(xlabel='Time since January 1, 2023 [days]',
       ylabel='Temperature [°C]', title='UW Weather Station — 2023',
       xlim=(0, (end - start).days))
ax.legend(loc='upper right', markerscale=4)
ax.grid(alpha=0.2)

Figure 10.11: UW Weather Station temperatures in 2023. Monthly means use only readings passing the stated validity rule; coverage is reported in the table above.

Comments

To export this figure from your notebook or script, use the Figure’s savefig method. Save before plt.show() in scripts, since some backends close the figure when the display window closes:

fig.savefig('waterloo-temperature-2023.png', dpi=200, bbox_inches='tight')
plt.show()

Jupyter normally displays figures automatically; a standalone script usually needs plt.show() to open the plot window. The exported image includes the title, units, and legend. Keep the data source and validity rule with the figure when sharing it.