Python lets us put objects of different types in the same list. Operations on those objects depend on their types, so the same operator can have different meanings. Consider the following case:
item*2 operation depends on the type of item: for example, numbers are multiplied while strings are repeated. Python must select and execute the appropriate operation for each object.
2
2.0
11
oneone
2
(2+0j)Processing millions of values one at a time in Python can be slow. The overhead includes interpreting the loop, selecting operations, and accessing individual Python objects. NumPy can store numerical values compactly and process whole arrays using compiled code, avoiding much of this per-element Python overhead.
Example 8.1 (Invert an image) Given the greyscale image below, invert the colors so that dark becomes light, and vice-versa.
Solution
For this image, each greyscale intensity is an 8-bit integer between 0 (black) and 255 (white). The JPEG file stores three equal color channels; selecting one gives a 2D array of intensities. Other image formats can use different ranges and data types. We can inspect a small piece of the array:
imread function fetches the image file and returns it as a numpy array, which we’ll learn about below.
[[198 198 198 199 199 199 199 199 197 198]
[198 198 198 199 199 199 199 199 197 198]
[198 198 198 199 199 199 199 199 198 198]
[198 198 198 199 199 199 199 199 198 198]
[198 198 198 199 199 199 199 199 198 198]
[198 198 198 199 199 199 199 199 198 199]
[198 198 198 199 199 199 199 199 198 199]
[198 198 198 199 199 199 199 199 198 199]
[199 199 199 199 198 198 198 198 197 198]
[199 199 199 199 198 198 198 198 198 199]]To invert the greyscale values in im using traditional for-loops, we need to do 2 nested loops to scan along each row. Just for fun, let’s also record the time it takes to do this:
from time import perf_counter
tstart = perf_counter()
im2 = im.copy()
im_max = 255
for i in range(im.shape[0]):
for j in range(im.shape[1]):
im2[i, j] = im_max - im[i, j]
tfinal = perf_counter() - tstart
print(f"This took {tfinal} seconds")This took 0.08523916400000076 secondsNow we can visualize the images one above the other to confirm we achieved our objective:
fig, ax = plt.subplots(2, 1)
ax[0].imshow(im, cmap=plt.cm.bone)
ax[1].imshow(im2, cmap=plt.cm.bone)
Comments
Note that indexing into this 2D numpy array is different than a 2D list.
arr[0][0]…we index into the “outer” list to get access to the “inner” list, so it’s like chaining the indices.numpy array we use arr[0, 0]. We access all the dimensions at the same time.numpy and the ndarraynumpy is not included in Python. We need to install it separately. If we used Anaconda to get ourselves started, then we get numpy included (along with a few hundred other packages). So despite the fact that numpy is pretty much mandatory for any numerical computing, it is still a separate package with its own “interface” that we must learn.
The main feature of numpy is that it provides a data type known as an ndarray, or “ND-array” meaning “N-Dimensional Array”. This array looks and acts a lot like a list but has one key difference:
Every element in an
ndarrayis stored using the same data type, called itsdtype.
For numerical dtypes, NumPy can select a compiled operation for the whole array instead of choosing a Python operation separately for each element. An advanced exception is dtype=object, which stores references to Python objects; those objects can have different types and do not provide the same numerical performance benefits.
Simply putting numbers into an array does not remove the overhead of a Python for-loop. To gain the performance benefit, we usually use NumPy’s array operations and functions to do the looping in compiled code.
The downside is that using numpy is like learning a language within a language.
Some numerical algorithms are easier to express with explicit loops. A package called Numba can compile supported Python functions into machine code, including functions with loops over numerical arrays. This is called “just in time” compilation. Compiled versions specialize to their argument types, and the initial compilation has a cost, so performance should be measured for the intended workload.
The Python type of an array object is ndarray. Its dtype is a separate attribute that describes how its elements are stored:
dtype, in this case int. If omitted, np.array infers a dtype from its input: [1, 2, 3] gives an integer array, while [1, 2, 3.0] gives a floating-point array. Other constructors have their own defaults; for example, np.zeros defaults to float.
type(arr) identifies the array object as an ndarray.
arr.dtype describes its elements. The default integer width depends on the platform and NumPy version. A dtype of int64 uses 64 bits per signed integer, with a finite range from -2**63 to 2**63 - 1.
<class 'numpy.ndarray'>
int64We can create arrays with all the usual types:
arr2 = np.array([1, 2, 3], dtype=float)
print(arr2)
arr3 = np.array([0, 1, 0], dtype=bool)
print(arr3)
arr4 = np.array([1, 2, 3], dtype=complex)
print(arr4)[1. 2. 3.]
[False True False]
[1.+0.j 2.+0.j 3.+0.j]Counting in binary is just like using familiar decimal numbers, but there are only 2 symbols (0 and 1) instead of 10 (0, 1, 2, 3, etc). This means we run out of symbols more often, so need to increment the placeholder more often. For example:
| Decimal | Binary |
|---|---|
| 0 | 0 |
| 1 | 1 |
| 2 | 10 |
| 3 | 11 |
| 4 | 100 |
| 5 | 101 |
| 6 | 110 |
| 7 | 111 |
| 8 | 1000 |
Bits represent 0’s and 1’s, so the more bits we use, the larger the number we can store.
14,013,403 vs 1.4013403 each have 8 digits. Adding more digits lets us either count higher or have more precision, depending on whether the number is an int or a float.Why not just use the maximum number of bits? Using fewer bits means:
NumPy lets us control the number of bits used to store data. For example, we can confirm that these arrays occupy different amounts of space:
import numpy as np
arr2 = np.array([1, 2, 3], dtype=np.int8)
print(arr2.nbytes)
arr3 = np.array([1, 2, 3], dtype=np.int16)
print(arr3.nbytes)
arr4 = np.array([1, 2, 3], dtype=np.float64)
print(arr4.nbytes)3
6
24We can convert between dtypes using the astype() method. Converting these finite floats to integers truncates toward zero; it does not round to the nearest integer:
arr = np.array([1.9, -2.9, 3.0])
integers = arr.astype(int)
print(integers)
print(arr) # astype created a new array; the original is unchanged[ 1 -2 3]
[ 1.9 -2.9 3. ]Choose a dtype that can represent the values you need. Unlike Python’s integers, NumPy’s fixed-width integers can overflow during arithmetic. Floating-point arrays also have finite precision and range; calculations can produce inf or nan. Use np.isfinite or np.isnan to check for these values when needed, rather than testing equality with np.nan.
One of the ways that numpy lets us avoid the use of for-loops is by changing the way mathematical operations like + and * work.
Recall that for lists, multiplication by 2 doubled the length of the list:
arr = [1, 2, 3] * 2
print(arr)[1, 2, 3, 1, 2, 3]With numpy, multiplication does actual math!:
import numpy as np
arr = np.array([1, 2, 3]) * 2
print(arr)[2 4 6]This is called “elementwise” operation because it operates on each element, rather than on the whole array.
numpy also allows for elementwise operations between 2 arrays:
arr1 = np.array([1, 2, 3])
arr2 = np.array([10, 20, 30])
print(arr1/arr2)[0.1 0.1 0.1]Here arr1 and arr2 have the same shape, so corresponding elements are paired. Arrays with different shapes can also work if their shapes are compatible with broadcasting. Having the same total number of elements alone does not guarantee compatible shapes.
The familiar arithmetic operators work elementwise for these numerical arrays:
arr1 = np.array([1, 2, 3])
arr2 = np.array([4, 5, 6])
print( arr1 + arr2)
print( arr1 - arr2)
print( arr1 * arr2)
print( arr1 / arr2)
print( arr1 // arr2)
print( arr1 % arr2)
print( arr1**arr2)[5 7 9]
[-3 -3 -3]
[ 4 10 18]
[0.25 0.4 0.5 ]
[0 0 0]
[1 2 3]
[ 1 32 729]In all cases, each of the above lines performs the stated operation using element i of arr1 and element i of arr2.
For numerical arrays, these operations run compiled loops chosen for the array dtypes. Avoiding a Python-level operation for every element can make them much faster.
In the above example, elementwise operations avoided a Python loop over individual pixels. Prefer NumPy operations for this kind of calculation on a large array, and measure performance when it matters. It is also common to loop over a batch of images, using NumPy functions inside the loop, like this:
ims = [im1, im2, im3, im4, im5]
mx = []
for im in ims:
mx.append(im.max()) The point is that looping over a small number of items and doing big calculations on each loop is fine, common, and usually unavoidable.
When we looked at Python containers like lists and strings, we used methods attached to them. A method operates on an object, but its behaviour may differ from a similarly named function. For example, vals.sort() changes the original list and returns None, while sorted(vals) returns a new sorted list.
ndarray objects also have many methods. They provide other attributes that describe or give access to aspects of the array, such as size and shape. Some useful attributes are listed below:
ndarrays
| Attributes | Description |
|---|---|
ndim |
Number of dimensions of the array |
size |
Number of elements in the array |
shape |
The size of the array in each dimension |
dtype |
Data type of elements in the array |
itemsize |
The size (in bytes) of each element in the array |
data |
The buffer containing actual elements of the array in memory |
T |
View of the transposed array |
flags |
Information about memory layout and properties such as whether the array is writable |
flat |
A 1-D iterator over the array |
imag |
The imaginary part of the array |
real |
The real part of the array |
nbytes |
Total bytes consumed by the elements of the array |
The shape, size and ndim attributes are especially useful when checking dimensions. Other attributes also have routine uses: dtype describes the stored values, T transposes an array, and real and imag access its real and imaginary components.
import numpy as np
arr = np.array([[2, 3, 4], [3, 4, 5]])
print(arr.shape)
print(arr.size)
print(arr.ndim)(2, 3)
6
2Attributes such as arr.size are accessed without calling them. Methods use parentheses and can perform operations or calculations, such as finding the maximum or minimum. The distinction is how we access them, rather than a general rule that attribute access never performs a computation.
arr = np.array([4, 3, 6])
print(arr.max())
print(arr.min())6
3Note that because these methods require an actual computation on the array, it is best to store their result if it needs to be used many times:
arr = np.array([4, 3, 6])
amax = arr.max()ndarrays which calculate some property based on the array contents
| Method | Description |
|---|---|
max |
Return the maximum along a given axis. |
min |
Return the minimum along a given axis. |
trace |
Sums a diagonal across two axes; by default, the main diagonal of axes 0 and 1. |
sum |
Return the sum of the array elements over the given axis. |
cumsum |
Returns cumulative sums along an axis, or over the flattened array if no axis is given. |
mean |
Returns the average of the array elements along given axis. |
var |
Returns the variance of the array elements, along given axis. |
std |
Returns the standard deviation of the array elements along given axis. |
prod |
Returns the product of the array elements over the given axis |
cumprod |
Returns cumulative products along an axis, or over the flattened array if no axis is given. |
all |
Returns True if all elements evaluate to True. |
any |
Returns True if any elements evaluate to True. |
Reductions such as max, sum, and mean use all elements by default and return a single value. Giving one axis reduces along that axis and normally removes it from the result’s shape. For example, the mean along axis 0 of a (2, 3) array has shape (3,); keepdims=True retains the reduced axis with length 1, giving (1, 3).
The other methods in Table 8.2 have different shape rules. cumsum and cumprod return running totals or products, preserving the shape when an axis is specified and flattening the array when it is omitted. trace sums a diagonal across two axes, removing both from the output shape; for a 2D array it returns one value.
Multidimensional arrays are very common in numerical programming. Some examples are:
(200, 200, 3).
As “simple humans” we can visualize 1D, 2D and 3D images and arrays quite well, but not higher. Figure 8.4 shows this progression. It also includes the definition of array “shape” as well:
It is actually possible to use the “list of lists” style indexing with ndarrays, as shown below:
import numpy as np
arr = np.random.rand(5, 5)
print(arr[0][0])
print(arr[0, 0])0.7915783410000388
0.7915783410000388But numpy offers several special features which require the [0, 0] style indexing, so it’s necessary to learn this. For instance, we can extract a subsection of an array using:
import numpy as np
arr = np.random.rand(9, 9)
arr2 = arr[:3, :3]
print(arr2)[[0.92264333 0.38219726 0.34837364]
[0.63291104 0.1466447 0.07590661]
[0.40004615 0.57845627 0.74558084]]ndarraysIndexing an ndarray behaves a bit differently than we have seen with nested lists. Recall that indexing into a nested list (i.e. a “list of lists”) worked as follows:
[0], we retrieve the list stored at location 0.
[0] we index into the list we obtained from the first [0].
[1, 2, 3]
1For an ndarray, the indexing is done within a single pair of brackets:
arr = np.array([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
1print(arr[0, 0])row and col simultaneously and receive the value stored at that location.
1Visualizing four or more dimensions is challenging. We can use nested diagrams or slices to represent the extra dimensions, as illustrated in the following example.
Example 8.5 (Thinking about higher dimensional arrays) Consider the array shown in Figure 8.5. How would you access the value of “31”?
Solution
1 through N, where N is the total number of elements in the final array. Like range, arange excludes its stopping value, N+1.
(3, 5, 3, 3, 4).
(0, 0, 2, 1, 2), one index per axis. In the first block of each of the first two axes, select the third 3-by-4 slice, its second row, and its third column.
31Comments
By default, reshape uses “C” index order: the last index changes fastest as values fill the result. With “F” (Fortran) index order, the first index changes fastest. Neither order determines where new axes are added or assigns them physical meanings such as x, y, or z. Axes are numbered 0, 1, and so on.
values = np.arange(1, 7)
print(values.reshape(2, 3, order='C'))
print(values.reshape(2, 3, order='F'))[[1 2 3]
[4 5 6]]
[[1 3 5]
[2 4 6]]It is sometimes helpful to think of a multidimensional array as a sequence of values with shape information attached. NumPy uses a data buffer and information describing how indices locate elements within it; a slice need not occupy one contiguous block. We can reshape an array as follows:
arr = np.arange(24)
print(arr)[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23]arr = arr.reshape([4, 6])
print(arr)[[ 0 1 2 3 4 5]
[ 6 7 8 9 10 11]
[12 13 14 15 16 17]
[18 19 20 21 22 23]]arr = arr.reshape([2, 12])
print(arr)[[ 0 1 2 3 4 5 6 7 8 9 10 11]
[12 13 14 15 16 17 18 19 20 21 22 23]]We can reshape between shapes as long as the total number of elements is conserved. Reshaping may share the existing data or make a copy, depending on the array’s layout. Transposing also commonly shares data rather than copying it.
A one-dimensional array with shape (n,) has no separate row or column axis, so .T leaves its shape unchanged. Use reshape(n, 1) for a 2D column or reshape(1, n) for a 2D row.
We can also retrieve subsections of an array using the slicing notation introduced for Python sequences:
arr = np.arange(25)
arr = arr.reshape([5, 5])
print(arr)
print(arr[0:3, 0:3])[[ 0 1 2 3 4]
[ 5 6 7 8 9]
[10 11 12 13 14]
[15 16 17 18 19]
[20 21 22 23 24]]
[[ 0 1 2]
[ 5 6 7]
[10 11 12]]We can also write to subsections of an array:
arr = np.arange(25)
arr = arr.reshape([5, 5])
arr[0:3, 0:3] = 0
print(arr)[[ 0 0 0 3 4]
[ 0 0 0 8 9]
[ 0 0 0 13 14]
[15 16 17 18 19]
[20 21 22 23 24]]The defaults also apply, so we can omit some values:
arr = np.arange(25)
arr = arr.reshape([5, 5])
1arr[:3, 3:] = 0
print(arr)3) onward. The start of a slice is included and its stop is excluded, so :3 selects rows 0, 1, and 2, while 3: selects columns 3 and 4 in this array.
[[ 0 1 2 0 0]
[ 5 6 7 0 0]
[10 11 12 0 0]
[15 16 17 18 19]
[20 21 22 23 24]]Unlike a list slice, a basic NumPy slice is a view: it shares data with the original array. As with list aliasing, alias = arr gives another name to the same object. Use .copy() when you want independent array data:
arr = np.arange(6)
alias = arr
view = arr[1:4]
independent = view.copy()
view[0] = 99
print(arr) # [0, 99, 2, 3, 4, 5]
print(alias) # the same array as arr
print(independent) # [1, 2, 3][ 0 99 2 3 4 5]
[ 0 99 2 3 4 5]
[1 2 3]As with mathematical functions, logical comparisons are also done elementwise:
[0, 1): 0 is included and 1 is excluded.
arr are less than 0.5
mask reveals that it contains True and False values, which are the result of the logical comparison.
[[False False False True True]
[False False True True False]
[False False True True False]
[False True False True True]
[ True True False False False]]We can use ndarrays filled with boolean values as “masks” to retrieve or write values from the locations where the mask is True.
mask will contain a True value in all locations where arr < 0.1.
arr that were < 0.1 using mask for the index.
[0.01446443 0.02580644 0.06326639]We can also use masks to write values:
import numpy as np
arr = np.random.rand(5, 5)
mask = arr < 0.5
1arr[mask] = 0.0
print(arr)numpy to put the value of 0.0 into arr at all locations where mask is True.
[[0. 0. 0. 0.65220071 0.60391204]
[0.96392761 0.65305643 0. 0. 0. ]
[0.65083299 0. 0. 0. 0. ]
[0. 0. 0.54014946 0.90251596 0.96426813]
[0. 0. 0. 0. 0.53346022]]Reading with a Boolean mask produces a copy: after vals = arr[mask], changing vals does not change arr. In contrast, assigning directly through arr[mask] = ... updates the selected positions of arr itself. Integer-array indexing, introduced below, also produces a copy when reading.
Integer-array indexing, often called fancy indexing, selects positions using integer indices:
[0. 0.20042145 0.62304241 0.14334264 0. 0.
0.41774954 0.23203804 0. 0.19034796]To select individual positions from a 2D array, we can provide one list of row indices and one of column indices. NumPy pairs corresponding entries, so the following lists select five positions rather than a five-by-five block:
import numpy as np
arr = np.random.rand(5, 5)
ind_x = [0, 0, 1, 3, 3]
ind_y = [0, 1, 3, 3, 4]
indices = (ind_x, ind_y)
arr[indices] = 0.0
print(arr)[[0. 0. 0.55424741 0.56308042 0.55829556]
[0.252363 0.54815816 0.27895998 0. 0.57926508]
[0.99794375 0.1070144 0.38386484 0.34533648 0.06638761]
[0.84995268 0.17923792 0.70424624 0. 0. ]
[0.10416918 0.49406742 0.94944668 0.74954711 0.26650565]]In all of the above code snippets and examples we have seen that ndarrays can be created by lists as follows:
vals = [1, 2, 3, 4, 5]
arr = np.array(vals, dtype=int)
print(arr)[1 2 3 4 5]It is also possible to convert an ndarray back to a list:
vals = [1, 2, 3, 4, 5]
arr = np.array(vals, dtype=int)
vals2 = arr.tolist()
print(type(vals2))<class 'list'>Often we want to create an ndarray directly. np.empty allocates an array of a given shape without initializing its numerical entries:
arr = np.empty((5, 2), dtype=float)
arr[:] = 0.0 # assign every entry before using it
print(arr)[[0. 0.]
[0. 0.]
[0. 0.]
[0. 0.]
[0. 0.]]Before assignment, the contents of an empty numerical array are arbitrary and must not be treated as valid data. When we want known initial values, np.ones or np.zeros is simpler:
arr1 = np.ones(5, dtype=int)
arr2 = np.zeros(5, dtype=int)
print(arr1)
print(arr2)[1 1 1 1 1]
[0 0 0 0 0]Often we want to combine two 1D arrays into a 2D array, or vice versa. numpy uses the term “stack” to refer to joining arrays. This refers to the fact that arrays “stack” like blocks, either beside or on top of each other.
Splitting arrays into smaller ones is also fairly common. We have already seen that slice indexing can be used for this, such as:
arr = np.arange(16).reshape([4, 4])
a1 = arr[:2, :]
a2 = arr[2:, :]
print(a1)
print(a2)[[0 1 2 3]
[4 5 6 7]]
[[ 8 9 10 11]
[12 13 14 15]]However, it is often more convenient to use numpy's built-in functions. This makes it less likely to make mistakes for instance, and it’s also fewer lines of code:
arr = np.arange(16).reshape([4, 4])
1a1, a2 = np.vsplit(arr, 2)
print(a1)
print(a2)2 requests two equal sections along axis 0. For this four-row array, each section has two rows. To specify a split position instead, use a list: np.vsplit(arr, [1]) splits before row index 1, giving sections with one and three rows. Both forms return views that share data with the original array.
[[0 1 2 3]
[4 5 6 7]]
[[ 8 9 10 11]
[12 13 14 15]]