6  Files and IO

Table of contents

• 6.1 Types of Memory
  • 6.1.1 Permanent Memory
  • 6.1.2 Temporary Memory
• 6.2 Files and Folders
  • 6.2.1 Finder/Explorer vs Command Line
• 6.3 Reading and Writing Text to a File
  • 6.3.1 Writing to a File
  • 6.3.2 Reading From a File
• 6.4 Installing External Packages
• 6.5 Assignment 5

ImportantLearning objectives for this lesson

Upon completion of this lesson you should:

• …appreciate the 2 main types of memory in a computer
• …be familiar with the file system, files, and folders
• …understand both relative and absolute file paths
• …know how to write simple text to a file, and read it back in
• …be able to format text to create desired output in files
• …be able to use the command line to perform basic tasks directly
• …know how to install external python libraries using the command line

Figure 6.1: We should probably know a bit more deeply about how a computer works

6.1 Types of Memory

NoteVideo: Computer memory and storage

TED-Ed · about 5 min. An animated explanation of how temporary memory and long-term storage differ in speed, capacity, and persistence.

Watch on YouTube.

We have not talked much about how computers work internally. As we write more complicated computer programs we need to understand more about how the computer deals with data because it impacts performance. Examples of this trend are advanced numerical algorithms such as machine learning or climate modeling.

The job of “data scientist” is a relatively new phenomena. It entails not only working with scientific data, but also applying scientific concepts and tools to data. It is considered separate from a software engineer or computer programmer. One of the main considerations when doing data science, and scientific programming in general, is ensuring good performance. Crunching a gazillion data points is only useful if it can be done quickly.

It is convenient to think of computer memory as being either permanent and slow or temporary and fast.

6.1.1 Permanent Memory

Computers commonly use a hard disk drive (HDD) or a solid-state drive (SSD) for long-term storage. Both retain data when power is removed, although neither lasts forever. Storage can be internal or external; some internal drives are replaceable, while others are built into the device. Several removable storage formats are shown in Figure 6.2.

Data is represented by bits, with values 0 and 1. Different devices store those bits differently:

1. An HDD stores magnetic patterns on rotating platters. A moving head writes patterns and senses magnetic changes when reading.
2. An SSD stores data electronically in flash-memory cells and has no moving parts. This makes them much more stable and faster.
3. Optical discs such as CDs, DVDs, and Blu-ray discs use a laser to read differences in reflected light from the surface of the disk. Etching a pattern on the disk is information is stored.

Storage access is generally slow, so avoiding unnecessary file reads and writes can improve performance, but saving results is still essential when we need to keep them.

NoteGithub Arctic Code Vault

Github has taken long-term storage to a new level.

Figure 6.2: The “save” icon in many programs is a picture of a removable floppy disk (shown in the upper-middle of the figure on the right), which is about as relevant to modern computers as an hour glass is to symbolize waiting.

6.1.2 Temporary Memory

Computers also have a second type of memory, usually called RAM (Random Access Memory). RAM is much like ‘short term’ memory in humans. It is where we keep information which we are actively working on. RAM is designed to be much faster than a disk, but normally there is much less of it. A typical computer might have ~10 GB of RAM, while having ~1000 GB of disk space.

We typically use RAM as follows:

1. Opening a file in Python creates a file object for accessing it; it does not automatically load the whole file into RAM.
2. Reading the file brings data into the program. We can read all of it or process smaller pieces. The resulting strings and other objects remain available in memory while the program uses them.
3. Writing sends data to a file, often through a buffer. Closing the file flushes Python’s pending output and releases the file resource. It does not erase variables containing data already read, and changing those variables does not automatically change the file.

Reading many large files into memory can put RAM under pressure. The operating system may move memory pages to storage (swap), which can slow the computer down. This is one possible cause of poor performance, not the only one.

NoteActive memory vs long-term storage

The sum it all up, think of a information stored on disk as long-term, semi-permanent storage; while anything that the computer is currently working with is moved into active memory.

Figure 6.3: A screenshot of the Windows Task Manager showing RAM usage vs time. The dip occurred after closing all instances of my web-browser.

6.2 Files and Folders

NoteVideo: Absolute and relative paths

Udacity · about 2 min. Understand how absolute paths locate a file from the root, while relative paths start from a current location.

Watch on YouTube.

When reading and writing data to the disk, the computer’s “operating system” (i.e. Windows, Mac, Ubuntu, etc.) presents us with a very familiar interface for dealing with stored data: files and folders.

NoteHidden file extensions

One extremely frustrating feature of Windows is that the file extension is hidden by default, so the files appear as “my_resume” instead of “my_resume.docx”. Enabling this visibility is a very good idea. This tutorial shows the process for several different versions Windows.

• Data is stored in files.
  • Common examples are are Word documents and images. We might have a Word document on our computer like my_resume.docx, or a picture like friends_at_pub.png.
  • The part after the last dot (.) is called the file extension.
  • Extensions indicate the intended format and help applications and the operating system choose how to handle a file. Renaming an extension does not convert the contents: renaming an image to .txt does not turn it into ordinary text. Some software also requires particular extensions.
• Files are stored in folders.
  • Common examples are your “downloads” or “documents” folder.
  • Folders can contain many files
  • Like real folders, folders can be stored inside other folders like /pictures/2020/January/PartyInNYC.
• Finally, folders are stored on a disk.
  • This is where the analogy to real world file systems breaks down because disks should be called “cabinets”.
  • A drive is a physical storage device where data can be written. It may be internal or external to the computer.
  • On Windows, mounted storage volumes commonly have drive letters, with C:\ usually containing the operating system. A physical drive can contain more than one volume.
  • On macOS, mounted volumes appear within one directory tree rooted at /, rather than using drive letters.
• If we combine all of these together we get a path.
  • On Windows the path looks like:

C:\users\jeff\photos\ufo_sightings\591.jpg

• And on Mac it looks like:

/Users/jeff/photos/bigfoot_sightings/431.jpg

These are absolute paths: they start at a drive’s root on Windows or / on macOS. A relative path is interpreted from the current working directory. For example, from C:\users\jeff, photos\ufo_sightings\591.jpg locates the same file as the absolute path above. Relative paths can also go upward: .. means the parent directory, so ../data/readings.csv means go up one level, then into data.

NoteForwards vs backward slashes

Note that Windows and Mac use different direction slashes. Recall the discussion about the use of escape characters when backslashes needed to be written (\\). This is particularly annoying on Windows since \ is used in the paths, while on Mac the / can be used without any escaping.

Below is a screenshot of a random folder deep within the Windows directory. Also shown is how title bar reports the absolute file path if you click it.

Figure 6.4: A screenshot of a file folder several levels deep into Windows

The point of storing files inside a hierarchical folder structure is for organization. Consider the file storage room in shown in Figure 6.5. If you are given a location (i.e. a path) in the form of aisle 8, section 4, shelf 4, box 11, page 10, line 55 you could find the exact sentence in a text file! The file system on a computer works exactly like this.

Figure 6.5: We could find a single sheet of paper in this pile, if given the right directions

6.2.1 Finder/Explorer vs Command Line

Now that we understand how our computer stores our data in the form of files, we can appreciate what is happening when we browse our files using the Explorer in Windows (or Finder on Mac).

However, there is an alternative way to navigate our files: using the command line. Consider the following comparison:

Figure 6.6: Comparison of the graphical explorer view (left) and the command line approach to navigating the file system.

A major milestone on the path to becoming a proficient programmer is learning to use the command prompt (sometimes known as the command line or terminal). Modern operating systems provide gorgeous and powerful graphical user interfaces (GUIs), but ultimately our button clicks and checkboxes get translated into commands to be run by the computer. The command prompt lets us call such commands directly, bypassing the GUI. There are 2 main scenarios where we may need to do this:

1. As “power-users”, we often need to interact with the computer more directly than the GUI allows. The GUI hides many features and options since the vast majority of computer users do not need them.
2. There are many essential utility applications which do not have a GUI so must be activated from the command prompt.

The following table illustrates an example of working from the command prompt to open a text editor, in both Windows and Mac.

Table 6.1: A simple example of opening a text editor from the command prompts in Windows and Mac.

Windows	Mac
Press Windows key to open Start Menu	Press Command-Space to open Spotlight
Type cmd and press enter	Type terminal and press enter
At the “prompt” (C:\) type notepad	At the “prompt” type open -a TextEdit
When you press enter, the NotePad application will open up	When you press enter, the TextEdit app should open up
This is equivalent to opening the Start menu, typing “notepad” and clicking the application.	This is equivalent to opening Spotlight, typing “TextEdit” and opening the application.

The number of commands that are available is enormous, and each one has different options. The following links provide “cheat sheets” for the most common and useful ones, to give a flavor of what is possible:

• A cheatsheet for Mac users is available here
• A cheatsheet for Windows users is available here

There are a few commands that are so common and useful that they are worth highlighting:

Listing contents of current directory	In Windows the command is dir, while in Mac it is ls.
Changing current directory	Use cd on Windows and macOS. cd .. moves up one level; cd <folder> moves into the specified folder. Quote paths containing spaces, for example cd "Course Notes".

Example 6.1 (Navigating through the file system from the command line) Open the command prompt on your computer and navigate to your “downloads” folder.

Solution

On Windows you can press the “Windows” key to get the menu, then type “cmd” or maybe “cmd.exe”, the click the result of the search (or press enter).

On Mac you can press “Command+space”, then type “terminal”, then click or press enter.

Once the terminal is open you can navigate using the following commands:

• cd .. moves up one level
• dir/ls will list all files and folders in current directory
• cd <path> will move into the specified folder. Note that you can start typing <something>, then press the tab key to get an autocomplete.

Comments

Knowing how to use the command line, even just a little, is mandatory for an aspiring programmer.

6.3 Reading and Writing Text to a File

Figure 6.7: Most people have pretty bad file system hygiene.

6.3.1 Writing to a File

NoteVideo: Writing text files

Bro Code · about 3 min. Use with open(...) and write() to save text, and compare writing with appending.

Watch on YouTube.

Writing text to a file is one of the ways we can make our data “permanent”, otherwise all of our calculations are lost when we shutdown our computer.

Because this is such an important task, Python has a built-in function for working with a file: open(). The open() command does not “open” a file physically for us to look at, they way Word does when we double-click a file name. Instead Python “opens” the door to the file so that data can flow in and out.

The general syntax is as follows:

1fname = 'data.txt'
2f = open(file=fname, mode='w')
3f.write('Hello world')
4f.close()

1

We define a string to use as our filename. The .txt extension conventionally identifies a text file; Python’s open() does not require it.

2

The mode 'w' creates a file or immediately truncates an existing file to zero length. open() returns a file object, which we store in f.

3

We can access a variety of methods attached to the file. The full list of methods is given in Table A.7, but a partial list is given below. In this case we use write() to write the given text.

4

Closing flushes buffered output and releases the file resource. Ordinary open() does not generally lock a file against other programs; sharing and locking are separate matters.

The usual way to ensure a file closes, even if an error occurs, is a with statement:

with open('data.txt', mode='w') as f:
    f.write('Hello world')

The indented block uses the open file; leaving it closes the file automatically. Common text modes are 'r' (read an existing file), 'w' (create or overwrite), and 'a' (append, creating the file if needed). Appending does not automatically insert a newline. See the Python file tutorial for more details.

NoteWhere is my file?

If you create a file solely using a file name (i.e. no directory information), then the file will be created in Python’s “current working directory”. Luckily this is pretty easy to find:

1import os
2print(os.getcwd())

1

The os module has lots of functions for navigating your file system, as well as general information like the number of cores available, etc.

2

Here cwd stands for “current working directory”

This will return something like C:\Users\jeff\projects. If you navigate to this directory in your explorer/finder you will see your file (i.e. 'data.txt')

The result of writing “Hello world” to this file is shown in Figure 6.8 on the left:

Figure 6.8: The contents of the file after writing “Hello world”

Successive write() calls on the same open file continue at its current position. The following creates one long line as shown in Figure 6.8 (middle). Opening with 'w' first erases any previous contents; it does not append to an existing file from an earlier run.

fname = 'data.txt'
with open(file=fname, mode='w') as f:
    f.write('Hello world')
    f.write('Hello world')
    f.write('Hello world')

And finally, using our knowledge of escape characters from last chapter we can add a new line after each write to get the result in Figure 6.8 (right)

fname = 'data.txt'
with open(file=fname, mode='w') as f:
    f.write('Hello world\n')
    f.write('Hello world\n')
    f.write('Hello world\n')

Example 6.2 (Write several lines of formatted text to a file) Write the current time and a random number to a file approximately once per second for 5 seconds.

Solution

To solve this problem we will use 2 modules from Python’s standard library. To get the current time we will use time:

import time
print(time.ctime())

Mon Nov  2 05:00:39 2026

time.ctime() returns a human-readable local date/time string.

The time module also has a function called sleep which we will use to control the timing of the program.

time.time() returns seconds since the Unix epoch (1970-01-01 00:00:00 UTC on the systems used here), which is not super useful. For measuring elapsed durations, use time.perf_counter(), whose clock is not affected by adjustments to the system’s date/time.

The random module contains functions for generating random numbers. (I wanted to use “real” data based on the temperature of your computer, but that requires installing external libraries).

1import time, random

2with open(file='data.csv', mode='w') as f:
3    init_time = time.perf_counter()
4    while (time.perf_counter() - init_time) < 5:
5        date_and_time = time.ctime()
6        value = random.randint(0, 1000)
7        s = date_and_time + ', ' + str(value) + '\n'
8        f.write(s)
9        time.sleep(1)

1

We import both time and random modules on the same line

2

CSV stands for “comma-separated values”. The with statement closes the file automatically when its block ends.

3

Record the starting value of the elapsed-time clock.

4

Here we compute the difference between the current time and the time the program started. As long as this is less than 5 seconds, the while-loop will keep running

5

We use the ctime() function to get a text representation of the current time and date.

6

random.randint(0, 1000) includes both endpoints.

7

Here we format our text by adding a comma between the time and the value, convert the value to a string, and add a new line character at the end.

8

We write the formatted string to our file.

9

sleep(1) pauses for at least about one second; scheduling and other work add delay, so this is approximate sampling, not precise timing.

Comments

We can open this file in Excel or another spreadsheet application. Double-clicking uses whichever application is associated with .csv files on your computer. Remember to use os.getcwd() if you can’t find the file.

NoteHow to do X?

Here is a good time to reveal how I know about all these hidden features buried under so many levels of abstraction:

I Google it!

I use a search term like “how to find the current time in python using standard library”. I usually look for the StackOverFlow links. Here is the page on StackOverflow. There are many answers, each proposing a different way to do it, complete with commentary by other coders and an upvote/downvote count to indicate how valuable the other users (like us!) found the answer.

6.3.2 Reading From a File

NoteVideo: Reading text files

Bro Code · about 4 min. Read text with with open(...), let the file close automatically, and see how a missing file can be handled.

Watch on YouTube.

Reading data from a file not only let’s us reload data we may have written to a file, but it is also common to store tabulated data like constants and fitting parameters in files.

Let’s start with something simple to see how it works:

1with open('file.txt', mode='w') as f:
    f.write('Hello world')
2with open('file.txt', mode='r') as f:
3    contents = f.read()
print("Result:", contents)

1

First we will open a new file and write “Hello world” in it to ensure the file is present.

2

Here we open the file in “read” mode ('r').

3

read() reads from the current position to the end. Here the file has just been opened, so it reads all the text. contents remains available after the file is closed. A second read() before closing would return '' because the position is already at the end.

Result: Hello world

Example 6.3 (Read complicated data from a file) Consider Antoine’s equation for predicting vapor pressure of a pure liquid as a function of temperature:

\[ log_{10}(P) = A - \frac{B}{T + C} \] where \(P\) is in units of \(mmHg\) and \(T\) is in units of \(^oC\). The values of \(A\), \(B\) and \(C\) are experimentally determined for each liquid, and their values are tabulated in various places. The data in Table 6.2 are an example from a file that was over 14 pages long.

Table 6.2: Antoine coefficient data for 6 species taken from a list over 14 pages long

ID	Formula	Compound Name	A	B	C	Tmin	Tmax
1	CCL4	carbon-tetrachloride	6.89410	1219.580	227.170	-20	101
2	CCL3F	trichlorofluoromethane	6.88430	1043.010	236.860	-33	27
3	CCL2F2	dichlorodifluoromethane	6.68619	782.072	235.377	-119	-30
4	CCLF3	chlorotrifluoromethane	6.35109	522.061	231.677	-150	-81
5	CF4	carbon-tetrafluoride	6.97230	540.500	260.100	-180	-125
6	CO	carbon-monoxide	6.24020	230.270	260.010	-210	-165

Let’s explore the process of reading this kind of data in from a file. We will load the data into dicts so that we can access it using the very convenient syntax of data['CCL4']['A'].

Solution

The first thing we need to do is create a file with formatted data. This way the file will exist in our current working directory and we can avoid the complication of searching for it (we’ll deal with that later).

So, let’s format the data as ‘comma separated values’:

1data = [
2    "ID, Formula, Compound Name, A, B, C, Tmin, Tmax\n",
3    "1, CCL4, carbon-tetrachloride, 6.89410, 1219.580, 227.170,-20, 101\n",
    "2, CCL3F, trichlorofluoromethane, 6.88430, 1043.010, 236.860, -33, 27\n",
    "3, CCL2F2, dichlorodifluoromethane, 6.68619, 782.072, 235.377, -119, -30\n",
    "4, CCLF3, chlorotrifluoromethane, 6.35109, 522.061, 231.677, -150, -81\n",
    "5, CF4, carbon-tetrafluoride, 6.97230, 540.500, 260.100, -180, -125\n",
    "6, CO, carbon-monoxide, 6.24020, 230.270, 260.010, -210, -165\n",
]

4with open('coefficients.csv', 'w') as f:
5    f.writelines(data)

1

We are putting each line as a str inside a list. This is a classic use of collections to hold many pieces of data at once.

2

This line is called the header. We will have to remember to remove this when we read the data back

3

Each line from the table is written as it’s own string, ending with a '\n'.

4

We open a new file in ‘write’ mode.

5

Here we are using a new function writelines() which write each str in a list to the file. We could have used a for-loop and called write() for each item in data.

Now we are ready to read this data in from the file we just created:

with open('coefficients.csv') as f:
1    header = f.readline()
2    database = dict()
    while True:
3        txt = f.readline()
4        if txt == '':
5            break
6        fields = txt.split(',')
        coeffs = {}
7        coeffs['A'] = float(fields[3].strip())
        coeffs['B'] = float(fields[4].strip())
        coeffs['C'] = float(fields[5].strip())
        coeffs['Tmin'] = float(fields[6].strip())
        coeffs['Tmax'] = float(fields[7].strip())
        chem_name = fields[1].strip()
8        database[chem_name] = coeffs

1

Read the first line to get past the header

2

Create an empty dict to hold the incoming data

3

Read a single line into txt

4

An empty string means end-of-file; a blank line containing a newline would instead be '\n'. This example file has no blank data rows.

5

break is a statement, not a function call. It exits the innermost loop.

6

Split at commas, then strip surrounding whitespace from each field we use. Splitting at ', ' would miss a comma without a following space, such as 227.170,-20 in the first row.

7

Convert coefficient and temperature fields to floats so they can be used in calculations. The chemical formula remains a string. Indices 3–5 hold A, B, and C, and indices 6–7 hold the valid temperature range.

8

We finally assign coeffs to the database under the chemical formula.

Now we can use our database as follows:

A = database['CCL4']['A']
B = database['CCL4']['B']
C = database['CCL4']['C']

We have done something a bit fancy here: we used nested dictionaries. The database variable is a dict, but the item stored in each element is also a dict. If this makes your brain hurt, we can do an equivalent look-up as follows:

coeffs = database['CCL4']
A = coeffs['A']
B = coeffs['B']
C = coeffs['C']

Now evaluate the vapor pressure at 25 °C, within the listed range for carbon tetrachloride:

T = 25.0  # degrees C
coeffs = database['CCL4']
if coeffs['Tmin'] <= T <= coeffs['Tmax']:
    P = 10**(coeffs['A'] - coeffs['B']/(T + coeffs['C']))
    print(f'Vapor pressure at {T} °C: {P:.2f} mmHg')
else:
    print('Temperature is outside the range of these coefficients')

Vapor pressure at 25.0 °C: 114.22 mmHg

Splitting by commas is sufficient for this simple file, but it is not a general CSV parser: quoted fields may themselves contain commas or newlines. Use Python’s csv module or pandas.read_csv() for those cases. A for line in f: loop is also a simpler way to read successive lines; the while version above illustrates the end-of-file test and break.

Comments

This example showed us quite a few new tricks in Python:

• Nested dictionaries
• break statements
• String splitting
• And of course reading from a rather complicated file

6.4 Installing External Packages

NoteVideo: Installing packages with pip

Bro Code · about 4 min. Install, list, and update packages with pip. For this course, use the python -m pip form shown below so pip runs with your intended Python interpreter.

Watch on YouTube.

We have already dabble in this briefly when installing streamlit to run the checker apps, but let’s take a closer look.

We have already seen that Python includes many additional packages as part of the “standard library”. These provide functionality that is often important, but not necessary for everyone. The programmer can therefore import the extra functionality when needed.

Although Python’s standard library is considered very good (compared to most languages), it cannot possibly consider everything. For this reason we can install “external packages” from a variety of sources. The most common source is The Python Package Index or PyPI.

Package installation is one reason to learn the command line. Run python -m pip install <package-name> with the Python interpreter used by your project. Replace <package-name> with the package name; the angle brackets are placeholders.

Anaconda Navigator normally manages conda packages from configured channels. pip normally installs from PyPI. These are different package sources and their selections need not be identical.

For our purposes, we can avoid complications by using the anaconda_prompt via the Anaconda Navigator.

The Anaconda Navigator

The Conda Prompt

Figure 6.9: The (Ana)Conda Prompt can be accessed via the Anaconda Navigator user interface.

The conda prompt configures the shell to use a conda environment; it does not provide a special kind of pip. Activate your intended environment (for example, conda activate ne111 if you have created an environment named ne111). Configure Spyder or your notebook kernel to use that same environment. Check the interpreter and its installed packages with:

python -c "import sys; print(sys.executable)"
python -m pip list

The first command displays the Python executable being used. The second runs that interpreter’s pip module and lists packages in its environment. If an installed package cannot be imported in the IDE, compare sys.executable there with the command-line result.

Example 6.4 (Install a package using pip) pandas is known as “Excel for Python”. It let’s us perform operations on tabular data. Probably the most common use of pandas is to read and write csv files.

Download the csv file found here and open it using pandas.

Solution

Clicking the above link will probably result in the file being downloaded to your computer, and placed in the downloads folder.

In Spyder, we will create the following script:

import pandas as pd

p = '/Users/jeff/Downloads/AntoineCoeffs.csv'  # Replace with your actual path
data = pd.read_csv(p)

On Windows, a path could instead be r'C:\Users\jeff\Downloads\AntoineCoeffs.csv'; the r makes this a raw string so backslashes are not interpreted as escapes.

If import pandas raises ModuleNotFoundError, install pandas into the environment used by Spyder:

1. Open the conda prompt and activate that environment.
2. Run python -m pip install pandas.
3. After installation finishes, rerun the script in the matching environment.

If pandas is already installed, no installation is needed. read_csv() does not require openpyxl; that is an engine used for some Excel workbook operations, such as reading .xlsx files with read_excel(). A FileNotFoundError instead means the filename or path needs checking.

data is a pandas DataFrame: a table with labelled rows and columns, not a subclass of dict. Selecting a column, as in data['Compound'], returns a Series. Some operations resemble dictionary access, but DataFrames have their own methods and indexing rules.

Try:

print(data.keys())

Which should give:

Index(['ID', 'Formula', 'Compound', 'A', 'B', 'C', 'Tmin', 'Tmax'], dtype='object')

Now select rows using pandas’ loc indexer. The comparison makes a Boolean condition for each row, and loc selects the rows where it is true:

vals = data.loc[data['Compound'] == 'cyclohexane']

vals looks like this:

      ID Formula     Compound       A         B        C  Tmin   Tmax
268  269   C6H12  cyclohexane  6.8413  1201.531  222.647     6  105.0

Comments

In this example pandas handles CSV parsing and provides a labelled table for selecting data. Installing a package may also install its dependencies, but additional Excel support is not needed for this CSV example.

Now that we have the ability to to install and use more complicated packages, we can revisit some tasks and do them differently.

Example 6.5 (Repeat Example 6.2 using pandas) Instead of writing data directly to a file line-by-line, collect the data in a dict, then write the entire dataset to a csv file using pandas.

Solution

import time, random
1import pandas as pd

2data = {'date_and_time': [], 'value': []}
init_time = time.perf_counter()
while (time.perf_counter() - init_time) < 5:
    data['date_and_time'].append(time.ctime())
    data['value'].append(random.randint(0, 1000))
    time.sleep(1)

3df = pd.DataFrame(data)
4df.to_csv('data2.csv', index=False)

1

Import pandas at the top since we know we’ll be using it

2

Initialize a dictionary with the desired keys, and set values to empty list which we will append data to on each loop.

3

Excel calls them ‘sheets’, pandas calls them ‘DataFrames’. Here we initialize a new DataFrame, and populate it with data. pandas will use the keys as column headers, and fill each column with the corresponding values.

4

pd.read_csv() is a pandas function; df.to_csv() is a DataFrame method. index=False omits the row index from the file, so reading it back yields just the two intended columns. The default index=True would write an extra column.

The DataFrame could look like this (timestamps and random values will differ). Its row index is shown at the left; index=False prevents that index from becoming an extra column in the CSV file:

	date_and_time	value
0	Sun Oct 6 19:46:58 2024	624
1	Sun Oct 6 19:46:59 2024	671
2	Sun Oct 6 19:47:00 2024	681
3	Sun Oct 6 19:47:01 2024	407
4	Sun Oct 6 19:47:02 2024	175

Comments

The overall message here is not “use pandas”, but more generally “there is probably a package that already does some tedious work that I can use instead of coding it by hand”.

6.5 Assignment 5

Submit one Jupyter notebook named A5.ipynb. Use one code cell for each question. The first non-blank line in each cell must be its indicated tag, such as # A5Q1. Write code directly in the cells and assign each result to answer. For questions that write or append to a file, assign None to answer after completing the file operation. Close files before the cell ends.

The checker supplies fresh input values each time it runs a cell. To test in Jupyter, you may place your own example input assignments at the start of the cell, after the tag. The checker replaces these initial assignments with its own values. Each question must work independently of the other question cells. Put shared imports in a cell whose first non-blank line is # setup, or import libraries in the question cell. Do not use notebook magics or shell commands.

Tip

Install and run the checker using the instructions in Appendix E, replacing A1 with A5. Save your notebook, then run python -m ne111_checker A5 on Windows or python3 -m ne111_checker A5 on macOS from any folder. Click Browse… in the checker to select your saved A5.ipynb. Save changes before running a question again; the checker rereads the selected file automatically.

For your own tests in Jupyter, download these files into the notebook’s working directory (right-click and choose Save link as if needed):

• data.csv
• file.txt

The checker creates its own copies of the input files in a temporary working directory for each test. Files written by the checker stay in that temporary directory; files written when you run a cell in Jupyter use Jupyter’s current working directory. Use the supplied filenames rather than absolute paths.

NoteQuestion 1: Count lines in a text file

The checker provides filename, the name of an existing text file in the current working directory. Open and read the file, then assign the number of lines to answer. Count blank lines too; a final line without a trailing newline still counts as a line.

Cell tag: # A5Q1

For example, a file containing "this\nis\na\n\n\nfile" has 6 lines.

NoteQuestion 2: Count words in a text file

The checker provides filename, the name of an existing text file in the current working directory. Open and read the file, then assign its number of words to answer. Words are separated by whitespace; spaces, tabs (\t), and newlines (\n) do not count as words.

Cell tag: # A5Q2

For example, a file containing "this\nis\na\n\n\nfile" has 4 words.

NoteQuestion 3: Write repeated lines to a text file

The checker provides filename, a string value, and a non-negative integer repetitions. Write value to the file repetitions times, with a newline after each copy. Replace any existing contents and store the file in the current working directory. You may assume value contains no newline characters. Assign None to answer after writing the file.

Cell tag: # A5Q3

For example, value = "line" and repetitions = 3 should produce "line\nline\nline\n".

NoteQuestion 4: Write columnar data to a CSV file

The checker provides filename and a dictionary data containing columnar data. Each key is a column heading and each value is a list of column values; all lists have the same length. Write the data to a CSV file in the current working directory, using the keys as column headings and omitting any row index. If filename does not end with .csv, add that extension. Assign None to answer after writing the file. You may use pandas.

Cell tag: # A5Q4

For example, filename = "data2" or filename = "data2.csv" should write data2.csv. For data = {"a": [9, 8, 7], "b": [6, 5, 4]}, its contents are:

a,b
9,6
8,5
7,4

NoteQuestion 5: Read a CSV file into a dictionary

The checker provides filename, the name of an existing CSV file in the current working directory. If the supplied name does not end with .csv, add that extension before opening the file. Read the data and assign a dictionary to answer, with each column heading as a key and its column values as a list. Keep numeric values as numbers. You may use pandas.

Cell tag: # A5Q5

For example, filename = "data" or filename = "data.csv" refers to data.csv. If the file contains "x,y\n1,4\n2,5\n3,6", the expected answer is {"x": [1, 2, 3], "y": [4, 5, 6]}.

NoteQuestion 6: Append a line to a text file

The checker provides filename, the name of an existing text file in the current working directory, and a string value with no newline characters. Append value on a new line, preserving all existing contents. The existing file may have no trailing newline. Assign None to answer after appending.

Cell tag: # A5Q6

For example, appending "new line" to a file containing "first line" should leave "first line\nnew line" (a final trailing newline is also allowed).