JIT — Intro to Data Science · Python Lab · Problem 21 of 30

Opening a file somebody else made

Four commands to run before you trust a new file — and the single empty cell that changes what a whole column is made of.

A file you did not create

Every table so far was typed out inside the programme, so you knew exactly what was in it. Real data arrives as a file from somebody else, and the first job is always the same: find out what you have actually been given, before you calculate a single thing.

New word — CSV A plain text file, one row per line, values separated by commas. The first line is usually the column headings. It is the most common way data is handed around because anything can read it — and because it carries no information about what the values mean, which is where the trouble starts.

The two files sitting next to your programme

attendance.csv — tidy:

name,city,classes_held,classes_attended Anjali,Nagpur,40,36 Farhan,Pune,40,31 Priya,Nagpur,40,40 Rohit,Mumbai,40,22 Meera,Nagpur,40,38 Imran,Pune,40,29 Sneha,Nagpur,40,35 Kabir,Mumbai,40,40

attendance_raw.csv — the same file as it actually arrived. Look at Rohit's row:

Rohit,Mumbai,40,← nothing after the last comma
New words — head, shape, dtypes, info read_csv opens the file and hands back a DataFrame. Then, in order:
.head() shows the first five rows, so you can see whether it looks like what you expected.
.shape says how big it is.
.dtypes says what kind of value pandas decided each column holds.
.info() puts all of that together and adds the one number that matters most on a new file: how many values in each column are not missing.
Careful — one empty cell changes a whole column A column of whole numbers is int64. Leave one cell blank and pandas has to put something in the gap, and the something it uses is NaN — not a number. There is no such thing as a missing whole number, so the entire column becomes float64 and every value in it grows a decimal point.
Nobody edited those other seven values. One blank cell in one row changed how all eight are stored, and .dtypes is where you find out.
info() shows its answer and hands back nothing .info() prints directly, the way the function in problem 16 did when it used print instead of return. So you call it on a line by itself. Wrap it in print() and you get the report followed by the word None — the same signature as before.

The output shown here is from pandas 3. Older versions print object where this shows str, and a longer class name at the top of the info report. The lesson is the same in both.

Coming later Finding the gap is this problem. Deciding what to do about it — fill it, or drop the row — is problem 23.

Build the programme

Your plan

The steps still to place

The whole programme is laid out below. Five pieces are missing. One of them changes what pandas thinks the first line of the file is, which quietly changes everything else.

Leave nothing on "choose…". A wrong pick does not always cause an error — sometimes it just prints something you did not expect, which is the harder kind of mistake to spot.

Run and read

Here is the whole programme. Switch between the two files and run it twice. The rows barely change; the report underneath changes a great deal.

Your programme

Output

Nothing has run yet.