JIT — Intro to Data Science · Python Lab · Problem 24 of 30

Marks worth a second look

Quartiles, the spread between them, and a pair of fences built out of the class itself.

Unusual, or just low?

Problem 23 dealt with values that were never recorded. This one deals with values that were recorded and look wrong. A mark of 12 in a class sitting in the seventies might be a real student who had a bad day, or a typing slip for 72, or a mark out of 20 that got into the wrong column. You cannot tell from the number alone — but you can tell that it deserves a second look, and that is a decision a rule can make for you.

New word — quartile Sort the marks and cut them into four equal groups. Q1 is the value a quarter of the way up, Q3 three quarters of the way up. Between them sits the middle half of the class.
marks.quantile(0.25) and marks.quantile(0.75). The number in the brackets is a proportion, so it runs from 0 to 1 — not a percentage.
New word — IQR The interquartile range: q3 - q1. The width of that middle half, and a measure of spread that one extreme value cannot disturb.
That is its advantage over the standard deviation from problem 19. A single mark of 12 pulls the average down and pushes the standard deviation up, so both are affected by the very value you are trying to judge. Q1 and Q3 barely move.
The fences q1 - 1.5 * iqr and q3 + 1.5 * iqr. Anything outside them is worth looking at.
The 1.5 is a convention, not a law of nature. It is wide enough that ordinary variation stays inside and narrow enough to catch genuine oddities, and it has been in use long enough that a reader will understand what you did without being told.

The class you are given

Nine marks between 62 and 81, and one of 12. Sorted onto a number line, with the middle half shaded and the two fences dashed:

020406080100 12 outside the fence middle half of the class Q1 68.5 Q3 75.5 58 86

The middle half is narrow — seven marks wide — so the fences sit at 58 and 86. Every ordinary mark falls comfortably inside. The 12 is nowhere near, and the rule found that without anybody deciding in advance what counted as too low.

Flagging is not deleting This programme picks out rows for a human to look at. It does not remove them. An unusual value may be an error, and it may be the most interesting thing in your data — quietly dropping whatever falls outside a fence is how real findings get thrown away.
Careful — combining two conditions on a column A value is unusual if it is below the low fence or above the high one. In pandas that is written with |, not the word or, and each condition needs its own brackets:
(marks < low_fence) | (marks > high_fence)
Use & and you have asked for marks that are below 58 and above 86 at the same time, which nothing can be. The answer comes back empty and perfectly calm.

What you are building

Q1: 68.5 Q3: 75.5 IQR: 7.0 Fences: 58.0 to 86.0 name marks 9 Nikhil 12 How many unusual: 1
Coming later This is the arithmetic a box plot draws. In problem 26 you will plot the same figures and see the box, the whiskers and the outlier as a picture rather than a set of numbers.

Build the programme

Your plan

The steps still to place

The whole programme is laid out below. Five pieces are missing. One of them stops the programme; the rest move the fences, and a fence in the wrong place either misses the odd value or flags half the class.

Leave nothing on "choose…". A wrong pick does not always cause an error — sometimes it just prints something you did not expect, which is the harder kind of mistake to spot.

Run and read

Here is the whole programme. Two files are available: one with an unusual mark in it, and one where the same ten students all sat in a narrow band. Run both — a rule that finds nothing when there is nothing to find is doing its job too.

Your programme

Output

Nothing has run yet.