JIT — Intro to Data Science · Python Lab · Problem 30 of 30

Who is at risk, and did we find them?

A classifier that is right three times out of four and catches none of the students it was built to catch.

Right most of the time, and useless

Problem 29 predicted a number and judged it with R². This one predicts a category — is this student at risk of failing, yes or no — and judging it turns out to need more care than a single score can give.

Forty students, ten of them genuinely at risk. The model is trained on twenty-eight and tested on twelve, of whom three are at risk.

New word — classifier KNeighborsClassifier(n_neighbors=3) predicts a label rather than a number. To classify a new student it finds the three most similar students it has seen and goes with the majority.
The idea is as simple as it sounds, and it is a reasonable first thing to try on almost any labelled data.
New word — confusion matrix Four numbers, because there are four things that can happen:
predicted safepredicted at risk
really safeTNFP — a false alarm
really at riskFN — a student missedTP — caught
confusion_matrix(y_test, predicted) prints them in that layout: truth down the side, predictions across the top. The order is truth first. Swap the arguments and the table comes back transposed, which turns every missed student into a false alarm and reads perfectly plausibly.
Careful — the order of tn, fp, fn, tp .ravel() flattens the four numbers into a row, reading left to right and top to bottom, so they come out as tn, fp, fn, tp. Unpack them in any other order and every line you write afterwards is confidently mislabelled — there is no error, only four numbers wearing each other's names.
Why accuracy is not enough Nine of the twelve test students are safe. A model that simply answers "safe" to everybody is right nine times out of twelve — 75% — while being no use to a single student who needed help.
That is roughly what happens here. Watch the accuracy and the number caught in the same output: the accuracy looks respectable and the model found nobody. Whenever one outcome is much rarer than the other — fraud, disease, students at risk — accuracy quietly rewards ignoring it.
The two questions worth asking instead Of the students who were really at risk, how many did we catch? — that is tp out of tp + fn, the bottom row of the table.
Of the students we flagged, how many really were at risk? — that is tp out of tp + fp, the right-hand column.
They are different questions with different answers, and which one matters depends entirely on what it costs to miss somebody versus what it costs to raise a false alarm.

What you are building

Confusion matrix: [[9 0] [3 0]] Accuracy: 0.75 Caught 0 of the 3 students who were really at risk
That last line is the whole course in miniature The programme ran, the accuracy was respectable, and the answer was worthless. Nothing in Python was going to tell you that. Every problem in this series has been a version of the same lesson: the machine does exactly what you asked, and checking whether you asked the right thing is your job.

Build the programme

Your plan

The steps still to place

The whole programme is laid out below. Five pieces are missing. Two stop the programme; the rest report numbers that are perfectly real and answer a question you did not ask.

Leave nothing on "choose…". A wrong pick does not always cause an error — sometimes it just prints something you did not expect, which is the harder kind of mistake to spot.

Run and read

Here is the whole programme. One file, one fixed split, so the answer is the same every run. Read the accuracy and the last line together — they are describing the same model.

Your programme

Output

Nothing has run yet.