JIT — Intro to Data Science · Python Lab · Problem 28 of 30

Is the relationship real?

Two numbers for the two questions a scatter plot raises — how strong, and how easily chance could have done it.

Putting a number on the cloud

In problem 27 you looked at two scatter plots and judged them by eye: one a tight rising band, the other a shapeless blob. That judgement was right, and it does not travel. You cannot put "looked fairly tight to me" in a report, and on a hundred pairs of columns you cannot look at every one.

Two numbers do the job, and they answer two different questions.

New word — correlation coefficient, r A number from −1 to +1 measuring how close the points lie to a straight line.
+1 perfectly rising, 0 no straight-line relationship at all, −1 perfectly falling. The sign is the direction; the size is the strength. An r of −0.9 is just as strong as +0.9 and points the other way, which is why strength is judged on abs(r) and never on r itself.
New word — p-value Suppose hours and marks had nothing to do with each other. How often would ten students, picked at random, happen to line up at least as well as yours did? That proportion is the p-value.
A small p means the pattern would be a surprising fluke, so it is probably not a fluke. A large p means what you are looking at is well within the range of ordinary coincidence. Below 0.05 is the usual line — a convention, like the 1.5 in problem 24, not a law.
They are not the same question, and you need both r says how strong. p says how easily chance could have produced it.
On a large enough sample a feeble r of 0.08 can come with a tiny p: real, and far too weak to be worth acting on. On ten students a strong-looking r can come with a large p: striking, and quite possibly luck. Reporting one without the other is how both mistakes get made.
Careful — r only sees straight lines An r near zero means no straight-line relationship. Points sitting in a perfect arch have a real and obvious pattern and an r of about zero. This is why problem 27 came first: look at the picture, then calculate the number. A number quoted without ever plotting the data is a number nobody has checked.

The two files, as numbers

FilerpWhat it means
study_strong.csv0.9920.0000000185very strong, and chance is not a plausible explanation
study_none.csv-0.1140.754almost nothing, and easily luck

That second p of 0.75 is worth sitting with. It says: if hours and marks were entirely unrelated, three times out of four you would see a pattern at least this convincing. There is nothing here.

What you are building

r = 0.992 p = 0.0 Strong enough? True Unlikely to be chance? True Worth reporting: the two move together.
Coming later r says the two move together. Problem 29 fits the line properly, uses it to predict, and measures how much of the variation it actually accounts for.

Build the programme

Your plan

The steps still to place

The whole programme is laid out below. Five pieces are missing. None of them will stop the programme — every one produces a verdict, and three of them produce a confident wrong one.

Leave nothing on "choose…". A wrong pick does not always cause an error — sometimes it just prints something you did not expect, which is the harder kind of mistake to spot.

Run and read

Here is the whole programme. Run it on both files. The second one is the test of whether your two conditions are written correctly, because that is the file where the honest answer is "no".

Your programme

Output

Nothing has run yet.