WEBVTT

1
00:00:00.200 --> 00:00:05.700
More data is not automatically more useful. Its value depends on what we are trying to

2
00:00:05.700 --> 00:00:07.160
predict or decide.

3
00:00:07.160 --> 00:00:12.110
We therefore need to separate how much remains unknown from whether that uncertainty

4
00:00:12.110 --> 00:00:13.693
matters for the task.

5
00:00:13.693 --> 00:00:19.540
We choose facts useful for the task while the card's Being stays fixed. Becoming would

6
00:00:19.540 --> 00:00:24.280
require a change in that state. Watch which distinctions the task needs.

7
00:00:25.520 --> 00:00:28.853
Episode three kept the distinctions needed for a prediction.

8
00:00:28.853 --> 00:00:35.246
To measure uncertainty, we need more than a list of possibilities: we need probabilities and a target.

9
00:00:35.246 --> 00:00:37.580
Return to the four-card example.

10
00:00:37.580 --> 00:00:45.120
The short existing overview names four meanings of information; then we will work one of its calculations step by step.

11
00:00:45.990 --> 00:00:49.398
The word information can point to several different things.

12
00:00:49.398 --> 00:00:51.590
Here are four possible states.

13
00:00:51.590 --> 00:00:54.650
A is the first digit; B is the second.

14
00:00:54.650 --> 00:00:57.750
Suppose an exact readout says A is zero.

15
00:00:57.750 --> 00:01:00.050
That answer is an observation record.

16
00:01:00.050 --> 00:01:02.630
Now ask what this test can distinguish.

17
00:01:02.630 --> 00:01:07.163
Reading A separates states starting with zero from states starting with one.

18
00:01:07.163 --> 00:01:09.510
It leaves the second digit unresolved.

19
00:01:09.510 --> 00:01:13.243
This is a pattern of distinctions, not yet an uncertainty score.

20
00:01:13.243 --> 00:01:17.323
To measure uncertainty, supply probabilities and name a target.

21
00:01:17.323 --> 00:01:23.870
If the four states are equally likely, reading A leaves one bit of uncertainty about the full state.

22
00:01:23.870 --> 00:01:29.080
That number summarizes the probability model; it does not list the missing distinctions.

23
00:01:29.080 --> 00:01:31.043
Finally, name the task.

24
00:01:31.043 --> 00:01:33.881
If we need A, this readout gives the answer.

25
00:01:33.881 --> 00:01:38.078
Change the target to B, and it gives no advantage under our independent,

26
00:01:38.078 --> 00:01:38.944
fair model.

27
00:01:38.944 --> 00:01:44.330
Record, distinctions, uncertainty, and task value are related questions.

28
00:01:44.330 --> 00:01:48.990
Keeping them separate helps us say exactly what changed when we gained information.

29
00:01:49.780 --> 00:01:53.000
Assign each card probability one quarter.

30
00:01:53.000 --> 00:01:55.260
Let X name the whole card.

31
00:01:55.260 --> 00:02:00.380
One fair zero-or-one choice has one bit of uncertainty.

32
00:02:00.380 --> 00:02:06.716
This card combines two independent fair choices, so its Shannon entropy is two bits.

33
00:02:06.716 --> 00:02:11.543
Entropy averages uncertainty under a specified probability distribution.

34
00:02:11.543 --> 00:02:14.903
It differs from the number of digits used to write the answer.

35
00:02:14.903 --> 00:02:20.716
We will use this fair case first; the optional entropy primer also works a biased example.

36
00:02:21.506 --> 00:02:24.386
Read A and suppose it is zero.

37
00:02:24.386 --> 00:02:26.893
Zero-zero and zero-one survive.

38
00:02:26.893 --> 00:02:34.556
Their original probabilities were one quarter each; the accepted event had total probability one half.

39
00:02:34.556 --> 00:02:39.970
Within it, each card therefore has conditional probability one half.

40
00:02:39.970 --> 00:02:45.810
One fair choice remains, so one bit of full-card uncertainty remains.

41
00:02:45.810 --> 00:02:49.910
The A-equals-one branch also leaves one bit.

42
00:02:49.910 --> 00:02:57.030
Conditional entropy averages these remaining uncertainties over the possible answers; here the average is one bit.

43
00:02:57.900 --> 00:03:00.090
Now the target is B alone.

44
00:03:00.090 --> 00:03:07.420
Reading A still leaves B equally likely to be zero or one, so one bit about B remains.

45
00:03:07.420 --> 00:03:11.200
Reading B itself leaves no uncertainty about B.

46
00:03:11.200 --> 00:03:16.727
The average reduction in uncertainty about a named target is called mutual information.

47
00:03:16.727 --> 00:03:22.807
In this distribution, A supplies zero bits about B, while B supplies one.

48
00:03:22.807 --> 00:03:25.820
These numbers depend on the target and the model.

49
00:03:26.610 --> 00:03:33.530
Suppose the task is to predict B correctly, scoring one for a correct answer and zero otherwise.

50
00:03:33.530 --> 00:03:40.870
Averaged over the four equally likely cards, the best accuracy using only A is fifty percent.

51
00:03:40.870 --> 00:03:46.210
Reading B gives one hundred percent accuracy under this exact model.

52
00:03:46.210 --> 00:03:50.870
Yet either single test leaves one bit about the whole card.

53
00:03:50.870 --> 00:03:56.677
Equal full-state residual entropy does not imply equal task value.

54
00:03:56.677 --> 00:04:01.043
Change the task to A, and which reader is useful reverses.

55
00:04:01.833 --> 00:04:07.353
The four probabilities were assumptions, not consequences of there being four cards.

56
00:04:07.353 --> 00:04:11.737
Changing them can change the numerical entropies and decision scores.

57
00:04:11.737 --> 00:04:19.873
Also, conditional entropy is an average across answers; a particular rare answer can leave more uncertainty than before.

58
00:04:19.873 --> 00:04:23.093
The optional foundations explain that case.

59
00:04:23.093 --> 00:04:31.433
Our core calculations used a fair distribution and exact observations, not a general rule that every question removes one bit.

60
00:04:32.303 --> 00:04:38.803
For a final check, the task is shape prediction and you may read one of the two card properties.

61
00:04:38.803 --> 00:04:45.016
Which do you choose?

62
00:04:45.016 --> 00:04:48.883
B, because it directly resolves the target under this model.

63
00:04:48.883 --> 00:04:55.672
To compare richer strategies, we must also name available tests, permitted timing and costs.

64
00:04:55.672 --> 00:04:57.856
That is the next episode’s job.

65
00:04:57.856 --> 00:05:05.256
Later, the noisy-reader lesson will announce its own prior instead of borrowing these four equal card probabilities.