According to the Wikimedia Commons description, this is a B-17 of the U.S. 398th Bomb Group whose nose was destroyed by flak during a bombing raid on Cologne, Germany, on October 15, 1944. The plane still made it back to base. Image: U.S. Air Force (National Museum of the U.S. Air Force), via Wikimedia Commons; license: public domain.

In 1943, Abraham Wald of Columbia University's Statistical Research Group (SRG) wrote a set of memoranda on how to estimate, using only the bullet holes on returning aircraft, the probability that a plane would be shot down after being hit in each of its parts. The popular story reduces this to a one-line quip: “reinforce wherever there are no holes.” Checked against the original memoranda and Mangel and Samaniego's 1984 interpretation, the actual reasoning was more subtle, and many of the details in circulation cannot be verified.

Columbia University's Statistical Research Group

The Statistical Research Group (SRG) was established on July 1, 1942, at Columbia University in New York. It was funded by the Applied Mathematics Panel (AMP) of the National Defense Research Committee (NDRC), which sat under the U.S. Office of Scientific Research and Development (OSRD). W. Allen Wallis, the group's director of research, recorded several figures in a 1980 retrospective article. Over its lifetime the group had 18 principal members and a total of 78 staff. At most about 50 people were on duty at once, including about 30 young women who worked only on computation. During the war the group produced 572 reports, memoranda and letters. Its members included Friedman, Savage, Mosteller, Wolfowitz and others who later became leading figures.

Wald was born in 1902 in Cluj, in present-day Romania. He worked on pure mathematics in Vienna in his early years and turned to mathematical statistics in the mid-thirties. Later, after losing his position in Austria because he was Jewish, he made his way to the United States. Wallis recalled that in the fall of 1939, when Wald had just begun teaching at Columbia, he and Wolfowitz often attended Wald's evening lectures. Wald was a part-time member of SRG and took part in its work for about 30 months in total. His best-known achievement there was sequential analysis, which originated at SRG in 1943. According to Wallis, the classification of sequential analysis as secret had a side effect: Wald's naturalization application was approved quickly, making him one of the few “enemy aliens” to obtain U.S. citizenship during the war.

Aircraft vulnerability was one of the topics that grew out of SRG's air-warfare research. In SRG's final report, written in 1945, Wallis wrote that this problem led the group to devise a method of “judging vulnerability from the damage sustained by our own planes.” Mangel and Samaniego summarize the operational problem of the time this way: commanders had to decide which tactics to use and which parts of the aircraft to give extra protection, but added armor increases weight and reduces a plane's ability to carry out its mission. The real difficulty was that no one could see how the bullet holes were distributed on the planes that failed to return.

The Reasoning and Numerical Example in the Memoranda

Wald's work took the form of a set of technical memoranda under the overall title “A Method of Estimating Plane Vulnerability Based on Damage of Survivors.” According to the Defense Technical Information Center's entry, these are eight memoranda that SRG wrote in 1943 for the National Defense Research Committee. They were not published during the war and were reprinted only in July 1980, by the Center for Naval Analyses (CNA), as CRC 432. The first few parts do not distinguish among parts of the aircraft. Using only the number of sorties, the number of planes that returned, and the number of hits on each returning plane, they derive the probability that a plane is downed by the i-th hit, along with upper and lower bounds on that probability. Part Five divides the aircraft into several “equal-vulnerability regions” and estimates the vulnerability of each part. Part Six deals with sampling error and gives 95% and 99% confidence limits.

The numerical example in Part Five shows the line of reasoning most clearly. Wald supposes that 400 planes fly a bombing mission and 359 return. Of the returning planes, 240 were not hit, while 68, 29, 12 and 10 planes took 1 to 4 hits respectively, for a total of 202 hits. He adopts simplifying assumptions, such as “every hit has the same probability of downing the plane, regardless of how many hits came before.” Under these assumptions he first solves for the probability that a plane survives a single hit, which comes to about 0.85. He then divides the aircraft into the engine, the fuselage, the fuel system and other parts. He assumes that the probability of a single hit landing on a given part equals that part's share of the projected area facing the direction of enemy attack: engine 26.9%, fuselage 34.6%, fuel system 15.4%, other parts 23.1%.

The hits on the returning planes, however, did not follow these proportions. The engine accounted for only 19.3% of them (39 hits), the fuselage 38.6% (78), the fuel system 15.4% (31), and other parts 26.7% (54). If a part's share of hits on surviving planes is lower than the share it “should” receive based on its area, then more of the planes hit there failed to return. By Wald's formula, the probability of surviving a single hit on a given part equals 0.85 multiplied by “actual share ÷ expected share.” That gives about 0.61 for the engine, about 0.95 for the fuselage, 0.85 for the fuel system, and about 0.98 for other parts. On this basis the memorandum states that, for the data of this hypothetical example, the engine area is the most vulnerable and the fuselage is relatively less vulnerable.

The hypothetical example in Part Five of Wald's memoranda. The engine's share of hits on returning planes is lower than the share its area would predict, and its estimated single-hit kill probability is the highest. Data from Wald's (1943) memoranda, CNA 1980 reprint; chart drawn with code for this article.

What Survivorship Bias Means and What It Assumes

This is what later came to be called survivorship bias: whether an object makes it into the sample depends on the very outcome we want to study. The returning planes are not a random sample of all the planes that flew. They are only the ones that “could still fly home after being hit.” If you count holes on them, the places with the most holes are often the places that can best withstand fire. In a short 1980 piece responding to readers' comments, Wallis put the point plainly. The military was inclined to add protection to the parts of returning planes that had been hit most often. Wald, however, had good reason to assume that hits in combat were spread roughly evenly over the whole aircraft. Hits on vital parts would therefore be underrepresented on the returning planes, because planes hit in those parts were less likely to come back and supply data.

Still, “reinforce where there are no holes” is only shorthand, and it holds only under one condition: we must first know how hits would have been distributed if no planes had been shot down. In Part Five, Wald specifically points out that the damage on returning planes alone cannot be used to estimate both the hit distribution and the vulnerability of each part. He gives an example. Suppose all the holes on returning planes fall in Region One. That could mean Region Two simply cannot be hit, or it could mean that any plane hit in Region Two goes down. The data cannot tell the two explanations apart. He therefore required the hit distribution to be known in advance. It could be inferred from the dispersion of enemy fire, or measured in firing tests with “dummy bullets.” In the numerical example, the proportions of projected area serve in its place.

In other words, Wald was not comparing “holes versus no holes.” He was comparing “how many fewer holes than expected.” In his example the engine still had 39 holes, just fewer than its area would predict. If a spot has few holes only because it is small and rarely hit, that is no reason to conclude that hits there are more deadly. In their 1984 article, Mangel and Samaniego re-examined these memoranda using modern statistical theory and discussed them within the framework of missing data and approximation methods. They concluded that Wald's treatment of these problems “was definitive.”

Exaggerations and Unsupported Details in Popular Versions

The popular version goes roughly like this. Looking at the pattern of holes on returning bombers, the military was about to reinforce the heavily hit fuselage. Wald pointed out on the spot that the sparsely hit engines were what needed reinforcing, and his advice was adopted. In 2016, the mathematician Bill Casselman checked this story point by point in his column for the American Mathematical Society. When it is compared with the original sources, its claims fall into three categories.

The documented parts: Wald did write a set of memoranda at SRG on inferring vulnerability from damage to surviving aircraft. Wallis's 1980 rejoinder did say that the military was inclined to reinforce the parts of returning planes that had been hit most often, and it stated that these methods were later used by the Navy and the Air Force. CNA's note to the reprint and Mangel and Samaniego's article both mention that the method was still used to analyze data from the Korean and Vietnam Wars.

The exaggerated parts: the memoranda are rigorous and highly technical mathematical derivations, and Casselman notes that they make no recommendation about what the military should do. The claim that “the engine is the most vulnerable” comes from the hypothetical data in Part Five. The text says explicitly that this conclusion applies to “this hypothetical example” and is not a finding measured in any actual campaign. Telling the story as a flash of insight also leaves out the assumptions Wald made to solve the identification problem, as well as the separate treatment he gave to sampling error.

The unsupported parts: no original record currently supports the conversations and meeting scenes between Wald and military officers, or the so-called “Wald's hand-drawn bullet-hole diagram” that circulates online. Casselman originally found that the only reliable material on this work was two brief passages in Wallis's retrospective article that do not mention Wald by name, plus the memoranda themselves. He later added Wallis's rejoinder in an addendum. The red-dotted airplane diagrams commonly seen online were all drawn by others later, and some even show a C-47 transport, a plane that rarely saw combat.

Chance also played a part in the memoranda becoming public nearly forty years late. Casselman quotes Phil DePoy of the Center for Naval Analyses on how this happened. When Wallis was moving house, he came across a box of loose papers left over from when SRG shut down in 1946 and handed it to DePoy. DePoy kept only one packet from it, Wald's material, edited it lightly, and published it under Wald's name in July 1980. Wald himself had died in 1950 in a plane crash in the mountains of southern India, and he left no recollections of his own about this work.

References

1. Wald, A. (1980). A Reprint of "A Method of Estimating Plane Vulnerability Based on Damage of Survivors" by Abraham Wald (CRC 432). Center for Naval Analyses. Original memoranda written in 1943. https://www.cna.org/reports/1980/0204320000.pdf

2. Defense Technical Information Center. Entry ADA091073. https://apps.dtic.mil/sti/citations/ADA091073

3. Mangel, M., & Samaniego, F. J. (1984). Abraham Wald's Work on Aircraft Survivability. Journal of the American Statistical Association, 79(386), 259–267. https://doi.org/10.1080/01621459.1984.10478038

4. Wallis, W. A. (1980). The Statistical Research Group, 1942–1945. Journal of the American Statistical Association, 75(370), 320–330. https://doi.org/10.1080/01621459.1980.10477469

5. Wallis, W. A. (1980). The Statistical Research Group, 1942–1945: Rejoinder. Journal of the American Statistical Association, 75(370), 334–335. https://doi.org/10.1080/01621459.1980.10477472

6. Casselman, B. (2016, June). The Legend of Abraham Wald. AMS Feature Column. https://www.ams.org/publicoutreach/feature-column/fc-2016-06

7. Research Methods · Rituals can be imitated; understanding cannot be faked

8. Compiled and published by Scholay, a one-stop academic platform. If this infringes your rights, please let us know and it will be taken down.

9. Also published on EnterScholar, a community for scholars in the age of intelligence

On October 20, 2026, Wedgewood Polls, an emerging American polling outfit, admitted on the social platform X that the vast majority of the polls it had published had no real survey data behind them. The admission was prompted by an investigation by the election-data site FiftyPlusOne: after obtaining its raw data, reporters found the sample was abnormally “clean,” and the respondent IDs and transaction IDs did not follow the patterns of the survey platform it claimed to use. This article reviews what happened, how the data were checked and how polling aggregators screen pollsters, and lists items readers can check for themselves.

The “Vote Here” sign outside the polling place at Woodbury City Hall, Minnesota, on U.S. Election Day, November 3, 2020. Photo: Tony Webster. Source: Wikimedia Commons. License: CC BY 2.0.

What Happened with Wedgewood

According to Newsweek and The Guardian, Wedgewood Polls appeared in June 2026, describing itself as an independent election pollster. One of its earlier polls, on Louisiana’s Republican Senate primary, showed John Fleming leading Julia Letlow by 4 percentage points, while in the actual result on June 27 Letlow won by nearly 14 percentage points. On July 10, Washington Monthly columnist Bill Scher pointed out that the new outfit’s first poll had been wildly wrong.

On August 20, Straight Arrow News reported on its background: its website was built on Google Sites and listed no staff names, phone number or location; the site was registered on June 26, and its only social media account, on X, was also created in June. Wedgewood described itself as a two-person team that had not yet incorporated as a company. It declined to reveal who its members were on the grounds that one of them had been harassed online, and it also refused video calls. At the time, The New York Times’ polling database already included its results, but with the note “this pollster does not meet our criteria for select pollsters.”

In mid-October, Wedgewood released a Mississippi Senate poll: conducted October 12 to 14 among 511 likely voters, it showed incumbent Republican Senator Cindy Hyde-Smith leading Democrat Scott Colom by only 52% to 48%. According to Magnolia Tribune, Colom’s campaign promptly cited the poll in a press release. FiftyPlusOne’s Mary Radcliffe, G. Elliott Morris and Cooper Burton had received multiple tips about Wedgewood before, but this time the tip came with evidence, so they asked Wedgewood for the poll’s raw data. On October 20, FiftyPlusOne sent its findings to Wedgewood for comment. Rather than answer the questions, Wedgewood posted a statement on X admitting that the vast majority of its polls were not backed by real data. It called this a “social experiment” to expose gaps in how media outlets and aggregators vet pollsters, and said that only one crowdfunded Nebraska poll was real. For now, both the “social experiment” motive and the claim that the Nebraska poll was real rest only on Wedgewood’s own account.

Afterward, The New York Times removed all of Wedgewood’s polls from its poll tracker, Decision Desk HQ deleted 17 of them from its race lists and averages, Silver Bulletin announced a permanent ban, and Newsweek retracted 3 articles and corrected other coverage. Michigan State Senator Mallory McMorrow said that a Wedgewood poll in late June showing her support falling into single digits dried up her campaign donations and hurt her volunteers’ morale; she withdrew from the Senate primary on July 5. This account of the poll’s impact is her own.

FiftyPlusOne’s Data Check

According to FiftyPlusOne’s report (as quoted on Gelman’s blog), the reporters’ first impression on obtaining the raw data was that it was “surprisingly clean.” The unweighted sample was 67% white and 31% Black, already quite close to reasonable estimates for Mississippi (about 60% and 37%); the poll’s design effect was only 1.18. The design effect measures “how hard” the weighting has to work to make the sample representative: the more skewed the sample, the more spread out the weights must be, the higher the design effect, and the smaller the effective sample size. For example, if a sample has far too few young people, the few young respondents must be given very large weights, which amounts to letting a handful of people speak for a large group. By the usual conversion, 511 respondents with a design effect of 1.18 amount to an effective sample of about 433, a very small loss for an online survey.

A low design effect does not by itself mean fabrication. If quotas are set during recruitment, requiring the platform to return respondents from each group in proportion to the population, the raw sample can also be very close to the population. FiftyPlusOne therefore asked Wedgewood whether it had used quotas, because as far as they understood, quotas seemed to be the only way to control sample composition on Pollfish while a survey is in the field. Wedgewood said it had not, which made the unusually low design effect harder to explain. But the reporters also stated that this was still not enough to prove the data were made up.

The decisive evidence came from IDs automatically generated by the survey platform. Wedgewood said the poll had been conducted through the online survey platform Pollfish, so FiftyPlusOne paid out of its own pocket to replicate the Mississippi survey on Pollfish as closely as possible and obtain real data for comparison. In the real data, about half of the respondent IDs were plain strings of digits (e.g., “1234567890”) and half had a hyphen prefix (e.g., “-1234567890”); the transaction IDs tracked the time respondents submitted the questionnaire almost perfectly (in the report’s words, “almost 1:1 correlated”). The reporters confirmed with Pollfish that this was exactly the expected behavior of these two columns, and checked data from another organization that had used the platform, finding the same patterns. The data Wedgewood provided matched neither pattern, nor the platform’s description of these two columns.

Example scatter plots for different correlation coefficients: at the far left of the first row the correlation is 1 and the points fall on a straight line; in the middle it is 0 and the points form a patternless cloud. According to the report, the relationship between transaction IDs and submission times in real Pollfish data is close to the former. This is a generic illustration, not data from FiftyPlusOne or Wedgewood. Image: DenisBoigelot (original uploader Imagecreator). Source: Wikimedia Commons. License: CC0.

It is a bit like checking a receipt: a forger can make the amount and items look very convincing, but is unlikely to think of imitating the fixed format of the cash register’s serial numbers or the way they increase over time. FiftyPlusOne stated that the ID patterns were only two of the “many factors” that raised its suspicions. The rest is in the paywalled part of the original article, which the author of this article was unable to read. In the freely available part, the report does not mention statistical tests such as checking the distribution of final digits.

Before this data check, FiftyPlusOne says it had already checked Wedgewood’s identity: Wedgewood provided two names, one of whom was independently confirmed to have worked in polling, and the reporters also read the methodology description on its website closely. In its statement, Wedgewood accused FiftyPlusOne, The New York Times and other aggregators of failing to vet properly; FiftyPlusOne says this claim is false. Silver Bulletin also says Wedgewood had previously answered questions from several aggregators about its identity, methods and funding, which is why it was included. Judging by the outcome, vetting based on self-reported materials was circumvented, and what ultimately worked was checking at the level of raw data.

Aggregators’ Screening Rules and the Blog Commentary

Polling aggregators usually do not audit raw data poll by poll; instead they decide whether to include a pollster, and how much weight to give it, based on the pollster’s characteristics. The explanation on The New York Times’ polling page says that pollsters that meet at least two of the following three criteria and conduct surveys for nonpartisan sponsors are classified as “select pollsters”: an accurate track record in recent elections; membership in professional polling organizations; and use of probability sampling. These factors affect each poll’s weight in the average. Wedgewood did not meet this standard, so although it was included, its polls always carried the “does not meet the criteria” notice.

Silver Bulletin’s explanation says its averages distinguish between registered voters and likely voters, adjust for each pollster’s systematic lean (house effect), and give more weight to more reliable polls; most of its election polls are drawn from The New York Times’ database. Rules like these rely mainly on pollsters’ self-reported information and track records. A new pollster has no track record yet and can fabricate its self-reported information, so this kind of screening can only do so much. FiftyPlusOne also warns in its report that AI tools make it increasingly easy to generate websites, PDFs and tables, so fakers can make their materials look as polished as those of legitimate research organizations.

On his blog, Gelman praised this as a successful application of data-journalism methods and offered two personal comments. First, Wedgewood said it was an “experiment” only on the very day it was exposed, which he considers “a bit suspicious.” Second, he reiterated a view he first put forward in 2004: there are too many polls in the United States, even counting only legitimate ones. The sheer number of polls wastes respondents’ time and also makes people less willing to take part in surveys that serve the public good, such as public health surveys. Polls and “horse-race” election coverage that cares only about who is ahead feed each other: polls supply headlines for such coverage, and such coverage spawns more polls. Fake polls mixed in only make things worse. These are Gelman’s views, not conclusions of the investigation itself.

Items Readers Can Check

The American Association for Public Opinion Research (AAPOR) Transparency Initiative lists the elements that should be disclosed when survey results are published; the initiative only requires disclosure and does not judge methodological quality. In light of this episode, when readers come across an unfamiliar poll, they can check the following items one by one. Note that these are only screening clues: meeting all of them does not prove the data are real. Wedgewood itself passed some vetting.

• Who conducted it and who paid: whether the sponsor and the organization that carried out the survey are named; whether it was commissioned by a candidate, party or advocacy group.
• Whether the organization can be verified: whether there are verifiable names of those in charge, an address and a phone number; when the website and social media accounts were created; whether it belongs to professional polling organizations or the AAPOR Transparency Initiative.
• Track record: how far the organization’s past polls were from actual election results. Wedgewood’s earlier Louisiana poll even got the leader wrong.
• Sampling method: whether it used probability sampling or an opt-in online sample; which platform, list or panel the sample came from.
• Questionnaire, timing and population: whether the exact question wording, interview dates, target population (all adults, registered voters or likely voters) and sample size are published.
• Weighting and error: whether it explains which variables were used for weighting; whether the margin of error accounts for the design effect. Under AAPOR’s requirements, a nonprobability sample that does not describe the model used should not report a measure of precision, and should instead explain why none is provided.
• Data quality control: whether it describes measures such as attention checks, removing respondents who answer too quickly, and preventing duplicate responses, and whether AI was used.
• Whether the results and timing are too convenient: whether the results happen to play into hot topics in public discussion, and whether the release timing is suspiciously convenient; when the raw data are available, whether the sample is abnormally “clean.” Clues like these only signal that further checking is needed; they are not, on their own, evidence of fabrication.

Generally only journalists and aggregators can check raw data, and only if they can obtain the data and have comparable real data as a reference. In this case, FiftyPlusOne ran its own survey on the same platform and compared the suspect data against the “fingerprint” of real data. For ordinary readers, the more practical course is to pay attention to how aggregators label pollsters and to treat polls of unclear origin with only a single result with caution.

References

1. Andrew Gelman. Real political reporters use data journalism techniques to catch fakers at “Wedgewood Polls”. Statistical Modeling, Causal Inference, and Social Science, 2026-10-20.

2. Mary Radcliffe, G. Elliott Morris, Cooper Burton. FiftyPlusOne just caught a pollster faking data — and they copped to it. FiftyPlusOne, 2026-10-20.

3. Lucy Campbell. Pollster admits to fabricating data for ‘social experiment’ to expose flaws of industry. The Guardian, 2026-10-21.

4. Newsweek Editors. Fake Polling Firm Duped Media in ‘Social Experiment’. Newsweek, 2026-10-21.

5. Azi Paybarah. Pollsters weather storm of fake data, secret sponsorships, with some afraid to give names. Straight Arrow News, 2026-08-20.

6. Frank Corder. Poll pushed by Colom claiming Senate race with Hyde-Smith is “tied” was fake, fabricated. Magnolia Tribune, 2026-10-20.

7. Nate Silver, Eli McKown-Dawson. 2026 Midterm Election Forecast: Senate & House. Silver Bulletin, updated 2026-10-21.

8. The New York Times. President Trump Approval Rating: Latest Polls (About our polling averages). Publication date not stated.

9. American Association for Public Opinion Research. Transparency Initiative. Accessed 2026-10-22.

10. Research Methods · Rituals can be imitated; understanding cannot be faked

11. Compiled and published by Scholay, the all-in-one academic platform. If any content infringes your rights, please let us know and it will be taken down.

12. Also published on EnterScholar, a community for scholars in the age of AI

Charles Sanders Peirce (1839–1914), photographed in 1891. Image: Napoleon Sarony; original held by the New York Public Library; source: Wikimedia Commons; license: public domain.

In October 1884, Peirce and his student Jastrow reported a set of weight-perception experiments to the US National Academy of Sciences: the experimenter used the color of cards from a shuffled pack to decide whether the pressure would be “first increased, then decreased” or “first decreased, then increased,” while the subject made judgments from behind a screen. Even when the subjects believed they had no confidence at all, their judgments were still clearly better than guessing. The experiment is often regarded as one of the earliest to use an explicit randomization scheme, but what it randomized was the order of trials, not group assignment.

The Dispute over the Difference Threshold, and the Apparatus

From the mid-19th century onward, the psychophysics of G. T. Fechner assumed the existence of a “difference threshold” (Unterschiedsschwelle): when the difference between two stimuli falls below a certain proportion, people can no longer tell them apart. Peirce, who had long studied observational error at the US Coast Survey, held that the errors made in comparing two sensations arise from a great many tiny chance disturbances along the path of nerve conduction and should follow the error curve (the normal distribution); in that case, there would be no threshold. The two views yield different testable predictions. If a threshold exists, then for differences below it, right and wrong judgments should each make up half in the long run. If the theory of errors holds, then no matter how small the difference, the proportion of correct answers should be above half, and that proportion can be calculated in advance from the theory of errors.

According to Hacking's research, the experiments were carried out between 10 December 1883 and 7 April 1884. Jastrow, then Peirce's student at Johns Hopkins University, recalled in 1914 that the problem, the apparatus, the method and the way of arriving at the results were all supplied by Peirce, and that the apparatus was set up in his own room. Both men served as subjects, each making 150 judgments in a session lasting about two hours.

The apparatus was adapted from a Fairbanks postal scale. One end of the beam carried a small block fitted with a rubber cap, which pressed up from below against a finger of the subject's left hand; with a 1-kilogram weight on the scale pan, the finger bore a pressure roughly equal to one quarter of that weight. A separate small pan holding the “difference weight” could be lowered on or lifted off by a fine rubber band, increasing or decreasing the pressure without any sound or vibration. The difference weights were 60, 30 and 15 parts per thousand of 1 kilogram; later on, Jastrow switched to 20, 10 and 5 parts per thousand. In each trial, the subject had to judge whether the pressure had “increased then decreased” or “decreased then increased,” and was not allowed to answer “can't tell.” At the same time, the subject reported confidence on a scale from 0 to 3: 0 meant no inclination at all, and 3 meant the strongest assurance one could have about a sensation of this kind.

Ordering by Playing Cards and Concealment by a Screen

The order in each trial was determined by playing cards. Each set of 25 trials used a pack of 25 cards, 12 red and 13 black, or the reverse, so that for a given difference weight, the 50 trials of one session had exactly 25 red cards and 25 black. Before each set, the pack was shuffled thoroughly, and the experimenter used the colors of the cards, turned up one after another, to decide whether each trial would be increase-then-decrease or decrease-then-increase. After the subject answered, the experimenter laid the card face down or face up depending on whether the answer was right or wrong, and, according to the confidence level, set it so that the corner bearing 0 to 3 punched holes pointed in a fixed direction. This meant no separate notes were needed and the pace was not slowed; the results were recorded all together once a set was finished.

A postal scale made by the Fairbanks company, in the collection of the Smithsonian National Postal Museum (date of manufacture unverified). Peirce and Jastrow modified a postal scale of the same brand; this image is for reference only and is not the scale used in the experiment. Image: Smithsonian National Postal Museum; license: CC0 1.0.

The reason given in the paper was that this prevented the subject from making any psychological guess about which change the experimenter “might be likely to choose.” The authors also wrote about a side effect of the random sequence: long runs of the same kind of change, occurring by chance, would confuse the subject. They believed, however, that if long runs were deliberately excluded and the subject knew this, the resulting disturbance would be even greater. Hacking points out that the problem of long runs remained contested later, when Fisher promoted randomization, and that Peirce's use of two small packs with opposite numbers of red and black cards had in effect already limited the imbalance between red and black within a session.

As for concealment, a screen hid all of the experimenter's movements, and the paper states that they took ample precautions against “unconsciously received hints.” It should be noted that this blinded only the subject. The experimenter could see the card colors and knew exactly how to add or remove weight each time, so he was not blind. The role of randomization was to take “which order to choose” out of the experimenter's hands, so that even if he fell into some rhythm, consciously or unconsciously, it could not affect the sequence of stimuli. Hacking's summary of the experiment is exactly this: weights were presented to subjects who could not see the manipulation, and randomization was used to guard against unconscious suggestion.

Accuracy of Low-Confidence Judgments

The paper compared the number of errors in each set with the expected value calculated from the theory of errors, and concluded that, as far as could conveniently be traced, the error counts all conformed to the probability curve, thereby “destroying every presumption in favor of a difference threshold.” Even more striking is the confidence record in footnote 4 of the original:

The last row is the most telling: when the difference weight was only 5 to 20 parts per thousand, Jastrow marked more than nine in ten of his judgments as “no inclination at all,” yet about two-thirds of those judgments were still correct. On this basis, the authors argued that confidence reflects a “secondary sensation” of the difference between the two sensations. This secondary sensation likewise has no threshold, but subjects often overlook it, so much so that confidence was still recorded as 0 even when the rate of correct answers reached three-fifths. They also found that average confidence roughly followed m = c log(p/(1−p)), where p is the probability of a correct answer. As cited by Hacking, Stigler considered this possibly the earliest attempt to elicit subjective probability experimentally.

The final paragraph of the paper pushed the conclusion further: people may gain knowledge of other people's minds through faint sensations of which they are not conscious, and “women's insight” and certain “telepathic” phenomena might perhaps be explained in this way. This paragraph is the authors' speculation; the experiment itself did not test it.

Relation to Fisher's Randomization, and Limitations

Calling it “one of the earliest randomized experiments” rests mainly on research in the history of science. As cited by Hacking, Stigler wrote in 1978 that this was the first experiment he knew of to be conducted according to a “precise, mathematically sound randomization scheme.” Hacking sums up its design as three innovations: deliberate randomization, not allowing subjects to answer “can't tell,” and confidence ratings from 0 to 3. By way of contrast, in Fechner's earlier weight-lifting experiments, Fechner himself served as the subject and knew in advance which box was heavier. Hacking also notes that the French physiologist Charles Richet independently used random card drawing in telepathy tests at about the same time, so “earliest” should be understood as among the earliest known.

The practice was not handed down all the way to Fisher. Hacking found that the later psychophysics literature almost never mentioned randomization again. Fullerton and Cattell's 1892 study says only that trials were conducted in an “irregular order”; when Jastrow later gave a systematic account of the method, he did not mention randomization either; and the debate over Peirce's method centered on “not allowing ‘can't tell’ answers.” Fisher first publicly argued for the central role of randomization in 1926 in “The Arrangement of Field Experiments,” and his 1935 The Design of Experiments made it textbook material. His reasons belonged to his own system of statistical inference: randomization gives error estimation and significance testing an objective basis and can improve the efficiency of experiments. Hacking states explicitly that he does not believe early psychical research had any important “influence” on Fisher; he points out only that the concern about chance long runs in the “lady tasting tea” example of The Design of Experiments had already been raised by Peirce.

Judged by the standards of modern design, the experiment has several limitations. First, it randomized the order of trials within the same subject; it did not randomly assign subjects to treatment and control groups. Dehue 1997 argues that the random group design took shape gradually in educational psychology during the 20s of the 20th century, and that McCall appears to have been the first to propose forming groups at random. Second, there were only two subjects, who were the authors themselves, and the experimenter was not blind. Third, the balanced packs themselves left a clue: a footnote in the paper admits that Jastrow's observational errors in the last two stages were abnormally small, partly because he unconsciously kept track of how many times each of the two changes had appeared in a set and answered accordingly. Fourth, practice effects made Jastrow's errors fall steadily, so the authors could only divide his data into four stages and compare them separately; and in the early experiments of December 1883 and January 1884, the procedure had not yet been perfected. Finally, the paper describes its data analysis in less detail than its data collection. Its inference relied mainly on comparing error counts with the theoretical values of the error curve, and the role of randomization in inference was not explicitly argued in the way Fisher would later argue it.

References

1. Peirce, C. S., & Jastrow, J. (1885). On Small Differences of Sensation. Memoirs of the National Academy of Sciences, 3(1), 73–83 (read 17 October 1884). PhilArchive scan; Full text at Classics in the History of Psychology

2. Hacking, I. (1988). Telepathy: Origins of Randomization in Experimental Design. Isis, 79(3), 427–451. doi:10.1086/354775

3. Stigler, S. M. (1978). Mathematical Statistics in the Early States. The Annals of Statistics, 6(2), 239–265. doi:10.1214/aos/1176344123

4. Dehue, T. (1997). Deception, Efficiency, and Random Groups: Psychology and the Gradual Origination of the Random Group Design. Isis, 88(4), 653–673. doi:10.1086/383850

5. The Collected Papers of Charles Sanders Peirce, Vol. 7, editors' notes (including Jastrow's 1914 recollection). Electronic edition

6. Research Methods · Rituals can be imitated; understanding cannot be faked

7. Compiled and published by Scholay, the one-stop academic platform. If any content infringes your rights, please let us know and it will be taken down.

8. Also published on EnterScholar, a community for scholars in the age of intelligence