CIA-RDP96-00789R002200610001-0

Journal of Parapsychology, Vol. 53, December 1989
.
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
"FUTURE TELLING": A META-ANALYSIS OF
FORCED-CHOICE PRECOGNITION
EXPERIMENTS, 1935-1987
BY CHARLES HONORTON AND DIANE C. FERRARI
ABSTRACT: We report a meta-analysis of forced-choice precognition experiments
published in the English-language parapsychological literature between 1935 and
1987. These studies involve attempts by subjects to predict the identity of target
stimuli selected randomly over intervals ranging from several hundred milli-
seconds to one year following the subjects' responses. We retrieved 309 studies
reported by 62 investigators. Nearly two million individual trials were contributed
by more than 50,000 subjects. Study outcomes are assessed by overall level of sta-
tistical significance and effect size. There is a small, but reliable overall effect (z
= 11.41, p = 6.3 x 10-25). Thirty percent of the studies (by 40 investigators) are
significant at the 5% significance level. Assessment of vulnerability to selective re-
porting indicates that a ratio of 46 unreported studies averaging null results would
be required for each reported study in order to reduce the overall result to nonsig-
nificance. No systematic relationship was found between study outcomes and eight
indices of research quality. Effect size has remained essentially constant over the
survey period, whereas research quality has improved substantially. Four moder-
ating variables appear to covary significantly with study outcome: Studies using
subjects selected on the basis of prior testing performance show significantly larger
effects than studies using unselected subjects. Subjects tested individually by an
experimenter show significantly larger effects than those tested in groups. Studies
in which subjects are given trial-by-trial or run-score feedback have significantly
larger effects than those with delayed or no subject feedback. Studies with brief
intervals between subjects' responses and target generation show significantly
stronger effects than studies involving longer intervals. The combined impact of
these moderating variables appears to be very strong. Independently significant
outcomes are observed in seven of the eight studies using selected subjects, who
were tested individually and received trial-by-trial feedback.
Precognition refers to the noninferential prediction of future
events. Anecdotal claims of "future telling" have occurred through-
out human history in virtually every culture and period. Today such
This work was funded by SRI International and the John E. Fetzer Foundation.
We wish to thank our PRL colleague George P. Hansen, who is primarily responsible
for retrieving the studies used in the meta-analysis. We are grateful to Edwin C. May,
Jessica Outs, and to five anonymous reviewers at SRI for valuable comments on an
carlier draft of this report. Valuable comments were also made by Ephraim Schechter
and by three anonymous referees. The division of authorship responsibility is as fol-
lows: Honorton is responsible for the design of the meta-analysis, definition of study
coding criteria, the actual analyses, and the report itself. Ferrari coded the individual
research reports in consultation with Honorton and/or Hansen.
Copied
6/30
250
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

· 282
The Journal of Parapsychology
claims are generally believed to be based on factors such as delusion,
igationality, and superstitious thinking. The concept of precogni-
tidn runs counter to accepted notions of causality and appears to
Chffict with current scientific theory. Nevertheless, over the past
lalf-century a substantial number of experiments have been re-
parted claiming empirical support for the hypothesis of precogni-
t@n. Subjects in forced-choice experiments, according to many re-
Arts, have correctly predicted to a statistically significant degree the
Entity (or order) of target stimuli randomly selected at a later
"We performed a meta-analysis of forced-choice precognition ex-
@Oriments published in the English-language research literature be-
Been 1935 and 1987. Four major questions were addressed
rough this meta-analysis: (1) Is there overall evidence for accurate
Øget identification (above-chance hitting) in experimental precog-
@tion studies? (2) What is the magnitude of the overall precognition
Dect? (3) Is the observed effect related to variations in methodo-
Magical quality that could allow a more conventional explanation? (4)
Des precognition performance vary systematically with potential
Moderating variables, such as differences in subject populations,
stimulus conditions, experimental setting, knowledge of results, and
The interval between subject response and target generation?
DELINEATING THE DOMAIN
Strieval of Studies
Approved For Release 2000/08/08 -: CIA RDP96-00789R002200610001-0
Parapsychological research is still academically taboo, and it is
likely that there have been many dissertations and theses in this
Rea that have escaped publication. Our retrieval of studies for this
feta-analysis is therefore based on the published literature. The
Oldies include all forced-choice precognition experiments appear-
Ifig in the peer-reviewed English-language parapsychology journals:
Journal of Parapsychology, Journal (and Proceedings) of the Society for
Bychical Research, Journal of the American Society for Psychical Research,
European Journal of Parapsychology (including the Research Letter of
Utrecht University Parapsychology Laboratory), and abstracts of
Cer-reviewed papers presented at Parapsychological Association
meetings published in Research in Parapsychology.
Criteria for Inclusion
Our review is restricted to fixed-length studies in which signifi-
cance levels and effect sizes based on direct hitting can be calcu-
283
A Meta-Analysis of Forced-Choice Precognition Experiments
lated. Studies using outcome variables other than direct hitting, such
as run-score variance and displacement effects, are included only if
the report provides relevant information on direct hits (i.e., numBr
of trials, hits, and probability of a hit). Finally, we exclude studies
conducted by two investigators, S. G. Soal and Walter J. Levy, whoe
work has been unreliable.
Many published reports contain more than one experiment1
experimental unit. In experiments involving multiple conditio
significance levels and effect sizes are calculated for each condition.
Outcome Measures
Significance level. Significance levels (z scores) were calculated for
each study from the reported number of trials, hits, and probabaty
of success using the normal approximation to the binomial disgi-
bution with continuity correction. Positive z scores indicate above-
chance scoring, and negative z scores reflect below-chance scorin.
Effect size. Because most parapsychological experiments, partic-
ularly those in the older literature, have used the trial rather than
the subject as the sampling unit, we use a trial-based estimatoff
effect size. The effect size (ES) for each study is the z score divided
by the square root of the number of trials in the study.'
General Characteristics of the Domain
We located 309 studies in 113 separate publications. These send-
ies were contributed by 62 different senior authors and were p@b-
lished over a 53-year period, between 1935 and 1987. Considergg
the half-century time-span over which the precognition experim@ts
were conducted, it is not surprising that the studies are very diveße.
The database comprises nearly two million individual trials and .
more than 50,000 subjects. Study sample sizes range from 25Oto
297,060 trials (median = 1,194). The number of subjects ranges
from 1 to 29,706 (median = 16). The studies use a variety of mesh-
odologies, ranging from guessing ESP cards and other card symbols
to automated random number generator experiments. The domain
encompasses diverse subject populations: the most frequently used
' Elsewhere (Honorton, 1985), we have used the effect size index Cohen's h
(Cohen, 1977), and one referee has asked that we explain why we are now using
z/N1/2. The answer is that h and z/N12 yield virtually identical results, and z/N1/2 is.
computationally simpler. For the present sample of 309 precognition studies, the
mean difference between the two indices is .00047, and the standard deviation of the
difference is .026: {(308) = 0.312, p = . 756, two-tailed. The correlation between the
two indices is .97.
96-00789R0022006100010
¿ Approved For Release 2000/08/08 ℃

284
The Journal of Parapsychology
A Meta-Analysis of Forced-Choice Precognition Experiments
285
TABLE 1
OVERALL. SIGNIFICANCE LEVEL AND EFFECT SIZE
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
2
ES
Mean
0.65
0.020
SD
2.68
0.100
Lower 95% confidence estimate
0.40
0.011
Combined z = 11.41, p = 6.3 × 10 -25
"Fail-safe N" = 14,268
((ES) = 3.51, 308 df, p = . 00025
population is students (in approximately 40% of the studies); the
least frequently used populations are the experimenters themselves
and animals (each used in about 5% of the studies).
Though a few studies tested subjects through the mail, more typ-
ically subjects were tested in person, either individually or in groups.
'Target selection methods included no randomization at all (studies
using "quasi-random" naturalistic events), informal methods includ-
ing manual card-shuffling or dice-throwing, and formal methods,
primarily random number tables or random number generators.
The time interval between the subjects' responses and target gen-
eration varied from less than one second to one year.
OVERALL CUMULATION
Evidence for an overall effect is strong. As shown in the top part
of Table 1, the overall results are highly significant.2 Lower bound
(one-tailed) 95% confidence estimates of the mean z score and ES
are displayed in the bottom portion of Table 1.
Ninety-two studies (30%) show significant hitting at the 5% level,
and significant outcomes are contributed by 40 different investiga-
tors. The z scores correlate significantly with sample size: r(307) =
.156, p = . 003. The mean number of trials for significant studies is
34% larger than the mean number of trials for nonsignificant stud-
ies.
2 The statistical analyses presented here were performed using SYSTAT (Wilk-
inson, 1988). When t tests are reported on samples with unequal variances, they are
calculated using the separate variances within groups for the error and degrees of
freedom following Brownlee (1965). Unless otherwise specified, p levels are one-
tailed. Combined z's are based on Stouffer's method (Rosenthal, 1984).
0
0
O
Investigator
0
C
0
0
O
O
0
O
0
01
-0.2
-0.1
-0.0
0.1
0.2
0.3
0.4
0.5
Mean effect size
Figure 1. Mean effect size by investigator. N = 62 investigators.
Replication Across Investigators
Virtually the same picture emerges when the cumulation is b
investigator rather than study as the unit of analysis; the combine
z is 12.13, and 23 of the 62 investigators (37%) have overall our
comes significant at the 5% level. The mean (investigator) effect size
is 0.033 (SD = . 093).
There is a significant difference in the mean ES across investi
gators, but it is surprisingly small: Kruskal-Wallis one-way ANOVA
by ranks, x2(61) = 82.71, p = . 034. The effect is clearly not due
a few major contributors. If investigators contributing more than
three studies are eliminated, leaving 33 investigators, the combine
z is still 6.00 (p = 1.25 × 10-9) and the mean ES is .028 (SD
.091). Figure 1 shows the mean effect sizes by investigator.
These results indicate substantial cross-investigator replicability
and directly contradict the claim of critics such as Akers (1987) that
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

:286
The Journal of Parapsychology
287
successful parapsychological outcomes are achieved by only a few
investigators.
The Filedrawer Problem
A well-known reporting bias exists throughout the behavioral
Dences favoring publication of "significant" studies (e.g., Sterling,
(59). The extreme view of this "filedrawer problem" is that "the
Wirnals are filled with the 5% of the studies that show Type I er-
Ors, while the filedrawers back at the lab are filled with the 95% of
e studies that show nonsignificance ... " (Rosenthal, 1984, p. 108).
recognizing the importance of this problem, the Parapsychological
Ssociation in 1975 adopted an official policy against selective re-
Brting of positive results.3 Examination of the parapsychological lit-
Hature shows that nonsignificant results are frequently published,
Qd, in the precognition database, 70% of the studies have reported
Bonsignificant results. Nevertheless, 75% of the precognition studies
were published before 1975, and we must ask to what extent selec-
de publication bias could account for the cumulative effects we ob-
serve.
The central section of Table I uses Rosenthal's (1984) "fail-safe
statistic to estimate the number of unreported studies with z
Cores averaging zero that would be necessary to reduce the known
Stabase to nonsignificance. The filedrawer estimate indicates that
Per 46 unreported studies must exist for each reported study to
aduce the cumulative outcome to a nonsignificant level.
Approved For Release 2000/08/08 : CFA RDP9600789R002200610001-05
A different approach to the filedrawer problem is described by
Rtwes, Landman, and Williams (1984; personal communication
Com Dawes to Honorton, July 14, 1988). Their truncated normal
parve analysis, like Rosenthal's "fail-safe N," is based on normal
urve assumptions. Their null hypothesis is that z scores above some
itical level (e.g., z = 1.65, 1.96, etc.) are randomly sampled from
4/(0,1) above that critical level. The alternative to the null hypothesis
D that, because there is some real effect, the distribution of z's is
bifted to the right of 0 and the z's will be larger than predicted by
Be null. For a critical level of z = 1.65, the expected mean z is 2.06
Ped the variance is .14. In the precognition database, there are 92
studies with z's > 1.65. Their average is 3.61, not 2.06 as predicted
A Meta-Analysis of Forced-Choice Precognition Experiments
by the null hypothesis. Since the variance of the normal truncated
above 1.65 is .14, the test z (using the Central Limit Theorem) com-
paring 3.61 to 2.06 is 39.84 [1.55 divided by (.14/92)1/2]. Here, p is
virtually zero. Similar results are found with cut points of 1.96, 2.33,
and 2.58.
On the basis of these analyses, we conclude that the cumulative
significance of the precognition studies cannot satisfactorily be ex-
plained by selective reporting.
OUTLIER REDUCTION
Although the overall z scores and effect sizes cannot reasonably
be attributed to chance, inspection of the standard deviations in
Table 1 indicates that the study outcomes are extremely heteroge-
neous. Given the diversity of methods, subject populations, and
other study features that characterize this research domain, this is
not surprising.
The study outcomes are in fact extremely heterogeneous. Al -;
though a major objective of this meta-analysis is to account for the
variability across studies by blocking on differences in study quality,
procedural features, and sampling characteristics, the databasec
clearly contains extreme outliers. The z scores range from - 5.1 too
19.6, a 25-sigma spread! The standardized index of kurtosis (g2)
is
9.47, suggesting that the tails of the distribution are much too long
for a normal distribution.
We eliminated the extreme outliers by performing a "10 percent@
trim" on the study z scores (Barnett & Lewis, 1978). This involvesa
eliminating studies with z scores in the upper and lower 10% of the
distribution, and results in an adjusted sample of 248 studies. The
trimmed z scores range from - 2.24 to 3.21 (g) = - 1.1). The re-
vised z scores and effect sizes are presented in Table 2.
Elimination of extreme outliers reduces the combined z scores by
approximately one half, but the outcomes remain highly significant
Twenty-five percent of the studies (62/248) show overall significant
hitting at the 5% level. Lower bound confidence estimates show that
the mean z's and effect sizes are above 0 at the 95% confidence level
Elimination of outliers reduces the total number of investigators
from 62 to 57, but the results remain basically the same when the
analyses are based on investigators rather than studies. The com-
bined z is 6.84; 18 of the 57 investigators (31.6%) have overall. sig-
" Analyses indicate no significant difference in the magnitude of reported study
outcomes before and after 1975. The mean ES for studies prior to 1975 is 0,021 (SD)
.099), and for studies reported thereafter the mean is 0.017 (SD) = . 106): ((307)
= 0.28, p = . 782, two-tailed.
"Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

.288
The Journal of Parapsychology
A Meta-Analysis of Forced-Choice Precognition Experiments
289
TABLE 2
SIGNIFICANCE LEVEL AND EFFECT SIZE FOR TRIMMED SAMPLE
2
ES
Pean
0.38
0.012
1.45
0.065
wer 95% confidence estimate
0.23
0.005
Combined z = 6.02, p = 1.1 × 10-9
((ES) = 2.90, 247 df, p = . 002
Picant outcomes at the 5% level. The mean (investigator) ES is
20 (SD = . 05).
? For the trimmed sample, the difference in ES across investiga-
's is not significant: Kruskal-Wallis one-way ANOVA by ranks,
(56) = 59.34, p = . 355. If investigators contributing more than
Free studies are eliminated, leaving 37 investigators, the combined
Tis still 5.00 (p = 3.0 x 10-7) and the mean ES is 0.022 (SD =
156). Figure 2 shows the mean effect size by investigator.
Thus, elimination of the outliers does not substantially affect the
inclusions drawn from our analysis of the database as a whole.
mere clearly is a nonchance effect. In the remainder of this report,
use the trimmed sample to examine covariations in effect size
d a variety of methodological and other study features.
STUDY QUALITY
Because target stimuli in precognition experiments are selected
y after the subjects' responses have been registered, precognition
adies are usually not vulnerable to sensory leakage problems.
Other potential threats to validity must, however, be considered.
Re problem of variations in research quality remains a source of
controversy in meta-analysis. Some meta-analysts advocate eliminat-
ing low quality studies whereas others recommend empirically ac-
cosing the impact of variations in quality on study outcome. Rosen-
dal (1984) points out that the practice of discarding studies is
equivalent to assigning them weights of zero, and he recommends
weighting study z scores in relation to ratings of research quality.
Study Quality Criteria
Ideally, the assessment of study quality should be performed by
knowledgeable specialists who are blind to the study outcomes. In
0
0
O
O
0
0
Investigator
0
0
0
O
0
0
O
-0.20
-0.15
-0.10
-0.05
0.00
0.05
0.10
0.15
0.20
Mean effect size
Figure 2. Mean effect size by investigator for trimmed sample. N = 57 if
vestigators.
practice, this is usually not feasible, particularly when, as in the pres
ent case, large numbers of studies are involved. For our analysis
study quality, statistical and methodological variables are defined
and coded in terms of procedural descriptions (or their absence)
the research reports. This approach was used in an earlier meta
analysis of psi ganzfeld research (Honorton, 1985), and it led K
study quality ratings that were generally in agreement, r(26) = . 766
p = 10-6, with independent "flaw" ratings by an outside critic (H)
man, 1985).
One point is given (or withheld) for each of the following eigh
criteria:
Specification of sample size. Does the investigator preplan the num-
ber of trials to be included in the study or is the study vulnerable
to the possibility of optional stopping? Credit is given to reports that
explicitly specify the sample size. Studies involving group testing, in
which it is not feasible to specify the sample size precisely, are also
FORRelease 2000/08/08: CIA-RDP96-00789R002200610001-0
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

290'
The Journal of Parapsychology
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
given credit. No credit is given to studies in which the sample size
is either not preplanned or not addressed in the experimental re-
port.
Preplanned analysis. Is the method of statistical analysis, including
the outcome (dependent variable) measure, preplanned? Credit is
given to studies explicitly specifying the form of analysis and the
outcome measure. No credit is given to those not explicitly stating
the form of the analysis or those in which the analysis is clearly post
hoc.
Randomization method. Credit is given for use of random number
tables, random number generators, and mechanical shufflers. No
credit is given for failure to randomize (i.e., use of "quasi-random
naturalistic events") or for informal methods such as hand-shuffling,
die-casting, and drawing lots.
Controls. Credit is given to studies reporting randomness control
checks, such as random number generator (RNG) control series and
empirical cross-check controls.
Recording. One point is allotted for automated recording of tar-
gets and responses, and another for duplicate recording.
Checking. One point is allotted for automated checking of
matches between target and response, and another for duplicate
checking of hits.
Study Quality Analysis
Each study received a quality weight between 0 and 8 (mean =
3.3, SD = 1.8). We find no significant relationship between study
quality and ES: r(246) = . 08], p = . 202, two-tailed. This tendency
for study outcomes to correlate positively with study quality has the
consequence that the quality-weighted z score of 6.26 is slightly
larger than the unweighted z of 6.02. Table 3 shows the correlations
between effect size and each of the eight individual quality meas-
ures." The mean effect sizes by quality level are displayed graphi-
cally in Figure 3.
4 The correlation between ES and study quality is also nonsignificant for the un-
trimmed sample of 309 studies: r(307) = -. 060, p = . 289. The quality-weighted z
score is 7.38: p = 2.32 x 10-13. However, three of the individual quality measures
are significantly related to performance. Controls and duplicate checking correlate
significantly positively with ES, and randomization correlates significantly negatively
with ES. These correlations appear to be due to a few studies with z scores that are
extreme outliers (z > 7). When the 10 studies with z > 7 are eliminated, the signifi-
cant correlations between quality and ES disappear.
A Meta-Analysis of Forced-Choice Precognition Experiments
291
TABLE 3
CORRELATIONS BETWEEN EFFECT SIZE AND QUALITY MEASURES
Quality measure
r(246)
Sample size specified in advance
- . 100
Preplanned analysis
.001
Randomization
.011
Controls
.058
Automated recording
.169
Duplicate recording
.047
Automated checking
.136
Duplicate checking
.078
Quality Extremes
Is there a tendency for extremely weak studies to show larger
effects than exceptionally "good" studies? Analysis on the extremes
of the quality ratings indicates that this is not the case.
This analysis, based on the untrimmed sample of 309 studies,
uses studies with quality ratings outside the interquartile range of
the rating distribution (median = 4, Q1 = 2, Q3 = 5). There are 56
"low-quality" studies (ratings of 0-1) and 35 "high-quality" studies
(ratings of 6-8). The high-quality studies have effect sizes that are
not significantly lower than the low-quality studies; the ES means
are 0.017 (SD = 0.063) and 0.037 (SD = 0.137), for the low- and.
high-quality studies, respectively: t(82) = -. 92, p = . 358, two-
tailed.
Quality Variation in Publication Sources
Precognition ES is not significantly related to source of publica-
tion: Kruskal-Wallis one-way ANOVA, x2(4). = 0.78, p = . 942.
However, the sources of publication differ significantly in study
quality: Kruskal-Wallis one-way ANOVA, x2(4) = 17.19, p = . 002.
This is due largely to the lower quality of studies published in the
Journal of the Society for Psychical Research and in Research in Parapsy-
chology.
Study Quality in Relation to Year of Publication
Precognition effect size has remained constant over a half-cen-
tury of research, even though the methodological quality of the re-
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

292
The Journal of Parapsychology
293
8
-
5
7
-
13
6
1 7
5
-
45
4
- 63
3
1 35
2
1 41
1
1 31
0
8
-0.10
-0.05
0.00
0.05
0.10
Mean effect size
FigDe 3. Precognition effect size in relation to study quality, with 95% con-
fidence limits. N = 248 studies.
seatoh has improved significantly during this period. The correla-
tionbetween ES and year of publication is -. 071: t(307) = - 1.25,
p =0.213, two-tailed. Study quality and year of publication are, how-
ev
, positively and significantly correlated: r(246) = . 282, p = 2 ×
10 6 two-tailed.
"Critics of parapsychology have long believed that evidence for
parapsychological effects disappears as the methodological rigor in-
creases. The precognition database does not support this belief.
"REAL-TIME" ALTERNATIVES TO PRECOGNITION
Investigators have long been aware of the possibility that precog-
nition effects could be modeled without assuming either time rever-
sal or backward causality. For example, outcomes from studies with
A Meta-Analysis of Forced-Choice Precognition Experiments
targets based on indeterminate random number generators (RNGs)
could be due to a causal influence on the RNG-a psychokinetic
(PK) effect-rather than information acquisition concerning its fu-
ture state. In experiments with targets based on prepared tables of
random numbers, the possibility exists that the experimenter or
other randomizer may be the actual psi source, unconsciously using
"real-time" ESP combined with PK to choose an entry point in the
random number sequence that will significantly match the "sub-
ject's" responses. While the latter possibility may seem far-fetched,
it cannot be logically eliminated if one accepts the existing evidence
for contemporaneous ESP and PK, and it has been argued that it is
less far-fetched than the alternative of "true" precognition.
Morris (1982) discusses models of experimental precognition
based on "real-time" psi alternatives and methods for testing "true"
precognition. In general terms, these methods constrain the selec-
tion of the target sequence so as to eliminate nonprecognitive psi
intervention. In the most common procedure, attributed to Mangan
(1955), dice are thrown to generate a set of numbers that are math-
ematically manipulated to obtain an entry point in the random num-
ber table. This procedure is sufficiently complex "as to be appar-
ently beyond the capacities of the human brain, thus ruling out PK
because the 'PKer' would not know what to do even via ESP" (Mor-
ris, 1982, p. 329).
Two features of precognition study target determination proce-
dures were coded to assess "real-time" psi alternatives to precogni-
tion: method of determining random number table entry point and
use of Mangan's method.
·
Methods of eliminating "real-time" psi alternatives have not been
used in studies with random number generators and have only been
used in a small number of studies involving randomization by hand-
shuffling. These analyses are therefore restricted to studies using
random number tables (N = 138).
Method of Determining RNT Entry Point
The reports describe six different methods of obtaining entry.
points in random number tables. If the study outcomes were due to
subjects' precognitive functioning rather than to alternative psi '
modes on the part of the experimenter or the experimenter's as-
sistants, there should be no difference in mean effect size across the
various methods used to determine the entry point. Indeed, our
analysis indicates that the study effect sizes do not vary systemati-
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
.
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

294
The Journal of Parapsychology
A Meta-Analysis of Forced-Choice Precognition Experiments
295
cally as a function of method of determining the entry point: Krus-
kal-Wallis one-way ANOVA by ranks: x2(5) = 7.32, p = . 198.
Use of Mangan's Method
We find no significant difference in ES between studies using
complex calculations of the type introduced by Mangan to fix the
random number table entry point and those that do not use such
calculations: ((45) = 0.38, p = . 370, two-tailed.
MODERATING VARIABLES
The stability of precognition study outcomes over a 50-year pe-
riod, which we described earlier, is also bad news. It shows that in-
vestigators in this area have yet to develop sufficient understanding
of the conditions underlying the occurrence (or detection) of these
effects to reliably increase their magnitude. We have identified four
variables that appear to covary systematically with precognition ES:
(1) selected versus unselected subjects, (2) individual versus group
testing, (3) feedback level, and (4) time interval between subject re-
sponse and target generation.
The analyses use the raw study z scores and effect sizes; we
found that this results in uniformly more conservative estimates of
relationships with moderating variables than when the analyses are
based on quality-weighted z scores and effect sizes.
Selected Versus Unselected Subjects
Our meta-analysis identifies eight subject populations: unspeci-
fied subject populations, mixtures of several different populations,
animals, students, children, "volunteers," experimenter(s), and se-
lected subjects.
Effect size magnitude does not vary significantly across these
eight subject populations: Kruskal-Wallis one-way ANOVA, x2(7) =
10.90, p = . 143. Effect sizes by subject population are displayed in
Figure 4.
However, studies using subjects selected on the basis of prior
performance in experiments or pilot tests show significantly larger
effects than studies using unselected subjects. As shown in Table 4,
60% of the studies with selected subjects are significant at the 5%
level. The mean z score for these studies is 1.39 (SD = 1.40). The
ES is significantly higher for selected-subjects studies than for stud-
Selected
25
Exptr
12
Volunteer
26
Subject population
Children
31
Students
-
108
Animals
10
Mixed
28
Unspec
8
1
-0.06
-0.04
-0.02
0.00
0.02
0.04
0.06
0.08
0.10
Mean effect size
Figure 4. Precognition effect size by subject population, with 95% confi-
dence limits. N = 248 studies.
ies with unselected subjects. The t test of the difference in mean ES
is equivalent to a point-biserial correlation of .198.
Does this difference result from less stringent controls in studies
with selected subjects? The answer appears to be "No." The average
quality of studies with selected subjects is higher than studies using
TABLE 4
SELECTED VERSUS UNSELECTED SUBJECTS
Selected
Unselected
N studies
25
223
Combined z
6.89
4.04
Studies with p. < . 05
60%
21%
Mean ES
.051
.008
SDES
.075
.063
((246) = 3.16, p = . 001
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

296
The Journal of Parapsychology
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
TABLE 5
INDIVIDUAL. VERSUS GROUP TESTING
Individual
Group
N studies
97
105
Combined z
6.64
1.29
Studies with p < . 05
30%
19%
Mean ES
.021
.004
SDES
.060
.066
t (200) = 1.89, p = . 03
unselected subjects: (27) = 1.51, p = . 142, two-tailed. This result
appears to reflect a general tendency toward increased rigor and
more detailed reporting in studies with selected subjects.
Individual Versus Group Testing
Subjects were tested in groups, individually, or through the mail.
Studies in which subjects were tested individually by an experimen-
ter have a significantly larger mean ES than studies involving group
testing (Table 5).
The t test of the difference is equivalent to a point-biserial cor-
relation of .132, favoring individual testing. Of the studies with sub-
jects tested individually, 30% are significant at the 5% level.
The methodological quality of studies with subjects tested indi-
vidually is significantly higher than that of studies involving group
testing: t(137) = 3.08, p = . 003, two-tailed. This result is consistent
with the conjecture that group experiments are frequently con-
ducted as "targets of opportunity" and may often be carried out
hastily in an afternoon without the preparation and planning that
go into a study with individual subjects that may be conducted over
a period of weeks or months.
Thirty-five studies were conducted through the mail. In these
studies, subjects completed the task at their leisure and mailed their
responses to the investigator. These correspondence studies yield
outcomes similar to those involving individual testing. The com-
bined z score is 2.66, with a mean ES of 0.018 (SD = . 082). Ten
correspondence studies (25.7%) are significant at the 5% level.
Eleven studies are unclassifiable with regard to experimental set-
ting.
A Meta-Analysis of Forced-Choice Precognition Experiments
297
TABLE 6
FEEDBACK RECEIVED BY SUBJECTS
Feedback of Results
None
Delayed
Run score Trial-by-trial
N studies
15
21
21
47
Combined z
- 1.30
2.11
4.74
6.98
Studies with p < . 05
0.0%
19.0%
33.3%
42.6%
Mean ES
-. 001
.009
.023
.035
SDES
.028
.036
.048
.072
Feedback
A significant positive relationship exists between the degree of
feedback subjects receive about their performance and precognitive
effect size (Table 6).
Subject feedback information is available for 104 studies. These
studies fall into four feedback categories: no feedback, delayed
feedback (usually notification by mail), run-score feedback, and
trial-by-trial feedback. We gave these categories numerical values
between 0 and 3. Precognition effect size correlates .231 with feed-
back level (102 df, p = . 009). Of the 47 studies involving trial-by-
trial feedback, 20 (42.6%) are significant at the 5% level. None of
the studies without subject feedback are significant.
Feedback level correlates positively though not significantly with
research quality: r(102) = . 173, p = . 082, two-tailed. Inadequate
randomization is the most plausible source of potential artifacts in
studies with trial-by-trial feedback. We performed a separate analy-
sis on the 47 studies in this group. Studies using formal methods of
randomization do not differ significantly in mean ES from those
with informal randomization: t(15) = 0.67, p = . 590, two-tailed.
Similarly, studies reporting randomness control data do not differ
significantly in ES from those not including randomness controls:
t(42) = 0.79, p = . 436, two-tailed.
Time Interval
The interval between the subject's response and target selection
ranges from less than one second to one year. Information about
the time interval is available for 144 studies. This information, how-
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

298
The Journal of Parapsychology
.
Months
7
Weeks
17
Precognition interval
Days
15
Hours
35
Minutes
19
Seconds
20
Millisec
-
31
-0.06
-0.04
- 0.02
0.00
0.02
0.0-1
0.0G
0,08
0.10
Mean effect size
Figure 5. Effect size by precognition interval, with 95% confidence limits.
N = 144 studies.
ever, is often imprecise. Our analysis of the relationship between
precognitive ES and time interval is therefore limited to seven broad
interval categories: milliseconds, seconds, minutes, hours, days,
weeks, and months. (Effect sizes by precognition interval are dis-
played in Figure 5.)
Although it is confounded with degree of feedback, there is a
significant decline in precognition ES over increasing temporal dis-
tance: r(142) = -. 199, p = . 017, two-tailed. The largest effects oc-
cur over the millisecond interval: N = 31 studies, combined z =
6.03, mean ES = 0.045, SD = . 073. The smallest effects occur over
periods ranging from a month to a year: N = 7, combined z =
0.53, mean ES = 0.001, SD = . 049.
Interestingly, the decline of precognition performance over in-
creasing temporal distances results entirely from studies using; un-
A Meta-Analysis of Forced-Choice Precognition Experiments
selected subjects: r(122) = -. 235, p = . 009, two-tailed. Studies with
selected subjects show a nonsignificant positive relationship between
ES and time interval: r(18) = . 077, p = . 745, two-tailed. Although
the difference between these two correlations is not significant (z =
1.24), this suggests that the origin of the decline over time may be
motivational rather than the result of some intrinsic physical bound-
ary condition. The relationship between precognition ES and feed-
back also supports this conjecture. Nevertheless, any finding sug-
gesting potential boundary conditions on the phenomenon should
be vigorously pursued.
Influence of Moderating Variables in Combination
The above analyses examine the impact of each moderating var-
iable in isolation. In this final set of analyses, we explore their joint
influence on precognition performance. For this purpose, we iden-
tify two subgroups of studies. One subgroup is characterized by the
use of selected subjects tested individually with trial-by-trial feed-
back. We refer to this as the Optimal group (N = 8 studies). The
second group is characterized by the use of unselected subjects
tested in groups with no feedback. We refer to this as the Suboptimal
group (N = 9 studies).
The Optimal studies are contributed by four independent inves-
tigators and the Suboptimal studies are contributed by two of the
same four investigators. All of the Optimal studies involve short pre-
cognition time intervals (millisecond interval); the Suboptimal stud-
ies involve longer intervals (intervals of weeks or months). All of the
Optimal studies and 5 of the 9 Suboptimal studies use RNG meth-
odology. The two groups do not differ significantly in average sam-
ple size. The mean study quality for the Optimal group is signifi-
cantly higher than that of the Suboptimal studies: Optimal mean =
6.63, SD = 0.92; Suboptimal mean = 3.44, SD = 0.53; t(10) =
8.63, p = 3.3 × 10-6, two-tailed.
The combined impact of the moderating variables appears to be
quite strong (Table 7). Seven of the 8 Optimal studies (87.5%) are
independently significant at the 5% level, whereas none of the Sub-
optimal studies are statistically significant. All four investigators con-
tributing studies to the Optimal group have significant outcomes.5
" In the untrimmed sample of 309 studies, there are a total of 17 Optimal studies.
The mean ES is 0.117 (SD = . 154), and the combined z is 15.84. The percentage of
independently significant studies is virtually the same as it is in the trimmed sample:
15 of the 17 studies (88.2%) are significant.
299
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

300
The Journal of Parapsychology
301
TABLE 7
IMPACT OF MODERATORS IN COMBINATION
"Optimal" studies
"Suboptimal" studies
N studies
8
9
Combined z
6.14
1.29
Studies with p < . 05
87.5%
0.0%
Mean ES
.055
.005
.045
.035
t(15) = 2.61, p = . 01
r = . 559
These results are quite striking and suggest that future studies
combining these moderators should yield especially reliable effects.
SUMMARY AND CONCLUSIONS
Our meta-analysis of forced-choice precognition experiments
confirms the existence of a small but highly significant precognition
effect. The effect appears to be replicable; significant outcomes are
reported by 40 investigators using a variety of methodological par-
adigms and subject populations.
The precognition effect is statistically very robust: it remains
highly significant despite elimination of studies with z scores in the
upper and lower 10% of the z-score distribution and when a third
of the remaining investigators-the major contributors of precog-
nition studies-are eliminated.
Estimates of the "filedrawer" problem and consideration of para-
psychological publication practices indicate that the precognition ef-
fect cannot plausibly be explained on the basis of selective publica-
tion bias. Analyses of precognition effect sizes in relation to eight
measures of research quality fail to support the hypothesis that the
observed effect is driven to any appreciable extent by methodolog-
ical flaws; indeed, several analyses indicate that methodologically su-
perior studies yield stronger effects than methodologically weaker
studies.
Analyses of parapsychological alternatives to precognition, al-
though limited to the subset of studies using random number tables,
provide no support for the hypothesis that the effect results from
A Meta-Analysis of Forced-Choice Precognition Experiments
the operation of contemporaneous ESP and PK at the time of ran-
domization.
Although the overall precognition effect size is small, this does
not imply that it has no practical consequences. It is, for example,
of the same order of magnitude as effect sizes leading to the early
ermination of several major medical research studies. In 1981, the
National Heart, Lung, and Blood Institute discontinued its study of
propranolol because the results were so favorable to the propranolol
reatment that it would be unethical to continue placebo treatment
Kolata, 1981); the effect size was 0.04. More recently, The Steering
Committee of the Physicians' Health Study Research Group (1988),
n a widely publicized report, terminated its study of the effects of
ispirin in the prevention of heart attacks for the same reason. The
ispirin group suffered significantly fewer heart attacks than a pla-
:ebo control group; the associated effect size was 0.03.
The most important outcome of the meta-analysis is the identi-
ication of several moderating variables that appear to covary sys-
ematically with precognition performance. The largest effects are
observed in studies using subjects selected on the basis of prior test
performance, who are tested individually, and who receive trial-by-
rial feedback. The outcomes of studies combining these factors con-
rast sharply with the null outcomes associated with the combination
f group testing, unselected subjects, and no feedback of results. Be-
ause the two groups of studies were conducted by a subset of the
ame investigators, it is unlikely that the observed difference in per-
ormance is due to experimenter effects. Indeed, these outcomes-
underscore the importance of carefully examining differences in
ubject populations, test setting, and so forth, before resorting to
acile "explanations" based on psi-mediated experimenter effects or
he "elusiveness of psi."
The identification of these moderating variables has important
nplications for our understanding of the phenomena and provides
clear direction for future research. The existence of moderating
ariables indicates that the precognition effect is not merely an
.nexplained departure from a theoretical chance baseline, but
: ather is an effect that covaries with factors known to influence
hore familiar aspects of human performance. It should now be pos-
ble to exploit these moderating factors to increase the magnitude
nd reliability of precognition effects in new studies.
REFERENCES
KERS, C. (1987). Parapsychology is science, but its findings are inconclusive.
Behavioral and Brain Sciences, 10, 566-568.
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

302
The Journal of Parapsychology
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
303
BARNETT, V., & LEWIS, T. (1978). Outliers in statistical data. New York: John
Wiley & Sons.
BROWNLEE, K. A. ( 1965). Statistical theory and methodology in science and engi-
herring. New York: John Wiley & Sons.
COHEN, J. (1977). Statistical power analysis for the behavioral sciences. New York:
Academic Press.
DAWES, R. M., LANDMAN, J., & WILLIAMS, J. (1984). Reply to Kurosawa. Amer-
ican Psychologist, 39, 74-75.
HONORTON, C. (1985). Meta-analysis of psi ganzfeld research: A response to
Hyman. Journal of Parapsychology, 49, 51-92.
HYMAN, R. (1985). The ganzfeld psi experiment: A critical appraisal. Journal
of Parapsychology, 49, 3-50.
KOLATA, G. B. (1981). Drug found to help heart attack survivors. Science, 214,
774-775.
MANGAN, G. L. (1955). Evidence of displacement in a precognition test. Journal
of Parapsychology. 19, 35-44.
MORRIS, R. L. (1982). Assessing experimental support for true precognition.
Journal of Parapsychology, 46, 321-336.
ROSENTHAL, R. (1984). Meta-analytic procedures for social research. Beverly Hills,
CA: Sage.
STEERING COMMITTEE OF THE PHYSICIANS' HEALTH STUDY RESEARCH GROUP.
(1988). Preliminary report: Findings from the aspirin component of the
ongoing Physicians' Health Study. New England Journal of Medicine, 318,
262-264.
STERLING, T. D. (1959). Publication decisions and their possible effects on
inferences drawn from tests of significance-or vice versa. Journal of the
American Statistical Association, 54, 30-34.
WILKINSON, 1 .. (1988). SYSTAT: The system for statistics. Evanston, IL: SYSTAT.
CHRONOLOGICAL LISTING OF STUDIES IN META-ANALYSIS
CARINGTON, W. (1935). Preliminary experiments in precognitive guessing. Jour-
nal of the Society for Psychical Research, 29, 86-104.
RHINE. J. B. (1938). Experiments bearing on the precognition hypothesis: I.
Pre-shuffling card calling. Journal of Parapsychology, 2, 38-54.
RHINE, J. B., SMITH, B. M., & WOODRUFF, J. L. (1938). Experiments bearing
on the precognition hypothesis: II. The role of ESP in the shuffling of
cards. Journal of Parapsychology, 2, 119-131.
HUMPHREY, B. M., & PRATT, J. G. (1941). A comparison of five ESP test
procedures. Journal of Parapsychology, 5, 267-293.
RHINE, J. B. (1941). Experiments bearing upon the precognition hypothesis:
III. Mechanically selected cards. Journal of Parapsychology, 5, 1-57.
STUART, C. E. (1941). An analysis to determine a test predictive of extra-chance
scoring in card-calling tests. Journal of Parapsychology, 5, 99- 137.
HUMPHREY, B. M., & RHINE, J. B. (1942). A confirmatory study of salience in
precognition tests. Journal of Parapsychology, 6, 190-219.
A Meta-Analysis of Forced-Choice Precognition Experiments
RHINE. J. B. (1942). Evidence of precognition in the covariation of salience
ratios. Journal of Parapsychology, 6, 111 - 143.
NICOL, J. F., & CARINGTON, W. (1947). Some experiments in willed die-throw-
ing. Proceedings of the Society for Psychical Research, 48, 164-175.
THOULESS, R. H. (1949). A comparative study of performance in three psi
tasks. Journal of Parapsychology, 13, 263-273.
BASTIN, E. W., & GREEN, J. M. (1953). Some experiments in precognition.
Journal of Parapsychology, 17, 137-143.
MCMAHAN, E. A., & BATES, E. K. (1954). Report of further Marchesi exper-
iments. Journal of Parapsychology, 18, 82-92.
MANGAN, G. L. (1955). Evidence of displacement in a precognition test. Journal
of Parapsychology, 19, 35-44.
Osis, K. (1955). Precognition over time intervals of one to thirty-three days.
Journal of Parapsychology, 19, 82-91.
NIELSEN, W. (1956). An exploratory precognition experiment. Journal of Para-
psychology, 20, 33-39.
NIELSEN, W. (1956). Mental states associated with success in precognition. Jour-
nal of Parapsychology, 20, 96-109.
FAHLER, J. (1957). ESP card tests with and without hypnosis. Journal of Para-
psychology, 21, 179-185.
MANGAN, G. L. (1957). An ESP experiment with dual-aspect targets involving
one trial a day. Journal of Parapsychology, 21, 273-283.
ANDERSON, M., & WHITE, R. (1958). A survey of work on ESP and teacher-
pupil attitudes. Journal of Parapsychology, 22, 246-268.
NASH, C. B. (1958). Correlation between ESP and religious value. Journal of
Parapsychology, 22, 204-209.
ANDERSON, M. (1959). A precognition experiment comparing time intervals of
a few days and one year. Journal of Parapsychology, 23, 81-89.
ANDERSON, M., & GREGORY, E. (1959). A two-year program of tests for clair-
voyance and precognition with a class of public school pupils. Journal of
Parapsychology, 23, 149-177.
NASH, C. B. (1960). Can precognition occur diametrically? Journal of Parapsy-
chology, 24, 26-32.
FREEMAN, J. A. (1962). An experiment in precognition. Journal of Parapsychology,
26, 123-130.
RHINE, J. B. (1962). The precognition of computer numbers in a public test.
Journal of Parapsychology, 26, 244-251.
RYzL., M. (1962). Training the psi faculty by hypnosis. Journal of the Society for
Psychical Research, 41, 234-252.
SANDERS, M. S. (1962). A comparison of verbal and written responses in a
precognition experiment. Journal of Parapsychology, 26, 23-34.
FREEMAN, J. (1963). Boy-girl differences in a group precognition test. Journal
of Parapsychology, 27, 175-181.
RAO, K. R. (1963). Studies in the preferential effect: II. A language ESP test
involving precognition and "intervention." Journal of Parapsychology, 27, 147-
160.
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

304
The Journal of Parapsychology
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
305
FREEMAN, J. (1964). A precognition test with a high-school science club. Journal
of Parapsychology, 28, 214-221.
FREEMAN, J., & NIELSEN, W. (1964). Precognition score deviations as related
to anxiety levels. Journal of Parapsychology, 28, 239-249.
SCHMEIDLER, G. (1964). An experiment on precognitive clairvoyance: Part I.
The main results. Journal of Parapsychology, 28, 1-14.
FREEMAN, J. A. (1965). Differential response of the sexes to contrasting ar-
rangements of ESP target material. Journal of Parapsychology, 29, 251-258.
Osis, K., & FAHLER, J. (1965). Space and time variables in ESP. Journal of the
American Society for Psychical Research, 59, 130-145.
FAHLER, J., & OsIs, K. (1966). Checking for awareness of hits in a precognition
experiment with hypnotized subjects. Journal of the American Society for
Psychical Research, 60, 340-346.
FREEMAN, J. A. (1966). Sex differences and target arrangement: High-school
booklet tests of precognition. Journal of Parapsychology, 30, 227-235.
ROGERS, D. P. (1966). Negative and positive affect and ESP run-score variance.
Journal of Parapsychology, 30, 151 - 159.
ROGERS, D. P., & CARPENTER, J. C. (1966). The decline of variance of ESP
scores within a testing session. Journal of Parapsychology, 30, 141 - 150.
BRIER, B. (1967). A correspondence ESP experiment with high-I.Q. subjects.
Journal of Parapsychology, 31, 143-148.
BUZBY, D. E. (1967). Subject attitude and score variance in ESP tests. Journal
of Parapsychology, 31, 43-50.
BUZBY, D. E. (1967). Precognition and a test of sensory perception. Journal of
Parapsychology, 31, 135-142.
FREEMAN, J. A. (1967). Sex differences, target arrangement, and primary men-
tal abilities. Journal of Parapsychology, 31, 271-279.
HONORTON, C. (1967). Creativity and precognition scoring level. Journal of
Parapsychology, 31, 29-42.
CARPENTER, J. C. (1968). Two related studies on mood and precognition run-
score variance. Journal of Parapsychology, 32, 75-89.
DUVAL, P., & MONTREDON, E. (1968). ESP experiments with mice. Journal of
Parapsychology, 32, 153-166.
FEATHER, S. R., & BRIER, R. (1968). The possible effect of the checker in
precognition tests. Journal of Parapsychology, 32, 167-175.
FREEMAN, J. A. (1968). Sex differences and primary mental abilities in a group
precognition test. Journal of Parapsychology, 32, 176-182.
NASH, C. S., & NASH, C. B. (1968). Effect of target selection, field dependence,
and body concept on ESP performance. Journal of Parapsychology, 32, 248-
257.
RHINE, L. E. (1968). Note on an informal group test of ESP. Journal of Para-
psychology, 32, 47-53.
RYZL, M. (1968). Precognition scoring and attitude toward ESP. Journal of Para-
psychology, 32, 1-8.
RYZI., M. (1968). Precognition scoring and attitude. Journal of Parapsychology,
32, 183-189.
A Meta-Analysis of Forced-Choice Precognition Experiments
CARPENTER, J. C. (1969). Further study on a mood adjective check list and
ESP run-score variance. Journal of Parapsychology, 33, 48-56.
DUVAL, P., & MONTREDON, E. (1969). Precognition in mice: A confirmation.
Journal of Parapsychology, 33, 71-72.
FREEMAN, J. A. (1969). The psi-differential effect in a precognition test. Journal
of Parapsychology, 33, 206-212.
FREEMAN, J. A. (1969). A precognition experiment with science teachers. Journal
of Parapsychology, 33, 307-310.
JOHNSON, M. (1969). Attitude and target differences in a group precognition
test. Journal of Parapsychology, 33, 324-325.
MONTREDON, E., & ROBINSON, A. (1969). Further precognition work with mice.
Journal of Parapsychology, 33, 162-163.
SCHMIDT, H. (1969). Precognition of a quantum process. Journal of Parapsy-
chology, 33, 99-108.
BENDER, H. (1970). Differential scoring of an outstanding subject on GESP and
clairvoyance. Journal of Parapsychology, 34, 272-273.
FREEMAN, J. A. (1970). Sex differences in ESP response as shown by the Free-
man picture-figure test. Journal of Parapsychology, 34, 37-46.
FREEMAN, J. A. (1970). Ten-page booklet tests with elementary-school children.
Journal of Parapsychology. 34, 192-196.
FREEMAN, J. (1970). Shift in scoring direction with junior-high-school students:
A summary. Journal of Parapsychology, 34, 275.
FREEMAN, J. A. (1970). Mood, personality, and attitude in precognition tests.
Journal of Parapsychology, 34, 322.
HARALDSSON, E. (1970). Subject selection in a machine precognition test. Journal
of Parapsychology, 34, 182-191.
HARALDSSON, E. (1970). Precognition of a quantum process: A modified rep-
lication. Journal of Parapsychology, 34, 329-330.
NIELSEN, W. (1970). Relationships between precognition scoring level . and
mood. Journal of Parapsychology, 34, 93-116.
SCHMIDT, H. (1970). Precognition test with a high-school group. Journal of
Parapsychology, 34, 70.
BELOFF, J., & BATE, D. (1971). An attempt to replicate the Schmidt findings.
Journal of the Society for Psychical Research, 46, 21-31.
HONORTON, C. (1971). Automated forced-choice precognition tests with a "sen
sitive." Journal of the American Society for Psychical Research, 65, 476-481.
MITCHELL, E. D. (1971). An ESP test from Apollo 14. Journal of Parapsychology,
35, 89-107.
SCHMIDT, H., & PANTAS, L. (1971). Psi tests with psychologically equivalentC
conditions and internally different machines. Journal of Parapsychology, 35,
326-327.
STANFORD, R. G. (1971). Extrasensory effects upon "memory." Journal of the
American Society for Psychical Research, 64, 161 - 186.
STEILBERG, B. J. (1971). Investigation of the paranormal gifts of the Dutch
sensitive Lida T. Journal of Parapsychology, 35, 219-225.
"Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

306
The Journal of Parapsychology
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
307
THOULESS, R. H. (1971). Experiments on psi self-training with Dr. Schmidt's
pre-cognitive apparatus. Journal of the Society for Psychical Research, 46, 15-
21.
HONORTON, C. (1972). Reported frequency of dream recall and ESP. Journal
of the American Society for Psychical Research, 66, 369-374.
JOHNSON, M., & NORDBECK, B. (1972). Variation in the scoring behavior of a
"psychic" subject. Journal of Parapsychology, 36, 122-132.
KELLY, E. E., & KANTHAMANI, B. K. ( 1972). A subject's efforts toward voluntary
control. Journal of Parapsychology, 36, 185-197.
SCHMIDT, H., & PANTAS, L. (1972). Psi tests with internally different machines.
Journal of Parapsychology, 36, 222-232.
CRAIG, J. G. (1973). The effect of contingency on precognition in the rat.
Research in Parapsychology 1972, 154-156.
FREEMAN, J. A. (1973). The psi quiz: A new ESP test. Research in Parapsychology
1972, 132-134.
ARTLEY, B. (1974). Confirmation of the small-rodent precognition work. Journal
of Parapsychology, 38, 238-239.
HARRIS, S., & TERRY, J. (1974). Precognition in a water-deprived Wistar rat.
Journal of Parapsychology, 38, 239.
RANDALL, J. L. (1974). An extended series of ESP and PK tests with three
English schoolboys. Journal of the Society for Psychical Research, 47, 485-494.
EYSENCK, H. J. (1975). Precognition in rats. Journal of Parapsychology, 39, 222-
227.
HARALDSSON, E. (1975). Reported dream recall, precognitive dreams, and ESP.
Research in Parapsychology 1974, 47-48.
HONORTON, C., RAMSEY, M., & CABIBBO, C. (1975). Experimenter effects in
extrasensory perception. Journal of the American Society for Psychical Research,
69, 135-149.
KANTHAMANI, H., & RAO, H. H. (1975). Response tendencies and stimulus
structure. Journal of Parapsychology, 39, 97-105.
LEVIN, J. A. (1975). A series of psi experiments with gerbils. Journal of Para-
psychology, 39, 363-365.
TERRY, J. C., & HARRIS, S. A. (1975). Precognition in water-deprived rats.
Research in Parapsychology 1974, 81.
DAVIS, J. W., & HAIGHT, J. (1976). Psi experiments with rats. Journal of Para-
psychology, 40, 54-55.
JACOBS, J., & BREEDERVELD, H. (1976). Possible influences of birth order on
ESP ability. Research Letter (Parapsychology Laboratory, University of
Utrecht), No. 7, 10-20.
NEVILLE, R. C. (1976). Some aspects of precognition testing. Research in Para-
psychology 1975, 29-31.
DRUCKER, S. A., DREWES, A. A., & RUBIN, L. (1977). ESP in relation to cognitive
development and IQ in young children. Journal of the American Society for
Psychical Research, 71, 289-298.
A Meta-Analysis of Forced-Choice Precognition Experiments
HARALDSSON, E. (1977). ESP and the defense mechanism test (DMT): A further
validation. European Journal of Parapsychology, 2, 104-114.
SARGENT, C. L. (1977). An experiment involving a novel precognition task.
Journal of Parapsychology, 41, 275-293.
BIERMAN, D. J. (1978). Testing the "advanced wave" hypothesis: An attempted
replication. European Journal of Parapsychology, 2, 206-212.
BRAUD, W. (1979). Project Chicken Little: A precognition experiment involving
the SKYLAB space station. European Journal of Parapsychology, 3, 149-165.
HARALDSSON, E., & JOHNSON, M. (1979). ESP and the defense mechanism test
(DMT) Icelandic study No. III: A case of the experimenter effect? European
Journal of Parapsychology, 3, 11-20.
O'BRIEN, J. T. (1979). An examination of the checker effect. Research in Para-
psychology 1978, 153-155.
CLEMENS, D. B., & PHILLIPS, D. T. (1980). Further studies of precognition in
mice. Research in Parapsychology 1979, 156.
HARALDSSON, E. (1980). Scoring in a precognition test as a function of the
frequency of reading on psychical phenomena and belief in ESP. Research
Letter (Parapsychology Laboratory, University of Utrecht), No. 10, 1-8.
SARGENT, C., & HARLEY, T. A. (1981). Three studies using a psi-predictive
trait variable questionnaire. Journal of Parapsychology, 45, 199-214.
WINKELMAN, M. (1981). The effect of formal education on extrasensory abil-
ities: The Ozolco study. Journal of Parapsychology, 45, 321-336.
NASH, C. B. (1982). ESP of present and future targets. Journal of the Society for
Psychical Research, 51, 374-377.
THALBOURNE, M., BELOFF, J., & DELANOY, D. (1982). A test for the "extra-
verted sheep versus introverted goats" hypothesis. Research in Parapsychology
1981, 155-156.
CRANDALL, J. E., & HITE, D. D. (1983). Psi-missing and displacement: Evidence
for improperly focused psi? Journal of the American Society for Psychical Re-
search, 77, 209-228.
SCHWARTZ, S. A., & DEMATTEI, R. J. (1983). The Mobius Psi-Q test: Prelim-
inary findings. Research in Parapsychology 1982, 103-105.
JOHNSON, M., & HARALDSSON, E. (1984). The Defense Mechanism Test as a
predictor of ESP scores: Icelandic studies IV and V. Journal of Parapsychology,
48, 185-200.
TEDDER, W. (1984). Computer-based long-distance ESP: An exploratory ex-
amination (RB/PS). Research in Parapsychology 1983, 100-101.
HESELTINE, G. L. (1985). PK success during structured and nonstructured
RNG operation. Journal of Parapsychology, 49, 155-163.
HARALDSSON, E., & JOHNSON, M. (1986). The Defense Mechanism Test (DMT)
as a predictor of ESP performance: Icelandic studies VI and VII. Research
in Parapsychology 1985, 43-44.
VASSY, Z. (1986). Experimental study of complexity dependence in precogni-
tion. Journal of Parapsychology, 50, 235-270.
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0

Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0
:
30
The Journal of Parapsychology
HONORTON, C. (1987). Precognition and real-time ESP performance in a com-
puter task with an exceptional subject. Journal of Parapsychology, 51, 291-
320.
Psychophysical Research Laboratories
P. O. Box 569
Plainsboro, NJ 08536
Approved For Release 2000/08/08 : CIA-RDP96-00789R002200610001-0