How Can You Increase External Validity

External validity is the validity of applying the conclusions of a scientific written report outside the context of that report.^[one] In other words, it is the extent to which the results of a study can be generalized to and beyond other situations, people, stimuli, and times.^[ii] In dissimilarity, internal validity is the validity of conclusions drawn within the context of a particular written report. Because general conclusions are almost always a goal in research, external validity is an important property of any study. Mathematical analysis of external validity concerns a conclusion of whether generalization across heterogeneous populations is feasible, and devising statistical and computational methods that produce valid generalizations.^[3]

Threats [edit]

"A threat to external validity is an explanation of how you might exist incorrect in making a generalization from the findings of a particular report."^[4] In well-nigh cases, generalizability is express when the effect of one factor (i.e. the independent variable) depends on other factors. Therefore, all threats to external validity can be described as statistical interactions.^[5] Some examples include:

Aptitude by treatment Interaction: The sample may have certain features that interact with the independent variable, limiting generalizability. For example, comparative psychotherapy studies ofttimes apply specific samples (e.thou. volunteers, highly depressed, no comorbidity). If psychotherapy is found effective for these sample patients, volition it besides exist effective for non-volunteers or the mildly depressed or patients with concurrent other disorders? If not, the external validity of the study would exist limited.
Situation by treatment interactions: All situational specifics (eastward.1000. treatment weather condition, fourth dimension, location, lighting, racket, treatment administration, investigator, timing, scope and extent of measurement, etc.) of a study potentially limit generalizability.
Pre-test by treatment interactions: If cause-effect relationships can only be establish when pre-tests are carried out, then this also limits the generality of the findings. This sometimes goes under the label "sensitization", because the pretest makes people more sensitive to the manipulation of the treatment.

Note that a study's external validity is limited by its internal validity. If a causal inference made within a report is invalid, then generalizations of that inference to other contexts will also be invalid.

Cook and Campbell^[6] made the crucial stardom between generalizing to some population and generalizing across subpopulations divers past unlike levels of some background factor. Lynch has argued that it is well-nigh never possible to generalize to meaningful populations except as a snapshot of history, but it is possible to examination the degree to which the effect of some cause on some dependent variable generalizes across subpopulations that vary in some background factor. That requires a test of whether the treatment event being investigated is moderated past interactions with one or more groundwork factors.^[5] ^[vii]

Disarming threats [edit]

Whereas enumerating threats to validity may assist researchers avoid unwarranted generalizations, many of those threats tin exist disarmed, or neutralized in a systematic way, and then every bit to enable a valid generalization. Specifically, experimental findings from i population can be "re-processed", or "re-calibrated" so as to circumvent population differences and produce valid generalizations in a 2d population, where experiments cannot be performed. Pearl and Bareinboim^[three] classified generalization problems into 2 categories: (1) those that lend themselves to valid re-calibration, and (two) those where external validity is theoretically impossible. Using graph-based calculus,^[8] they derived a necessary and sufficient condition for a problem instance to enable a valid generalization, and devised algorithms that automatically produce the needed re-calibration, whenever such exists.^[nine] This reduces the external validity trouble to an exercise in graph theory, and has led some philosophers to conclude that the trouble is now solved.^[10]

An important variant of the external validity problem deals with selection bias, also known every bit sampling bias—that is, bias created when studies are conducted on non-representative samples of the intended population. For example, if a clinical trial is conducted on college students, an investigator may wish to know whether the results generalize to the entire population, where attributes such as age, pedagogy, and income differ substantially from those of a typical student. The graph-based method of Bareinboim and Pearl identifies weather nether which sample pick bias can be circumvented and, when these conditions are met, the method constructs an unbiased estimator of the average causal consequence in the entire population. The chief difference between generalization from improperly sampled studies and generalization across disparate populations lies in the fact that disparities amid populations are usually acquired by preexisting factors, such as age or ethnicity, whereas selection bias is ofttimes caused past post-treatment conditions, for instance, patients dropping out of the study, or patients selected by severity of injury. When pick is governed by post-handling factors, unconventional re-calibration methods are required to ensure bias-gratis estimation, and these methods are readily obtained from the problem'south graph.^[11] ^[12]

Examples [edit]

If age is judged to be a major factor causing treatment effect to vary from individual to individual, then age differences between the sampled students and the general population would lead to a biased estimate of the boilerplate treatment effect in that population. Such bias can be corrected though by a simple re-weighing procedure: We take the age-specific effect in the student subpopulation and compute its boilerplate using the age distribution in the general population. This would give us an unbiased guess of the average treatment effect in the population. If, on the other mitt, the relevant factor that distinguishes the report sample from the general population is in itself affected by the treatment, and so a different re-weighing scheme need be invoked. Calling this factor Z, we again average the z-specific effect of X on Y in the experimental sample, but at present we weigh it by the "causal issue" of X on Z. In other words, the new weight is the proportion of units attaining level Z=z had treatment X=x been administered to the unabridged population. This interventional probability, often written^[13] $P(Z=z|practise(10=10))$ , can sometimes be estimated from observational studies in the full general population.

A typical example of this nature occurs when Z is a mediator betwixt the treatment and outcome, For instance, the treatment may exist a cholesterol-reducing drug, Z may be cholesterol level, and Y life expectancy. Hither, Z is both affected by the treatment and a major gene in determining the outcome, Y. Suppose that subjects selected for the experimental study tend to have college cholesterol levels than is typical in the general population. To gauge the boilerplate effect of the drug on survival in the entire population, nosotros first compute the z-specific handling consequence in the experimental study, and so boilerplate it using $P(Z=z|do(X=x))$ as a weighting function. The estimate obtained will be bias-free even when Z and Y are confounded—that is, when there is an unmeasured mutual factor that affects both Z and Y.^[fourteen]

The precise atmospheric condition ensuring the validity of this and other weighting schemes are formulated in Bareinboim and Pearl, 2022^[14] and Bareinboim et al., 2022.^[12]

External, internal, and ecological validity [edit]

In many studies and research designs, at that place may be a merchandise-off betwixt internal validity and external validity:^[15] ^[16] ^[17] Attempts to increase internal validity may also limit the generalizability of the findings, and vice versa. This state of affairs has led many researchers call for "ecologically valid" experiments. By that they hateful that experimental procedures should resemble "real-world" weather. They criticize the lack of ecological validity in many laboratory-based studies with a focus on artificially controlled and constricted environments. Some researchers think external validity and ecological validity are closely related in the sense that causal inferences based on ecologically valid research designs often permit for higher degrees of generalizability than those obtained in an artificially produced lab surround. However, this again relates to the stardom betwixt generalizing to some population (closely related to concerns about ecological validity) and generalizing across subpopulations that differ on some background factor. Some findings produced in ecologically valid research settings may hardly be generalizable, and some findings produced in highly controlled settings may claim near-universal external validity. Thus, external and ecological validity are contained—a report may possess external validity but not ecological validity, and vice versa.

Qualitative research [edit]

Inside the qualitative inquiry epitome, external validity is replaced by the concept of transferability. Transferability is the ability of research results to transfer to situations with similar parameters, populations and characteristics.^[xviii]

In experiments [edit]

It is common for researchers to merits that experiments are by their nature low in external validity. Some merits that many drawbacks tin can occur when following the experimental method. Past the virtue of gaining enough control over the state of affairs so equally to randomly assign people to atmospheric condition and dominion out the furnishings of inapplicable variables, the situation can become somewhat artificial and distant from real life.

At that place are two kinds of generalizability at issue:

The extent to which we can generalize from the situation synthetic by an experimenter to real-life situations ( generalizability across situations ),^[two] and
The extent to which nosotros tin can generalize from the people who participated in the experiment to people in full general ( generalizability beyond people )^[2]

Withal, both of these considerations pertain to Cook and Campbell'southward concept of generalizing to some target population rather than the arguably more primal job of assessing the generalizability of findings from an experiment across subpopulations that differ from the specific situation studied and people who differ from the respondents studied in some meaningful way.^[6]

Critics of experiments suggest that external validity could be improved past the use of field settings (or, at a minimum, realistic laboratory settings) and past the apply of true probability samples of respondents. Nonetheless, if one's goal is to understand generalizability across subpopulations that differ in situational or personal background factors, these remedies do non have the efficacy in increasing external validity that is commonly ascribed to them. If background gene X treatment interactions exist of which the researcher is unaware (equally seems probable), these research practices tin mask a substantial lack of external validity. Dipboye and Flanagan, writing virtually industrial and organizational psychology, note that the evidence is that findings from one field setting and from one lab setting are equally unlikely to generalize to a 2nd field setting.^[19] Thus, field studies are not by their nature high in external validity and laboratory studies are not by their nature low in external validity. Information technology depends in both cases whether the particular treatment effect studied would change with changes in background factors that are held constant in that study. If one's written report is "unrealistic" on the level of some background factor that does non collaborate with the treatments, it has no effect on external validity. It is just if an experiment holds some background factor constant at an unrealistic level and if varying that background factor would have revealed a strong Treatment x Groundwork gene interaction, that external validity is threatened.^[v]

Generalizability across situations [edit]

Research in psychology experiments attempted in universities is ofttimes criticized for existence conducted in artificial situations and that it cannot be generalized to existent life.^[20] ^[21] To solve this problem, social psychologists effort to increase the generalizability of their results by making their studies as realistic equally possible. Every bit noted above, this is in the hope of generalizing to some specific population. Realism per se does not help the make statements about whether the results would change if the setting were somehow more than realistic, or if written report participants were placed in a different realistic setting. If only one setting is tested, it is non possible to make statements about generalizability beyond settings.^[5] ^[7]

Even so, many authors conflate external validity and realism. There is more than than i way that an experiment can be realistic:

The similarity of an experimental situation to events that occur ofttimes in everyday life—it is clear that many experiments are decidedly unreal.
In many experiments, people are placed in situations they would rarely meet in everyday life.

This is referred to the extent to which an experiment is similar to real-life situations every bit the experiment'due south mundane realism.^[20]

It is more than important to ensure that a study is high in psychological realism—how similar the psychological processes triggered in an experiment are to psychological processes that occur in everyday life.^[22]

Psychological realism is heightened if people detect themselves engrossed in a real upshot. To reach this, researchers sometimes tell the participants a cover story—a false description of the study'south purpose. If however, the experimenters were to tell the participants the purpose of the experiment then such a procedure would be depression in psychological realism. In everyday life, no 1 knows when emergencies are going to occur and people do not accept time to plan responses to them. This means that the kinds of psychological processes triggered would differ widely from those of a real emergency, reducing the psychological realism of the report.^[two]

People don't always know why they do what they practice, or what they exercise until it happens. Therefore, describing an experimental situation to participants and then request them to respond normally volition produce responses that may not match the beliefs of people who are actually in the same situation. We cannot depend on people's predictions about what they would do in a hypothetical situation; we can only find out what people will really exercise when nosotros construct a situation that triggers the same psychological processes as occur in the existent globe.

Generalizability across people [edit]

Social psychologists written report the mode in which people, in general, are susceptible to social influence. Several experiments have documented an interesting, unexpected example of social influence, whereby the mere cognition that others were present reduced the likelihood that people helped.

The only mode to be certain that the results of an experiment represent the behaviour of a item population is to ensure that participants are randomly selected from that population. Samples in experiments cannot be randomly selected simply as they are in surveys because it is impractical and expensive to select random samples for social psychology experiments. It is difficult plenty to convince a random sample of people to agree to answer a few questions over the telephone as part of a political poll, and such polls can price thousands of dollars to deport. Moreover, fifty-fifty if one somehow was able to recruit a truly random sample, there can be unobserved heterogeneity in the effects of the experimental treatments... A handling tin can have a positive effect on some subgroups but a negative effect on others. The effects shown in the treatment averages may non generalize to any subgroup.^[5] ^[23]

Many researchers address this problem by studying bones psychological processes that brand people susceptible to social influence, assuming that these processes are and so fundamental that they are universally shared. Some social psychologist processes practise vary in different cultures and in those cases, diverse samples of people have to be studied.^[24]

Replications [edit]

The ultimate test of an experiment's external validity is replication — conducting the written report over once more, generally with different discipline populations or in different settings. Researchers volition oftentimes employ dissimilar methods, to meet if they still become the same results.

When many studies of 1 problem are conducted, the results tin can vary. Several studies might discover an effect of the number of bystanders on helping behaviour, whereas a few practice not. To brand sense out of this, there is a statistical technique called meta-analysis that averages the results of 2 or more studies to encounter if the outcome of an contained variable is reliable. A meta analysis substantially tells us the probability that the findings across the results of many studies are owing to chance or to the contained variable. If an independent variable is establish to have an effect in only ane of 20 studies, the meta-analysis will tell you that that one study was an exception and that, on average, the independent variable is not influencing the dependent variable. If an independent variable is having an result in near of the studies, the meta-analysis is likely to tell us that, on average, it does influence the dependent variable.

There can be reliable phenomena that are not express to the laboratory. For instance, increasing the number of bystanders has been found to inhibit helping behaviour with many kinds of people, including children, university students, and future ministers;^[24] in Israel;^[25] in pocket-size towns and big cities in the U.Southward.;^[26] in a diverseness of settings, such as psychology laboratories, metropolis streets, and subway trains;^[27] and with a variety of types of emergencies, such as seizures, potential fires, fights, and accidents,^[28] also as with less serious events, such as having a flat tire.^[29] Many of these replications take been conducted in real-life settings where people could non possibly have known that an experiment was being conducted.

[edit]

When conducting experiments in psychology, some believe that at that place is always a trade-off betwixt internal and external validity—

having enough control over the state of affairs to ensure that no extraneous variables are influencing the results and to randomly assign people to atmospheric condition, and
ensuring that the results can be generalized to everyday life.

Some researchers believe that a good way to increase external validity is by conducting field experiments. In a field experiment, people's beliefs is studied outside the laboratory, in its natural setting. A field experiment is identical in design to a laboratory experiment, except that it is conducted in a real-life setting. The participants in a field experiment are unaware that the events they experience are in fact an experiment. Some claim that the external validity of such an experiment is high because information technology is taking place in the real world, with existent people who are more than various than a typical university pupil sample. However, as real-world settings differ dramatically, findings in one existent-world setting may or may not generalize to another real-world setting.^[xix]

Neither internal nor external validity is captured in a single experiment. Social psychologists opt first for internal validity, conducting laboratory experiments in which people are randomly assigned to unlike weather condition and all extraneous variables are controlled. Other social psychologists prefer external validity to command, conducting most of their inquiry in field studies, and many practice both. Taken together, both types of studies meet the requirements of the perfect experiment. Through replication, researchers can study a given research question with maximal internal and external validity.^[30]

Notes [edit]

^ Mitchell, M. & Jolley, J. (2001). Research Blueprint Explained (4th Ed) New York:Harcourt.
^ ^a ^b ^c ^d Aronson, E., Wilson, T. D., Akert, R. One thousand., & Fehr, B. (2007). Social psychology. (4 ed.). Toronto, ON: Pearson Education.
^ ^a ^b Pearl, Judea; Bareinboim, Elias (2014). "External validity: From exercise-calculus to transportability across populations". Statistical Science. 29 (iv): 579–595. arXiv:1503.01603. doi:10.1214/14-sts486. S2CID 5586184.
^ Trochim, William One thousand. The Research Methods Cognition Base, 2nd Edition.
^ ^a ^b ^c ^d ^e Lynch, John (1982). "On the External Validity of Experiments in Consumer Research". Journal of Consumer Inquiry. ix (3): 225–239. doi:ten.1086/208919. JSTOR 2488619.
^ ^a ^b Melt, Thomas D.; Campbell, Donald T. (1979). Quasi-Experimentation: Design & Analysis Problems for Field Settings . Chicago: Rand McNally Higher Publishing Company. ISBN978-0395307908.
^ ^a ^b Lynch, John (1999). "Theory and External Validity". Periodical of the Academy of Marketing Science. 27 (3): 367–76. CiteSeerXx.one.1.417.8073. doi:10.1177/0092070399273007. S2CID 145357923.
^ Pearl, Judea (1995). "Causal diagrams for empirical enquiry". Biometrika. 82 (4): 669–710. doi:10.1093/biomet/82.4.669.
^ Bareinboim, Elias; Pearl, Judea (2013). "A general algorithm for deciding transportability of experimental results". Journal of Causal Inference. ane (1): 107–134. arXiv:1312.7485. doi:10.1515/jci-2012-0004. S2CID 13325846.
^ Marcellesi, Alexandre (December 2022). "External validity: Is there however a problem?". Philosophy of Science. 82 (5): 1308–1317. doi:10.1086/684084. S2CID 125072255.
^ Pearl, Judea (2015). Generalizing experimental findings. Periodical of Causal Inference. Vol. 3, no. 2. pp. 259–266.
^ ^a ^b Bareinboim, Elias; Tian, Jin; Pearl, Judea (2014). Brodley, Carla E.; Stone, Peter (eds.). "Recovering from Selection Bias in Causal and Statistical Inference". Proceedings of the Twenty-eighth AAAI Conference on Bogus Intelligence: 2410–2416.
^ Pearl, Judea; Glymour, Madelyn; Jewell, Nicholas P. (2016). Causal Inference in Statistics: A Primer. New York: Wiley.
^ ^a ^b Bareinboim, Elias; Pearl, Judea (2016). "Causal inference and the data-fusion problem". Proceedings of the National University of Sciences. 113 (27): 7345–7352. doi:ten.1073/pnas.1510507113. PMC4941504. PMID 27382148.
^ Campbell, Donald T. (1957). "Factors relevant to the validity of experiments in social settings". Psychological Message. 54 (4): 297–312. doi:10.1037/h0040950. ISSN 1939-1455. PMID 13465924.
^ Lin, Hause; Werner, Kaitlyn Grand.; Inzlicht, Michael (2021-02-16). "Promises and Perils of Experimentation: The Mutual-Internal-Validity Problem". Perspectives on Psychological Science. 16 (4): 854–863. doi:x.1177/1745691620974773. ISSN 1745-6916. PMID 33593177. S2CID 231877717.
^ Schram, Arthur (2005-06-01). "Artificiality: The tension between internal and external validity in economic experiments". Journal of Economical Methodology. 12 (two): 225–237. doi:ten.1080/13501780500086081. ISSN 1350-178X. S2CID 145588503.
^ Lincoln, Y. S.; Guba, E. M. (1986). "Just is it rigorous? Trustworthiness and actuality in naturalistic evaluation". In Williams, D. D. (ed.). Naturalistic Evaluation. New Directions for Programme Evaluation. Vol. xxx. San Francisco: Jossey-Bass. pp. 73–84. ISBN0-87589-728-2.
^ ^a ^b Dipboye, Robert L.; Flanagan, Michael F. (1979). "Research Settings in Industrial and Organizational Psychology: Are Findings in the Field More Generalizable than the Laboratory". American Psychologist. 34 (2): 141–150. doi:ten.1037/0003-066x.34.2.141.
^ ^a ^b Aronson, East., & Carlsmith, J.M. (1968). Experimentation in social psychology. In 1000. Lindzey & East. Aronson(Eds.), The Handbook of social psychology. (Vol. 2, pp. i–79.) Reading, MA: Addison-Wesley.
^ Yarkoni, Tal (2020-12-21). "The generalizability crunch". Behavioral and Brain Sciences: 1–37. doi:10.1017/S0140525X20001685. ISSN 0140-525X. PMID 33342451.
^ Aronson, E., Wilson, T.D., & Brewer, m. (1998). Experimental methods. In D. Gilbert, S. Fiske, & G. Lindzey (Eds.), The handbook of social psychology. (4th ed., Vol. 1, pp. 99–142.) New York: Random Firm.
^ Hutchinson, J. Wesley; Kamakura, Wagner A.; Lynch, John G. (2000). "Unobserved Heterogeneity as an Alternative Explanation for "Reversal" Effects in Behavioral Research". Journal of Consumer Research. 27 (three): 324–344. doi:ten.1086/317588. JSTOR 10.1086/317588. S2CID 16353123.
^ ^a ^b Darley, J.M.; Batson, C.D. (1973). "From Jerusalem to Jericho: A written report of situational and dispositional variables in helping behaviour". Periodical of Personality and Social Psychology. 27: 100–108. doi:x.1037/h0034449.
^ Schwartz, South.H.; Gottlieb, A. (1976). "Bystander reactions to a tearing theft: Crime in Jerusalem". Journal of Personality and Social Psychology. 34 (half-dozen): 1188–1199. doi:ten.1037/0022-3514.34.half-dozen.1188. PMID 1003323.
^ Latane, B.; Dabbs, J.Chiliad. (1975). "Sex, group size, and helping in three cities". Sociometry. 38 (2): 108–194. doi:10.2307/2786599. JSTOR 2786599.
^ Harrison, J.A.; Wells, R.B. (1991). "Bystander effects on male person helping behaviour: Social comparison and diffusion of responsibility". Representative Inquiry in Social Psychology. 96: 187–192.
^ Latane, B.; Darley, J.M. (1968). "Grouping inhibition of eyewitness intervention". Journal of Personality and Social Psychology. ten (3): 215–221. doi:x.1037/h0026570. PMID 5704479.
^ Hurley, D.; Allen, B.P. (1974). "The upshot of the number of people present in a nonemergency state of affairs". Journal of Social Psychology. 92: 27–29. doi:x.1080/00224545.1974.9923068.
^ Latane, B., & Darley, J.1000. (1970). The unresponsive bystander: Why doesn't he help? Englewood Cliffs, NJ: Prentice Hall