A study can tick every box on the feature list and still miss the psychological process it claims to target. Yet ask a student how to judge a psychology experiment, and the answer is often exactly that list: random assignment, a control group, standardized instructions. Those features matter, but spotting them is not the same as evaluating a study.
Rigor is a judgment about how three demands are held in balance. Does the study test what it claims to test? That’s validity. Would the same procedures and measures give similar results if repeated under the same conditions? That’s reliability. And are participants treated according to ethical standards? Strengthening one can strain the others, so a serious evaluation weighs all three together.
Validity Has Several Layers, and They Can Work Against Each Other
That gap has a name. A review of construct validation of experimental manipulations in social psychology, published in Perspectives on Psychological Science, argues that a manipulation is only useful to the extent that it has construct validity, meaning it actually engages the process it is meant to engage. Even a perfectly controlled procedure fails without it: it may be spotless but aimed at the wrong thing. The authors add that constructs sit in a network of lawful relations with other constructs, so manipulations and measures should be judged against that theory.
Internal validity asks a different question: did the manipulation, rather than confounds such as experimenter bias or demand characteristics, produce the effect? External validity asks whether the effect holds across settings, populations, times, and cultures. A manipulation can be clean on the first count and still tell us little about the second.
A methodological review in the Journal of Graduate Medical Education adds ecological validity, a subtype of external validity concerned with how well the experimental situation resembles real life. It notes that tightly controlled laboratory tasks often have strong internal validity and reliability but poor ecological validity because they do not resemble everyday demands. The layers are related but separable, and a design can score well on one while performing poorly on another.
Reliability Has to Be Checked, Not Assumed
Reliability asks whether the same procedure or measure would give similar results again under the same conditions. The catch is that the answer belongs to the sample at hand. The PLOS ONE analysis of measurement reliability, construct validity, and transparent reporting in original and replication psychological research found that a measure’s reliability can change from one sample to the next. So can its structure, meaning whether a scale is really measuring one underlying thing.
A different structure is a separate reason not to carry psychometric assumptions over from an earlier paper. The authors recommend recalculating reliability and checking the scale’s structure in each new dataset.
The same analysis of multi-lab replication projects found that many published studies inadequately report their measurements, omitting details needed to evaluate and reconstruct instruments. That makes reliability harder to verify than it should be. Reliability also has to be kept separate from the two validity questions. Internal validity asks whether something other than the intended cause produced the effect, and construct validity asks whether the measure or manipulation reflects the named construct.
The PLOS ONE authors argue that replication results cannot be confidently linked to the underlying psychological phenomenon unless measurement is valid and reliable in each new dataset. Consider why. A task can give very consistent scores while tapping the wrong thing or a stable confound. Consistency tells you the scores repeat, not what they are scores of.
A consistent manipulation may never engage the intended process, and demand characteristics or other confounds can still shape the effect. A write-up gives you ways to tell. Look for reliability that was calculated for the current dataset rather than borrowed from earlier uses of the measure, and for measures and materials described or shared clearly enough that others can see what was actually measured or manipulated and reconstruct the procedure for replication.
Raising the reliability of an experiment usually means tightening its procedures. Tightening isn’t free, though, and some of what it costs is set by ethics rather than by method.
Ethical Constraints Are Part of the Design Problem, Not External to It
Ethical requirements shape what a study can be. Limits on deception, informed consent rules, and participant welfare standards can rule out the more realistic or more internally valid version of a design. They also touch demand characteristics. If participants know they are being studied, or know what a study is after, their behavior can shift, and disclosure rules limit how much of that awareness a design can avoid.
So when a study scores lower on validity, an evaluator should ask whether that reflects a legitimate boundary rather than carelessness. These limits are appropriate, not failures of design.
That places ethical soundness inside the evaluation of rigor. A design that reaches strong validity by ethically unacceptable means is not a better design with a side problem. It is a different kind of compromise.
The Evaluative Skill Is Seeing What a Design Traded Away
Every rigor safeguard pushes a design somewhere, and the push costs something elsewhere. Imagine a study asking whether social pressure increases conformity.
Variant A uses a tightly scripted lab task with standardized instructions and timing. That boosts consistency and supports clearer causal inference. The Journal of Graduate Medical Education review observes that tightly controlled laboratory tasks often have strong internal validity and reliability but poor ecological validity. For the conformity task, the likely price is reduced ecological validity and limited generalization beyond the lab.
The construct-validation review adds a caution. Even a tightly standardized manipulation can fail if it does not truly engage the psychological process that the theory describes.
Variant B places participants in a more naturalistic group setting. That strengthens ecological validity and relevance. The price is weaker standardization, with more noise, more potential confounds, and harder causal attribution.
Ethics acts as a separate constraint on either variant. Limits on harsher or deceptive manipulations can rule out the version that would sharpen causal inference or realism, so researchers may accept smaller effects or less realism as the ethical cost.
Students can mirror one sentence shape: “This design prioritizes X by doing Y, accepting Z as the cost.” The weak alternative is “it was controlled, so it’s reliable and valid,” which sounds like analysis and isn’t. A stronger account of the controlled design would read: “This design prioritizes consistency and internal validity by standardizing conditions, accepting reduced ecological validity and narrower generalizability as the cost.” The field-like version prioritizes realism and ecological validity by placing participants in a group setting, accepting weaker control, more noise, and harder replication as the cost.
Every study, strong or weak, is a set of choices, ethical ones included, and good evaluation names what each choice cost.
Carrying the Trade-Off Lens Forward
The study that ticks every box and still misses its target only fools readers who stop at the list. Pick up any paper and ask what its design chose, what it gave up across validity, reliability, and ethics, and whether that balance suited the research question.
That habit carries over to topics and methods you haven’t met yet, and well beyond any particular exam.