4 Principles of Experimental Design
- Comparison: Use a design that compares two or more treatments
- Random Assignment: Use chance to assign experimental units to treatments. Doing so helps create roughly equivalent groups of units by balancing the effects of other variables among the treatment groups
- Control: Keep other variables that might affect the response the same for all groups
- Replication: Use enough experimental units in each group so that the differences in the effects of the treatments can be distinguished from chance differences between the groups
In an experiment, blinding is done to reduce any outside influence from participants or researchers on the results.
In a Double-blind, neither the subjects nor those who interact with them and measure the response variable know which treatment a subject received. In a Single-blind, one side (the subjects) or the other (those who interact with them) knows which treatment they are receiving.
When a result is said to be statistically significant, it is an observed effect so large that it would rarely occur by chance. We will learn more about how to check to see if a result is statistically significant in a later chapter. For now, be aware of when you state “significant” when interpreting your results.
Blocking
A block is a group of experimental units that are known before the experiment to be similar in some way that is expected to affect the response to the treatments. For example, when measuring how a certain nutritious diet affects height, you should block by gender because females are typically shorter than males.
A Randomized Block Design is the random assignment of experimental units to treatments carried out separately within each block.
The purpose of a sample is to give us information about a larger population. We are trying to make inferences about the population from what we know about the sample. Larger random samples give better information about the population than smaller random samples.
Important note: Sample results will differ somewhat, just by chance because rarely is data ideal in real life.
2 Reasons we can trust random samples:
- The results don't change haphazardly (randomly) from sample to sample (reliability)
- The laws of probability allow trustworthy inference about the population (validity)
Types of Bias in experimental design
In undercoverage bias, some members of the population cannot be chosen in a sample and so not everyone is represented. For example, a telephone survey would not include people who don’t have access to a phone.
In nonresponse bias, an individual chosen for the sample cannot be contacted or refuses to participate. If a sample is sent to a SRS of the school and only some students choose to participate, it would not be representative of the entire school population.
In response bias, there is a systematic pattern of inaccurate answers to a survey. This could be due to how the survey is phrased, the setting at which it’s given, or who gives the survey. If a police officer asks you how many times you’ve driven over the speed limit in the past month, you might respond lower than the actual number.
The wording of questions is the most important influence on the answers given to a sample survey
If we cannot do an experiment, the criteria for establishing causation is:
- The association is strong
- The association is consistent (multiple studies verify)
- Larger values of the explanatory variable are associated with stronger responses
- The alleged cause precedes the effect in time
- The alleged cause is plausible