Great news about online learning from the Evaluation of Evidence-Based Practices in Online Learning report released by the U.S. Department of Education in June — students perform better in online environments than in traditional face-to-face instruction. But, how do we know this? The study is based on a meta-analysis of roughly 99 studies published between 1997 and 2008. What is unclear from the report is (1) what types of learning were measured in the included studies — comprehension, retention, transfer, none, all of the above? (2) duration of studies — were these effects consistent over time? (3) how learning was measured — e.g., multiple choice, essay, one test or multiple tests? Most importantly, the report says nothing about the consistency of measures across the included studies.
The majority of studies considered in the meta-analysis were split between undergraduate education and graduate schools or professional training programs. Professional training programs cover a broad area of learning, much of which has been shown to benefit from online education. Therefore, the inclusion of professional training programs may positively skew the results.
Interestingly, although much of the publicity surrounding the report applies the findings to K-12 learning environments, only 5 of the 99 studies focused on K-12 students. The majority of the research considered learning outcomes for college students, graduate students, and adults enrolled in professional training programs (participant ages ranged from 13-44). By its own admission, “most of the studies were modest in scope,” with sample sizes ranging from 16 to 1,857 (less than 5 of the studies had sample sizes over 400). While effect sizes were weighted based on sample sizes, these low sample sizes are worth noting when applying the findings to practice.
Search for Conclusive Evidence
Meta-analysis serves as an informative first pass, but should not be the final destination. National studies that systematically measure effects of technology engagement on learning over time, such as the UK’s CIBER report on the “Google Generation’s” information use or “UK Children Go Online,” provide strong examples for large-scale evaluation of the effects of technology on learning. The results of the Department of Education’s meta-analysis should not result in wholesale adoption of online curricula, but instead inform future large-scale comparative studies of online education.
