February 2nd, 2010
My dissertation is now available online, thus increasing the chance that more than three people will read it. Here is the abstract:
In university settings, students are increasingly required to conduct online research to complete course-related assignments, yet often receive little instruction in the skills necessary to proficiently locate, evaluate, and use the information they find. By comparing the processes of 150 graduate and undergraduate students during a 50 minute online academic research task, this study examined the role of prior knowledge and cognitive processing in proficient online literacy practice. The findings of this study challenge the assumption that technology alone is all that is needed to effectively complete online academic research. Results of this research indicate that students who bring academic experience to an online academic research task are more likely to succeed than those with technical expertise alone. Furthermore, analyses of students’ cognitive processes yielded insight into online literacy proficiency, defined as the ability to select sources relevant to the research task, synthesize multiple perspectives to build understanding, and effectively communicate that understanding. While certainly requiring medium-specific adaptations, online literacy is not very different from offline literacy. Without the essential literacy skills of gauging credibility and synthesizing materials to form and communicate an understanding, the ease of information access afforded by the online environment does not matter. Findings from this research additionally show that deliberate practice afforded through years of schooling more significantly contributes to online literacy proficiency than short-term instruction.
Further, this research presents and tests a cognitive process model for online literacy proficiency. The model illustrates the interrelated cognitive processes of online literacy while additionally demonstrating the significant contributions of expertise to proficiency. While the scope of this study is limited to college students completing an academic task, the model has implications for other online literacy practices.
Share on Facebook
March 31st, 2008
Online literacy is currently a moving target. Whenever a new online communication tool emerges, it seems that those who study it attempt to re-define literacy practice in terms of the new tool’s features…for example, engagement, participation, dynamic content, data filtering, hyperlinks. True, a moving target is hard to study, but if we know a little about targets and a little about motion, we have a starting point. When I first started my study, I tried to compose a definition of literacy, based on the literature I read. I cast a broad net and studied perspectives from library science, communication, education, educational technology, composition, media studies, computer science, psychology, art, new media, sociology, even geography. What I realized is there are things we know about literacy and how to study it that transcend the medium. We know that learning can be measured using established retention and transfer tests. We also know that literacy practices can be studied using think aloud protocols and writing assessments. While these methods are neither infallible nor comprehensive, they provide a starting point to anchor the target and to understand what it is we’re aiming at. The next step is to test these measures in preliminary experiments by first narrowing the type of literacy practice we wish to study and then leveraging the combined strength of quantitative and qualitative approaches.
Share on Facebook
February 14th, 2008
The Trouble with Information: How students gather and evaluate online resources.
Intro: Increasingly, university students are required to find and synthesize online resources to complete academic assignments. I’m interested in studying the process students use to complete these assignments…where do they start, what are their priorities, where do they go and how much time do they spend on the search process versus the composing process?
To answer these questions, I compared undergraduate and graduate student performance on a writing task that required them to gather information online and briefly respond to a writing prompt.
Drawing on Flowers and Hayes’ (1981) cognitive model of composition, I am studying the gather, evaluation, and integration processes involved in writing academic texts. I’m using an expert-novice comparison to get at differences in source use (Wineburg, 1991). To cast a broad net, I’m using a combination of established qualitative (Coiro, 2007) and quantitative (Azevedo & Cromley, 2004; Brand-Gruwel, et al., 2005; Holscher & Strube, 2000; Lazonder, 2000; Metzger, Flanagin, & Zwarun, 2003) models of studying online literacy practices. I started my study with two main questions:
1) how do experts and novices differ in their overall process when engaging in an online academic research task?
2) which, if any, of these practices predict the quality of the final product?
I define experts and novices by years spent in school. Ideally, I would like to compare faculty and/or researchers with undergraduates, but for this study, my experts are pre-service teachers (graduates) and my novices are first-week freshman (undergraduates).
I use a mediational model that first examines how the two groups differ and then identifies predictors of high performance.
Method:
data collected during Fall quarter 2007
154 participants
65 experts (TEP students enrolled in Copeland’s “Teaching with Technology” course)
89 novices (first quarter freshmen enrolled in Writing 2 and 2E courses)
Procedure and materials:
Held in Phelps computer labs, each session lasted 70 minutes
Participants were first given pre-questionnaire that included questions about domain knowledge and interest, technical skills, and general demographic information. [show sample]
Participants were then given a prompt and told they had 50 minutes to write a 1-2 page response using information they found online. [show prompt]
During the gathering and composing phase, students were told when they had 30 and 10 minutes remaining.
After students submitted their work, they completed a post-questionnaire which included questions about their process (which sites they used, how they evaluate credibility) as well as follow-up questions about domain knowledge and interest. [show example]
Once students left, log files were collected from each computer. [show example] Log files included information about computer actions: how many sites they visited, how many times they revised their search term, how many times they returned to a site, how many links they followed within sites, and keystrokes (e.g., text entries and copy/paste).
Analysis:
Developed rubric to score student written responses. [show rubric] The main challenge was figuring out how to measure use of source materials, specifically, how to measure critical engagement with these materials. Used a combination of counting (quantitative) and holistic (qualitative) scoring.
Findings in progress
I am in the process of analyzing my data. What do you think will be a difference in the way experts and novices begin their search? My hope was that experts would use a database or at least Google Scholar or Eric Digests to begin their search. My preliminary results show that 82% of the expert participants and 72% of the novices started with Google. Less than five participants in each group started with Wikipedia, Dogpile, or ask. One expert started with Eric and one novice started with Google Scholar.
Furthermore, as I start the paper scoring, I’m finding that a majority of participants in both groups used the top five Google results for their first or second search terms. At first, this finding was a bit disheartening, but then I found that the preliminary differences seem to lie in how each group uses the source. For example, experts tend to consider the implications of the source material rather than simply inserting it into their texts.
Participants in both groups use personal experience and observation, but differ in the ways they use it — a few of the experts evaluate their experience in terms of other source materials, while most of the novices use personal experience/observation to make unsupported generalizations.
Future directions:
–Complete data analysis
–Conduct textual analysis using Pairwise to quantify degree of re-mixing (Jenkins, 2007) occurring in student academic texts (compare student texts with web pages they visited).
–Study usage patterns in non-academic settings: task-specific and recreational browsing among different age groups.
Share on Facebook
February 5th, 2008
When I started my dissertation research, I struggled to describe what I was doing. Was I studying online reading, digital literacy, information processing? While the exact phrasing is still up for grabs, I understand the process, the experience I’m studying. I’m studying how students use online resources to complete academic assignments. Specifically, I’m trying to measure the cognitive processes involved in gathering, evaluating, and integrating online sources to compose academic texts. This first step is part of a larger agenda. I am starting with students in a relatively controlled environment to develop baseline measures to later study how this process plays out in other scenarios, such as task-oriented searches or recreational browsing. I’m using a cognitive science model to identify differences in the processes of domain expert and novice students’ approaches to this task. However, this model is just the starting point. I am also blending qualitative methods of textual analysis from the fields of Education and English to look more deeply at the practices behind the process. Specifically, I’m interested in quantifying Jenkins’ theories of re-mixing. In Convergence Culture, Jenkins (2007) describes re-mixing as a process by which users blend a variety of media from multiple sources to create their own work. Basically, this process is what new media is: combining media from multiple sources and modes to create a new whole. Certainly, new technologies fit into this paradigm, but more importantly, how the users appropriate technologies to suit their needs is what interests me. While media theories can certainly describe trends in use, cognitive science provides an actual method of applying and testing these theories.
Share on Facebook
January 31st, 2008
At this point, my blog is likely a case of a tree falling in the woods, but I have to start somewhere. My interests are in the pedagogic implications of educational technologies; simply put, I’m interested in identifying instructionally sound methods of using technologies in the classroom.
Why? I spent three years as a Lecturer in the UCSB Writing Program, bookended by one-year stints as a TA. I was recruited because of my technical writing background to teach the Engineering Writing sequence (three-quarter curriculum specifically designed for first-year engineering freshman). At the time, my colleagues had to reapply for their jobs every year until they reached their sixth year: if accepted for “tenure,” they had to reapply every three years. They were under intense pressure to incorporate technology into their instruction. The administrators didn’t really understand the technologies, so there were no models of effective pedagogical use. Thus, I observed many classes where the instructor simply held their lecture-based course in a computer lab and used the board or an overhead projector to post notes. Overwhelming research on the pitfalls of lecture-based instruction in writing courses aside, this practice placed students in front of a highly seductive distraction, and most couldn’t resist checking e-mail or visiting websites.
When I’d argue against this practice in faculty meetings, I got blank stares. In the late 90’s, educational uses of technology were a bit fuzzy. I knew I’d need solid research to back my claims if I wanted to effect change.
I believed that technology was simply a tool and that we should make it fit within our pedagogical goals and not vice-versa. I attended conferences in my field that touted flashy best practices of flash animation creation and website use. While I saw the need for students to be able to communicate with new media, I also saw the content of student writing deteriorating. Sure, the design looked good, but the students couldn’t write. So, what next? We needed a way to better understand how to integrate technology into our instruction in a pedagogically sound way: we needed to determine whether we could use technology to improve learning outcomes.
I took a class with Richard Mayer in 2004 and his research examined exactly the same issue. With a focus on student understanding as the ultimate learning outcome, Mayer’s research considers how multi-media affects student learning retention and transfer.
His research provided a foundation and framework for me to explore issues of educational technology use in writing courses.
In the past four years, I have studied student engagement, instructor engagement, learner expertise, and, currently, effects of delivery modes on learner comprehension. While in each of these studies, I’ve encountered best practice examples, my goal is to build an empirically-based understanding of effective uses of technology in the classroom. At the same time, I’ve been involved in teacher training through the South Coast Writing Project, so I’ve been able to apply this research to training local K-College teachers in effective uses of technology for writing instruction.
Stay tuned…
Share on Facebook
