This challenge in measuring impact was a main point of conversation on Friday when the OII hosted the judging panel for our first-time European Competition for Best Innovations in University Outreach and Public Engagement, supported by the European Commission as part of the ULab project. The panel consisted of experts in public engagement from the ULab partner countries. Judges represented a range of backgrounds, including funding officers, policymakers, and those engaged in award-winning outreach.

When reporting impact, the majority of entries cited audience size, whether measured by attendance, web traffic, web clicks, or distribution of printed materials. Yet, this reporting usually did not directly measure progress on the stated aims of the projects. For example, if a project aims to change perceptions, or increase engagement with science, simple attendance at a fair or visits to a website cannot show whether these objectives were met. Tracking whether the person then attends another science event, enrolls in a science class, or undertakes a degree in science may provide a stronger measure, but still the extent of influence of a particular event is unknown. Of course, the problem is that indicators of input are more easily and economically gathered, than of impact, which are costly and an additional activity, which may not yield definitive answers.

However, some entries used participant surveys to gather more details about impact and experience. These ranged in depth from like/dislike buttons to short-answer questions about learning. Some projects engaged their participants in focus group discussions aimed at improving their offerings and better meeting the needs of their target audience. A few looked at impacts over time, showing shifts in community engagement, or increases in student enrollment.

The question of depth versus breadth of impact was debated among our judging panel, showing that even experts have diverse opinions of what constitutes models of good practice in public engagement. We found that entries from a number of European countries focused on the numbers of people reached as a measure of success. For example, events such as the Researcher’s Nights hosted throughout Europe aim to engage large numbers of community members in short events and therefore measure success by attendance levels. By way of contrast, several entries from the UK place an emphasis on depth of engagement. In other words, a project may have less than 20 participants, but have weeks or months of interaction that leads to a change in behavior, for example, a demonstrated interest in science, or a better understanding of a different social perspective. Some judges placed more emphasis on breadth, while others focused on depth.

Overall, even in top-ranked entries, the judges found that impact measures were not always ideal or fully developed and recommended that the projects should consider developing more evidence of impact. One of the winning entries, Active Science, demonstrated a multi-faceted approach to impact measures. In addition to reporting on audience attendance, the programme also demonstrated its success through tracing its expansion from a local to nation-wide project. To measure whether the project was successful in shifting perceptions about science, participants were given pre- and post-activity questionnaires that measured their attitudinal approaches.

Matching clearly defined objectives to appropriate measures may improve reporting and evaluation of impact. The wealth of activities and approaches reported in the EngageU entries provide a repository for strong examples of impact measures as well as a means of comparisons for how projects with similar objectives measure their impact.

Of course, you may have very different views on the centrality of impact assessment for outreach. We would appreciate hearing from you.

Share on Facebook

To organize ourselves, we are focusing on empirical studies to tease out how and whether harm is operationalized, looking for documented incidents of harm or measures used by practitioners to identify harm. Early on, we realized that interviewing experts in law, law enforcement, psychology, social work, education, and outreach, would lessen our chances of missing key work in the field. In addition, our expert contacts provide context and overviews as we move across different disciplines and perspectives.

Our work has taken us into the areas of cyberbullying, self-harm, and sexual abuse, certainly the darker side of the Internet. We are also exploring current legislation and interventions in several countries to get a sense of successes and challenges in approaching this issue.

At this point, we are 3/4 of the way through our large research corpus and look forward to sharing early observations very soon.

Share on Facebook

Share on Facebook

Cathy Davidson recently described an idyllic experience of teaching a course in which she and the students shared in the discovery of new applications of technologies for learning. She describes the process of developing the course, the thrill when the students actually invited and facilitated a guest lecture, and the ways in which the students challenged her to really be collaborative, even in grading.

If we step back for a moment, though, and consider a class with Davidson and those same students without the new technologies, what would the learning experience be like? I imagine it would still be exceptional, because Davidson is an obviously engaged teacher and the students are obviously engaged learners. She employs teaching strategies that were effective before the new technologies she describes. In particular, she encourages students to take ownership of their learning experience and creates a flexible environment to support whatever direction they take. When developing assignments, Davidson incorporates research in motivation, particularly students’ likelihood to put more effort into writing for an authentic audience. She also has deep experience with her topic and an obvious enthusiasm for both the content and the teaching. These factors are consistently linked to positive learning experiences in educational research. Additionally, the students clearly seem motivated to learn. She describes the class list as a diverse collection of disciplines, so the students appear to be choosing the course. They demonstrate active involvement with the assignments and content and even provide substantive feedback for future courses.

Davidson’s approach to her class corresponds with much of the research on good teaching. Now, if we imagine the same syllabus and same access to technologies, but with a different teacher, what happens? The course might still be exceptional, or it might not.

A common theme when addressing technology in education is a focus on the particular technology and the success or failure of its use. In David Risher’s recent article about educational technologies in developing countries, he urges consideration of ‘the triangle that connects students, teachers, and ideas.’ The way in which a teacher incorporates a technology, designs the learning environment, and promotes learning determines the ultimate effectiveness of the technology. Returning to Davidson’s example, the students are described as knowing they can take risks through the support and encouragement she provides. The technologies are secondary to the empowering environment she creates. In other hands, the students may have been focused on their screens, updating their Facebook profiles while the teacher lectured at the front of the room…a forgettable experience.

When describing the participatory culture new technologies afford, equally important is the teacher who brings these tools into the classroom — the tool merely plays a supporting role.

Share on Facebook

The German universities’ decision to accept the funding highlights a global struggle for universities generally. As governments continue to drastically cut budgets of higher education institutions, administrators are forced to make tough decisions that involve seeking external funding and/or making programmatic reductions. For example, the University of California system, after facing nearly a decade of budgetary shortfall was hit this month with additional deep cuts. The university system in California has already made deep reductions and is now asked to cut further. At what point do these decreases in government funding threaten the fabric of the university? Perhaps they already do. So, how do universities maintain their academic integrity and continue to pursue and preserve their research and teaching agendas with continually decreasing resources?

What if the collaboration involving Deutsche Bank becomes a standard model for survival?

Share on Facebook

In this presentation to our Summer Doctoral Program, I use Mayer’s (2001) multimedia learning theory to promote more meaningful (and memorable) presentations. I encourage our visiting students to consider the human element of the presenter-audience relationship by thinking about audience need and how to best communicate their message.

Share on Facebook

Google’s Big Tent Event

I wonder what could be possible if teachers had classrooms that functioned as well as Google’s Big Tent? If teachers had the technical and administrative support that benefited yesterday’s speakers, how could students’ learning experiences be improved?

Do the quality of bathrooms reflect the health of an institution? MP Jeremy Hunt may say yes. In his address to the Big Tent group yesterday, Jeremy Hunt drew connections between the vision demonstrated when developing London’s sewage system to current efforts to improve broadband infrastructure in the UK. The secret is the size of the pipes, apparently, and larger ones will ensure preparation for future data demands. Mr. Hunt used South Korea, who is #1 in OECD’s educational rankings, as an example of the success possible with super-fast broadband. While discussing funding models for the project, including private support, Mr. Hunt did not address what would seem an obvious part of the equation: education funding. UK schools have been hard hit by recent budget cuts, including canceling the Building Schools for the Future Scheme and recent £155m additional cuts in the standards fund. If the education rankings of South Korea are serving as justification for investing in infrastructure for faster broadband, it would seem that simultaneously cutting funding for education serves cross-purposes.

While much public debate surrounds the quality of education, often solely blaming teachers, the quality of educational environments, including support at the technical, facilities, and administrative levels need more attention. Let’s use Google’s Big Tent, rather than makeshift shelters, as a model for classrooms and start directing funding to supporting students and their teachers, rather than forcing them to make due without.

Share on Facebook

The scholars we interviewed described the tradition of collaboration within their respective disciplines. Unlike the sciences, in which research frequently involves large teams and multi-authored articles, collaboration in the humanities is more nuanced. One of our case studies, The Digital Republic of Letters, traces correspondences during the Enlightenment. These correspondences include letters from Descartes, Van Gogh, and Grotius, among others. The centuries-old collaboration methods examined by this group underlie current practice. Then, letters sent back and forth reported, unpacked, tested, and developed theories. Sound familiar? The description could easily be applied to e-mail, seminars, conference presentations, or hallway discussions. Research then and now begins with the sharing of ideas.

While not overtly collaborative in the scientific practice of the term, humanities scholars engage in research that “is done in conversation.” In addition to the above examples, scholars engage this conversation through their work in archives, when they prepare materials to be digitally accessed, when they report on rare materials, making previously obscure knowledge available to a larger public. They support each other in their work by talking through ideas and texts, presenting preliminary ideas that later become papers or monographs. Primarily, their research practices are source-intensive, but the sense-making process is very much accomplished in community.

Share on Facebook

For more information, I recommend Nancy Duarte’s slide:ology and Garr Reynold’s Presentation Zen.

Share on Facebook

Abstract: In university settings, students are increasingly required to conduct online research to complete course-related assignments, yet often receive little instruction in the skills necessary to proficiently locate, evaluate, and use the information they find. By comparing the processes of 150 graduate and undergraduate students during a 50-minute course-related Internet research and writing task, this study examined the roles of prior knowledge and cognitive processing in digital literacy practice. Framed within an expert-novice comparative design, this research combined qualitative and quantitative measures including questionnaire, behavioral analysis (log file data), and content analysis (search terms, URLs, and essays). Outcomes were measured by demonstration of synthesis, comprehension, and cohesion in students’ resulting essays. Results show that students who bring greater academic experience to a course-related Internet research task are more likely to succeed than those with technical expertise alone. Analysis of students’ cognitive processes show that deliberate practice afforded through years of schooling contributes to digital literacy more significantly than short-term instruction. The findings of this study challenge the assumption that ease of access to information afforded by the Internet equals skill in using information.

Share on Facebook