Making digital exemplars

In addition to my usual classroom use of exemplars as a means of familiarising students with the assessment requirements of a specific module, this year I have created a video walk through of an exemplar. Initially this was to enable those who missed the relevant class to catch up on the session, but the approach was welcomed by students who attended the exemplars activity session, as well as those who did not. 

How to create a digital exemplar walk through: 

• Bring up the exemplar on screen after selecting a ‘good’ piece of work

• Read through and use comments in word to annotate the work with points which would be surfaced in feedback, particularly comments related to the assessment criteria (of course!). Comments include things done well, things done less well which could be avoided and opportunities for further detail and development. This tagging process acts only as an aide memoire so that as I create  feedback video I am aware of what I wanted to include. 

• Open Screencast-o-Matic to very easily screen record the work as a video as I re-read it through and talk to each of the tags. ‘This work includes this … which is useful because…’ ‘this work used literature in this way …. It might be better to …. Because ….’. None of this is rehearsed; that would be too time consuming. The resultant video is a commentary on performance.

• The video is uploaded and made available to students.

After using the resource there was some consensus amongst my students that the value was ONLY in listening to the assessment commentary and not specifically in looking at the work. One student described how they listened but did not watch. They then recorded notes about what they should include, remember and avoid. They avoided looking at the work for fear of having their own ideas reshaped. If assessment judgments are ‘socially situated interpretive act[s]’ then the digitised marking commentary may be a useful way of making that process more transparent for students, and indeed for other staff.

I will definitely be including this in future modules.

Handley, K., den Outer, B. & Price, B. (2013) Learning to mark: exemplars, dialogue and participation in assessment communities. Higher Education Research & Development Vol. 32 , Iss. 6.

Using technology for student feedback: Lecturer perspectives. In their words

The document posted is a collection of short narrative portraits that has been constructed during my doctoral research, titled, ‘Using technology for student feedback: Lecturer perspectives’. Within the study, fifteen participants were interviewed. Each told their story of how and why they used technology in feedback. This illuminated challenges in the development of academic practice, it uncovered some of the ways in which feedback practice is formed, and it showed some of the ways in which lecturers internally mediate technology selection.

Individual interview transcripts were reduced to portraits (essentially these are mini accounts). This was done using a systematic and reflexive process articulated by Seidman (2013). The portraits themselves, and the process of data reduction, provided learning which fed in to the wider analytical process. These portrait stories are not all included in the final thesis in their full form, however given that narratives can provide instant knowledge (Webster & Mertova, 2007) I wanted to publish the collection. The participant portraits are presented here because they stand alone as insights in to the formation of academic practice.

DOWNLOAD Participant stories – in their words

Seidman, I. (2013). Interviewing as qualitative research: A guide for researchers in education and the social sciences (4th ed.). New York: Teachers College Press.

Webster, L., & Mertova, P. (2007). Using narrative inquiry as a research method: An introduction to using critical event narrative analysis in research on learning and teaching. Abingdon: Routledge.

Efficiency, technology and feedback

In considering staff experiences of choosing and using feedback technology, one of the emerging themes has been the differing views on feedback technologies and efficiencies. While the jury is still out on the data and the process is incomplete, my observations are that efficiency can be conceived in different ways in the negotiation of technology. For some efficiency is a primary driver in the decision making process. The search for technology and the refinement of its use is motivated and shaped by the quest for efficiencies. For others efficiencies are a welcome benefit of technology – they are almost an unexpected gift – welcome, but not necessary. Efficiencies also appear to be conceived relatively; rarely are efficiencies discussed without a reference to the relative enhancement gains that can be made through a technology. Wherever there is a time saving there is a tendency to ‘re-spend’ the saved time making still more enhancements to the feedback – adding detail and depth for example. In this way efficiencies become difficult to identify as they are theoretically achievable but in reality they are trumped by the possibility for improvement. Efficiency also seems to be a veto concept for some; it is not a particular concern in the run of practice but is triggered only when a particular technology is likely to encroach other activities or provide an intolerable stress.

Thesis research: Are you using technology to provide student feedback?

My ‘work in progress’ thesis for my doctoral studies at University of Liverpool is entitled Faculty experiences of feedback technology: A critical realist perspective. I have had a personal interest in feedback technology for some time, and through a process of practice based research and reviews of the literature it became clear that the lecturer or faculty voice is under-represented. Often feedback technology appears to be evaluated in terms of the student benefit rather than the experience of staff engagement. While there is a lot going on in the sector about making digital forms of feedback systematic, I was keen to discover what happens when staff have choice about the technology employed in feedback:

My research is therefore asking

  • What influences the choice and use of technologies for feedback?
  • What are the reflective processes through which practices develop?
  • What is the impact of faculty engagement with feedback technology?

Beyond answering these questions the research also seeks to shed light on beliefs about feedback and faculty relationships with technology for pedagogic practice.

So far I have undertaken ten narrative style interviews where staff engaged with a range of technologies (inc. Jing, GradeMark, Dragon, Pebblepad, Track Changes, audio) have shared their motivations for engagement, some of the barriers to practice, and some of their underpinning beliefs about both feedback and technology. They have shared the deliberative process about how technologies were chosen and they have shed light on the institutional factors which shape practice.

The critical realist approach to analysis (particularly using the work of Margaret Archer as a theoretical framework) is particularly revealing the compexity of interaction between individuals,  institutions and the wider environment.

 I will release parts of the research on this blog as they develop, but I am hoping to interview a few more individuals who are involved in using technology for feedback within UK HEI’s (and will offer a token voucher as a token of appreciation). If anyone can help I would be very pleased to hear from you at larnold@liverpool.ac.uk and I can provide more details.

Jing-tastic; Audio visual tool

During a summer of local and international educational development workshops ‘Jing’ has had many outings. I was struck to see how this really simple facility never fails to make people say – “wow, I can really use that”. It’s a low ceiling technology with transformative potential. From my summer ‘tour’ here are some thoughts on how Jing can be used productively by those involved in teaching and supporting learning.

Formative feedback – an assignment walk through
As described here and also by Russell Stannard , Jing can be used to offer feedback on actual assignment work by enabling a visual-voice combination to be used. Feedback can be given and related to the assignment on screen. Early signs are that this approach is widely enjoyed by students who particularly value the ability to play and replay the feedback; the personal tones of the feedback; and, the privacy and convenience of getting the feedback in a location that suits them.

Best bits and “no no’s”(one to many feedback)
To feed forward and enable one group to learn from another, Jing can be a way of presenting good practice and things to avoid. This needs a little care to avoid showing individuals up, but with careful doctoring any ethical issues can be avoided! This can be used as a group feedback method, and can be a useful interim form of feedback when individual comments can’t be provided in time to be useful for the next assessment. Such videos can be added to the VLE or sent direct to students.

Correction
In addition to providing feedback, Jing can assist with directly facilitating corrections. This, I find, is particularly helpful with very specific and detailed tasks. The visual element can help enable the recipient to use tools to make future changes. An example would be a student who has issues with alignment being shown how to use the facility in Word, which shows the spaces and tab marks. In discussion with colleagues I am advised that the same principles may carry to correction of language or sentence structure.

Peer feedback
Lots of attention s being given to teacher led Jing feedback, but this is freeware and as a result can be easily utilised for students giving peer to peer feedback. This might even help with communication skills and confidence.

Summative feedback – a tour of the mark sheet
Students have fed back their desire to know how marks have been allocated. One way this can be brought about is through the use of Jing to discuss the mark itself; perhaps by the tutor talking through the feedback sheet, one section or outcome at a time. In this way Jing can be a useful complementary technology.

A reflective tool for students (an audio layer in the battle against plagiarism)
One of the ways we can mitigate plagiarism, and encourage learners to reflect on their learning processes, is through the inclusion of an annotated bibliography in any assignment. As an alternative, perhaps catering for different styles and preferences, students could review their own assignment and create a walk through of any difficult points, any areas that they feel could be improved and any things they would do different in future. They could also comment on how they found particular readings cited in their work.

Recalling assumptions (project management tool)
As part of my role is project management, Jing also helps with remembering what we did and why. A two minute voice over on a spreadsheet means that when we go back and think how on earth did we arrive at x, y or z, that we have the detail captured from the moment. Jing is now, therefore, becoming a favourite of accountants and data managers as well as teachers!

Aspiring to meaningful feedback

Having explored a number of Jisc and independent feedback projects on assessment and feedback I have selected ten points which emerge as being critical for making feedback work.  

  1. Timeliness is critical if feedback is to be useful; consider whether quality management systems are blocking the use of feedback.  4 weeks is too slow!
  2. Criteria are important but they can also promote tactical learning as students learn to be selectively negligent (Gibbs, 2012). Use criteria but don’t be locked in by them.    
  3. Curriculum design is important. Too many assessments and too many small modules do not encourage deep engagement, and the associated assessment does not capture sufficient study time so as to be meaningful.
  4. Feedback should feed forward. It is important that staff are able to know something about what comes next for each student. Without any understanding the opportunity to be purposeful in feedback is severely limited.
  5. Little and often is better than more and infrequent.
  6. There is only so much individual tutors can do – there is a need to consider programmes and course suites.
  7. Feedback that refers to material that will not be studied again tends to be ignored.
  8. Where feedback stimulates dialogue about learning, the feedback is perceived as being more useful; this is where feedback crosses in to the line of personal development and students taking control.
  9. Reflection on feedback makes learner more autonomous. It reduces dependence on the teacher as giver and makes the student become an active part of the feedback process.
  10. Assessment diversity is good, but too much renders the feed forward process meaningless. Limit variety so progression is meaningful.

 

The Jing feedback experiment

Since the last post on Jing (screen capture) I have tried it out more intensively by making 45 videos for formative feedback on personal development. I received draft submissions from students, opened them on the screen, started the video capture and recorded as I went.

Lessons learnt …

  • Read through once only and highlight in yellow any areas where a comment should be made (a higher level of scripting than that means you may as well write the feedback first )
  • Live with imperfection. Unless you edit the feedback in an audio editor, Jing is one take only. Live with the odd, ‘errr…..’ … pause or stumble or else the videos will take a ridiculous amount of time.
  • Manage expectations: Jing feedback was sought once word got around, this created a rush at the last minute. For the sake of workload give cut offs, and only feedback on a pre-determined amount of work.
  • Opt out not in. Given the openness of feedback, being technically accessible by others and given the alternative nature of the approach brief students and tell them what you are doing and why, and offer an opt out. No-one chose this.
  • Practice makes efficient. The first handful of videos took forever. Had I not made a public commitment to do this I would have ditched it out of sheer frustration. It did get better.
  • Using other types of video in class meant that this was a familiar approach to students. It was in synch with classroom methods. For example, I used video feedback to playback a critique of a case study.
  • It saved an awful amount of time by removing the need for proofing my own feedback.

While it may seem labour intensive to offer 45 verbal feedbacks I was secure in the knowledge that 45 written feedback attempts would take an awful lot longer. The depth of the feedback was also more than could have been realistically achieved on paper. You can say a lot in 5 minutes.

What did the students think …

  • Students thought this was fantastic!
  • ‘Like a conversation’
  • Personalised
  • ‘It was like having a one to one tutorial’
  • Enabled them to work through changes one at a time with the video open and their work open at the same time
  • Only one technical glitch was reported
  • Lots of feedback is possible in this way

Other Jing ideas…

An alternative approach I saw recently was a tutor talking through the grade sheet. Giving a verbal commentary on why decisions were made as they were. A different take on Jing.

As a spin off from this work, experimentation shows Jing can work well with White Board technology too, so that in-class examples can be used and taken away. A blue tooth mic and you’re away …

(How to make a Jing feedback video is outlined here http://www.techsmith.com/education-tutorial-feedback-jing.html )