Preliminary findings – Lecturer experiences of choosing and using technology in assessment feedback

Part way through my thesis research I have stood back to ask what is all this data saying? To this end I have produced a pause for thought document about the emerging findings. This is not the finished output, but in creating it I managed to consolidate my thoughts, and in sharing it I hope for any comments that may help refine further analysis or additional data collection.

Please feel free to leave any questions or comments in response.

And thank you to those who have assisted me in setting up the next round of interviews after a recent plea for help.

Preliminary Research Findings available here.

 

Marrying mixed methods and critical realism

As  I edit through my thesis I needed to cut out a short section that I had written about how mixed methods can be highly compatible with critical realism. As my work emerged I did not, in the end, use mixed methods.  This section was hard won and so simply deleting it felt like a waste. Therefore, attached to this post is a short piece about the compatibility of mixed methods and critical realism, in hope that it may be useful to someone else.

Download the short article here: Marrying mixed methods and critical realism

Efficiency, technology and feedback

In considering staff experiences of choosing and using feedback technology, one of the emerging themes has been the differing views on feedback technologies and efficiencies. While the jury is still out on the data and the process is incomplete, my observations are that efficiency can be conceived in different ways in the negotiation of technology. For some efficiency is a primary driver in the decision making process. The search for technology and the refinement of its use is motivated and shaped by the quest for efficiencies. For others efficiencies are a welcome benefit of technology – they are almost an unexpected gift – welcome, but not necessary. Efficiencies also appear to be conceived relatively; rarely are efficiencies discussed without a reference to the relative enhancement gains that can be made through a technology. Wherever there is a time saving there is a tendency to ‘re-spend’ the saved time making still more enhancements to the feedback – adding detail and depth for example. In this way efficiencies become difficult to identify as they are theoretically achievable but in reality they are trumped by the possibility for improvement. Efficiency also seems to be a veto concept for some; it is not a particular concern in the run of practice but is triggered only when a particular technology is likely to encroach other activities or provide an intolerable stress.

Thesis research: Are you using technology to provide student feedback?

My ‘work in progress’ thesis for my doctoral studies at University of Liverpool is entitled Faculty experiences of feedback technology: A critical realist perspective. I have had a personal interest in feedback technology for some time, and through a process of practice based research and reviews of the literature it became clear that the lecturer or faculty voice is under-represented. Often feedback technology appears to be evaluated in terms of the student benefit rather than the experience of staff engagement. While there is a lot going on in the sector about making digital forms of feedback systematic, I was keen to discover what happens when staff have choice about the technology employed in feedback:

My research is therefore asking

  • What influences the choice and use of technologies for feedback?
  • What are the reflective processes through which practices develop?
  • What is the impact of faculty engagement with feedback technology?

Beyond answering these questions the research also seeks to shed light on beliefs about feedback and faculty relationships with technology for pedagogic practice.

So far I have undertaken ten narrative style interviews where staff engaged with a range of technologies (inc. Jing, GradeMark, Dragon, Pebblepad, Track Changes, audio) have shared their motivations for engagement, some of the barriers to practice, and some of their underpinning beliefs about both feedback and technology. They have shared the deliberative process about how technologies were chosen and they have shed light on the institutional factors which shape practice.

The critical realist approach to analysis (particularly using the work of Margaret Archer as a theoretical framework) is particularly revealing the compexity of interaction between individuals,  institutions and the wider environment.

 I will release parts of the research on this blog as they develop, but I am hoping to interview a few more individuals who are involved in using technology for feedback within UK HEI’s (and will offer a token voucher as a token of appreciation). If anyone can help I would be very pleased to hear from you at larnold@liverpool.ac.uk and I can provide more details.

Visit to Huazhong, Wuhan

Here’s a selection of pictures from our recent trip to Wuhan, China, where we went to share and learn about different teaching approaches in HE.

Active Lectures

After attending a workshop on Smarter Lectures with Phil Race I went away and tried to gather together some resources which gave practical lists of tips and tricks to use in live, large group teaching situations. While I rarely offer large lectures, the principles of engagement are no different for large groups than they are for small – summarised as participative, active, and focussed on learning not on teaching and with a requirement for feedback, however the tools available to us to bring these principles to life may be different. I have attempted to gather up some practical reference points to help.

Quick reference activities for lectures (from Sussex University)

Four simple class structures for planning active lectures

Ten ways to work with millennial students and defeat lecture-induced mind paralysis by Brad Garner

The value of gapped handouts in science and content rich classes

Six ways to make lectures in a large enrollment course more manageable and effective

As a precondition to make these approaches work though it seems that an internal shift is needed – this is captured quite well in this short YouTube clip. It’s quite powerful when an educator says they wouldn’t have wanted their own child to be educated in the way that she was educating others. Paradigm shift in action!

 

 

 

Conditions for employer engagement

Reflecting upon what makes a university operationally ready to engage with employers in collaborative provision, I have pulled together a list of “good conditions for employer engagement“. Its purpose is simply to provide prompts  for discussion about how institutions can prepare to engage with employers to meet the development needs of the workforce.  This list assumes that there is already a strategic level commitment in place and it is of course not exhaustive.

Creating case studies for the HEA Senior Fellow Application

I have been involved in informally supporting a number of HEA Senior Fellowship applicants.  I thought it helpful to share my thoughts and feedback on their draft case studies, in hope that it might also be useful for others.

Common issues have resulted in the following advice:

Consider the impact of each case with evidence – so for example if someone is describing their pro-active and innovative approaches to feedback, it is necessary to say how they know their approach is working well.

• Consider the impact of the case on your own values and attitudes –  as a result of involvement in the case did you rethink any assumptions about your role or your approach to supporting learning? By example, the person using innovative approaches to feedback as their case might say that student engagement led them to change their assumptions about students’ expectations; they discovered their students valued less feedback in a timely way, more than they valued extensive feedback which takes longer to deliver, in turn this led them to adjust their feedback practice.

• Consider different types of impact. The impact of the case may be on students, colleagues, the institution and its policies/systems, the discipline community or network or the sector as a whole. A Senior Fellowship application should really include evidence of institutional impact.

• Give detail about individual contributions when working on joint projects.

• Give detail about the specific context; sometimes there is a need to make the institutional and discipline context more explicit. Sticking with the feedback example, there might be a need to describe how the practical emphasis of the curriculum acts as an influence on the approach taken to feedback.

People approach the case studies in very different ways but I would recommend some kind of structure to ensure going beyond description. A proposed structure would be something similar to the following:

  • Describe the activity and an outline of the rationale
  • Consider what worked well and what not so well in practice
  • Identify what you learnt in involving in this activity and note any changes to your prior assumptions
  • Describe the impact but ensure that impact points to sources of evidence – student feedback, colleague feedback, or external feedback are all ok
  • Describe what the next steps might be to advance this area of practice even further.

This of course is just one way to tackle the task and I’m sure there are many more!

Jing-tastic; Audio visual tool

During a summer of local and international educational development workshops ‘Jing’ has had many outings. I was struck to see how this really simple facility never fails to make people say – “wow, I can really use that”. It’s a low ceiling technology with transformative potential. From my summer ‘tour’ here are some thoughts on how Jing can be used productively by those involved in teaching and supporting learning.

Formative feedback – an assignment walk through
As described here and also by Russell Stannard , Jing can be used to offer feedback on actual assignment work by enabling a visual-voice combination to be used. Feedback can be given and related to the assignment on screen. Early signs are that this approach is widely enjoyed by students who particularly value the ability to play and replay the feedback; the personal tones of the feedback; and, the privacy and convenience of getting the feedback in a location that suits them.

Best bits and “no no’s”(one to many feedback)
To feed forward and enable one group to learn from another, Jing can be a way of presenting good practice and things to avoid. This needs a little care to avoid showing individuals up, but with careful doctoring any ethical issues can be avoided! This can be used as a group feedback method, and can be a useful interim form of feedback when individual comments can’t be provided in time to be useful for the next assessment. Such videos can be added to the VLE or sent direct to students.

Correction
In addition to providing feedback, Jing can assist with directly facilitating corrections. This, I find, is particularly helpful with very specific and detailed tasks. The visual element can help enable the recipient to use tools to make future changes. An example would be a student who has issues with alignment being shown how to use the facility in Word, which shows the spaces and tab marks. In discussion with colleagues I am advised that the same principles may carry to correction of language or sentence structure.

Peer feedback
Lots of attention s being given to teacher led Jing feedback, but this is freeware and as a result can be easily utilised for students giving peer to peer feedback. This might even help with communication skills and confidence.

Summative feedback – a tour of the mark sheet
Students have fed back their desire to know how marks have been allocated. One way this can be brought about is through the use of Jing to discuss the mark itself; perhaps by the tutor talking through the feedback sheet, one section or outcome at a time. In this way Jing can be a useful complementary technology.

A reflective tool for students (an audio layer in the battle against plagiarism)
One of the ways we can mitigate plagiarism, and encourage learners to reflect on their learning processes, is through the inclusion of an annotated bibliography in any assignment. As an alternative, perhaps catering for different styles and preferences, students could review their own assignment and create a walk through of any difficult points, any areas that they feel could be improved and any things they would do different in future. They could also comment on how they found particular readings cited in their work.

Recalling assumptions (project management tool)
As part of my role is project management, Jing also helps with remembering what we did and why. A two minute voice over on a spreadsheet means that when we go back and think how on earth did we arrive at x, y or z, that we have the detail captured from the moment. Jing is now, therefore, becoming a favourite of accountants and data managers as well as teachers!

Empathetic validity and action research in educational development

As part of my doctoral studies I have recently undertaken an action research project relating to strengthening approaches to feedback practice. Informal reconnaissance led me to believe that feedback practice is very siloed. At the same time in the planning process I encountered a paper by Ball  (2009) who showed that collaborative practitioner centred action research  in itself can bring about the questioning of ones own practice, put simply, discussing the feedback practice of others shines a light on the way that each of us works and we then ask questions of ourselves and review how we might act differently.

Informed by this, my action hunch was that the development of an electronic sharing resource for good practice could be a mechanism for modelling good practice and bringing about transparency. Influenced by Ball, I envisaged that ensuring ownership of this resource by those who would use and populate it could act as a catalyst for critical dialogue around practice.

In seeking exemplary practice to populate the resource it became clear that there were some issues to address first. The project got messy in the way described by Cook (2009).  The intended action therefore was put on hold, and became a second project phase. This was clearly emergent work in progress.

The action research methodology permits these off-piste directions and in my search for good practice to populate the resource, I generated four spin off cycles to explore what is good feedback in this context? How can feedback practice be developed?  What are the barriers to developing good feedback practice? What conditions might be needed for the benefits of practitioner sharing to be realised?

I have take away learning about all of these points but by far the greatest learning from this research has been  the development of empathy with those engaged in the process. I have a much better sense of their experience and in staff development terms this is important for productive ways forward. My data was not vast and my conclusions didn’t add a lot to the already overflowing pool of literature on this topic, but it felt valuable. Trying to justify your research in terms couched in feelings is something that even I, as a self-confessed navel gazer, am not used to doing. In reading around this I was drawn to the work of  Dadds  who described a phenomenon called empathetic validity which refers to “the potential of the research in its processes and outcomes to transform the emotional dispositions of people towards each other, such that more positive feelings are created between them in the form of greater empathy” (2008, p208)

Whether empathy can really be incorporated a project aim I am not clear, I imagine it either happens or it doesn’t, but its benefit for me has trumped any of the the intended consequences of this project.

Ball, E. (2009). A participatory action research study on handwritten annotation feedback and its impact on staff and students. Systemic Practice and Action Research, 22(2), 111-124. doi: 10.1007/s11213-008-9116-6

Cook, T. (2009). The purpose of mess in action research: Building rigour though a messy turn. Educational Action Research, 17(2), 277-291. doi: 10.1080/09650790902914241

Dadds, M. (2008). Empathetic validity in practitioner research. Educational Action Research, 16(2), 279-290. doi: 10.1080/09650790802011973