Empathetic validity and action research in educational development

As part of my doctoral studies I have recently undertaken an action research project relating to strengthening approaches to feedback practice. Informal reconnaissance led me to believe that feedback practice is very siloed. At the same time in the planning process I encountered a paper by Ball  (2009) who showed that collaborative practitioner centred action research  in itself can bring about the questioning of ones own practice, put simply, discussing the feedback practice of others shines a light on the way that each of us works and we then ask questions of ourselves and review how we might act differently.

Informed by this, my action hunch was that the development of an electronic sharing resource for good practice could be a mechanism for modelling good practice and bringing about transparency. Influenced by Ball, I envisaged that ensuring ownership of this resource by those who would use and populate it could act as a catalyst for critical dialogue around practice.

In seeking exemplary practice to populate the resource it became clear that there were some issues to address first. The project got messy in the way described by Cook (2009).  The intended action therefore was put on hold, and became a second project phase. This was clearly emergent work in progress.

The action research methodology permits these off-piste directions and in my search for good practice to populate the resource, I generated four spin off cycles to explore what is good feedback in this context? How can feedback practice be developed?  What are the barriers to developing good feedback practice? What conditions might be needed for the benefits of practitioner sharing to be realised?

I have take away learning about all of these points but by far the greatest learning from this research has been  the development of empathy with those engaged in the process. I have a much better sense of their experience and in staff development terms this is important for productive ways forward. My data was not vast and my conclusions didn’t add a lot to the already overflowing pool of literature on this topic, but it felt valuable. Trying to justify your research in terms couched in feelings is something that even I, as a self-confessed navel gazer, am not used to doing. In reading around this I was drawn to the work of  Dadds  who described a phenomenon called empathetic validity which refers to “the potential of the research in its processes and outcomes to transform the emotional dispositions of people towards each other, such that more positive feelings are created between them in the form of greater empathy” (2008, p208)

Whether empathy can really be incorporated a project aim I am not clear, I imagine it either happens or it doesn’t, but its benefit for me has trumped any of the the intended consequences of this project.

Ball, E. (2009). A participatory action research study on handwritten annotation feedback and its impact on staff and students. Systemic Practice and Action Research, 22(2), 111-124. doi: 10.1007/s11213-008-9116-6

Cook, T. (2009). The purpose of mess in action research: Building rigour though a messy turn. Educational Action Research, 17(2), 277-291. doi: 10.1080/09650790902914241

Dadds, M. (2008). Empathetic validity in practitioner research. Educational Action Research, 16(2), 279-290. doi: 10.1080/09650790802011973

Accreditation in upgoer5

Yesterday I struggled to explain accreditation services and the point of them. So thought it worth a little time to think about my clarity of message. My verbal inadequacy coincided with me stumbling upon a post from Ruth which showed the importance of scientists and academics writing and communicating in an accessible way. I have had fun exploring the quirky writing tool that forces us to communicate in less complex ways by using only the 1000 most used words. The beauty of this approach is neatly captured by the explanation of the Saturn 5 rocket in words that help even me to understand such complexity.

So here is my attempt at explaining the point of accreditation using theupgoer5 method.

To help people learn so that they can be better at their jobs we can make courses so people can leave behind their work for a day or two and spend time together, learning. They come along and share ideas and have a go at doing stuff, listening to others or talking together. After the course they might do something which shows that they have learnt things. That way others will know they were paying attention.

We have some courses that people can join or we can make one just for your work place, for people who do the same things or work in a team together. To do that we will need to know what stuff you want to learn, what ways you want people to act when they have done the course and how you want people to actually learn together (you might want to sit in a room together and talk, you might want to go out to new places or you may want to use computers). This helps make a plan which can be taken away be people who know about stuff that you want to know about and be turned in to an amazing course. But you might not want us to make a course because you might be pretty good at helping people to learn too.

Another way people learn is when they take courses at work. They learn by listening to important people telling them things, by trying out new ways of doing things, by thinking through how things might be done better, by reading, playing games and by completing things on the computer. Instead of saying this learning is not as good as learning done here we can take a look and see if it is like the learning we do but just in a different place, with different people. If this is the case then we will be able to recognize your learning so that you have a certificate like students who do courses here.

Checking out your course can be hard – it means we need to send each other lots of paper and it can make people annoyed when we have to ask so many questions. It is however really important because when we are done, you know that the people doing the courses really should have their reward.

It’s really important for us to work in these different ways because we know that there are lots of people who do really good work and are really great at learning. We don’t always like the way we have to check on the way your courses are going because it can feel like we are trying to put our ideas on to you, but this also makes sure that we can show the outside world how good people are at learning.

Whilst my text here is highly imperfect it does help me to reflect on the words I use and the way that working groups assume a common language that is impenetrable to others.

Senior Fellowship: Reflections on the process

Today I received confirmation that my Senior Fellowship application to the HEA had been accepted. I thought it helpful if I shared a few insights in to the application process with others who might be considering individual entry route.

There are three ways to achieve recognition with the HEA –

1. Individual entrance route
2. In-house accredited CPD schemes
3. Through recognised qualifications (e.g. a PgC which leads automatically to Fellowship)

In 2011 the HEA launched new levels to their recognition scheme  (there are now four levels of recognition: Associate Fellow, Fellow, Senior Fellow and Principal Fellow). I suspect many institutions will take a little while longer to develop internal CPD systems for the two new levels of recognition and so for now the individual route may remain the favoured approach for some  colleagues.

I completed a Fellowship by individual entry in 2007 and have taken the same route for the Senior Fellowship. The application comprises a 6-7000 word reflection on practice and two ‘case studies’ of practice. It needs to demonstrate all of the areas of activity, all the professional values and all areas of knowledge as described by the Professional Standards Framework. It is not enough to list how you meet each requirement – there is a need to show how you apply the knowledge and values in practice. The biggest challenges in the application process were:

• Managing the time needed (which is significant)
• Going beyond description in the account to ensure sufficient reflection
• Selecting areas of practice on which to reflect

The process requires some detailed planning. The approach I took was to begin with the activities. I simply plotted out what I did against each of the headings listed. So, against ‘assessment and feedback’ I located my roles, projects and practices (present and past), likewise I asked myself what had I done in the area of learning design and developing student guidance and so on for each activity area. Simply creating a list provides the raw material for the reflection.

The next thing I did was to place the ideas in to chronological order so they made sense in terms of my personal progression – of course doing this showed up duplications and sparked additions. Initially I planned to create matrix to ensure I had all values and areas of knowledge covered, but this was quite limiting and made a tick box exercise of the process. Instead I took each activity on my list and reflected by asking a series of questions around each point, including:

• What did I do? (description of the activity)
• Why did I choose to work this way? What shaped the decision? (was it the influence of a colleague, a particular belief, a policy, an engagement with a particular academic idea or theory or case studies from elsewhere). What knowledge and understanding informed this way of working? **
• How was this way of working beneficial to students, colleagues and/or others (including industry partners)?
• What was the impact? How do I know this approach was working well?
• What was learned about working this way? Are there things in future that need to be done to refine this approach further?

**these were the most important questions as they gave opportunity to review  both values and knowledge

I then tagged the emerging narrative against each of the framework requirements by adding  “(v1, k3)” – these are the labels given to the framework requirements (k = knowledge, v = value). These tags were added where I believed my reflection demonstrated the criteria. By doing this I was able to see where the gaps were. My original draft was lacking in v4 for example and so I was able to track back and ask where, in my activities, did I draw upon this this value?

The case study elements (perhaps these should be re-named since they are more like illustrations of practice) are a thicker, and more focused, description of things that you have done. I considered these to be a zoom lens on two areas of my practice. I could adopt the same approach as for the general narrative but had more space to provide more detailed description and reflection.

During the process it was really important for me to have a critical friend who could chat through the sticking points and offer me feedback. I had hoped the application would take a day if I chained myself to my desk but in the end it was much longer. Overall the process has been valuable (if not a little intense) – it provides a useful opportunity to look back at what has been achieved and I was particularly pleased to see that my values had not waned too much! It was quite motivational to retrace my steps over years of practice and it was also helpful in informing planning for new CPD. While I was frustrated by the time commitment needed, without this dedicated ‘thought space’ the benefits of the reflective process would not have been realised.

Aspiring to meaningful feedback

Having explored a number of Jisc and independent feedback projects on assessment and feedback I have selected ten points which emerge as being critical for making feedback work.  

1. Timeliness is critical if feedback is to be useful; consider whether quality management systems are blocking the use of feedback.  4 weeks is too slow!
2. Criteria are important but they can also promote tactical learning as students learn to be selectively negligent (Gibbs, 2012). Use criteria but don’t be locked in by them.    
3. Curriculum design is important. Too many assessments and too many small modules do not encourage deep engagement, and the associated assessment does not capture sufficient study time so as to be meaningful.
4. Feedback should feed forward. It is important that staff are able to know something about what comes next for each student. Without any understanding the opportunity to be purposeful in feedback is severely limited.
5. Little and often is better than more and infrequent.
6. There is only so much individual tutors can do – there is a need to consider programmes and course suites.
7. Feedback that refers to material that will not be studied again tends to be ignored.
8. Where feedback stimulates dialogue about learning, the feedback is perceived as being more useful; this is where feedback crosses in to the line of personal development and students taking control.
9. Reflection on feedback makes learner more autonomous. It reduces dependence on the teacher as giver and makes the student become an active part of the feedback process.
10. Assessment diversity is good, but too much renders the feed forward process meaningless. Limit variety so progression is meaningful.

 

Apps 2012

As I have progressed through my EdD my ways of working have got a little smarter. There are four apps that have served me well in 2012 for supporting my studies …

1. Reminders (so in the hours where I have too much to do I can remember what they were!)

2. Good Reader – managing my online library downloads and annotating my reading without reams of paper. By far the best reading app I have found (still).

3. Good notes – high levels of functionality, a great jotter and annotator – good for generating diagrams and mapping out thoughts.

4. Splashtop – allows my desktop (including Endnote) to be fully functional from my ipad or phone. Excellent when not wanting to be stuck at my desk.

Two more fab apps (not study related)  for 2012 have been

5. Screenchomp – Jing for the ipad – great for audio visual feedback for students and again this means there is no need to be desk bound.

6. Spelling – my best parenting app! So the kids can input the spelling list for the week and then run the tests until the spellings stick. Very motivational for kids who hate spelling.

Five quick ways to write reflectively

1. Imagine an audience for your musings. It’s hard to write without an audience. Write like you are talking to someone that you trust and connect with, and to extend your thoughts imagine their probing questions when you hit natural pauses.
2. Talk, don’t just write. Use voice memos on your phone to capture thoughts in the moment and then write them down when back at base. Some of the most reflective thoughts happen in the car – catch them! This model is effective with adults and children alike.
3. Use a model … a blank page can be daunting, use a reflective model to provide a writing frame for your reflections. Gibbs is my favourite but there are others too …
4.  Go beyond describing what happened in an event or situation. Always follow up with the question, so what? (so what …. For me, for my students, for my colleagues, for my CPD needs, for my confidence, for my progression , for my efficiency, for my well-being?*).
5. Write quickly, naturally and without concern for prose. This is a first layer of reflection. Then  a) develop the text and tidy it up and b) add comments or text boxes to annotate and add further observations on your initial thoughts. Comments or annotations can add major depth compared to a first attempt – ‘when I wrote this, I was thinking …. And I thought this because … but now I have discussed it with my colleague/friend and have revised my original understanding’ or ‘ I can see the choices I made here were limited by ….’. Adding layers to a reflection in this way can be very productive and can help us to question how we see things in the moment.

* Delete and expand!

Other ideas welcome.

Diversifying assessment (and assessment generally)

After a inspiring Learning & Teaching Forum lead by Professor Chris Rust of Oxford Brookes, I pledged my post session action would be to capture my best bits from the day. So … Some  take away points from today’s session ….

1. Authentic assessment is an excellent way to encourage engagement by students as it helps to personalise the student’s approach to the task and generates buy in. So rather than offering abstract tasks, like produce an essay on leadership styles, frame it to have a sense of audience and so that it emulates real world situations that the student may encounter. This could be as simple as changing an essay on business planning principles into a presentation of a business proposal to a prospective funder.
2. Spot review class activities and feedback to the group – a major efficiency saver and a good way of making feedback a routine. So in a class of 40 students pick out five pieces of work to review and send group feedback based on those which have been seen.
3. Getting back to first principles of constructive alignment guarantees some variety. If the learning outcomes are sufficiently varied and, if the assessment lines up to ensure that the actual outcomes are being assessed, this should in itself offer a degree of variety.
4. Diversity is good, but care needs to be taken to ensure that the student journey does not become disjointed by variety. Having some repetition of assessment approaches at the programme level ensures that there is opportunity for students to make use of feedback. A balance is to be struck between variety and the opportunity to facilitate growth in students.
5. Tutor time is disproportionately spent on housekeeping feedback – Are headings present? Are tables labelled? Is evidence offered when requested?  etc etc. A super simple tip might be to have a housekeeping checklist that students complete before submission to deal with all of these aspects, thus allowing time to be better spent on more substantial points of feedback. It works on the principle that answering ‘no’ to any of the housekeeping question prompts a response such that issues are dealt with before submission. Using such checklists, as a routine, ensures that the student takes greater responsibility for their own learning and tutor feedback can deal with more substantial issues. It must be an integral part of the assignment, i.e. will not mark without it, or else it will not be adopted.
6. Less is more. With a greater number of summative assessments the opportunity to give feedback which can feed forward is limited by processes, effort spent on the justification of grades and administration. Instead, lose an assessment and gain the opportunity to utilise feed forward on a piece of work. One assignment, but constructed drawing upon feedback along the way. Simple, but brilliant (and a bit more like real life where review on a document would be an entirely sensible step).
7. Reviewer is king! It is the act of reviewing more than the act of receiving feedback that can spur interest, new insights and leaps in understanding. Getting peer review embedded within courses is an excellent way of raising the presence and effectiveness of the feedback process. To buy into this we need to lay aside fears around peer feedback meaning a lack of parity in the quality and quantity of feedback received (which may be inevitable), and appreciate the value of the experience of reviewing as where the learning is really at. Liken this to being a journal reviewer – how much is learned by engaging with a review whether good or awful? (Analogy courtesy of Mark).
8. Group vivas. Liking this lot and not something I had previously encountered. So simply a group project and attribution of marks depends in some part on a group viva where honesty is, in theory, self-regulating.
9. 24 hours to act. In considering the value of formative quizzes, computer aided or class based as an opportunity to engage with knowledge received, we were reminded of the benefits of engaging sooner rather than later. Engagement with formative quizzes (or indeed reflective processes) within 24 hours of a class is much more effective that if left.
10. Use audio feedback - it doubles feedback and makes production smoother.
11. Future proof feedback plans. Think how SMART devise ubiquity will play out in future. Formative in-class tests may be more efficient on paper for now, but insure efforts by taking a dual pronged approach (online and paper).
12. Pool efforts. Whether across course teams, departments or with colleagues nationally, look for efficiency gains in providing formative question banks. Open educational resource banks (e.g. Open jorum), subject centres and commercial textbooks with CDs of instructor question banks may all be sources to consider.
13. The uber simple approach of asking students what the strengths and weaknesses of their assignment are can focus minds. Additionally asking them on which aspect they would like feedback creates a learning dialogue and ensures feedback is especially useful.

While all of these points matter it remains that it is most important to review the bigger picture. A major barrier to diversifying assessment and capitalising (in learning terms) of feedback opportunities can be modular structure to programmes. The TESTA Project revealed that  “the volume of feedback students receive does not predict even whether students think they receive enough feedback, without taking into account the way assessment across the programme operates”.  Volume of feedback or assessment will not improve student perceptions of feedback. Point 4. Above leads to the assumption also that timeliness alone is not enough either. While it is good for module level assessment and feedback to be considered in relation to the ideas above  a holistic look at the programme level helps us to understand the assessment journey of the student. How can feedback feed forward? Is their sufficient variety across a programme? Is their repetition? All questions worth asking beyond the module level.