Making digital exemplars

In addition to my usual classroom use of exemplars as a means of familiarising students with the assessment requirements of a specific module, this year I have created a video walk through of an exemplar. Initially this was to enable those who missed the relevant class to catch up on the session, but the approach was welcomed by students who attended the exemplars activity session, as well as those who did not. 

How to create a digital exemplar walk through: 

• Bring up the exemplar on screen after selecting a ‘good’ piece of work

• Read through and use comments in word to annotate the work with points which would be surfaced in feedback, particularly comments related to the assessment criteria (of course!). Comments include things done well, things done less well which could be avoided and opportunities for further detail and development. This tagging process acts only as an aide memoire so that as I create  feedback video I am aware of what I wanted to include. 

• Open Screencast-o-Matic to very easily screen record the work as a video as I re-read it through and talk to each of the tags. ‘This work includes this … which is useful because…’ ‘this work used literature in this way …. It might be better to …. Because ….’. None of this is rehearsed; that would be too time consuming. The resultant video is a commentary on performance.

• The video is uploaded and made available to students.

After using the resource there was some consensus amongst my students that the value was ONLY in listening to the assessment commentary and not specifically in looking at the work. One student described how they listened but did not watch. They then recorded notes about what they should include, remember and avoid. They avoided looking at the work for fear of having their own ideas reshaped. If assessment judgments are ‘socially situated interpretive act[s]’ then the digitised marking commentary may be a useful way of making that process more transparent for students, and indeed for other staff.

I will definitely be including this in future modules.

Handley, K., den Outer, B. & Price, B. (2013) Learning to mark: exemplars, dialogue and participation in assessment communities. Higher Education Research & Development Vol. 32 , Iss. 6.

The tricky issue of word count equivalence

The challenges of managing media rich assessments, or managing student choice in assessment, has been evident in higher education for as long as I have been employed in the sector, and probably a lot longer. Back in 2004, when I worked on the Ultraversity Programme, the course team had an underpinning vision which sought to: enable creativity; encourage negotiation of assessment formats such that the outputs were of use; and, develop the digital capabilities of students (a form of assessment as learning). We encouraged mixed media assessment submissions for all modules. At this time we debated ‘the word count issue’ and emerged with a pragmatic view that alternative media should be broadly equivalent (and yes that is fuzzy, but ultimately this helps develop judgment skills of students themselves).

In the HEA accredited PgC in Teaching and Supporting Learning that I now manage, we assess using a patchwork media portfolio. Effectively there are five components (including an evaluation of assessment and feedback practices, a review of approaches used in teaching or supporting learning and a review of inclusive practices used) plus there is a stitching piece (a reflection on learning). The assessment brief describes what the students should show, but it is not prescriptive on the precise format. Each element has a word guide, but this should be used by those working with alternative media as a guide to the size of the output and the effort they apply.

wordcount

Where students opt for media rich formats, they are asked to decide on equivalence. Close contact in class sessions provides a guiding hand on judgment, critically with peer input (‘yes, that sounds fair’). Techniques to assess equivalence include taking a rough ‘words per minute’ rate and then scaling up. I have had other items such as posters and PowerPoints, again, I ask them to use their own approximation based on effort. Because the students in this particular programme are themselves lecturers in HE, there is a degree of professional reflection applied to this issue. We don’t ask for transcripts or supplementary text when an individual submits an audio or video format, because it can add considerable work and it may be a deterrent to creativity.

Media experimentation within this programme is encouraged because of the transformative effect it can have on individuals who then feel free to pass on less traditional, more creative methods to their students. I asked one of my students to share their thoughts having just submitted a portfolio of mixed media. Their comments are below:

My benefits from using media were;

  • Opportunity to develop skills
  • Creativity
  • More applied to the role I have as a teacher than a written report would have been
  • Gave ideas to then roll out into my own assessment strategies, to make these more authentic for students
  • Enjoyable and I felt more enthused to tackle the assignment elements

But I wouldn’t say it was quicker to produce, as it takes a lot of advanced planning. And, it was tricky to evidence / reference, which is a requisite for level 7. This is where I fell down a little.

I judged equivalence with a 60-100 words per minute time frame for narrative, and / or, I wrote the piece in full (with word count) and then talked it through. I think the elements that I chose to pop into video were those that were more reflective, and lent themselves better to this approach. With the more theoretical components, where I wasn’t feeling creative or brave enough to turn it into something spangly, I stuck with the written word. The exception to this was the learning design patch, where I wanted to develop particular skills by using a different approach.

This student’s comments very much match up with comments made back in 2009, by Ultraversity students who reported “without exception, felt that they had improved their technical skills through the use of creative formats in assessment” (Arnold, Thomson, Williams, 2009, p159).   Looking back at this paper I was reminded that a key part of managing a mixed picture of assessment is through the criteria, we said “In looking at rich media, the assessor needs be very clear about the assessment criteria and the role that technology has in forming any judgments, so as to avoid the ‘wow’ factor of quirky technology use. At the same time he/she must balance this with the reward of critical decision-making and appropriateness in the use of technology. Staff and student awareness of this issue as well as internal and external quality assurance guards against this occurrence” (p161). This is exactly the approach taken within the PgC Teaching and Supporting Learning. Tightly defined assessment criteria have been very important in helping to apply consistent assessment judgments across different types of submission.

If we want to receive identically formatted items, which all address the learning outcomes using the same approach, then of course mandating a single format with a strict word count is the way to go. But if we want to encourage an attitude to assessment which encourages creativity in new lecturers, and which acts as a development vehicle for their own digital skills, then we must reduce concerns about word counts and encourage junior colleagues to develop and use their professional judgment in this matter. The student quote above shows the thoughtful approach taken by one student to address the issue for themself.

Frustratingly, even by using word count as the reference point for parity we may ‘other’ some of the more creative approaches that we seek to encourage and normalize, but ultimately wordage has long been the currency of higher education. It is good to see some universities being pro-active in setting out a steer for equivalence so that individual staff do not feel that they are being maverick with word counts when seeking to encourage creativity.

9 Things to do with Assessment Rubrics

I’ve used rubrics in assessment marking since I first held an academic role some fifteen-ish years ago. For me, rubrics are an essential tool in the assessment toolkit. It’s important to recognize that they are not a ‘silver bullet’ and if not integrated in to teaching and support for learning, they may have no impact whatsoever on student engagement with assessment. I am therefore trying to collate a list of the ways in which rubrics can be used with students to enhance their performance, help them grow confidence and to demystify the assessment process. My top nine, in no particular order, are as follows:

  1. Discuss the rubric and what it means. This simply helps set out expectations and requirements, and provides opportunities for clarification.
  2. Encourage students to self-assess their own performance using the rubric, so that they engage more deeply with the requirements of the assessment.
  3. Encourage students to peer assess each other’s performance using the rubric, leading to further familiarization with the task, as well as the development of critical review and assessment judgment skills. This also allows the seeding of further ideas in relation to the task, through exposure to the work of others.
  4. Get students to identify the mark that they are aiming for and re-write the criteria in their own words. This sparks discussion about the requirements, flushes out any issues needing clarity and can result in students raising their aspirations (as the ‘assessment code’ is decrypted there are moments of “If that’s what it mean’s … I can do that”).
  5. Facilitate a negotiation of the rubric. Where full student led creation of a rubric is impractical, or not desirable, a tentative rubric can be presented and negotiated with the class. Students can have an influence on the coverage, the language, and the weightings. As well as familiarizing with the requirements, this allows a sense of ownership to develop. In my own experience rubrics are always better for student negotiations.
  6. Undertake a class brainstorm as the basis for the rubric design. Ask what qualities should be assessed e.g. report writing skills, then identify what this means to students themselves e.g. flow, use of literature to support argument. Then use this list to develop a rubric. It is a form of negotiation, but specifically it allows the rubric to grow out of student ideas. By using student language, the criteria are already written in a form that is accessible to the group (after all they designed the key components).
  7. Simply use the rubric as the basis for formative feedback with students to aid familiarity.
  8. Use the criteria to assess exemplars of previous students’ work. This will have the benefits of familiarity, developing assessment judgment as well as sparking new ideas from exposure to past students work. Of course this can be further developed with full sessions or online activities built around exemplar review, but the rubric can be central to this.
  9. A rubric can be partially written to offer space for choice. Leaving aspects of the rubric for students to complete leaves room for students to show their individuality and to customize tasks. Rubrics don’t box-us in to total uniformity. Recently I created a rubric for a research project and left space for students to articulate the presentational aspects of the criteria. Some students filled in the rubric to support the production of a report, others a poster and others a journal article.
img_20160908_152105
Using a class brainstorm to form the basis of a rubric with criteria relating to reflection 

I have only included approaches that I have used first hand. I’d like to build this up with the experiences of others; if you have additional suggestions please do let me know.

International staff development in teaching and learning: Lessons learnt

cstaAt the end of another period of working with lecturers from overseas I thought it would be useful to pause for thought and identify lessons from working in the area of transnational staff development. I have just completed my third international staff development summer school, and here is what I found. These points come with the caveat that they are only my learning, and others may, of course, have different views of what works.

Be conscious of your assumptions.

Before meeting academics from other continents it’s easy to let assumptions about what they may or may not be doing in practice, or what they may or may not believe, creep in to your thinking and planning. By example with tutors from China a common belief is that they are always involved in a transmission modes of education; this is simply not my experience and the view relates to outdated assumptions. While some Anglicized techniques may be new to colleagues from other locations my experience tells me we have a shared passion for making learning better, and some overlap already in our methods such as employing a flipped classroom.

Explore biography.

It’s always useful to start any transnational academic staff development by exploring the experiences and biography of the individuals involved. Those involved in supporting transnational staff development can then be agile and responsive to the specific needs of the group. Things that I have tried are very simple: Starting the development programme with a list of questions and concerns that the group would like to address, shared via a post-it-note board, hosting a session with no plan and offering the hot seat format where we simply respond to the questions of the group in an effort to take stock and allow international colleagues to add context to their growing understanding, and walking together to listen more informally to the needs of the group.

Ensure that learning is always two way.

One directional international staff development appears like neo-colonial self-righteousness. We need to employ empathetic methods wherein we offer our own practices, identify some of the limitations of these approaches, and also invite teachers from other locations and nations to do the same. It feels disrespectful to do anything else.

Get the basic accessibility matters right

Take care of the basics to ensure provision is accessible. Particularly it’s important to make sure that delivery is slow enough to allow the digestion of material; if you’re not sure, keep checking with the group. Also, ensure that resources (slides, papers, etc.) are available before the class so that translation apps can be used to familiarise with any tricky words.

Team teach, always.

Team teaching allows a richness which is not possiblealone; I tend to work with people who can offer a very practical take on the theoretical ideas and research that I am exploring. As well as simply providing more experience in the room, teaching in pairs provides an opportunity to model professional differences, which are inevitable in teaching. So by example when working with a colleague in a session about teaching evaluation, my view was that it is okay to identify some of the areas upon which you specifically want feedback from teaching observation, but my colleague’s view was that this might limit the range of feedback and prevent previously unnoticed habits or issues from surfacing. Our nuanced differences were explored publicly as a model of divergent views; this raises the exposure of the group to detailed discussion. Team teaching also provided in the moment opportunities for peer observation and debrief; this should be routinely incorporated through the provision of collegial feedback on what worked well, and what less so.

Use staff development as a vehicle for the host organisation’s staff development.

By keeping international staff development locked within one or two people, there is limited benefit. Encouraging colleagues who may be outside of the usual staff development circuit to come and join in, can, I think, have a considerable impact, in growing capacity and confidence for this type of work. It can also provide experience for aspiring Senior Fellows within the UK Professional Standards Framework. It can also provide challenge and a refreshing set of ideas, for staff who often manage staff development.

 

Be confident about lessons on technology transferability

Within international staff development programmes we may wish to explore tools for learning and teaching. Things like Twitter, Padlett, Nearpod, Facebook and Kahoot. A first thought may be, well what’s the point as we have access to different apps, and some of our apps may not be appropriate in a different cultural context. However, our experiences are that sharing is still good, even when the tools are not immediately transferable, as we can learn by return about alternative apps. Most importantly, since technology and the way we use it reflects much about the underpinning power, beliefs and values in learning, then exploring technology is a much more valuable experience than just swapping ideas on apps we like.

Hold discussion using first language

Group work with international, especially Chinese staff, is always very productive, but it is incredible demanding and limiting to make that group work happen in a second language (e.g. English). Permitting sense-making in the first language allows rich debate and discussion to evolve, rather than slowing down the pace and adding another cognitive load. This approach loses the opportunity for input to the discussion from the session host, but in my experience this is a process worth paying to enable lively group discussion, and in any case group précis in English can give the headlines.

Lead by self-exposure on the difficult discussions.

Asking questions, or discussing topics, which require some exposure of personal fragilities, can be tricky to get started. Working with my colleague, Jane Headley, we found that by sharing something of ourselves before we ask others to do the same was helpful in creating an open forum. So for example, when discussing technology, I shared some adverse feedback that I had received on my own approach; I then told the story of what happened next, and I identified the decisions I made that lead to a less than perfect learner experience. This lighthearted ‘fessing up’ made others comfortable to share their own critical incidents and learning from it. As well as showing openness, this deconstruction of practice also models reflection in action.

Use many examples and stories

Using real examples of situations, challenges and successes can really aid understanding, though remembering that some political context may be important to explain why one or another decision was taken at the time, e.g. we had funding for this type of work, or we are preparing for TEF.

Encourage journal writing

Finally, by encouraging some reflective writing after each topic or session, international staff can form their own ‘take away’ record of i) what were the key learning points and ii) what next to research, extent or apply aspects of learning? A simplified, structured formal of learning journals can promote consolidation and impact from learning.

This list is no doubt incomplete, so please do add any other points that you might have.

Creative commons image sources:

https://commons.wikimedia.org/wiki/File:Globe_terrestre_Orange_te_Bleu.svg

 

(My) Lessons from the flipped classroom

In September 2015 I committed to deliver a thirty-credit module, called The Teaching Practitioner, using A flipped classroom pedagogy. The module is the first of two in a PgC Teaching and Supporting Learning in HE; it is associated with Associate Fellowship of the Higher Education Academy.

My motivation for flipping the classroom was three fold:

  1. My contact time was limited and therefore moving ‘delivery of content’ outside of the classroom was an answer to a specific timetable challenge.
  2. In learning and teaching provision of this type I wanted to actively avoid ‘preaching’ or appearing as the ‘authority’. Everyone, without exception, on a work-based programme brings experience and the class dynamic is much more about guiding equals and facilitating mutual learning.
  3. I would rather place my energies in to discursive, challenging and unexpected contact time, rather than repeat sessions of transmitting content, which can be accessed in other ways.

The pattern of delivery was simply that each week I shared materials to work through, including narrated presentations, videos (commissioned and existent), reading, reflective tasks and then we would gather to discuss. The discussions varied in formality, structure and style as the module progressed. Over the course of the module I learnt a great deal, the key points from my mental list of lessons are shared below.

Screen Shot 2016-03-17 at 08.46.13
To do list example (click to view)

Essential to do list: Each week I published what needed to be done in advance of the face-to-face class. Importantly the list split out what was essential and what was optional. Participants reported that this was a helpful organizing distinction and allowed better management of their activity. This is something that I would definitely adopt in future modules of any type to act as a pacesetter. Simple, perhaps obvious, but actively encouraging participants to make choices about the level of engagement they can make is a pragmatic way of supporting work based practitioners who have so many competing demands on their time.

Slides not videos: I experimented with the media format of presentational material (pre-class content). The staple across most weeks was the narrated PowerPoint. I found more editing control by using Audacity to record the audio and then drag and drop in to PowerPoint, compared to recording direct in to PowerPoint. Audacity gave me opportunity to edit out any major interruptions with ease (phone calls, door knocks etc). I included some video lectures of studio production quality however participants found them relatively less engaging, with a preference for visuals and audios mixed in together with the ability to more easily navigate the presentation. I was surprised by this preference, but there is no doubt narrated presentations are easier to create.

Don’t force theory: We took a discursive approach to our face-to-face time (which was usually two hours per week). I provided questions and starters and then tried to guide the discussion. At first the conversation was loose, multi-directional, on and off-topic. I worried that we were not being ‘very level seven’ and the participants shared some of these concerns. However an under the surface, a process of sense making was going on; each person, in their own language and terms, through sharing and reflecting on their own experience got chance to reconceive, affirm and evaluate their practice. The explicit linking to theory was a more private activity, which seemed to occur in response to assessment. It was only obvious that this had taken place at the end of the module as discussion and theory were fused. Perhaps the discussions were a shared liminal space in which we muddled through difficult issues, then we went away to individually reflect and make clear.

Screen Shot 2016-03-17 at 12.13.31.png
A conception of flipped learning as a three stage process

 Facilitation skills matter more than online production skills: My role can be linked to all the activities of a facilitator, including:

  • Summarizing

    Screen Shot 2016-03-17 at 09.01.39
    A discussion summary in progress
  • Questioning
  • Providing occasional expertise
  • Sharing anecdotes
  • Signposting
  • Collating the issues that we couldn’t solve and referring them to other forums, or mentally ‘parking them’ as knowingly messy
  • Archiving ideas (e.g. photographing shared lists and posting them online for future reference)
  • Providing clarity as needed
  • Providing confidence
  • Managing the group dynamics
  • Modeling active listening

As we progressed through the weeks, methods for each of these aspects became more developed e.g. creating graphics for summaries, defining the discussion purpose to keep us mainly on task. One thing I did from time to time was add a summary of the discussion as a resource for reference so that everyone had opportunity to revisit key points. This involved simply using my mobile phone and talking through the diagrams that we had created in class such that everyone had a record. This was not onerous at all if done straight after the session while fresh in memory.

Quick and dirty production process: If the model of delivery is going to be sustainable then resources need to be produced within a realistic time frame. By taking a quick and dirty approach to development, those on the programme see the approach as achievable and replicable; it provides accessible modeled practice. For me there is also a really clear sign in this approach that the value of the learning experience is the interaction and not a resource. To avoid perfectionism I never listened to my own presentations after they were recorded other than for a quick sound check.

Shared endeavor: While new roles were not formally defined, we fell in to a more even relationship. I sensed that we were co-researchers (in to the effectiveness of the pedagogy) and co-learners (about all aspects of the programme). We were facilitators and facilitated, rather than ‘teacher and student’. To reinforce this role equality, I tried to be very open about when I was learning too.

Allow choice about levels of engagement: As grown ups, participants face a simple rational choice about whether to engage or attend; sometime this choice is made in light of personal life and professional workload. In the weeks where individuals had not done the preparation for class, no action was taken or penalty applied. This approach relies on a commitment to engage and the rewards are implicit in the design. It also reflects the idea of running a community of equals. The group dynamic needs to be honest about the need for preparation, but pragmatic when this slips. If the facilitation works well then even those who have not prepared should be invited and able to contribute experience, and hopefully then inspired to retrospectively visit the online class.

A human process not a technical one: Flipped classroom may evoke thoughts about complex online tools and an unfathomable methodology of teaching promoted by centres of e-learning and academic development, but for me the experience of flipped classroom is a fundamentally human process which involves a respect the opportunity to explore individual experience and knowledge. It allows social learning and creates space for the discussion of any issues arising that matter to the group. I hope the language around this practice, and the identity of the learning model as slightly exotic, does not take away from the collegial simplicity, which resonates with traditional seminar based learning.

Support for the flipped approach from participants was demonstrated in three distinct ways: i) the adoption of flipped classroom by some group members ii) protest when classes are not flipped iii) outstanding, highly personalized, deeply connected assignments to demonstrate the culmination of meaningful engagement (though I am a little bias on the last point).

If I had a point nine on my list, it would be to keep faith that the approach will pay off, even when there is angst about its effectiveness. That said, when I saw in the module assessments that we had reached our destination (albeit a fleeting one on the way to the next module) I was very relieved!

 

 

Lecturers: Find your rockstar!

Much has been written about the value of lectures: It is not clear whether they will endure, or if instead they are dated and doomed. Either way though I don’t see them going away any time soon. While the lecture has remained core in many universities, we know that lecture content (i.e. higher level knowledge) has gone from being underpinned by privileged information locked down within the academic community, to in many cases nothing more than ‘stuff’ that might be gleaned in a decent web search. How many colleagues Google source at least some of their content (be honest now!)?

The nature of knowledge has changed rapidly, but still the main mode of operation for ‘oh so many’ courses is transmission. When I hear others say ‘why would students come to lectures if there is … no assessment incentive … when there is lecture capture …or, when there are full notes on the virtual learning environment’, I can’t help feeling it’s the wrong question. We don’t want to dupe students in to coming to lectures by denying these beneficial actions just to sustain the status quo; this is a defensive approach, which devalues the time of students and the professionalism of lecturers. Instead it is perhaps better to ask ‘why would students come to class at all?’ What are we offering that is good enough and useful enough for students to want come along and engage?

Screen Shot 2016-02-03 at 19.45.40

There are clear synergies between higher education and contemporary trends in music. Commercial music, like knowledge, has gone from a prized product, to being cheap and accessible, and then to being free and abundant. Artists are succeeding by not holding on the old modes of distributing, but by making their product open and available, to whet the appetite of fans who will then travel, pay and commit time to go to the live event; to share an experience and share in something which cannot be consumed by other means. Often the benefits of live events lie in the way they make you feel rather than in a quantifiable outcome.

I am advising colleagues who are new to teaching to ‘find their rockstar’. This doesn’t mean becoming an edutainer; which I take issue with as it trivializes scholarship. It rather means locating the reasons that make the live coming together of people to learn, a valuable and meaningful experience. If the transmission mode is used, then what extra does the live performance add? When I watch a band the value for me is ‘feeling’ the bass, making memories, and being part of an audience. When I watch a speaker, I enjoy the focus, the interaction and the stories, which feel are told for the specific audience.

Screen Shot 2016-02-04 at 10.59.59.png

When asked by a researcher, who is new to teaching, what is my ‘rockstar’ and how do I find it? My answer … you already have it …

‘your unpublished and ongoing research’

‘your empathy with the students you truly appreciate their angst because a similar experience is not so far in your own past’

‘your calm and considered outlook coupled with boundless enthusiasm that makes your content far richer than any download’ (your newness means your passion is undiminished). This can be felt by those around you; it can’t be measured’.

If I think about brilliant lecturers they all have their own attributes that make attendance worthwhile – these are always not displays of radical teaching or ‘fizz-bomb’ personalities; but there is always something more than transmission. Examples include anecdotes, stories, empathy, humour, and enthusiasm, and research. Rather than assume students are disengaged we should make sure when they come to class they have something more than a download… it’s potentially a useful, confidence generating staff development reflection to #FindYourRockstar.

 

 

Living scholarship

During a workshop on teaching recognition at Harper Adams University, I was involved in discussion with colleagues about ‘what is scholarship in the context of teaching and supporting learning?’ This discussion is not new of course. Boyer’s four types of scholarship provide a common reference point to answer this recurring question. Locating types of scholarship seemed not to fully capture our group’s perceptions of their own scholarship though; what about the informal, the discursive, the self-review and inquiring mind? What about the underpinning, hard to quantify, desire to enhance and learn? What about growing and using social capital or networks to respond to events emerging (something akin to collective reflection in action)? What about scholarship as routine, habit or modus operandi? What about scholarship as critical and thoughtful engagement. All of this is very tricky to measure or locate. It is more wrapped together as a package we called ‘living scholarship’. Although with some danger of being self congratulatory about finding a label for scholarship as being, I rather liked it 🙂

Living Scholarship: Combining aspects of scholarship, with a persistent, passionate and committed search for enhanced practice; Scholarship which can be sensed as well as seen; Scholarship which is private as well as public, natural rather than additional, not always necessarily explicit, and which is underpinned by thoughtfulness and self awareness.