The tricky issue of word count equivalence

The challenges of managing media rich assessments, or managing student choice in assessment, has been evident in higher education for as long as I have been employed in the sector, and probably a lot longer. Back in 2004, when I worked on the Ultraversity Programme, the course team had an underpinning vision which sought to: enable creativity; encourage negotiation of assessment formats such that the outputs were of use; and, develop the digital capabilities of students (a form of assessment as learning). We encouraged mixed media assessment submissions for all modules. At this time we debated ‘the word count issue’ and emerged with a pragmatic view that alternative media should be broadly equivalent (and yes that is fuzzy, but ultimately this helps develop judgment skills of students themselves).

In the HEA accredited PgC in Teaching and Supporting Learning that I now manage, we assess using a patchwork media portfolio. Effectively there are five components (including an evaluation of assessment and feedback practices, a review of approaches used in teaching or supporting learning and a review of inclusive practices used) plus there is a stitching piece (a reflection on learning). The assessment brief describes what the students should show, but it is not prescriptive on the precise format. Each element has a word guide, but this should be used by those working with alternative media as a guide to the size of the output and the effort they apply.

wordcount

Where students opt for media rich formats, they are asked to decide on equivalence. Close contact in class sessions provides a guiding hand on judgment, critically with peer input (‘yes, that sounds fair’). Techniques to assess equivalence include taking a rough ‘words per minute’ rate and then scaling up. I have had other items such as posters and PowerPoints, again, I ask them to use their own approximation based on effort. Because the students in this particular programme are themselves lecturers in HE, there is a degree of professional reflection applied to this issue. We don’t ask for transcripts or supplementary text when an individual submits an audio or video format, because it can add considerable work and it may be a deterrent to creativity.

Media experimentation within this programme is encouraged because of the transformative effect it can have on individuals who then feel free to pass on less traditional, more creative methods to their students. I asked one of my students to share their thoughts having just submitted a portfolio of mixed media. Their comments are below:

My benefits from using media were;

  • Opportunity to develop skills
  • Creativity
  • More applied to the role I have as a teacher than a written report would have been
  • Gave ideas to then roll out into my own assessment strategies, to make these more authentic for students
  • Enjoyable and I felt more enthused to tackle the assignment elements

But I wouldn’t say it was quicker to produce, as it takes a lot of advanced planning. And, it was tricky to evidence / reference, which is a requisite for level 7. This is where I fell down a little.

I judged equivalence with a 60-100 words per minute time frame for narrative, and / or, I wrote the piece in full (with word count) and then talked it through. I think the elements that I chose to pop into video were those that were more reflective, and lent themselves better to this approach. With the more theoretical components, where I wasn’t feeling creative or brave enough to turn it into something spangly, I stuck with the written word. The exception to this was the learning design patch, where I wanted to develop particular skills by using a different approach.

This student’s comments very much match up with comments made back in 2009, by Ultraversity students who reported “without exception, felt that they had improved their technical skills through the use of creative formats in assessment” (Arnold, Thomson, Williams, 2009, p159).   Looking back at this paper I was reminded that a key part of managing a mixed picture of assessment is through the criteria, we said “In looking at rich media, the assessor needs be very clear about the assessment criteria and the role that technology has in forming any judgments, so as to avoid the ‘wow’ factor of quirky technology use. At the same time he/she must balance this with the reward of critical decision-making and appropriateness in the use of technology. Staff and student awareness of this issue as well as internal and external quality assurance guards against this occurrence” (p161). This is exactly the approach taken within the PgC Teaching and Supporting Learning. Tightly defined assessment criteria have been very important in helping to apply consistent assessment judgments across different types of submission.

If we want to receive identically formatted items, which all address the learning outcomes using the same approach, then of course mandating a single format with a strict word count is the way to go. But if we want to encourage an attitude to assessment which encourages creativity in new lecturers, and which acts as a development vehicle for their own digital skills, then we must reduce concerns about word counts and encourage junior colleagues to develop and use their professional judgment in this matter. The student quote above shows the thoughtful approach taken by one student to address the issue for themself.

Frustratingly, even by using word count as the reference point for parity we may ‘other’ some of the more creative approaches that we seek to encourage and normalize, but ultimately wordage has long been the currency of higher education. It is good to see some universities being pro-active in setting out a steer for equivalence so that individual staff do not feel that they are being maverick with word counts when seeking to encourage creativity.

Published … Advancing the Patchwork Text: The Development of Patchwork Media Approaches

Following on from the July 2009 learning Conference “The” paper is now published in the International Journal of Learning and available here:

http://ijl.cgpublisher.com/product/pub.30/prod.2169

Learning & Teaching Forum, Harper – Patchwork media for practice.

Patchwork_- Harper Teaching & Learning Forum

On Thursday 9th I gave a presentation on Patchwork Media at the Teaching & Learning Forum at Harper Adams. The presentation is available as a PDF download from the link above. This dovetailed nicely with the work I had done in Barcelona. The Harper presentation was more about the practicalities of patchwork whereas the Barcelona one focused on unpacking the online mechanisms for supporting patchwork.

Barcelona : 16th International Conference on Learning


5415_1196398912148_1293728604_30551015_474520_nBetween Thursday and Saturday last week I was in beautiful Barcelona at the 16th International Conference on Learning where I presented a paper on patchwork media. The conference was an eclectic mix of practitioners bound together by themes in teaching and learning. The topics under consideration ranged from origami for maths education to e-learning in HE. There are pros and cons of course to such a wide ranging coverage mix. Sometimes though inspiration in practice can come from the most unlikely sessions, perhaps sometimes we try to hard to listen to a narrow band of peers afterall, learning is learning wherever it is found.

My own session was entitled Patchwork media: Advancing the patchwork text. Essentially the session focused upon how technology can enhance the patchwork text approach. Three main ways were considered: through resource creation, through online community and through media rich assessment products. The research was undertaken with Kev Thompson & Tim Williams. The presentation is here: Patchwork in BARCELONA. 

A particularly interesting session was on the promotion of professional learning in to undergraduate courses at Griffin University, Australia. Essentially, within undergraduate courses learners are being professionally sensitized,; they are  developing a sense of professionalism and professional identity and they are being encouraged to understand the industry that they will work in through direct contact sessions. The role of industry in this development was high, though not in placement the industry is adding authenticity to the campus. A super model which fuses knowledge types in ways envisaged by Barnett (1999).  [The presentation, was by Craig Cameron and Brett Freudenberg].  Presently I am involved in developing a professional development module for post graduate learners from various professions. I wonder <note to self> how for small groups of learners or cohorts from varied professional backgrounds, how online open resources can most effectively help inject professional sensitization and awareness.

So, what’s special about patchwork?

The patchwork approach to assessment at first view could be seen as ‘just a portfolio’  so it helps to just pin down what makes this approach particularly valuable.  

1. Patches are aligned with learning outcomes through learning the design of experiences/activities. Each patch captures the learning in a quite specific area. Patchwork is ‘assessment as learning’ by design. It is a clear method achieving constructive alignment.

2. Essentially, patches are shared between peer groups for critique, formative feedback and discussion. Patches are thereafter refined. The patchwork is a social portfolio.

3. Patchwork allows the learner to feel comfortable that they are on a journey as an apprentice, they are not setting out to show mastery through a single academic voice but rather they are demonstrating learning.

4. Patchwork media encourages the experimentation with the ‘voice’ used to show learning. This can be particularly useful for learning in the workplace where learners use their professional voice and media which is in-synch with workplace activity. Equally, learners can benefit from playful genre which is very much removed from their areas of practice.  

5. Patchwork requires an discussion of learning after individual parts are produced. The discussion allows ideas to be joined up but also provides an areans for learners to reflect upon their own learning jouney and how they may develop their future approaches. the approach embraces learning about learning with this additional layer.

A tension: peer review and patchwork (some of my own learning from treating data collected)

 Whilst peer review is a highly valued process (by students and facilitators alike) there is a danger of a contradiction occurring between this process and the original patchwork vision that the review seeks to serve. Winter’s original view was that peer review should help to refine understanding, that it should enable students to ‘come to know’ and to grow through their apprenticeship in their field, in effect the patch should represent a learning journey and should not claim mastery. It is also well charted that students are mainly highly motivated to develop their assessment products and that assessment is highly valued. The contradiction occurs because students have sometimes, in the experience of the BA LTR facilitation team, seen the perfection of the patch as the greatest priority, sometimes over and above the gains of learning. The mindset that sets the patch as perfection can be evidenced in the community; students post a draft, then a re-draft then more drafts for feedback. Ways in which this is being dealt with in practice include the limitation of facilitator review and the prioritisation of attention to dialogue around issues and concepts. Whilst the online community enables feedback at the convenience of the participants, it also means that feedback can be sought at will, a balance between the influencing forces of assessment and learning, mastery and apprenticeship, needs to be managed.

Challenges and benefits of the patchwork media approach as experienced by learners

Emerging from recently collected data and the emerging analysis is a portrait of some of the benefits of operating a digital and media rich version of Winter’s patchwork text approach. 

Benefits for learners

• The use of media to provide opportunities for seeing events and issues from other perspectives.
• Intrinsic enjoyment from using creative media
• The opportunity to develop technical skills and to become a critical user of technology
• The opportunity to use media native to the work setting
• The opportunity to articulate and develop understandings through considered asynchronous peer review
• The opportunity to share assessment products beyond he peer group (through the world wide web) and thus gain authentic feedback. 
• The opportunity to create assessment products in ways that are natural to an individual learning style.

Challenges for learners

• The need to develop strategies for managing the quantity and quality of review messages.
• Negotiating the use of media which is both creative, authentic to the workplace and helps to develop the learners technology use but that does not distract from the learning content/message. 
• Managing the balance between seeking to perform at a high level for assessment and accepting the apprenticeship mindset inherent to the patchwork process (particularly this means managing the number of times that feedback is sought). 
  Finding creative solutions to enable the submission of new file types as they emerge.