The tricky issue of word count equivalence

The challenges of managing media rich assessments, or managing student choice in assessment, has been evident in higher education for as long as I have been employed in the sector, and probably a lot longer. Back in 2004, when I worked on the Ultraversity Programme, the course team had an underpinning vision which sought to: enable creativity; encourage negotiation of assessment formats such that the outputs were of use; and, develop the digital capabilities of students (a form of assessment as learning). We encouraged mixed media assessment submissions for all modules. At this time we debated ‘the word count issue’ and emerged with a pragmatic view that alternative media should be broadly equivalent (and yes that is fuzzy, but ultimately this helps develop judgment skills of students themselves).

In the HEA accredited PgC in Teaching and Supporting Learning that I now manage, we assess using a patchwork media portfolio. Effectively there are five components (including an evaluation of assessment and feedback practices, a review of approaches used in teaching or supporting learning and a review of inclusive practices used) plus there is a stitching piece (a reflection on learning). The assessment brief describes what the students should show, but it is not prescriptive on the precise format. Each element has a word guide, but this should be used by those working with alternative media as a guide to the size of the output and the effort they apply.

wordcount

Where students opt for media rich formats, they are asked to decide on equivalence. Close contact in class sessions provides a guiding hand on judgment, critically with peer input (‘yes, that sounds fair’). Techniques to assess equivalence include taking a rough ‘words per minute’ rate and then scaling up. I have had other items such as posters and PowerPoints, again, I ask them to use their own approximation based on effort. Because the students in this particular programme are themselves lecturers in HE, there is a degree of professional reflection applied to this issue. We don’t ask for transcripts or supplementary text when an individual submits an audio or video format, because it can add considerable work and it may be a deterrent to creativity.

Media experimentation within this programme is encouraged because of the transformative effect it can have on individuals who then feel free to pass on less traditional, more creative methods to their students. I asked one of my students to share their thoughts having just submitted a portfolio of mixed media. Their comments are below:

My benefits from using media were;

  • Opportunity to develop skills
  • Creativity
  • More applied to the role I have as a teacher than a written report would have been
  • Gave ideas to then roll out into my own assessment strategies, to make these more authentic for students
  • Enjoyable and I felt more enthused to tackle the assignment elements

But I wouldn’t say it was quicker to produce, as it takes a lot of advanced planning. And, it was tricky to evidence / reference, which is a requisite for level 7. This is where I fell down a little.

I judged equivalence with a 60-100 words per minute time frame for narrative, and / or, I wrote the piece in full (with word count) and then talked it through. I think the elements that I chose to pop into video were those that were more reflective, and lent themselves better to this approach. With the more theoretical components, where I wasn’t feeling creative or brave enough to turn it into something spangly, I stuck with the written word. The exception to this was the learning design patch, where I wanted to develop particular skills by using a different approach.

This student’s comments very much match up with comments made back in 2009, by Ultraversity students who reported “without exception, felt that they had improved their technical skills through the use of creative formats in assessment” (Arnold, Thomson, Williams, 2009, p159).   Looking back at this paper I was reminded that a key part of managing a mixed picture of assessment is through the criteria, we said “In looking at rich media, the assessor needs be very clear about the assessment criteria and the role that technology has in forming any judgments, so as to avoid the ‘wow’ factor of quirky technology use. At the same time he/she must balance this with the reward of critical decision-making and appropriateness in the use of technology. Staff and student awareness of this issue as well as internal and external quality assurance guards against this occurrence” (p161). This is exactly the approach taken within the PgC Teaching and Supporting Learning. Tightly defined assessment criteria have been very important in helping to apply consistent assessment judgments across different types of submission.

If we want to receive identically formatted items, which all address the learning outcomes using the same approach, then of course mandating a single format with a strict word count is the way to go. But if we want to encourage an attitude to assessment which encourages creativity in new lecturers, and which acts as a development vehicle for their own digital skills, then we must reduce concerns about word counts and encourage junior colleagues to develop and use their professional judgment in this matter. The student quote above shows the thoughtful approach taken by one student to address the issue for themself.

Frustratingly, even by using word count as the reference point for parity we may ‘other’ some of the more creative approaches that we seek to encourage and normalize, but ultimately wordage has long been the currency of higher education. It is good to see some universities being pro-active in setting out a steer for equivalence so that individual staff do not feel that they are being maverick with word counts when seeking to encourage creativity.

E-assessment for work based learning: Functionality v ideals

There are very close ties, or at least there can be, between e-learning, e-assessment and work-based learning. The compatibility of e- methods and work-based, work-located studies is in many instances because of:

· Pragmatic considerations

o Access anytime, anywhere using asynchronous technologies.

· Quality concerns

o E-learning allows the HEI to contain a direct link in to provision that may otherwise be entirely delivered by a partner organisation.

Where e-assessment is used for these reasons only (without consideration of the wider learning design) there may be limited benefits, a reductionist approach results. What is lost when only a narrow rationale is used for choosing e-assessment for work-based learning?

· Association with authenticity If assessment is a bolt-on, a means to an end, then the opportunity to enable work-based learners to use and build upon their day-to-day practices may be lost in the rush to simply weigh knowledge.

· Association with social justice E-assessment offers a chance to level the playing field a little more. In getting away from essay writing and enabling the creation of multi-media artefacts for assessment, learners can play to their strengths and enhance authenticity. However, to enhance the chance of such an approach succeeding the use of media playfulness needs to be engrained into delivery/the learning journey, the e-infrastructure, the human support and the assessment success criteria. This is not a concept easily bolted on!

· Disjunction – Constructive alignment remains widely accepted good practice for all learning and teaching, based on this accepted wisdom alignment between assessment tasks and learning should remain clear. Whilst the content of learning can form an alignment, even when using assessment as a bolt-on, there may be disjunction when the means of assessment is remote from the learning experience. It could be argued that the assessment instrument should be synergistic with the journey.  For example a student sitting down to take a computer aided test with scenario based question and answers when none of the delivery has been in this way may result in feelings of separateness between learning and assessment; likewise for a summative portfolio to be online after a face to face delivery which did not in any way utilise technology erodes the possibility of full and deep engagement and potentially acts to make the technology alien and intimidating under pressure.

As we design work-based learning initiatives with an e- element in either delivery or assessment or both care must be taken to be holistic in looking at how to support, how to maximise the benefit and how to ensure that for the learners a sense of journey is maintained. There are potentially lost opportunities and also dangers of disjuncture in being overly focussed on finding a means assessment that ‘will do the job’ of providing some measure of learning without considering such designs more holistically with reference to integration with delivery, support (tutor or peer) and [work-based, local] context.  This tension then in the relationship between e-learning and work-based learning equates to function v. ideal design.

Patchwork discussions

Really interesting to catch up with Kevin Brace earlier in the week. Kevin, based at Aston, and I met up to explore ideas about patchwork text (or, more accurately patchwork media) on the back of some ALT mail list exchanges.

Thoughts arising and articulated through our discussions ….

·      Patchwork media is about process and not just product (a focus on the produce dis-aggregates the learning).

·      Patchwork approaches are inseparable from the belief that learning is a journey constructed and are inseparable from social processes and dialogue.

·      Curriculum design and assessment design are inseparably interwoven for maximum impact.

·      Grading reflective portfolios is a messy business but can be made more simple by focusing on the meta-level attributes.

·      Buy-in to patchwork needs to be [sometimes, heavily] facilitated and does not just happen.

·      Templates for patchwork can be practical and solve a whole range of potential issues around logistics, hand-ins, learner scaffolding and perceived parity; however templating approaches may be as stifling as they are enabling.

·      Do not get consumed by the media! Encourage patch creators to think critically about their media choices and critically review choices in the light of experience. (How does this stack up with some marking ruberics?)

·      Attempting to create or re-create patchwork approaches from other study programmes may be extremely difficult as the nature of patchwork is that it is very much shaped by the subtleties of implementation (e.g. strategies for peer review, the place of technology, the immersiveness of facilitators, the use of structured vs. open patches). Perhaps better is to create a context relevant and practically achievable version of the approach.

·      Despite very many worthy efforts the use of rather cumbersome e-portfolio tools can perhaps, sometimes, act as a barrier to plain old discussion and sharing – simplicity in tools may be under-rated.

patchwork elephant

 

Word count

At various points (forever) I have been pressed on the issue of wordcount, by students and colleagues. In the past, an issue that was frequently of concern to others was around equivalence – when multi-media is used how do we weigh the words? How many words does a picture count for, how many does a table or an animation count for. The conversation around this always felt arbitrary and very stifling especially when credit volume is measured by hours not words.

How many words is reasonable for an assignment?

Whether we suffix with,

i. to show learning

ii.  to appease systems and norms

… will inevitably generate different answers.
In working with different partners, colleagues, systems and students over a period of years it is plain to see that  the word count issue appears to be ingrained in perceptions, beliefs and practice and across the board. New outlooks which do away with governance by word count will be hard fought.

In grappling with this over-emphasised issue and ignoring the futility of the question (to self) I take comfort from Phil Race’s mantra …

There would be no more silly relationships between word counts and credit points.
For example, no more regulations stating that a given number of credit points = 3000 words. Such ‘equivalences’ encourage low-level ‘word-spinning’, and give advantage to students who are good at ‘waffling’, and disadvantage students who are learning in a 2nd language. Shorter word-constrained tasks (such as a 200-word – exactly – summary, or a 150-word argument against something, or a 300-work review of three sources, and so on) generate student work (and thinking) of a higher quality, and take far less time to mark, and make marking much more reliable, and just about eliminate plagiarism possibilities.


5 Thoughts from the Jisc event – Employer Responsive Provision: Designing, delivering and supporting flexible learning opportunities

A productive day at Aston University spent with colleagues sharing thoughts and practices around employer engagement design, technology, assessment and policy. The main emerging points for me from today….

  1. The HE sector is rising to the challenge of working with business in many ways, whilst there may be very many unanswered questions, we are increasingly aware of the issues facing us, which is surely the first step to solving them!We are beginning to get to know the unknowns.
  2. The pump-prime funding of employer engagement has been a useful, perhaps essential stimulus, it would be helpful to know if the end-game is sustainability and within what time frame we are working to achieve self-sustaining engagement.
  3. The place of interoperability and sensible systems architecture has emerged as key to unlock a smooth online learner experience. To avoid bitty, disjointed experiences systems must talk. Today I realised that this was a very real issue if HEI’s are to face outwards, however pursuit of perfect interoperability must not distract from the job of getting on with engagement.
  4. Patchwork text has never been so relevant. Artefacts make learning authentic, a wrap around commentary adds value.  The artefacts and commentary can be a way of separating (for expertise alignment purposes) the competence, work based and technical aspects from the reflective, extended and meta- learning aspects.
  5. A running theme in discussions was around the language of employer engagement – is HE talking in the same language as business? How much of a problem is this?  …  My own view is that the challenges of language are far from insurmountable and can mainly be solved through mutual understanding and engagement.

Published … Advancing the Patchwork Text: The Development of Patchwork Media Approaches

Following on from the July 2009 learning Conference “The” paper is now published in the International Journal of Learning and available here:

http://ijl.cgpublisher.com/product/pub.30/prod.2169

Learning & Teaching Forum, Harper – Patchwork media for practice.

Patchwork_- Harper Teaching & Learning Forum

On Thursday 9th I gave a presentation on Patchwork Media at the Teaching & Learning Forum at Harper Adams. The presentation is available as a PDF download from the link above. This dovetailed nicely with the work I had done in Barcelona. The Harper presentation was more about the practicalities of patchwork whereas the Barcelona one focused on unpacking the online mechanisms for supporting patchwork.