Chapter 8 Case study on representation: Making designs explicit through visualisation
This chapter presents a case study on the development, use and evaluation of tools for representing learning designs, CompendiumLD. A rationale for the development of the tool will be provided, along with a description of the tool, its functionality and use. A comparison will be provided of related visualisation tools. The chapter will conclude with a discussion of the benefits of such tools, along with the challenges they present.
Application of the empirical findings to the development of design tools
The extensive range of data collected in the OULDI described elsewhere in this book provides a rich body of empirical evidence to inform our thinking and the development of appropriate tools for design. In summary we have conducted a series of interviews, workshops and focus groups with practitioners to elicit their approach to design and any associated challenges. In terms of guiding and representing learning designs, which have adapted an argumentation and visualisation tool (Compendium) to create a visualising tool for design, CompendiumLD. This section provides a summary of some of the key findings from the empirical data, in terms of how it has informed and developed our thinking in the development of the CompendiumLD tool, a more detailed discussion of some of the findings from the interviews with teachers/designers in provided elsewhere Cross et al. (2008)) and Clarke and Cross (2010).
The empirical data provided a rich picture of the way in which teachers design. It was evident from the data that there was no one perfect tool for design and that individuals had different preferences for how they went about the design process – some sketching ideas out and linking them, others working systematically from learning outcomes, others using the subject content as a baseline for development. The interviews and case studies provided valuable insights into the design process that cluster into five overarching themes (Figure 1).
Figure 1: The five overarching themes on design approaches and methods
The most prominent finding from the interviews was that design is a messy, creative and interactive process, and that even when working in teams there is a large element of individuality in the design process. Teachers design at different levels of granularity and focus on different aspects of design over the curriculum design lifecycle. Both the interviews and the workshops gave us a clearer understanding of the design strategies that teachers adopt. Foci for design include: looking at learning outcomes and mapping these to assessment strategies, integrating the use of externally resources with locally authored materials, designing activities to test understanding, integrating a range of tools and approaches, addressing different learner preferences and levels of competence, mapping to externally prescribed professional requirements.
It’s not in one direction. Not sure if I always start with aims, sometimes I do! Broad aims, then thinking about the mix, go to the palette and look at existing resources, what will the budget allow us to do (chairs hat on), what additional resources do we need, which would be most effective to teach certain things. For example, we need this software to help teach linguistic analysis. We might want some video analysis, so think about how to bring in video sequences, what videoing needs to be done. Then start writing. It’s chicken and egg. Sometimes start with study guide and then think about activities, and then think I need this bit of video. But you don’t always have luxury of working in this direction or budget to do filming so start looking for other sequences and build activities around those. [Interview 160607]
The following quote gives an example of how a teacher iteratively develops their concept of the course over a period of time and how they kept an evolving record of relevant resources and materials for the course.
I was building a sense of what the new course might be … we must remember to do x, or a url of relevance [Interview 160607]
It was also evident that design for a new course is very different to design when redeveloping a course based on interpretation of student feedback and evaluation. The interviews revealed that there was no simple route to teachers accessing support and guidance on the design process; little use appeared to be made of online resources and networks – most adopted a serendipitous approach, rel
ying on peer practitioners and close colleagues for ideas. One interviewee from the case studies conducted by Wilson said:
This says more about me than it does about the stuff really but I preferred the corridor conversations. It was a way of … I had invested quite a lot of money in coffee and so there were a whole set of people across the university who I took to coffee and pumped them for what I could really. [Case study Interview, 210107]
Those interviewed recognised the value of sharing and reuse, but little evidence emerged of a significant amount of sharing and reuse. Different forms of representation of learning activities (textual, visual, etc.) all had different pros and cons and there was evidently a distinction between the process of producing a design and design as an artefact. When shown visual presentations of learning activities for example, many of those interviewed found it difficult to interpret them, to apply/adapt them to their own context. However, on further probing they could see a genuine benefit in using visual tools as a means of mapping their own practice, as is evident in the following quote from one of the interviews.
[On the value of a visual representation] It always needs to be brought to life, to have some form of enactment… Would I want to see what someone else has done, yes I suppose so. [Interview, 141107]
The conflict between the process of dynamic creation of an activity and the associated sense of ownership the designer has in the process, contrasts with design as a product, a static artefact. For example one interviewee struggled to see the benefit of a visual representation of someone else’s design, even though it was an activity in her subject area. She continued later in the interview to argue for the need for a mediation role to help interpret designs and as she says ‘make them come alive’:
[One being shown a visual representation of a learning activity] It’s such a different context and level. This is language teaching rather than linguistic teaching. And there isn’t the contextual information, even with you having just explained a little, which helped, without you there I’d be looking at this and thinking… I think there’d be too much work to look in to this plus the recontextualisation. I wouldn’t spend the time to be honest.
I really think you need someone who goes to the course team, although not necessarily staying with them. And sits down, not right at the start but a little way in, and asks what are you teaching and what resources are you going to use alone or in combination and that person would go away do some work and come back – have some insight into bringing together their knowledge of the technologies available and which would best fit your intention and provide you with a map – that’s when a map would work, they’d be bringing it alive. [Interview, 141107]
The interactive and holistic nature of the design process came out strongly across the data:
One of the difficulties is mapping the whole process I have tried to approach course design using a holistic approach [Interview, 121107]
Teachers differed in the extent to which they worked visually or textually, although evidently the institutional quality audit and validation processes require textual representations of some description for courses. Some used software, others sketched or wrote ideas, one teacher had a scrapbook which he used as he was developing his design ideas:
It’s in words, not diagrams a dumping ground for thoughts – [to] capture thoughts [Interview, 121107]
Others used visualisation as a means of mapping different elements of the design process:
List of words clustered into blocks, arrows…can you have clusters link to TMAs [Assignments] [Interview, 141107]
Start from assessment strategies and learning outcomes and get an alignment [Interview, 151007]
I tend to sit and doodle a map – will draw the logic and flow of the course on paper and then go to Compendium. Then the problem is sharing it [Interview, 291107]
The interviews also highlighted a number of contradictions about the process of design, forms of representation for design and the nature/type of support, which teachers wanted:
· A tension between design as process and design as artefact
· The difficulty of capturing what is inherently an implicit process
· The demand for subject-specific case studies and examples, which are then not used and adapted
· The variety of influences on the different forms of representation and individuals interpretations of them
· The desire for specific, just-in-time help and support and the difficulty of capturing support in an online tool
· How to map the evolving, dynamic and changing nature of design.
Repurposing OER through making inherent designs explicit
In related work, we explored teachers conceptions of design in terms of how they might repurpose stand-alone Open Educational Resources (OER) for collaborative learning activities (G. Conole, McAndrew, & Dimitriadis, 2010). The findings from this work were similar to the examples provided above. Analysis of the data revealed a number of themes that are discussed here. Part of our approach is predicated on the notion that OER have inherent designs and that if we can make those designs more explicit this will aid repurposing. A number of themes emerged with respect to this, which are discussed in this section. In the following sections participants are represented as P1, P2, etc. while the workshop facilitators are indicated as F1, F2, etc. In the following section references to CSCL patterns are indicated in italics.
It was evident that there were a number of ways in which textual representations could emphasise different aspects of the design – some may be description, others more metaphorical and others still more operational – for example a bullet list articulating steps in a learning sequence. A common approach adopted by the participants was to have a temporal sequence. Another strategy was to focus mainly on the content and associated resources. Participants started from different perspectives; some began by considering the learning objectives, whilst others started with the content or activities.
P2: “My resource is a design by itself. So, it is the design of an activity, it is the representation of that, a few bullet points and then a graphical representation. …. So the resource basically represents arrows pointing into a sequence of the activities.”
It was interesting to see the extent to which each of the representations was easily sharable with others. More often than not a dialogic engagement was necessary to help make meaning of the design and to clarify misunderstandings. The exercise and subsequent discussion enabled us to tease out both the main facets of design and participants’ different perspectives and approaches. In addition to articulating objectives, content and tasks, some of the participants evidenced a subtler level of design – associated with the inherent principles of the design.
P3: “My resource is task-driven, so that is the principle and also it integrates many pedagogies into the content, so, and also it is question based.”
In terms of principles we explored a little whether or not they had articulated a principle around individuality/collaboration. A range of characteristics was identified as being associated with the design – the objectives, generic characteristics, sequence of tasks undertaken, an individual or collaborative focus. Participants recognised that it was important to focus in terms of clarifying what information was essential to communicate so that the activity could be subsequently taken up and adapted by others.
F1: “Just try to think again of what elements you wrote down and what elements you used when you tried to explain it to your neighbour and try to think whether they were mainly based on objectives, mainly based on the characteristics of the activities, of a temporal sequence or …”
One of the participants suggested that it would be valuable to have multiple views of the same design each view representing a different aspect.
P7: “So probably having different layers of visualization of the same structure could help filter the relevant information if you are looking at the learning objectives, or if you are looking at interactions, something like that, so, other thing that we were thinking about it probably what is missing is a legend of the different items, because we understood that there is a mixing of 2 layers, one is devoted to the designer, for example, all the questions in blue are annotations for the designers while for example it is very clear that the sequence for students is talking to the student verbally, it is talking to him, so probably having the legends saying ok, question mark annotation for the designer and the red bits are feedbacks we had from one evaluation and then filtering visually this information according to the task you are following.”
This participant also argued that visualisation potentially has additional power, if a semantic dimension is included.
P7: “A semantic of visualizations, really we understood that some of the connection are more related to cognitive activities of the design where as others are tactical activities of the use (missing comment) and cause and some other connection are like database connections with the resources and what they are looking, so probably having different semantic of the connections and representations.”
Another aspect of importance that participants mentioned was identifying the quality and provenance of the resource; i.e. designs need to do more that display the sequence of activities, users need some indication of how effective and fit for purpose it is. There are two ways in which this can be included. Firstly, in the design representation itself, however the more detail that is included in the design the more complex it is. Secondly, an alternative is to have a wrap-around dialogue about the resource and its design, in a tool such as Cloudworks.
The data revealed that deconstruction and subsequent reconstruction of OER is complex, indeed it is possible to identify four layers that need to be considered to make most effective repurposing of an OER:
1. Visual representation of the design – how can the implicit OER design be made more explicit and hence shareable?
2. Opinion of goodness – how appropriate is the OER for different contexts?
3. Transferability through pedagogical pattern – how can generic patterns be applied to specific contexts?
4. Layer of discussion, critique and contextualisation – how can sites like Cloudworks act as a supporting structure to foster debate between those using the same OER?
In conclusion describing design was seen as a difficult and unfamiliar task:
· having multiple solutions;
· many options for what to include;
· being hard to interpret in a consistent way;
· only able to capture partial details in the example representations; and,
· needing additional information for clarification.
Development of CompendiumLD
CompendiumLD has been developed out of our interpretation of the empirical data we have collected and a realisation that visualisation is underutilised as an approach to adopting a creative approach to the design process. Brasher et al. (2008) and Conole et al. (2008) provide more detailed information on the tool and associated technical development; only the salient features are described here.
We wanted to use a flexible tool as the basis for our initial prototype. We considered various drawing packages, as well as more specialised mind mapping tools (such as Inspiration and MindManager). In the end we choose to use Compendium (http://compendium.open.ac.uk/i
nstitute/), a visual representation tool, originally developed for enabling group argumentation, which was produced by researchers at our own institution. We selected Compendium for a number of reasons. Firstly because it was produced at the Open University, we felt there was more opportunity for further tool development specifically in terms of learning design requirements. Secondly, Compendium supports the creation of a range of visual mapping techniques, including mind maps, concept maps, web maps and argumentation maps (2008), which we felt offered the potential for a range of flexible approaches to the design process. Compendium comes with a predefined set of icons (question, answer, map, list, pros, cons, reference, notes, decision, and argument). The creation of a map is simple; users drag icons across and drop them onto the main window thus creating a node. Relationships between the nodes are built up by dragging between nodes thus creating a connecting arrow. Each node can have an associated name attached and displayed; if a more detailed textual description is associated with the node an asterisk appears next to the node. If the user hovers their mouse over this the content inside the node is revealed. Other types of electronic files can also be easily incorporated into the map such as images, Word files or PowerPoint presentations. The reference node enables you to link directly to external websites. Icons can also be meta-tagged using either a pre-defined set of key words or through user generated terms. Maps can be exported in a variety of ways from simple diagrammatic jpeg files through to inter-linked websites.
Compendium provides a utility by which users can create and share new sets of icons, for use as nodes. These sets, know as ‘stencils’, contain ‘items’ where an item defines certain properties of a potential node such as its image icon and label. In the standard version of Compendium, each item inherits the behaviour of one of the standard node types. These standard node types are node which has an icon, text label and other descriptive textual information, link node which links from a node to another node, and view which is a collection of nodes and can be displayed either as a map or a list. There are several different mechanisms by which a user can interact with nodes. These include drag and drop (e.g. to instantiate a node as described in the preceding paragraph), double-clicking (e.g. to display and edit details of a node including its text), right clicking (to display a menu offering actions and operations to apply to the node), left-click (to select a node, or allow other menu driven operations to be executed on the node). We adapted Compendium to make it more explicit in terms of its use for learning design and this version of the tool is referred to as CompendiumLD – it includes additional functionality such as tailored LD stencil sets and in situ help. In CompendiumLD, behaviour specific to learning design has been implemented for these modes of interaction as explained in the next few paragraphs. Figure 2 is a screenshot of CompendiumLD, showing the LD-OU stencil towards the left hand side, and a map describing each item in the main window.
Figure 2: A screenshot of CompendiumLD
In addition to the standard icon set available in Compendium, we have created a series of stencils specifically for learning design:
· Approaches to learning design
· LD-Conditional stencil
· Learning Design icons
· Learning Design templates
· Sequence mapping – a stencil to help with laying ut learning activities
CompendiumLD enables its users to visually represent learning activities in a flexible way. They can map connections between tutors and learners, tasks, resources and tools, and a variety of notes and links to external websites or documents. The process of mapping a learning activity in this way involves the user in a cognitive process of externalising their understanding of the learning activit
y. This facilitates and drives development of their own understanding of the nature of the activity, and the map facilitates communication of this understanding with colleagues. We contend that the process of mapping using CompendiumLD can improve the quality of activities that will be realised.
In addition to providing a visual representation of the design process, we also wanted the tool to provide some form of in-built scaffolding and support to guide decisions at various points in the process. This we have achieved in a number of ways – by providing suggestions for each of the different types of nodes, additional resources and examples, and access to a restricted searchable set of additional help features. As an example of the first kind, when a user drags and drops a “role” node onto the main design area, they are presented with a menu to select the type of role as shown in figure 3(a). Therefore this simple prompt reminds them of typical kinds of roles which they might want to include in their design sequence. The users are not restricted to these roles however and can choose to type in an alternative role of their specification. This sensitive balance between guided scaffolding and user flexibility/creativity is an important design principle for our development of CompendiumLD. A similar form of scaffolding is available for the “tool’ mode. When a user drags and drops a tool node onto the main design area, they will be presented with a menu to select the type of tool as shown in figure 3(b). Note that the options for tools include ‘Other’, which enables users to specify a tool of their own choosing. The other tool types available for selection are those currently available with the Open University’s VLE. The “Other” type allows the designer to specify a tool for face-to-face interactions, or a tool not currently supported by the VLE. The tool type selected is stored in CompendiumLD’s data model, and tools to query the contents of this data model could be used to examine tool usage.
Figure 3: Prompts presented for role and tool nodes
In terms of provided additional help, users of the system have the option of letting CompendiumLD offer context-sensitive help. For example, as the designer types into a task description label, the words typed are scanned and help related to selected verbs (e.g. collaborate, consider. discuss, reflect etc.) pops up. An example of such a help window is shown in Figure 4. In this example, the designer has typed ‘Discuss’ into the task label: this prompts the application to pop up a window showing tools to support discussing and existing activities that include tasks which include the word ‘discuss’. The set of tools shown in this help window are selected using a verb-to-tool look-up table based on verbs within a task taxonomy similar to that described by Falconer et al. (2006); the set of activities is generated by searching the database maintained by CompendiumLD for activities including tasks with ‘discuss’ in their label. Further help is provided by the ‘About..’ buttons. These buttons initiate a customised Google search of selected web sites (http://www.google.com/coop/cse?cx=000971387191123125524%3Alworuyth0qs). The web sites were chosen because of the quantity and quality of the information they provide about use of tools in learning and include sites such as http://www.learningdesigns.uow.edu.au/ and http://www.educause.edu/. We adopted this pragmatic approach for a number of reasons. To create our own hand crafted text would not only be time consuming but would suffer from quickly becoming dated. However the alternative of a free Google search arguably produced a daunting and untargeted set of resources. The middle approach we have adopted enables us to focus in on a small set of quality assured sites, which we have checked for relevance and which are likely to be sustained and updated in the near future. Using a customised search allows means that potentially other institutions installing versions of CompendiumLD could choose to select and include their own tailor made set of resources, which might include institution-specific examples. In our own case we have a set of tailored resources on tools and their uses within the OU context – ‘the learn about guides’, as well as a set of institutional cases studies on specific uses of VLE tools.
Figure 4: Help relevant to a particular activity
Help related to tools that the designer drags and drops onto the window may also be shown. Figure 5 shows an example of help presented when the designer selects ‘Wiki’ for the tool type.
Figure 5: Help relevant to a particular tool
Figure 6 represents a screen shot of part of the learning activity associated with a third-level environmental course (i.e. equivalent to the final year of a full-time, three year degree course). Two roles are shown (student and tutor) along with their respective tasks. Tools, resources and outputs (i.e. assets) associated with each task are shown alongside, with arrows indicating connections.
Figure 6: Visual representation of part of a collaborative role play activity
Our ultimate goal is to provide adaptive and contextualised information on different aspects of the design process, tailored to individual needs and delivered on a just-in-time basis. We have now undertaken an extensive number of workshops enabling practitioners to explore CompenidumLD. These have included workshops within the OU, as well as externally (including the University of Porto, the University of Cyprus, for the EdTech community in Canada, and at numerous conferences). Evaluation of feedback on the use of the tool has enabled us to improve it. We were surprised at how far the participants got in representing their designs and it did seem during the sessions that CompendiumLD acted as a useful tool to help them articulate and share their thought processes. A few participants however commented that they did not find representing their designs visually helpful, stating that, for them, pencil and paper/discussion would be preferable. It is likely that such a focus on the visual aspects of the design process will not suit everyone, but overall most participants were positive both during the session and in their evaluation feedback.
Feedback was also positive about our approach to helping teachers/designer consider in more detail the general issues and use of visualisation and its value in improving the practice of design. There were some disadvantages noted regarding visualisation but these were ones we anticipated and provide further valuable food for thought (i.e. someone said ‘some designs may be difficult to describe using this visualisation’). Much of the focus of our use of CompendiumLD during the workshops has been designing at the level of an individual learning activity, whereas a number of attendees also saw the value in stressing course-level design techniques and process as much as for individual activities and felt that this would have a lot of appeal to teachers. However, whilst the principles were appreciated those new to CompendiumLD did encounter some usability issues and asked for more guidance and support. In view of this, it is planned that further support will be provided within the application. For example, a movie with a commentary describing the basics of creating a learning activity will be provided with the next release.
The empirical evidence we gathered on practitioners’ design practice has informed our development of the CompendiumLD tool. We believe that there is no one perfect ‘tool’ for design and instead prefer to adopt a pick and mix approach to the design process. Our initial findings are positive; however it is clear that there is a need for further research – practitioners are crying out for examples of good practice and guidance in design. However previous research shows that representing learning design practice and providing appropriate support for learning designers is both difficult and contested. By bringing together both narrative accounts of learning designs with notational maps showing the design visually, we hope to address and find practical ways of approaching the key issues in this area. CompendiumLD seems to provide an easy to use visual tool to help represent different learning designs.
However, it is also evident that there are a number of drawbacks with a tool like CompendiumLD. It is available to download to PC, Mac and Linux platforms. However the tool is relative difficult to learn and it not always intuitive to use. We have had considerable success in recent workshops using paper-based print outs of the icons, rather than the software per se. Another issue is that CompendiumLD is not able to represent the full range of design representations, which were discussed in Chapter 5. A better solution would be to have a web-based tool, which enables users to oscillate easily between the different design views. In addition, despite our best efforts to include scaffolded guidance and support, the help facility at the moment is limited and is not as comprehensive as that available in pedagogical planner tools such as DialogPlus, Phoebe and the LDSE discussed elsewhere in this book. Using Cloudworks as a form of pedagogical wrapper around CompendiumLD is one way of addressing this shortcoming and has been used successful in a number of our workshops. For example in a workshop at Brunel University on 9th November 2009, participants shared and discussed the designs they created using CompendiumLD (http://cloudworks.ac.uk/cloud/view/2639).
Brasher, A., Conole, G., Cross, S., Weller, M., Clark, P., & White, J. (2008). CompendiumLD – a tool for effective, efficient and creative learning design.
Clark, P., & Cross, S. (2010). Findings from a series of staff interviews about learning design, representations of design, design process, evaluation and barriers, Appendix 2, The OU Learning Design Initiative Project Phase Two Report, Embedding learning design and establishing a reversioning culture. Mltion Keynes: The Open University.
Conole, G., Brasher, A., Cross, S., Weller, M., Clark, P. and White, J. (2008). Visualising learning design to foster and support good practice and creativity. Educationa
l Media International.
Conole, G., McAndrew, P., & Dimitriadis, Y. (2010). The role of CSCL pedagogical patterns as mediating artefacts for repurposing Open Educational Resources’ in F. Pozzi and D. Persico (Eds), Techniques for Fostering Collaboration in Online Learning Communities: Theoretical and Practical.
Cross, S., Conole, G., Clark, P., Brasher, A., & Weller, M. (2008). Mapping a landscape of Learning Design: Identifying key trends in current practice at the Open University.
Falconer, I., Conole, G., Jeffrey, A., & Douglas, P. (2006). Learning Actiivty Reference Model pedagogy, report for the JISC LADIE project. Southampton: Southampton University.
Okada, A., Shum, S. B., & Sherborne, T. (2008). Knowledge Cartography: Software Tools and Mapping Techniques: Springer %@ 1848001487 %7 1.