Designing effective MOOCs
I am currently writing a paper which I hope to submit to Educational Multimedia International which looks at effective ways of designing MOOCs. Here is the draft to date, comments welcome!
Massive Open Online Courses (MOOCs) have expanded significantly in recent years and are challenging traditional educational fee-paying offerings. The advantages of MOOCs are cited as the fact that they are free, that they enable participants to be part of a global community of peers and to have the experience of learning through social media, and that they offer the potential for opening up educational and facilitating social inclusion. Nonetheless there are challenges associated with MOOCs. Firstly, most have very high dropout rates. Secondly, there are challenges with approaches to recognition of learning and issues with learner authentication and cheating. Thirdly, there are issues with providing support at scale. This paper argues that effective design of MOOCs is key. It focuses on the description of a twelve-dimensional classification schema for describing and designing MOOCs (Massive Open Online Courses). It then uses the schema to describe five MOOCs, which are respectively based on associative, cognitive, constructivist, situative and connectivist pedagogies. It then describes the 7Cs of Learning Design framework and discusses how it can be used to design MOOCs.
A Massive Open Online Course (MOOC) is an online course taken by a significant number of participants. MOOCs have gained traction in education in recent years and a number of evaluations have been undertaken to ascertain how learners are participating in them and how they are learning. The Open Education Europa ‘European MOOCs Scoreboard’, indicates that there are currently over 800 MOOCs being offered by Europe-based institutions.
The original vision behind MOOCs was the idea of harnessing the potential of digital technologies to learn at scale through a network of peers. The original MOOC, Connectivism and Connected Knowledge (CCK08), promoted a Connectivist approach to learning. Learners were encouraged to create their own Personalised Learning Environment and to engage in dialogue and exchange of ideas with their peers through appropriate social media. There was no formal learning pathway or correct set of prescribed tools; each learner adopted their own approach. This type of MOOC has been labelled a cMOOC. More recently we have seen the emergence of more individually focused, didactic MOOCs, where the emphasis is on learning primarily through content and videos, supported by e-assessment elements. These MOOCs have been labelled xMOOCs. The number of MOOCs increased dramatically following the first MOOC. Providers include: edX, Udacity, Coursera and FutureLearn. MOOCs are very much a disruptive technology (Christensen 1997), as they re challenging traditional educational business models. The rapid growth in the number of MOOCs lead to the New York Times naming 2012 ‘the year of the MOOC’. A recent survey suggests that MOOCs are supplementing traditional Higher Educational offerings and democratising learning. The survey looked at 13 MOOCs offered on the Coursera platform. Findings included:
- Students in the younger group often reported taking MOOCs in topics not taught in their schools.
- Students in the youngest group also reported taking MOOCs to explore different disciplines to help weigh academic and career choices.
- Those in the over-65 group reported taking MOOCs to pursue lifelong learning, to keep their minds active and to mentor younger students in their own professional field.
- Students who said they enrolled in MOOCs because of limited access to higher ed chose them because MOOCs were available despite their financial or mobility limitations.
Veltsianos et al. (Veletsianos, Collier et al. 2015) argue that in order to fully understand participation in MOOCs it is important to look at more than just log files and online interactions. They were interested in how people experience MOOCs and why they engage in particular activities in the way they do. The findings of their study included the following:
- Successful learners have highly developed study habits.
- Students take notes, if they take more than one MOOC on a similar topic they combine the notes.
- There is evidence of off platform participation via social media or face to face.
- Online learning is an emotional experience; both in terms of excitement and disappointment.
- Life’s daily routines shape the way in which people participate in online courses, in other words the courses need to fit in with other activities individuals are involved with.
- Finally, drop out rates are not necessarily negative, some learners choose to only do part of a course for a reason.
Opinions as to the value of MOOCs are divided. On a positive note they are free and hence are seen as potentially supporting social inclusion and providing an opportunity for participants to experience being part of a global community of peers. Kopp et al. conclude that there are good reasons for HE institutions to offer MOOCs in such a context: firstly, to ‘fulfil their obligations in the field of lifelong learning by providing scientific content to the general public’ (Kopp, Ebner et al. 2014) and secondly to enable students to accumulate credits towards their qualifications in a more flexible way by learning online.
On a negative note, many point to the high drop out rates and low levels of participation and some feel that MOOCs are more about ‘learning income’ rather than ‘learning outcomes’, and that they are merely a marketing exercise. Indeed many institutions who have developed MOOCs state that their main reason is to get a feel for the MOOC experience and look at how it sits alongside their traditional educational offerings.
A key issue with MOOCs is the way in which participants can achieve recognition for their learning. A number of models have emerged. Participants might pay to get a certificate of attendance or participation. Alternatively they may be awarded digital badges for an achievement or the gaining of particular competences. The OERu is an international consortium of institutions. Learners can approach one of the members and ask for recognition from that institution of their learning for studying through OER or MOOCs.
The OpenCred project explored the range of ways in which non-formal open learning could be recognised. Non-formal learning is defined as learning which is embedded in planned activities not always explicitly designated as learning (in terms of learning objectives, learning time or learning support), but which contain an important learning element. Non-formal learning is intentional from the learner’s point of view. The OpenCred project developed a typology around four characteristics of open educational courses:
- Formality of recognition
- Robustness of assessment
- Eligibility for assessment/recognition
- Affordability for learners.
The executive summary of the final OpenCred reports states:
The development of Massive Open Online Courses (MOOCs) in the Higher Education sector is reaching a point of maturity, where they are no longer an experimental form of learning but moving towards being mainstream. Although few providers of open education have so far awarded credentials, this is beginning to change.
The key findings from the project were:
- Robust assessment is central to recognition of open learning, and that the need to pay for robust assessment leads institutions to either pass on the cost of assessment to learners, or require learners to enrol at the institution – in both cases reducing the openness of the assessment and recognition process.
- The recognition of open learning is, at present, limited to components of programmes rather than full qualifications. Some organisations are working towards this objective but such initiatives are in their infancy and, as yet, there are no examples available of an individual receiving a full qualification via open learning.
- There are some common perceptions which impact on the sector’s ability to provide recognition. These relate to the way online learning in general is perceived, and online assessment in particular; the value of ECTS credits as a currency, and the value of badges as motivational tools in open education.
- There are pockets of experimental practice in workplace settings, but these are currently very limited.
Another issue is how to support learners participating in a MOOC, with often tens of thousands of participants, providing tailored individual support is not possible. There are a number of alternatives. The first, prevalent in cMOOCs, is to encourage participants to create their own Personal Learning Environment (PLE) of tools and peers to support their learning. This might include use of hashtags on social media to filter information or the creation and use of a network of peers to interact with. The second is to provide tutors, who summarise key elements of learning at key points in the MOOC.
EFQUEL facilitated a 12-week series of blog posts exploring the issue of quality and MOOCs. Selwyn and Bulfin (2014) did a meta analysis of the MOOC literature and identified the following themes: the fact that MOOCs vary in size and scale, the issue as to whether they are free or not, how they sit alongside traditional educational offerings, the issue as to whether they are transformative, the way in which they are challenging traditional educational business models, the types of pedagogical approaches that are facilitated, the nature of the content and degree of interactivity and communication, the relationship between the participants and the tutors, the forms of recognition of learning and assessment, and the ways in which technologies are used.
A classification schema for MOOCs
The classification of MOOCs as either xMOOCs or cMOOCs is arguably too simple given the diversity of MOOCs we are now seeing. Conole (2014) has developed a classification schema for MOOCs, which consists of twelve dimensions (Tables 1). Each dimension can be seen as a spectrum, from little or no evidence of that dimension through to a significant amount.
Table 1: A twelve dimensional classification schema for MOOCs
The degree to which the MOOC is open, ranging from closed Learning Management System courses which require the users to login and potentially pay for access through to completely open courses that use open source tools, where participants are encouraged to share their learning outputs using a creative commons license.
How large the MOOC is in terms of the number of participants, which may range from a few hundred to thousands.
How diverse the participant population is; a small specialised course for local doctors for example is likely to be fairly homogenous in terms of the background and experience of the participants, in contrast a large MOOC on Art Aesthetics is likely to have a diverse participant population
In terms of how much and what type of multimedia is used. Some MOOCs are primarily text-based whereas others make significant use of multimedia and have a high degree of interactivity.
This dimension describes the way in which participants are encouraged to communicate with their peers and their tutors. This might range from limited use of discussion forums (which may or may not be moderated by the tutors), through to significant use of communication through a variety of social media tools.
This dimension refers to the ways in which participants are encourage to collaborate together, this might range from no collaboration (particularly in xMOOCs where participants primarily work through the materials on their own) through to significant collaboration with participants working in groups on a shared set of activities.
Reflection is an important facet of learning (Dewey 1916). This dimension reflects the extent to which participants are encouraged to reflect on (and perhaps apply) their learning. Some MOOCs will not explicitly state this, whilst others might include statements such as ‘reflect on what you have learnt to date’ or ‘apply your understanding to your context’. Alternatively the participants might be encouraged to write reflective blogs and comment on the blog posts of other participants.
Some MOOCs, such as cMOOCs, deliberately do not have a specified learning pathway through the materials; the emphasis is on participants creating their own learning pathway and Personal Learning Environments. Other MOOCs may have a prescribed learning pathway to guide the learners. Other still might have alternative learning pathways through the materials for example in the form of a ‘MOOC-lite’ route or a more extensive route through the materials.
This dimension evidences the degree to which the MOOC, when it is being designed and in the evaluation of its delivery, has an associated quality assurance process. This might range from no quality assurance, where the MOOC is developed by an individual teacher, through to some form of relatively informal peer review through to high quality assurance through a formal review process and a number of iterations and improvements.
This ranges from no certification associated with the MOOC, through to the assignment of badges on completion or different aspects of the MOOC or achievement of particular competences, through to certificates for participation or completion.
This is concerned with whether or not the MOOC is linked to a formal educational offering. This can range from the MOOC being informal and optional through to perhaps being part of a formal educational offering, where MOOC participants learn alongside fee-paying students on a course.
This is the extent to which participants are expected to work individually through the MOOC and take control of their learning with little or no tutor support through to the participants being given a certain degree of tutor support. This might include forum moderation, or formative assessment on artefacts the participants produce.
This classification schema has a number of uses. It can be used to describe a MOOC in terms of these twelve dimensions, and hence provide a mechanism to compare different MOOCs.
Using the classification schema to describe different MOOCs
This section describes how the classification schema can be used to describe five different MOOCs. Each MOOC is based on a particular pedagogical approach; namely: associative, cognitive, constructivist, situative or connectivist. Pedagogical approaches (Mayes and De Freitas 2004, Conole 2010) can be classified as:
· Associative – where the focus is on the individual. It is about associating a stimulus with a response or in other words operant conditioning. Examples of ways in which technologies can facilitate associated pedagogies include drill and practice, and e-assessment. An example of an associative MOOC is a course on Chinese provided by the Open University UK. The MOOC is based around a series of podcasts and interactive assessment elements to test knowledge and understanding.
· Cognitive – where the emphasis is on learning by experiencing a stimuli, with learners encouraged to contemplate on their learning. An example of a cognitivist MOOC is a coursera’s course on songwriting. The MOOC starts from the learner’s current level of experience.
· Constructivist – where the emphasis is on building on prior knowledge; i.e. applying a meaning to and building on what the learner already knows. This is more active and task orientated. Example of ways in which technologies can facilitate constructivist pedagogies include: problem-based or inquiry learning. An example of a construcivist MOOC is a course on Learning Design run by the Open University, UK. The course begins by examining participants existing level of knowledge of teaching and of design and builds on this as the course progresses.
· Situative – where the emphasis is on learning in a context and through dialogue. Examples of how situative pedagogies can be facilitated include learning through virtual worlds. An example of a situative MOOC is a coursera course on Clinical Neurology. It is an applied, contextual course, intended to provide continuing professional development to professionals working in the field.
· Connectivist – where the emphasis is on learning in a networked context, through a distributed community of peers. Learners create their own personal learning environment and repertoire of digital tools. This encourages reflective, personalised learning. An example of a Connectivist MOOC is the Connectivism and Connective Knowledge course.
Table 2 provides a comparison of these five MOOCs using the twelve dimensional classification schema, with quantification along each of the dimensions to indicate the degree to which they are present.
Table 2: Comparison of five different MOOC courses
The classification schema can also be used to design a MOOC, using the criteria for each of the dimensions as a starting point. This can be used in conjunction with the resources associated with the 7Cs of Learning Design framework, which considers seven aspects of the design process.
The importance of Learning Design
Designing for learning is arguably one of the key challenges facing education today. Digital technologies offer a plethora of ways in which learners can interact with rich multimedia resources and ways in which they can communicate and collaborate with peers. Despite this technologies are not used extensively and teachers lack the necessary digital literacies (Jenkins 2006, Jenkins 2009) to make effective use of technologies in their teaching. A 2014 survey of the effective use of learning technology in education lists the following as key barriers to the uptake of learning technologies:
- Lack of resource to provide release and support for staff to enable them to incorporate technology in their practices.
- Reliance on individuals to champion innovation and exploitation of their willingness to support colleagues.
- Lack of direction at a strategic level resulting in fragmentation of practice across provider curriculum areas and levels of work.
- Lack of credit and recognition for innovative uses of technology by key influencers such as government agencies, awarding bodies, governing bodies.
- Lack of headroom for managers to support innovation and risk taking.
- Focus on omission and error in inspection and quality assurance, which does not encourage experimentation and exploration of the potential of technology.
- Little recognition that learning technologies are diverging between central sensitive data for administration, and independent and collaborative use by teachers and learners.
- Lack of funding to purchase technology.
- Lack of guidance on what would constitute outstanding and good practice across the range of technology use.
- Funding methodologies that are inimical to technology supported learning e.g. focusing on classroom activity.
- Staff not encouraged to use technology so many focus on the mandated administrative processes (e.g. quality assurance requirements, registers, outcomes, summative assessment).
In other words the key barriers can be grouped under the following five themes: lack of staff time and support, lack of support at senior level, lack of leadership in effective use of technology, lack of incentives, and lack of funding for technology.
A survey by Houghton Mifflin Harcourt surveyed more than 1,000 teachers and administrators on the use of technologies, and found the following were the top needs in classrooms:
- More engagement from parents of students
- More time with students to cover the curriculum
- Training on the use of technologies devices in the classroom
- More money for classroom materials
- More training on technology implementation
- New standards aligned to instructional materials
- More job embedded professional development
- More stand alone professional development.
Key findings were that 97% of teachers use some form of digital content, 51% of teachers use Apps/digital games in the classroom, only 36% use laptops or desktops and 61% do not use tablets, 60% of teachers stated that they saw increased student engagement when using technologies such as digital content, online applications and games.
Learning Design aims to help bridge this gap and guide teachers in their design practice. Learning Design as a research field has emerged in the last fifteen years or so, driven mainly by academics in Europe and Australia. It is an alternative to the more established field of Instructional Design. It aims to provide an holistic overview of the whole design process and helps teachers make more effective design decisions that are pedagogically informed and make appropriate use of digital technologies. Conole (2013, pg. 7) defines Learning Design as:
It is a methodology for enabling teachers/designers to make more informed decisions in how they go about designing learning activities and interventions, which is pedagogically informed and makes effective use of appropriate resources and technologies. This includes the design of resources and individual learning activities right u to curriculum-level design. A key principle is to make the design process more explicit and shareable. Learning Design as an area of research and development includes both gathering empirical evidence to understand the design process and the development of a range of Learning Design resources, tools and activities.
There are three facets to Learning Design: guidance (to help the teacher make design decisions), visualisation (to help teachers represent their designs visually from different perspectives) and sharing (with peers). It is based on a socio-cultural theoretical approach, and in particular the concept of what mediating artefacts teachers use in the design process (Conole 2013). Learning Design helps teachers think beyond content to the activities learners will engage with and the learner experience. The Larnaca Declaration on Learning Design provides an overview of the emergence of Learning Design and the key concepts.
Frey suggests that a:
Good Learning Experience Design can create relevant, engaging and memorable educational experiences that successfully address the specific challenges of these adult learners.
He suggests the following facets of design:
- The importance of designing a purposeful journey and making this clear to the learners.
- Making efficient use of the limited time available to learners.
- Directly linking learning goals to activities.
- Building on existing understanding and addressing any gaps in understandings
- Providing immersive, real-world simulations or experiences.
MOOCs provide an alternative to campus-based courses, and emphasis the power of harnessing a global, distributed community of peers for learning. However, the design of MOOCs is challenging and to date most have been developed on a fairly ad hoc basis. This paper describes how Learning Design can be used to design more effective MOOCs and in particular the use of the 7Cs of Learning Design framework, which is described in the next section.
The 7Cs of Learning Design
This section describes the 7Cs of Learning Design framework, which can be used to help teachers/designers to make more informed design decisions. Figure 1 illustrates the 7Cs of Learning Design framework. For each C there are a range of resources and activities, some of which are described here.
Figure 1: The 7Cs of Learning Design Framework
The first C, Conceptualise, is about creating a vision for the course or module being designed. It helps the teacher/designer think about the nature of the learners who are likely to take the course or module, their age range, diversity, characteristics, skills, perceptions and aspirations. It is also about articulating the core principles associated with the course or module. In terms of a MOOC this is about identifying the type of learners who are likely to participate, along with some understanding of their perceptions about MOOCs and their reasons for wanting to participate. It is also about establishing their level of experience with using technologies and their educational background. This can be achieved through the creation of ‘personas’ describing the types of learners who are likely to take the course. Personas are a tool for the MOOC designers to articulate their understanding of expected participants, as a starting point for design. In addition the Course Features activity helps the designers to articulate their vision for the MOOC in terms of:
- The pedagogical approaches used
- The core principles of the MOOC
- The nature of the guidance and support provided
- The types of content and activities
- The forms of communication and collaboration that are encouraged
- The ways in which reflection is encouraged and how the participants can demonstrate achievement of the learning outcomes.
The next four Cs are concerned with designing the resources and activities that the learners will engage with.
The Create C helps the teacher/designer articulate what learning materials need to be created, whether these are text-bases, interactive materials, podcasts or videos. In addition, it covers the use or repurposing of Open Educational Resources. Finally, the teacher/designer might also create some activities, which require the learners to create their own content. Depending on the subject nature of the MOOC there may be many existing OER which can be reused; for more specialised MOOCs it is likely that the designers will need to create the majority of the resources.
The Communicate C is concerned with methods to facilitate communication, between the learner and the tutor, the learner and their peers, and the broader community through social media. This might range from effective mechanisms for fostering discussion in a forum, through effective moderation, or looser communication through social media. In cMOOCs there is likely to be a significant emphasis on communication and encouraging participants to communicate with their peers, particularly through social media. This might include the use of a hashtag for the MOOC on Twitter or the setting up of a facebook page for the MOOC. In xMOOCs the focus is more on the individual so communication might be mainly restricted to interaction with the tutors via email.
Similarly, the Collaborate C is about fostering mechanisms to enable collaboration or group work. Again this is likely to be more prevalent in cMOOCs than xMOOCs. This could include the use of the jigsaw pedagogical pattern (Hernándex-Leo, Asensio-Pérez et al. 2010). In this participants are put into groups of four. Each person is given a topic to research, they then get together with others who have been researching the same topic and pool their collective knowledge. Finally, they return to their home team and share their combined understanding of the four topics assigned.
Finally, the Consider C, is concerned with ways in which reflection can be encouraged and the ways in which the learners can demonstrate the achievement of learning outcomes. Assessment might be diagnostic, formative or summative. Reflection in cMOOCs might be through interaction with other participants or might include encouraging participants to write and share reflective blogs on their learning. xMOOCs usually include formative assessment elements, through for example interactive formative quizzes. Participants might also get recognition of their learning through certificates of participation or attendance, or the awarding of digital badges for particular achievements or development of competences.
The Combine C enables the teacher/designer to step back and reflect on the design process to date and look at the design from different perspectives. This includes creativity an activity profile showing how much time participants are spending on the following types of tasks:
- Assimilative activities – such as reading, listening or viewing
- Information handling – such as finding and collating resource or manipulating data in a spreadsheet
- Communication – for example in forums or through social media
- Productive – creating learning artefacts, for example a chemical compound or an architectural model
- Adaptive – interacting with modelling or simulation tools
- Experimental – such as drill and practice or practising skills in a particular context or undertaking an investigation
- Assessment – recognition of learning through either formative or summative assessment.
It is also possible to create a Course View Map which describes in details the nature of the MOOC in terms of:
- What guidance and support is provided
- The nature of the content and activities the participants will engage with
- The way in which communication and collaboration is encouraged
- The way in which reflection is encouraged and how participants can demonstrate achievement of the learning outcomes.
The storyboard enables the designer to map out what the participants are doing over time. Figure 2 shows an example of a story board. At the top the weeks and topics are listed. On the left hand side the learning outcomes are listed. In the centre are the activities the participants will engage with. For example in week one they are reading an article and listening to a video. In week two they listen to a podcast and read a document. In the final weeks they listen to a podcast, read a document and watch a video. In week one, the output is for them to write an essay. In week two they write a blog post. In the final week they do a group presentation and write a reflective essay on their learning. Under this the assessment elements are listed. So in week one the tutor provides formative feedback on the assay. In week two participants provide feedback on some of their peers’ blog posts. In the final week there is a summative assessment of a group presentation and a reflective review of their learning. Finally, the learning outcomes are mapped to the assessment elements, to ensure constructive alignment (Biggs n.d.).
Figure 2: An example of a storyboard
Finally, the Consolidate C is about implementing the design in a real-life context and evaluating its effectiveness. This might include the development of an evaluation rubric. Evaluation methods might include analysis of participants’ interactions with the MOOC, a survey or interviews. In addition the MOOC classification schema can be used to evaluate how effective a MOOC is in terms of the extent to which the design is effectively implemented.
This paper has described a twelve dimensional classification schema for MOOCs, which can be used to design, describe and evaluate MOOCs. Five examples of different pedagogical MOOCs have been mapped against the schema.
MOOCs represent a sign of the times; they instantiate an example of how technologies can disrupt the status quo of education and are a forewarning of further changes to come. Whether or not MOOCs will reach the potential hype currently being discussed is a mote point, what is clear is that we need to take them seriously. More importantly, for both MOOCs and traditional educational offerings we need to make more informed design decisions that are pedagogically effective, leading to an enhanced learner experience and ensuring quality assurance.
The key value of MOOCs is that they are challenging traditional educational institutions and having to make them think, in particular on their educational business model and the distinctiveness of their offering. As a recent article states MOOCs are challenging traditional institutional business models about what they are offering, how it is distinctive and what the unique learner experience will be at their institution. As Cormier (2013) states:
When we use the MOOC as a lens to examine Higher Education, some interesting things come to light. The question of the ‘reason’ for education comes into focus.
Furthermore, UNESCO estimates that more than 100 million children can’t afford formal education. MOOCs provide them with a real lifeline to get above the poverty line. This, and the fact that MOOCs provide access to millions.
As Creelman (2013) notes:
Whatever you think of them they are opening up new learning opportunities for millions of people and that is really the main point of it all.
So for me the value of MOOCs to promote social inclusion, coupled with them making traditional institutions reconsider the educational offering they are providing and what is distinctive about their institution. These factors signify the importance of MOOCs as a disruptive technology, challenging existing educational business models and, hence, whether they survive or not is not important, if they result in an opening up of education and a better quality of the learner experience that has got to be for the good.
Biggs, J. (n.d.). “Constructive alignment.” John Biggs http://www.johnbiggs.com.au/academic/constructive-alignment/.
Christensen, C. (1997). The innovator’s dilemma: When new technologies cause great firms to fail. Harvard, Harvard University Press.
Conole, G. (2010) “Review of pedagogical frameworks and models and their use in e-learning.”
Conole, G. (2013). Designing for learning in an open world. New York, Springer.
Conole, G. (2014). “A new classification schema for MOOCs.” INNOQUAL 2(3).
Cormier, D. (2013). “Week 3 – Forget the learners, how do I measure a MOOC quality experience for ME! By Dave Cormier.” MOOC Quality Project http://mooc.efquel.org/week-3-forget-the-learners-how-do-i-measure-a-mooc-quality-experience-for-me/.
Creelman, A. (2013). “Make hay whilt the sunshines.” The corridor of uncertainty http://acreelman.blogspot.co.uk/2013/05/make-hay-while-sun-shines.html.
Dewey, J. (1916). Experience and Nature. New York, Dover.
Hernándex-Leo, D., J. I. Asensio-Pérez, Y. Dimitriadis and E. D. Villasclaras-Fernández (2010). Generating CSCL Scripts: From a Conceptual Model of Pattern
Languages to the Design of Real Scripts. E-learning design patterns. P. Goodyear and S. Retalis. Rotterdam, Sense Publishers: 49-64.
Jenkins, H. (2006). Convergence culture: Where old and new media collide, NYU Press.
Jenkins, H. (2009). Confronting the challenges of participatory culture: Media education for the 21st century, Mit Pr.
Kopp, M., M. Ebner and A. Dorfer-Novak (2014). “Introducing MOOCs to Austrian Universities: is it worth it to accept the challenge?” The International Journal for Innovation and Quality in Learning, INNOQUAL November 2014.
Mayes, T. and S. De Freitas (2004). Review of e-learning frameworks, models and theories, JISC e-learning models desk study.
Selwyn, N. and S. Buffin (2014). MOOC research initiative – final report.
Veletsianos, G., A. Collier and E. Schneider (2015). “Digging deeper into learners’ experiences in MOOCs: Participation in social networks outside of MOOCs, notetaking and contexts surrounding content consumption.” British Journal of Educational Technology 46(3): 570 – 587.
 The full set of resources and activities associated with the framework are available from here http://www2.le.ac.uk/projects/oer/oers/beyond-distance-research-alliance/7Cs-toolkit
 The LTDI evaluation cookbook is a useful resource describing different evaluation techniques – see http://www.icbl.hw.ac.uk/ltdi/cookbook/