|
Quality Assurance in Online Learning at the University of the West Indies:A Baseline Survey of Online Courses Anna-May Edwards-Henry, Instructional Development Unit, UWI, St. Augustine Abstract
The University of the West Indies (UWI) currently operates in a context of increasing demand for higher education. There is an immediate need to widen access to UWI programmes throughout the region, in order to increase access to higher education. To live up to this responsibility, as a premier higher education institution, we must also be concerned about the quality of the education we offer the region for its development. In the quest to satisfy increasing demands for higher education opportunities, it is even more imperative to ensure that quality assurance mechanisms are built into the university’s operations and offerings from the onset. Thus, to effectively promote The UWI as a leader and flagship institution in higher education, an emphasis on quality and attention to quality assurance mechanisms must be seen as integral to the University’s operations and offerings. It is against this background, with the support of the Pro Vice Chancellor, Non-campus Countries and Distance Education, that three units on the St. Augustine Campus undertook a project to evaluate online courses offered at The UWI. The three units are the Instructional Development Unit (IDU), The University of the West Indies Distance Education Centre (UWIDEC), and Campus IT Services (CITS). The project focuses on an evaluation of online courses offered by The UWI with respect to course configuration, course intentions, provision of learning opportunities and how well courses meet their learning intentions. The aims of the evaluation are to:
The evaluation will include a number of components, one of which will be a survey of online course offerings aimed at classifying the courses, as they exist. Another component will be a detailed analysis of representative samples of courses to determine fitness for their stated purposes. This paper will focus on the survey of existing online courses. For this study, we interpret an online course as any course, which includes a web-based component.
The UWI Background The University of the West Indies (UWI) is an independent University, which is supported by and serves sixteen (16) English-speaking territories of the West Indies. These territories are Anguilla, Antigua & Barbuda, Bahamas, Barbados, Belize, British Virgin Islands, Cayman Islands, Dominica, Grenada, Jamaica, Montserrat, St. Kitts and Nevis, St. Lucia, St.Vincent and The Grenadines, the Republic of Trinidad and Tobago and the Turks and Caicos Islands. The University began teaching in 1948 as a University College affiliated with the University of London, and became an independent University in 1962. The UWI now has Campuses at Cave Hill in Barbados, St. Augustine in Trinidad and Tobago and Mona in Jamaica. The total student body which was approximately 32,000 in 2005, is distributed amongst the Faculties of Law, Humanities, Science and Technology, Social Sciences, and the School of Clinical Medicine and Research at Cave Hill; Arts and Education, Medical Sciences, Social Sciences, and Pure and Applied Sciences at Mona; and, Engineering, Humanities and Education, Medical Sciences, Science and Agriculture, and Social Sciences at St. Augustine. In addition to the three main campuses, the University has centres in all of its non-campus Caribbean countries, which form part of a network making up The University of the West Indies Distance Education Centre (UWIDEC).
The UWI is currently a dual mode institution offering teaching by distance education as well as face-to-face. Gift, Moniquette and Perry (2004, p.4) postulate that The UWI is being transformed by “the increasing presence of universities from outside the region”, which employ aggressive marketing and recruitment strategies and are more flexible in their course and programme offerings. In an environment where The UWI is no longer the sole provider of higher education, the institution has been forced to become more competitive and to widen access to its programmes throughout the region, while at the same time pay more attention to quality assurance. The UWI now aspires to offer teaching and learning through a variety of modalities that will make it a truly open access, “anytime, anywhere” institution of higher learning that will better meet the needs of a heterogeneous and developing region. Quality Assurance at The UWI “Quality” has been defined in many ways including: zero defects, excellence, transformation/empowerment, value for money, fitness for purpose, and a holistic view that includes several of these factors (Harvey and Green 1993). The definition that is used by many tertiary education institutions, including The UWI, is “fitness for purpose” – though different stakeholders may have very different views of both “purpose” and “fitness”. At The UWI, the entity responsible for planning and directing the system of quality audit and quality assurance is the Quality Assurance Unit of the Board for Undergraduate Studies (BUS). “Quality is judged in terms of the extent to which a product or service meets its stated purposes. This allows decisions as to the aims and objectives of the teaching of a discipline, the content of the programmes and courses, teaching methods, assessment practices, etc. to reside with the teaching staff, while an evaluation of the results may be performed by others.” (Gift, S. 2005)
The incorporation of more Internet-based, asynchronous methods of teaching and learning has been hailed as one of the means to widen access to UWI programmes. The desire to move rapidly into Internet-based technologies has resulted in a mandate in the University’s strategic plan for each campus to have online material to support a considerable percentage of their courses. The result of this mandate has been the uncoordinated offering of a number of online courses and components of courses of possible varying quality, with no articulated and publicised institutional standard in place for quality assurance of these courses. In 2005 there were 444 ‘on-line’ courses listed at The UWI, offered at both the undergraduate and post-graduate levels. The list included125 on-line courses at Cave Hill, 150 at Mona and 169 at the St Augustine campus, spanning a range of departments and faculties. METHODOLOGY Theoretical Framework The theoretical framework adopted for this evaluation is essentially consumer-oriented. The consumer-oriented approach to evaluation is predominantly summative according to Worthen, Sanders and Fitzpatrick (1997). In this approach a number of checklists and criteria are defined by which it is determined whether educational products meet the needs and requirements of the user. The need for consumer-oriented evaluation emerged out of the blossoming trade in educational products that started in the 60s and which is even more evident today. Out of this process, product developers soon realized that using the checklists and criteria of the consumer/consumer advocate while the product is being created is the best way to prepare for subsequent public scrutiny. This benefit supports the need in this instance to develop standards for future development and implementation of online courses. Specifically this evaluation builds on and incorporates insights of Scriven (1967, 1974) and the Product Analysis Systems approach which emerged from evaluations conducted by the Educational Products Information Exchange (EPIE) that promote the work of evaluators like Robert Stake, as well as the Curriculum Materials Analysis System (CMAS) whose checklists were developed by Morrisett and Stevens (1967). It will also draw on the work of the Institute for Higher Education Policy (IHEP 2000), which has identified seven quality indicators for online learning:
The point made by Herrington, et. al. (2001) is that various institutions develop online learning evaluation instruments to serve a variety of functions which may include an exploration of the potential effectiveness of online courses, “a comparison of online courses, as a formative tool to guide development of learning materials or for summative purposes associated with establishing quality of existing materials.” (p. 264) The online courses will be evaluated on context and environment, structure and content, pedagogy, assessment, and overall success/failure. Data collection strategies will incorporate a mix of survey questionnaires, review and analysis of course documents including course material published online and interviews with course developers. The baseline survey is designed to specifically inform the evaluation process. The Baseline Survey Preliminary to the evaluation is a baseline survey to determine what courses exist online, their configuration and use. The baseline survey relates to the importance of establishing quality assurance mechanisms from the onset of course development and implementation. The survey, which commenced in May 2006, targeted courses that were placed online through the assistance of campus IT Services present on each campus. It is known that not all courses are made available to students online with the assistance of the campus IT services. Lecturers are currently being individually surveyed to determine the extent and quality of their courses available online including those without the assistance of the campus IT Services. The baseline survey aimed to collect data on course activity/inactivity, learning management system used, faculty, programme, level of course and delivery mix. A matrix was developed incorporating these criteria (see Appendix I). This was the main evaluation instrument for this component of the evaluation project. Data Collection The matrix was distributed to each of the three campus Information Technology Services (ITS) – Mona, Cave Hill and St Augustine for completion. Although the matrix was sent to each campus, we only received the complete data set requested from the St. Augustine ITS. While Cave Hill’s ITS submitted data, the set was incomplete in that it did not include information on the delivery mix. To date, we have not received the required data from Mona’s ITS. At this time, the data from the lecturers have not all been collected. Data Analysis Only data obtained from St. Augustine’s ITS were analysed at this time since these were the only full set of data submitted. While 169 online courses were listed by campus ITS in 2005, data from St. Augustine indicate that there are now 393 courses with online components. We felt that the large volume of courses that exist at St. Augustine would allow us to make certain assumptions about online learning. However, in the absence of input from the other campuses, assumptions and recommendations will of necessity be confined to the St. Augustine context. An SPSS template was developed based on the matrix supplied. This allowed for the summary and cross tabulation of the data. FINDINGS From the data collected from the St. Augustine ITS a number of trends and characteristics have already become evident. These are indicated in the following summaries. Activity/Inactivity, Level of Courses, Learning Management Systems (LMS) Currently at St. Augustine, there are a total of 393 courses with online components supported by the learning management system LMS Web Course Tools (WebCT). In 2005, 444 online courses were listed for all three campuses. Of the 393 courses listed at St. Augustine, 136 courses are inactive test courses or “shells” which are courses that have not been populated with or used by students. 181 courses are undergraduate courses and 76 are postgraduate courses. There are a total of 9 courses with online components supported by the Moodle LMS. This indicates that there has been an upsurge in the number of online courses between January 2005 and May 2006. It is highly likely that the mandate for all lecturers to have at least one online course could be responsible There are 127 courses in WebCT with no users, 85 courses with one user and 12 courses with two. The users range from three students in three courses to 10781 users in one course. More than half of the courses have either one or zero users. The majority of courses have fewer than two users. These data establish that the online modality is still a fledgling activity on the St. Augustine Campus. Users of the Moodle LMS courses range from a minimum of 11 students to a maximum of 301 students. Moodle is currently being used by The UWIDEC and the courses developed at St Augustine were developed initially as part of a blended learning project. This perhaps accounts for the activity in all the courses using Moodle, as opposed to the ones using WebCT. Faculties using Online Learning The early adopters generally appear to be from the Faculty of Engineering indicated by the 68 courses in WebCT from that faculty. The Faculty with the second highest number of courses in WebCT is the Faculty of Science and Agriculture with 23. The UWIDEC has a total of 19 courses in WebCT, while the Faculty of Social Sciences has 17, the Faculty of Medical Sciences, 15 and the Faculty of Humanities and Education has 14. Figure 1. Number of Courses with Online Components Pedagogical Tools Incorporated into Online Courses Two-thirds of the courses make use of one or more of WebCT’s pedagogical tools. Some effort seems to be made to incorporate pedagogy in the online modality; although, how the tools are used is still to be addressed. One-third of the courses do not make use of any of the pedagogical tools. In these cases, having an online presence seems to be the driving force for the use of the online modality. This is further supported by the significant correlation (p = 0.01) between the date on which the course is created and the last access date, meaning that the date on which the course is created is the last day it is accessed. With regard to the specific tools incorporated into the WebCT courses, the
Calendar tool is the most popular. This may be due to the ease of use of the
Calendar and its usefulness to students and lecturers in terms of disseminating
information such as schedules, 49% of the online courses utilise the Calendar
tool. 48% use E-mail. 43% use the Discussion tool. 33% use the Quiz tool. The
least used tool is the chat and assignment with 31% using the Chat tool and
28% using the Assignment tool. A comparison of pedagogical tools used at the different programme levels indicate that 40% of the postgraduate courses use six online pedagogical tools while only 6% of the undergraduate courses use as many of the online pedagogical tools. At the undergraduate level, with a more diverse population whose needs are more disparate, advantage is not taken of the opportunity to enhance learning through the use of the variety of pedagogical resources available within the learning management system. RECOMMENDATIONS Based on the findings, it is felt that a series of actions need to be taken in order to ensure that the University’s aspiration to become an open-access institution of quality higher education for the region is realized:
Beck, S. (2005). Evaluation Criteria: The good, the bad & the ugly: or, why it’s a good idea to evaluate web sources. Institute for Technology-Assisted Learning, New Mexico State University. [Online] available: http://lib.nmsu.edu/instruction/eval.html Accessed January 24th 2006 Gift, S. (2005). ‘Quality at UWI-How is it managed?’ In UWI Today. Sunday June 12th 2005. Gift, S., Moniquette, J and Perry, A. (2004). ‘The University of the West Indies: Challenges, Dilemmas and Quality Assurance.’ Paper presented at the Biennial Conference of the International Network for Quality Assurance Agencies in Higher Education (INQAAHE), Dublin, Ireland, and 14-17 April 2003. Harvey, L., and Green, D. (1993) ‘Defining Quality’, Assessment and Evaluation in Higher Education. Volume 18 No 1. Herrington, A., Herrington, J., Oliver, R., Stoney S. & Willis, J. (2001). ‘Quality guidelines for online courses: The development of an instrument to audit online units.’ In G. Kennedy, M. Keppell, C. McNaught & T. Petrovic (Eds.) Meeting at the crossroads: proceedings of ASCILITE 2001, (pp 263-270). Melbourne: The University of Melbourne. Institute for Higher Education Policy. (2000) Quality on the Line: Benchmarks for Success in Internet-based Distance Education. Washington, D.C. IHEP Morrisett. I. & Stevens, W. W. (1967). Steps in curriculum analysis outline. Boulder: University of Colorado, Social Science Education Consortium. Office of the Board of Undergraduate Studies. (2000) The UWI Quality Strategy: The Quality Assurance System at the University of the West Indies. Mona: OBUS. Reeves, T.C. & Laffey J.M. (1999) ‘Design, assessment and evaluation of a problem-based learning environment in undergraduate engineering.’ Higher Education Research and Development Journal, Volume 18, No. 2, pp. 219-232. Scriven, M. (1967). ‘The methodology of evaluation.’ In R.E. Stake (ed.), Curriculum evaluation. American Educational Research Association Monograph Series of Evaluation, No. 1. pp. 39-83). Chicago: Rand McNally. Scriven, M. (1974a). ‘Evaluation perspectives and procedures’. In W. J. Popham (ed.), Evaluation in education, Berkeley, CA: McCutchan. Worthen, B. R., Sanders, J.R. and Fitzpatrick, J. L. (1997). Program evaluation: Alternative approaches and practical guidelines. 2nd ed. NY: Longman Publishers.
APPENDIX I GUIDELINES FOR CITS STAFF The attached matrix is designed to obtain feedback on courses with an online component. This information will assist us in gauging the characteristics and activity of online offerings at The UWI. This will serve as the first stage in establishing standards in online teaching and the provision of support for online course developers. INSTRUCTIONS
Please return completed matrix by April 30th 2006 in the envelope provided or via email to: Instructional Development Unit APPENDIX 1A
Appendix II WEBCT COURSE EVALUATION & APPROVAL FORM
Course Designer: __________________________ Faculty: __________________________
**Minimum of a 3 rating required in MOST categories for Approval
Figures | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
