It is against this background, with the support of the Pro Vice Chancellor, Non-campus Countries and Distance Education, that three units on the St. Augustine Campus undertook a project to evaluate online courses offered at The UWI. The three units are the Instructional Development Unit (IDU), The University of the West Indies Distance Education Centre (UWIDEC), and Campus IT Services (CITS). The project focuses on an evaluation of online courses offered by The UWI with respect to course configuration, course intentions, provision of learning opportunities and how well courses meet their learning intentions. The aims of the evaluation are to:

• classify/categorize existing online courses offered at The UWI
• establish standards for developing and implementing effective online courses
• establish a monitoring mechanism for online course development and use.

The evaluation will include a number of components, one of which will be a survey of online course offerings aimed at classifying the courses, as they exist. Another component will be a detailed analysis of representative samples of courses to determine fitness for their stated purposes. This paper will focus on the survey of existing online courses. For this study, we interpret an online course as any course, which includes a web-based component.

CONTEXT

The UWI Background

The University of the West Indies (UWI) is an independent University, which is supported by and serves sixteen (16) English-speaking territories of the West Indies. These territories are Anguilla, Antigua & Barbuda, Bahamas, Barbados, Belize, British Virgin Islands, Cayman Islands, Dominica, Grenada, Jamaica, Montserrat, St. Kitts and Nevis, St. Lucia, St.Vincent and The Grenadines, the Republic of Trinidad and Tobago and the Turks and Caicos Islands.

The University began teaching in 1948 as a University College affiliated with the University of London, and became an independent University in 1962. The UWI now has Campuses at Cave Hill in Barbados, St. Augustine in Trinidad and Tobago and Mona in Jamaica. The total student body which was approximately 32,000 in 2005, is distributed amongst the Faculties of Law, Humanities, Science and Technology, Social Sciences, and the School of Clinical Medicine and Research at Cave Hill; Arts and Education, Medical Sciences, Social Sciences, and Pure and Applied Sciences at Mona; and, Engineering, Humanities and Education, Medical Sciences, Science and Agriculture, and Social Sciences at St. Augustine.

In addition to the three main campuses, the University has centres in all of its non-campus Caribbean countries, which form part of a network making up The University of the West Indies Distance Education Centre (UWIDEC).

The UWI’s Changing Teaching and Learning Environment

The UWI is currently a dual mode institution offering teaching by distance education as well as face-to-face. Gift, Moniquette and Perry (2004, p.4) postulate that The UWI is being transformed by “the increasing presence of universities from outside the region”, which employ aggressive marketing and recruitment strategies and are more flexible in their course and programme offerings. In an environment where The UWI is no longer the sole provider of higher education, the institution has been forced to become more competitive and to widen access to its programmes throughout the region, while at the same time pay more attention to quality assurance. The UWI now aspires to offer teaching and learning through a variety of modalities that will make it a truly open access, “anytime, anywhere” institution of higher learning that will better meet the needs of a heterogeneous and developing region.

Quality Assurance at The UWI

“Quality” has been defined in many ways including: zero defects, excellence, transformation/empowerment, value for money, fitness for purpose, and a holistic view that includes several of these factors (Harvey and Green 1993). The definition that is used by many tertiary education institutions, including The UWI, is “fitness for purpose” – though different stakeholders may have very different views of both “purpose” and “fitness”.

At The UWI, the entity responsible for planning and directing the system of quality audit and quality assurance is the Quality Assurance Unit of the Board for Undergraduate Studies (BUS). “Quality is judged in terms of the extent to which a product or service meets its stated purposes. This allows decisions as to the aims and objectives of the teaching of a discipline, the content of the programmes and courses, teaching methods, assessment practices, etc. to reside with the teaching staff, while an evaluation of the results may be performed by others.” (Gift, S. 2005)

Online Learning and Quality Assurance at The UWI

The incorporation of more Internet-based, asynchronous methods of teaching and learning has been hailed as one of the means to widen access to UWI programmes. The desire to move rapidly into Internet-based technologies has resulted in a mandate in the University’s strategic plan for each campus to have online material to support a considerable percentage of their courses. The result of this mandate has been the uncoordinated offering of a number of online courses and components of courses of possible varying quality, with no articulated and publicised institutional standard in place for quality assurance of these courses. In 2005 there were 444 ‘on-line’ courses listed at The UWI, offered at both the undergraduate and post-graduate levels. The list included125 on-line courses at Cave Hill, 150 at Mona and 169 at the St Augustine campus, spanning a range of departments and faculties.

METHODOLOGY

Theoretical Framework

The theoretical framework adopted for this evaluation is essentially consumer-oriented. The consumer-oriented approach to evaluation is predominantly summative according to Worthen, Sanders and Fitzpatrick (1997). In this approach a number of checklists and criteria are defined by which it is determined whether educational products meet the needs and requirements of the user. The need for consumer-oriented evaluation emerged out of the blossoming trade in educational products that started in the 60s and which is even more evident today. Out of this process, product developers soon realized that using the checklists and criteria of the consumer/consumer advocate while the product is being created is the best way to prepare for subsequent public scrutiny. This benefit supports the need in this instance to develop standards for future development and implementation of online courses.

Specifically this evaluation builds on and incorporates insights of Scriven (1967, 1974) and the Product Analysis Systems approach which emerged from evaluations conducted by the Educational Products Information Exchange (EPIE) that promote the work of evaluators like Robert Stake, as well as the Curriculum Materials Analysis System (CMAS) whose checklists were developed by Morrisett and Stevens (1967). It will also draw on the work of the Institute for Higher Education Policy (IHEP 2000), which has identified seven quality indicators for online learning:

• Institutional Support - administrative and financial commitment institutions make towards online learning, including: the maintenance of programmes; incentives for professionals; the equitable treatment of learning done on-campus and at a distance; a technology infrastructure plan defining the technical and related requirements needed to support the learning activities.
• Course Development - components such as a peer review process; minimum standards for course development; and, a team approach;
• Course Structure/design - a clear statement of intended learning outcomes; assessment of learner progress by reference to these outcomes; appropriate selection and application of media; learning activities responsive to the learning needs of individual learners; learner autonomy in terms of time, place and pace; team approach to content creation; and, continuous evaluation for effectiveness;
• Teaching/learning process - learner-centred versus instructor centered, peer interaction, self-help;
• Student Support - a learner support structure of counseling, tutorials, and administration that helps learners in accessing a wide range of required information, various resources including library and other technical facilities, learning assets to suit their learning styles;
• Faculty Support- appropriate training and relief provided to enable staff involvement;
• Evaluation and Assessment - system in place for evaluation of effectiveness of programmes/courses.

The point made by Herrington, et. al. (2001) is that various institutions develop online learning evaluation instruments to serve a variety of functions which may include an exploration of the potential effectiveness of online courses, “a comparison of online courses, as a formative tool to guide development of learning materials or for summative purposes associated with establishing quality of existing materials.” (p. 264)

The online courses will be evaluated on context and environment, structure and content, pedagogy, assessment, and overall success/failure. Data collection strategies will incorporate a mix of survey questionnaires, review and analysis of course documents including course material published online and interviews with course developers. The baseline survey is designed to specifically inform the evaluation process.

The Baseline Survey

Preliminary to the evaluation is a baseline survey to determine what courses exist online, their configuration and use. The baseline survey relates to the importance of establishing quality assurance mechanisms from the onset of course development and implementation. The survey, which commenced in May 2006, targeted courses that were placed online through the assistance of campus IT Services present on each campus. It is known that not all courses are made available to students online with the assistance of the campus IT services. Lecturers are currently being individually surveyed to determine the extent and quality of their courses available online including those without the assistance of the campus IT Services.

The baseline survey aimed to collect data on course activity/inactivity, learning management system used, faculty, programme, level of course and delivery mix. A matrix was developed incorporating these criteria (see Appendix I). This was the main evaluation instrument for this component of the evaluation project.

Data Collection

The matrix was distributed to each of the three campus Information Technology Services (ITS) – Mona, Cave Hill and St Augustine for completion.

Although the matrix was sent to each campus, we only received the complete data set requested from the St. Augustine ITS. While Cave Hill’s ITS submitted data, the set was incomplete in that it did not include information on the delivery mix. To date, we have not received the required data from Mona’s ITS.

At this time, the data from the lecturers have not all been collected.

Data Analysis

Only data obtained from St. Augustine’s ITS were analysed at this time since these were the only full set of data submitted. While 169 online courses were listed by campus ITS in 2005, data from St. Augustine indicate that there are now 393 courses with online components. We felt that the large volume of courses that exist at St. Augustine would allow us to make certain assumptions about online learning. However, in the absence of input from the other campuses, assumptions and recommendations will of necessity be confined to the St. Augustine context.

An SPSS template was developed based on the matrix supplied. This allowed for the summary and cross tabulation of the data.

FINDINGS

From the data collected from the St. Augustine ITS a number of trends and characteristics have already become evident. These are indicated in the following summaries.

Activity/Inactivity, Level of Courses, Learning Management Systems (LMS)

Currently at St. Augustine, there are a total of 393 courses with online components supported by the learning management system LMS Web Course Tools (WebCT). In 2005, 444 online courses were listed for all three campuses. Of the 393 courses listed at St. Augustine, 136 courses are inactive test courses or “shells” which are courses that have not been populated with or used by students. 181 courses are undergraduate courses and 76 are postgraduate courses. There are a total of 9 courses with online components supported by the Moodle LMS.

This indicates that there has been an upsurge in the number of online courses between January 2005 and May 2006. It is highly likely that the mandate for all lecturers to have at least one online course could be responsible

There are 127 courses in WebCT with no users, 85 courses with one user and 12 courses with two. The users range from three students in three courses to 10781 users in one course. More than half of the courses have either one or zero users. The majority of courses have fewer than two users. These data establish that the online modality is still a fledgling activity on the St. Augustine Campus.

Users of the Moodle LMS courses range from a minimum of 11 students to a maximum of 301 students. Moodle is currently being used by The UWIDEC and the courses developed at St Augustine were developed initially as part of a blended learning project. This perhaps accounts for the activity in all the courses using Moodle, as opposed to the ones using WebCT.

Faculties using Online Learning

The early adopters generally appear to be from the Faculty of Engineering indicated by the 68 courses in WebCT from that faculty. The Faculty with the second highest number of courses in WebCT is the Faculty of Science and Agriculture with 23. The UWIDEC has a total of 19 courses in WebCT, while the Faculty of Social Sciences has 17, the Faculty of Medical Sciences, 15 and the Faculty of Humanities and Education has 14.

Figure 1. Number of Courses with Online Components

Pedagogical Tools Incorporated into Online Courses

Two-thirds of the courses make use of one or more of WebCT’s pedagogical tools. Some effort seems to be made to incorporate pedagogy in the online modality; although, how the tools are used is still to be addressed. One-third of the courses do not make use of any of the pedagogical tools. In these cases, having an online presence seems to be the driving force for the use of the online modality. This is further supported by the significant correlation (p = 0.01) between the date on which the course is created and the last access date, meaning that the date on which the course is created is the last day it is accessed.

With regard to the specific tools incorporated into the WebCT courses, the Calendar tool is the most popular. This may be due to the ease of use of the Calendar and its usefulness to students and lecturers in terms of disseminating information such as schedules, 49% of the online courses utilise the Calendar tool. 48% use E-mail. 43% use the Discussion tool. 33% use the Quiz tool. The least used tool is the chat and assignment with 31% using the Chat tool and 28% using the Assignment tool.

A comparison of pedagogical tools used at the different programme levels indicate that 40% of the postgraduate courses use six online pedagogical tools while only 6% of the undergraduate courses use as many of the online pedagogical tools. At the undergraduate level, with a more diverse population whose needs are more disparate, advantage is not taken of the opportunity to enhance learning through the use of the variety of pedagogical resources available within the learning management system.

RECOMMENDATIONS

Based on the findings, it is felt that a series of actions need to be taken in order to ensure that the University’s aspiration to become an open-access institution of quality higher education for the region is realized:

• A large number of courses are online but are not available to the students. As such, guidelines and procedures need to be developed for the removal of test courses after a specified period.
  
  It is recommended that a course be removed after two successive semesters of inactivity. This should apply to test courses or shells as well as courses that may have been inactive.
  
• Many courses are currently online but do not use any pedagogical tools. This has to be addressed at several levels.
  • Standards need to be instituted for developing and placing a course online.
    
    It is recommended that these standards address course structure, learning activities, access by students, availability
    
  • Standards need to be developed for the review and evaluation of courses.
    
    It is recommended that these standards address feedback; review process; timing and extent of evaluations
    
• Appropriate assistance needs to be provided to lecturers for the development, maintenance, and evaluation of courses. This may involve providing guidelines, checklists, and support through the IDU, UWIDEC, and CITS.
  
  A model for online course development should embrace the team approach as illustrated in Figure 3.
  

• A procedure needs to be developed for evaluating the readiness of courses for placement online. The procedure should include minimum requirements for populating courses – that is, making them available to students.
  
  A suggested checklist for approval of online courses, in which minimum requirements are implied, is found at Appendix II.

REFERENCES

Beck, S. (2005). Evaluation Criteria: The good, the bad & the ugly: or, why it’s a good idea to evaluate web sources. Institute for Technology-Assisted Learning, New Mexico State University. [Online] available: http://lib.nmsu.edu/instruction/eval.html Accessed January 24th 2006

Gift, S. (2005). ‘Quality at UWI-How is it managed?’ In UWI Today. Sunday June 12th 2005.

Gift, S., Moniquette, J and Perry, A. (2004). ‘The University of the West Indies: Challenges, Dilemmas and Quality Assurance.’ Paper presented at the Biennial Conference of the International Network for Quality Assurance Agencies in Higher Education (INQAAHE), Dublin, Ireland, and 14-17 April 2003.

Harvey, L., and Green, D. (1993) ‘Defining Quality’, Assessment and Evaluation in Higher Education. Volume 18 No 1.

Herrington, A., Herrington, J., Oliver, R., Stoney S. & Willis, J. (2001). ‘Quality guidelines for online courses: The development of an instrument to audit online units.’ In G. Kennedy, M. Keppell, C. McNaught & T. Petrovic (Eds.) Meeting at the crossroads: proceedings of ASCILITE 2001, (pp 263-270). Melbourne: The University of Melbourne.

Institute for Higher Education Policy. (2000) Quality on the Line: Benchmarks for Success in Internet-based Distance Education. Washington, D.C. IHEP

Morrisett. I. & Stevens, W. W. (1967). Steps in curriculum analysis outline. Boulder: University of Colorado, Social Science Education Consortium.

Office of the Board of Undergraduate Studies. (2000) The UWI Quality Strategy: The Quality Assurance System at the University of the West Indies. Mona: OBUS.

Reeves, T.C. & Laffey J.M. (1999) ‘Design, assessment and evaluation of a problem-based learning environment in undergraduate engineering.’ Higher Education Research and Development Journal, Volume 18, No. 2, pp. 219-232.

Scriven, M. (1967). ‘The methodology of evaluation.’ In R.E. Stake (ed.), Curriculum evaluation. American Educational Research Association Monograph Series of Evaluation, No. 1. pp. 39-83). Chicago: Rand McNally.

Scriven, M. (1974a). ‘Evaluation perspectives and procedures’. In W. J. Popham (ed.), Evaluation in education, Berkeley, CA: McCutchan.

Worthen, B. R., Sanders, J.R. and Fitzpatrick, J. L. (1997). Program evaluation: Alternative approaches and practical guidelines. 2nd ed. NY: Longman Publishers.

APPENDIX I

GUIDELINES FOR CITS STAFF
RE: COMPLETION OF THE
ONLINE COURSE CHARACTERISTICS MATRIX

The attached matrix is designed to obtain feedback on courses with an online component. This information will assist us in gauging the characteristics and activity of online offerings at The UWI. This will serve as the first stage in establishing standards in online teaching and the provision of support for online course developers.

INSTRUCTIONS
Please complete the matrix using the explanation of each category provided below. Provide information for all courses with an online component that are facilitated through your campus IT services.

Return date and address

Please return completed matrix by April 30th 2006 in the envelope provided or via email to:

Instructional Development Unit
The University of the West Indies
St. Augustine

APPENDIX 1A

Appendix II

WEBCT COURSE EVALUATION & APPROVAL FORM

Course Code/Title: _________________________

Course Designer: __________________________ Faculty: __________________________

**Minimum of a 3 rating required in MOST categories for Approval