The University of the West Indies (UWI) currently operates in a context of
increasing demand for higher education. There is an immediate need to widen
access to UWI programmes throughout the region, in order to increase access
to higher education. To live up to this responsibility, as a premier higher
education institution, we must also be concerned about the quality of the education
we offer the region for its development. In the quest to satisfy increasing
demands for higher education opportunities, it is even more imperative to ensure
that quality assurance mechanisms are built into the university’s operations
and offerings from the onset. Thus, to effectively promote The UWI as a leader
and flagship institution in higher education, an emphasis on quality and attention
to quality assurance mechanisms must be seen as integral to the University’s
operations and offerings.
It is against this background, with the support of the Pro Vice Chancellor,
Non-campus Countries and Distance Education, that three units on the St. Augustine
Campus undertook a project to evaluate online courses offered at The UWI. The
three units are the Instructional Development Unit (IDU), The University of
the West Indies Distance Education Centre (UWIDEC), and Campus IT Services (CITS).
The project focuses on an evaluation of online courses offered by The UWI with
respect to course configuration, course intentions, provision of learning opportunities
and how well courses meet their learning intentions. The aims of the evaluation
- classify/categorize existing online courses offered at The UWI
- establish standards for developing and implementing effective online courses
- establish a monitoring mechanism for online course development and use.
The evaluation will include a number of components, one of which will be a
survey of online course offerings aimed at classifying the courses, as they
exist. Another component will be a detailed analysis of representative samples
of courses to determine fitness for their stated purposes. This paper will focus
on the survey of existing online courses. For this study, we interpret an online
course as any course, which includes a web-based component.
The UWI Background
The University of the West Indies (UWI) is an independent University, which
is supported by and serves sixteen (16) English-speaking territories of the
West Indies. These territories are Anguilla, Antigua & Barbuda, Bahamas,
Barbados, Belize, British Virgin Islands, Cayman Islands, Dominica, Grenada,
Jamaica, Montserrat, St. Kitts and Nevis, St. Lucia, St.Vincent and The Grenadines,
the Republic of Trinidad and Tobago and the Turks and Caicos Islands.
The University began teaching in 1948 as a University College affiliated with
the University of London, and became an independent University in 1962. The
UWI now has Campuses at Cave Hill in Barbados, St. Augustine in Trinidad and
Tobago and Mona in Jamaica. The total student body which was approximately 32,000
in 2005, is distributed amongst the Faculties of Law, Humanities, Science and
Technology, Social Sciences, and the School of Clinical Medicine and Research
at Cave Hill; Arts and Education, Medical Sciences, Social Sciences, and Pure
and Applied Sciences at Mona; and, Engineering, Humanities and Education, Medical
Sciences, Science and Agriculture, and Social Sciences at St. Augustine.
In addition to the three main campuses, the University has centres in all of
its non-campus Caribbean countries, which form part of a network making up The
University of the West Indies Distance Education Centre (UWIDEC).
The UWI’s Changing Teaching and Learning Environment
The UWI is currently a dual mode institution offering teaching by distance
education as well as face-to-face. Gift, Moniquette and Perry (2004, p.4) postulate
that The UWI is being transformed by “the increasing presence of universities
from outside the region”, which employ aggressive marketing and recruitment
strategies and are more flexible in their course and programme offerings. In
an environment where The UWI is no longer the sole provider of higher education,
the institution has been forced to become more competitive and to widen access
to its programmes throughout the region, while at the same time pay more attention
to quality assurance. The UWI now aspires to offer teaching and learning through
a variety of modalities that will make it a truly open access, “anytime,
anywhere” institution of higher learning that will better meet the needs
of a heterogeneous and developing region.
Quality Assurance at The UWI
“Quality” has been defined in many ways including: zero defects,
excellence, transformation/empowerment, value for money, fitness for purpose,
and a holistic view that includes several of these factors (Harvey and Green
1993). The definition that is used by many tertiary education institutions,
including The UWI, is “fitness for purpose” – though different
stakeholders may have very different views of both “purpose” and
At The UWI, the entity responsible for planning and directing the system of
quality audit and quality assurance is the Quality Assurance Unit of the Board
for Undergraduate Studies (BUS). “Quality is judged in terms of the extent
to which a product or service meets its stated purposes. This allows decisions
as to the aims and objectives of the teaching of a discipline, the content of
the programmes and courses, teaching methods, assessment practices, etc. to
reside with the teaching staff, while an evaluation of the results may be performed
by others.” (Gift, S. 2005)
Online Learning and Quality Assurance at The UWI
The incorporation of more Internet-based, asynchronous methods of teaching
and learning has been hailed as one of the means to widen access to UWI programmes.
The desire to move rapidly into Internet-based technologies has resulted in
a mandate in the University’s strategic plan for each campus to have online
material to support a considerable percentage of their courses. The result of
this mandate has been the uncoordinated offering of a number of online courses
and components of courses of possible varying quality, with no articulated and
publicised institutional standard in place for quality assurance of these courses.
In 2005 there were 444 ‘on-line’ courses listed
at The UWI, offered at both the undergraduate and post-graduate levels. The
list included125 on-line courses at Cave Hill, 150 at Mona and 169 at the St
Augustine campus, spanning a range of departments and faculties.
The theoretical framework adopted for this evaluation is essentially consumer-oriented.
The consumer-oriented approach to evaluation is predominantly summative according
to Worthen, Sanders and Fitzpatrick (1997). In this approach a number of checklists
and criteria are defined by which it is determined whether educational products
meet the needs and requirements of the user. The need for consumer-oriented
evaluation emerged out of the blossoming trade in educational products that
started in the 60s and which is even more evident today. Out of this process,
product developers soon realized that using the checklists and criteria of the
consumer/consumer advocate while the product is being created is the best way
to prepare for subsequent public scrutiny. This benefit supports the need in
this instance to develop standards for future development and implementation
of online courses.
Specifically this evaluation builds on and incorporates insights of Scriven
(1967, 1974) and the Product Analysis Systems approach which emerged from evaluations
conducted by the Educational Products Information Exchange (EPIE) that promote
the work of evaluators like Robert Stake, as well as the Curriculum Materials
Analysis System (CMAS) whose checklists were developed by Morrisett and Stevens
(1967). It will also draw on the work of the Institute for Higher Education
Policy (IHEP 2000), which has identified seven quality indicators for online
- Institutional Support - administrative and financial commitment
institutions make towards online learning, including: the maintenance of programmes;
incentives for professionals; the equitable treatment of learning done on-campus
and at a distance; a technology infrastructure plan defining the technical
and related requirements needed to support the learning activities.
- Course Development - components such as a peer review process;
minimum standards for course development; and, a team approach;
- Course Structure/design - a clear statement of intended learning
outcomes; assessment of learner progress by reference to these outcomes; appropriate
selection and application of media; learning activities responsive to the
learning needs of individual learners; learner autonomy in terms of time,
place and pace; team approach to content creation; and, continuous evaluation
- Teaching/learning process - learner-centred versus instructor centered,
peer interaction, self-help;
- Student Support - a learner support structure of counseling, tutorials,
and administration that helps learners in accessing a wide range of required
information, various resources including library and other technical facilities,
learning assets to suit their learning styles;
- Faculty Support- appropriate training and relief provided to enable
- Evaluation and Assessment - system in place for evaluation of effectiveness
The point made by Herrington, et. al. (2001) is that various institutions develop
online learning evaluation instruments to serve a variety of functions which
may include an exploration of the potential effectiveness of online courses,
“a comparison of online courses, as a formative tool to guide development
of learning materials or for summative purposes associated with establishing
quality of existing materials.” (p. 264)
The online courses will be evaluated on context and environment, structure
and content, pedagogy, assessment, and overall success/failure. Data collection
strategies will incorporate a mix of survey questionnaires, review and analysis
of course documents including course material published online and interviews
with course developers. The baseline survey is designed to specifically inform
the evaluation process.
The Baseline Survey
Preliminary to the evaluation is a baseline survey to determine what courses
exist online, their configuration and use. The baseline survey relates to the
importance of establishing quality assurance mechanisms from the onset of course
development and implementation. The survey, which commenced in May 2006, targeted
courses that were placed online through the assistance of campus IT Services
present on each campus. It is known that not all courses are made available
to students online with the assistance of the campus IT services. Lecturers
are currently being individually surveyed to determine the extent and quality
of their courses available online including those without the assistance of
the campus IT Services.
The baseline survey aimed to collect data on course activity/inactivity, learning
management system used, faculty, programme, level of course and delivery mix.
A matrix was developed incorporating these criteria (see Appendix I). This was
the main evaluation instrument for this component of the evaluation project.
The matrix was distributed to each of the three campus Information Technology
Services (ITS) – Mona, Cave Hill and St Augustine for completion.
Although the matrix was sent to each campus, we only received the complete
data set requested from the St. Augustine ITS. While Cave Hill’s ITS submitted
data, the set was incomplete in that it did not include information on the delivery
mix. To date, we have not received the required data from Mona’s ITS.
At this time, the data from the lecturers have not all been collected.
Only data obtained from St. Augustine’s ITS were analysed at this time
since these were the only full set of data submitted. While 169 online courses
were listed by campus ITS in 2005, data from St. Augustine indicate that there
are now 393 courses with online components. We felt that the large volume of
courses that exist at St. Augustine would allow us to make certain assumptions
about online learning. However, in the absence of input from the other campuses,
assumptions and recommendations will of necessity be confined to the St. Augustine
An SPSS template was developed based on the matrix supplied. This allowed for
the summary and cross tabulation of the data.
From the data collected from the St. Augustine ITS a number of trends and characteristics
have already become evident. These are indicated in the following summaries.
Activity/Inactivity, Level of Courses, Learning Management Systems (LMS)
Currently at St. Augustine, there are a total of 393 courses with online components
supported by the learning management system LMS Web Course Tools (WebCT). In
2005, 444 online courses were listed for all three campuses. Of the 393 courses
listed at St. Augustine, 136 courses are inactive test courses or “shells”
which are courses that have not been populated with or used by students. 181
courses are undergraduate courses and 76 are postgraduate courses. There are
a total of 9 courses with online components supported by the Moodle LMS.
This indicates that there has been an upsurge in the number of online courses
between January 2005 and May 2006. It is highly likely that the mandate for
all lecturers to have at least one online course could be responsible
There are 127 courses in WebCT with no users, 85 courses with one user and
12 courses with two. The users range from three students in three courses to
10781 users in one course. More than half of the courses have either one or
zero users. The majority of courses have fewer than two users. These data establish
that the online modality is still a fledgling activity on the St. Augustine
Users of the Moodle LMS courses range from a minimum of 11 students to a maximum
of 301 students. Moodle is currently being used by The UWIDEC and the courses
developed at St Augustine were developed initially as part of a blended learning
project. This perhaps accounts for the activity in all the courses using Moodle,
as opposed to the ones using WebCT.
Faculties using Online Learning
The early adopters generally appear to be from the Faculty of Engineering indicated
by the 68 courses in WebCT from that faculty. The Faculty with the second highest
number of courses in WebCT is the Faculty of Science and Agriculture with 23.
The UWIDEC has a total of 19 courses in WebCT, while the Faculty of Social Sciences
has 17, the Faculty of Medical Sciences, 15 and the Faculty of Humanities and
Education has 14.
Figure 1. Number of Courses with Online Components
Pedagogical Tools Incorporated into Online Courses
Two-thirds of the courses make use of one or more of WebCT’s pedagogical
tools. Some effort seems to be made to incorporate pedagogy in the online modality;
although, how the tools are used is still to be addressed. One-third of the
courses do not make use of any of the pedagogical tools. In these cases, having
an online presence seems to be the driving force for the use of the online modality.
This is further supported by the significant correlation (p = 0.01) between
the date on which the course is created and the last access date, meaning that
the date on which the course is created is the last day it is accessed.
With regard to the specific tools incorporated into the WebCT courses, the
Calendar tool is the most popular. This may be due to the ease of use of the
Calendar and its usefulness to students and lecturers in terms of disseminating
information such as schedules, 49% of the online courses utilise the Calendar
tool. 48% use E-mail. 43% use the Discussion tool. 33% use the Quiz tool. The
least used tool is the chat and assignment with 31% using the Chat tool and
28% using the Assignment tool.
Figure 2. Percent of Courses Using Pedagogical Tools
A comparison of pedagogical tools used at the different programme levels indicate
that 40% of the postgraduate courses use six online pedagogical tools while
only 6% of the undergraduate courses use as many of the online pedagogical tools.
At the undergraduate level, with a more diverse population whose needs are more
disparate, advantage is not taken of the opportunity to enhance learning through
the use of the variety of pedagogical resources available within the learning
Based on the findings, it is felt that a series of actions need to be taken
in order to ensure that the University’s aspiration to become an open-access
institution of quality higher education for the region is realized:
Figure 3. Model of an Online Course Preparation Team
- A large number of courses are online but are not available to the students.
As such, guidelines and procedures need to be developed for the removal of
test courses after a specified period.
It is recommended that a course be removed after two successive semesters
of inactivity. This should apply to test courses or shells as well as courses
that may have been inactive.
- Many courses are currently online but do not use any pedagogical tools.
This has to be addressed at several levels.
- Standards need to be instituted for developing and placing a course
It is recommended that these standards address course structure, learning
activities, access by students, availability
- Standards need to be developed for the review and evaluation of courses.
It is recommended that these standards address feedback; review process;
timing and extent of evaluations
- Appropriate assistance needs to be provided to lecturers for the development,
maintenance, and evaluation of courses. This may involve providing guidelines,
checklists, and support through the IDU, UWIDEC, and CITS.
A model for online course development should embrace the team approach
as illustrated in Figure 3.
- A procedure needs to be developed for evaluating the readiness of courses
for placement online. The procedure should include minimum requirements for
populating courses – that is, making them available to students.
A suggested checklist for approval of online courses, in which minimum
requirements are implied, is found at Appendix II.
Beck, S. (2005). Evaluation Criteria: The good, the bad & the ugly:
or, why it’s a good idea to evaluate web sources. Institute for Technology-Assisted
Learning, New Mexico State University. [Online] available: http://lib.nmsu.edu/instruction/eval.html
Accessed January 24th 2006
Gift, S. (2005). ‘Quality at UWI-How is it managed?’ In UWI
Today. Sunday June 12th 2005.
Gift, S., Moniquette, J and Perry, A. (2004). ‘The University of the
West Indies: Challenges, Dilemmas and Quality Assurance.’ Paper presented
at the Biennial Conference of the International Network for Quality Assurance
Agencies in Higher Education (INQAAHE), Dublin, Ireland, and 14-17 April
Harvey, L., and Green, D. (1993) ‘Defining Quality’, Assessment
and Evaluation in Higher Education. Volume 18 No 1.
Herrington, A., Herrington, J., Oliver, R., Stoney S. & Willis, J. (2001).
‘Quality guidelines for online courses: The development of an instrument
to audit online units.’ In G. Kennedy, M. Keppell, C. McNaught & T.
Petrovic (Eds.) Meeting at the crossroads: proceedings of ASCILITE 2001,
(pp 263-270). Melbourne: The University of Melbourne.
Institute for Higher Education Policy. (2000) Quality on the Line: Benchmarks
for Success in Internet-based Distance Education. Washington, D.C. IHEP
Morrisett. I. & Stevens, W. W. (1967). Steps in curriculum analysis
outline. Boulder: University of Colorado, Social Science Education Consortium.
Office of the Board of Undergraduate Studies. (2000) The UWI Quality Strategy:
The Quality Assurance System at the University of the West Indies. Mona:
Reeves, T.C. & Laffey J.M. (1999) ‘Design, assessment and evaluation
of a problem-based learning environment in undergraduate engineering.’
Higher Education Research and Development Journal, Volume 18, No. 2,
Scriven, M. (1967). ‘The methodology of evaluation.’ In R.E. Stake
(ed.), Curriculum evaluation. American Educational Research Association
Monograph Series of Evaluation, No. 1. pp. 39-83). Chicago: Rand McNally.
Scriven, M. (1974a). ‘Evaluation perspectives and procedures’.
In W. J. Popham (ed.), Evaluation in education, Berkeley, CA: McCutchan.
Worthen, B. R., Sanders, J.R. and Fitzpatrick, J. L. (1997). Program evaluation:
Alternative approaches and practical guidelines. 2nd ed. NY: Longman Publishers.
GUIDELINES FOR CITS STAFF
RE: COMPLETION OF THE
ONLINE COURSE CHARACTERISTICS MATRIX
The attached matrix is designed to obtain feedback on courses with an online
component. This information will assist us in gauging the characteristics and
activity of online offerings at The UWI. This will serve as the first stage
in establishing standards in online teaching and the provision of support for
online course developers.
Please complete the matrix using the explanation of each category provided below.
Provide information for all courses with an online component that are facilitated
through your campus IT services.
|1. Course code/Title
||The 8-digit alphanumeric code and full course title
(e.g. EDME2006 – Classroom Testing and Evaluation)
|2. Start date
The date when the course first became available to students OR when
it will become available
If the course is a test course only or for any other reason may not become
available to students, indicate this by writing N/A
||The semester(s) when the course is available to students
||Campus on which the course was created
||Indicate the faculty with academic responsibility for the course
|| The academic programme of which the course is a component (e.g. BSc Management
Studies, MSc Environmental Management)
||Staff member who designed or created the online course component
||Certificate, Undergraduate, Diploma, Postgraduate
||Indicate whether the online course is one of The UWIDEC’s course
offerings or not
|10. Learning Management
|| Identify the learning management tool through which
the online components are offered (e.g. WebCT, Moodle, Blackboard, faculty
created website/ web page/learning management system)
Return date and address
Please return completed matrix by April 30th 2006 in the envelope provided
or via email to:
Instructional Development Unit
The University of the West Indies
WEBCT COURSE EVALUATION & APPROVAL FORM
Course Code/Title: _________________________
Course Designer: __________________________ Faculty: __________________________
|Use of correct colour (faculty appropriate)
- Technical quality
- Appropriateness for example, use of icons
- Guidelines/helpfulness to students
|Appropriateness of tools
|Level/quality of use of tools
|Availability of Compile tool
- Ease of use for students
- Variety of options in moving through course material
- Material appropriate to delivery strategy
- Quality of diagrams
- Links (within and beyond course)
|Guidance to students in moving through course material
**Minimum of a 3 rating required in MOST categories for Approval
APPROVED‚ NOT APPROVED‚