APQN conference – Siem Reap, Cambodia

2012 Feb. 29

APQN conference - Siem Reap, Cambodia

Summary of IDRC-supported QA in DE studies

Insung Jung

National quality assurance regulatory framework covering distance education in Asian countries

The study was carried out between January and December 2010 and employed three data collection steps: 1) 11 cases (10 countries and one SAR) from East, South, and Southeast Asia were selected to include those with relatively well-established QA systems, those just introducing QA systems, and others still in the process of developing QA concepts in DE; 2) formal documents published by QA agencies (research institutes and governments in the selected countries/territories) and other references were analyzed to delineate DE development, QA policies, procedures, standards, and criteria for higher education in general and DE specifically; and 3) face-to-face, email, or telephone interviews with local experts working in the QA agencies and DE institutions were conducted to verify the data obtained.

This study revealed that QA in DE is still at an early stage of development compared with QA in conventional higher education. QA in DE is still a relatively new concept in some countries/territories, and the different QA approaches described above reflect the differences in cultures, expectations, and stages of development. At the same time, it found that there are common features in the approach taken by the different QA agencies in the cases examined, including: 1) working towards promoting a culture of quality within QA agencies and DE institutions, 2) positioning QA in the pursuit of self-improvement and public accountability of DE institutions, 3) considering distinctive features of DE in QA frameworks or during evaluation processes, 4) linking QA results to direct or indirect funding, levels of autonomy, or other supports,( only applies to public funded institutions) 5) adopting both internal and external assessments, and 6) making QA results public.

In light of these findings, several suggestions are made for the improvement of QA systems in Asian DE.

All nations should see QA in DE as an integral part of broader national, regional, and international QA frameworks.
Concerted efforts are needed from governments, national and regional QA bodies, and institutions regarding cross-border accreditation and QA in Asian DE.
With the convergence of conventional and distance educational methods, there should be no distinctions between QA in DE and conventional higher education or between e-learning and face-to-face teaching. However, having said that, specific QA criteria and standards are needed for the various modes of delivery.
The existence of a national QA framework would facilitate DE institutions to make QA an integral part of their institutional missions with respect to teaching and research and to promote a culture of quality in their institutions.
Understanding the relative and value-laden nature of QA (Dondi, Moretti, & Nascimbeni, 2006), Asian governments, DE providers, and assessors should make an effort to understand distance learners’ concerns and integrate their views when developing national and institutional QA policies since the success of DE typically relies to a greater extent on learners’ motivation and engagement.
As COL (2009) and Shale and Gomes (1998) argue, quality key performance indicators (Q-KPIs) will help DE institutions monitor their performance against institutional objectives.

Details of this study can be found in: Jung, I.S., Wong, T.M., Li, C., Baigaltugs, S., & Belawati, T. (2011). Quality assurance in Asian distance education: Diverse approaches and common culture. The International Review Of Research In Open And Distance Learning, 12(6), 63-83. Retrieved from http://www.irrodl.org/index.php/irrodl/article/view/991/1953

Quality assurance in Asian distance education institutions

A survey was sent out to over 40 DE providers in Asia between January and December 2010 to collect information about institutional QA mechanisms including procedures, methods and criteria, and accreditation and quality certification. Eventually 26 institutions responded to the survey.

When asked about the meaning of quality and QA in DE, the participating DE institutions saw quality and QA differently, depending on the needs of stakeholders, their priorities and perceptions. Despite differences in the meaning of quality and QA, they share common underpinning concepts relating to quality, such as continuous improvement, quality culture and quality standards. Some see QA as a comprehensive continuous improvement effort in terms of inputs, process, outputs and outcome, and others see it as effort to maintain, reach or comply with standards externally set, normally by their governments or quality agencies. To most DE providers, a good QA system should make efforts to meet customers’ and stakeholders’ satisfactions of distance and e-learning programmes and courses, and QA effort should be applied at planning and implementation stages to ensure quality outputs. Most DE providers indicate the important role of leadership in encouraging member participation throughout the QA process and effort is important so that everyone in the institution shares common awareness, understanding and spirit of QA.

Overall, large DE and e-learning institutions in Asia have a formal QA policy framework. Generally smaller institutions and institutions with small DE and e-learning operations state that there is no formal QA policy, despite the fact that they try to implement QA systems for their DE and e-learning programmes and courses. Also, DE and e-learning institutions in Asia vary in terms of the establishment of a formal QA Unit or Office, depending on size and priorities of the institution.

Several DE institutions in Asia also employ an external QA system, one referring to adopting benchmarks and good practices worldwide, another relating to the use of external peer review. Internal QA processes in Asian DE and e-learning institutions share common practices despite the fact that they vary slightly in terms of implementation, and these include such processes as assessment, monitoring, and review of programmes and courses, feedback consultation with stakeholders and employers, managements, reports and self-assessment. Some Asian DE and e-learning institutions have developed their own guidelines, manuals, procedures and forms for their own internal QA system.

Major challenges indicated by the DE institutions include: 1) inclusion of external reviewers with knowledge and experience in DE and e-learning as this will result in fair quality assessment, 2) applying indicators and standards that take into account the special characteristics of DE institutions, 3) securing well qualified and trained manpower in QA for DE, 4) improving awareness among faculty members and administration that the QA process needs to be implemented for quality DE and e-learning, and 5) establishing reliable technology infrastructure and receiving enough funding and support from governments or other supporters in the case of smaller DE institutions.

Details of this study can be found in: Zuhairi, A., Belawati, T., Nugraheni, E., Sadjati, I.M., PKH, Y., & Isman, S.M. (2011). A provider survey on the quality of distance and e-learning in Asia. A paper presented at the 25th AAOU Annual Conference. September 28 – 30, 2011. Penang, Malaysia.

Asian distance learners’ perceptions of quality in distance education

The study identified ten quality dimensions and built a QA model on three domains: supportive, pedagogical and environmental. The three domains are used to categorize and organize the ten dimensions in a meaningful and efficient manner.

Supportive domain refers to an assistive quality aspect that helps learners carry out distance learning effectively and efficiently and includes three quality dimensions – Faculty Support, Student Support, and Information & Publicity.
Pedagogical domain refers to a core quality aspect in DE that helps learners develop and change their knowledge, skills and attitude both independently and collaboratively, and include four quality dimensions – Course Development, Teaching & Learning, Interactive Tasks, and Evaluation & Assessment.
Environmental dimension refers to a contextual quality aspect that creates distance teaching and learning environment where learners work productively and flexibly with high confidence in DE, and includes three quality dimensions – Infrastructure, Internal QA Mechanism, and Institutional Credibility.
In the Supportive Domain, while Faculty Support appeared to be less important, Student Support and Information and Publicity dimensions were quite influential when assessing the quality of the support aspect of DE in the eyes of the Asian learners. In particular, the distance learners perceived a DE institution or programme that provides social and psychological supports and clear course information to be of high quality.
In the Pedagogical Domain, Evaluation & Assessment was the most powerful dimension in evaluating DE quality. The learners saw fair and clear learning assessment guidelines and periodic students’ evaluation of teaching and learning to be particularly important. The dimension of Course Development was also influential in assessing the quality of the academic aspect of DE. “Well-structured course materials,” “Clear guidelines for course development,” and “Course content adaptability to students needs” appeared strong in explaining the Course Development dimension—meaning that Asian learners perceive a DE programme that provides well-structured course materials that follow clear development procedures and are considerate of learners’ needs to be of high quality.
In the Environmental Domain, Infrastructure appeared to be most important in assessing the quality of DE. Items such as “Reliable media/technology infrastructure,” “Reliable learning management system,” and “Security of student data system” affected strongly in explaining the Infrastructure dimension. But “Physical classrooms” did not appear to be important at all in assessing the quality of DE. Internal QA Mechanism and Institutional Credibility also appeared to be influential environmental aspects in assessing the quality of DE, which was supported in Jung (2011) in the context of South Korea. “Existence of quality standards specifically for DE” appeared to be highly influential in explaining the Internal QA Mechanism dimension, and “Strong leadership” and “External accreditation at the national level” were the two most important items in explaining the Institutional Credibility dimension.

This study concludes that Asian DE providers should consider learners’ perspective in establishing or improving their QA framework since their views highlights important quality criteria which are not always reflected in the providers’ QA guidelines, and that DE providers should consider the gender differences when designing a learner support system. Some gender-considerate support strategies suggested in the study include: offering flexible schedules that help distance learners, especially females, avoid time conflicts with other responsibilities in taking exams and attending face-to-face meetings, setting up virtual or face-to-face office hours to provide academic or learning content-related support to distance learners, offering both personalized counseling and tutoring services especially to female students and taking advantages of online technologies in providing such services, and providing information on values of the degree earned through DE based on policies and factual evidences specific to male students.

Best practices in quality assurance in Asian distance education institutions

This book attempts to document the existing regulatory framework covering QA in higher education in a number of Asian countries and how 16 DE/e-learning providers/programmes have developed their QA systems and procedures to address the regulatory requirements. It presents good practices in QA for DE/ e-learning, analyzes challenges in assuring the quality of DE and e-learning products and services, offer possible solutions to meet those challenges and presents lessons for other DE and e-learning providers. The 16 institutions include: four mega open universities (India’s Indira Gandhi National Open University, Open University of China, Thailand’s Sukhothai Thammathirat Open University, and Indonesia’s Universitas Terbuka), two dedicated public open universities (Open University of Sri Lanka and University of Philippines Open University), four dedicated private open universities (Malaysia’s Open University of Malaysia and Wawasan Open University, Open University of Hong Kong, and Singapore’s SIM University), two online programmes offered by conventional universities (China’s Peking University and Japan’s Kumamoto University), two new virtual universities (Korea’s Hanyang Cyber University and Virtual University of Pakistan), an NGO (Mongolian e-Knowledge), and a for-profit corporation (Korea’s AutoEverSystems).

The above selection of cases is not exhaustive but ensures that a wide range of QA systems and perspectives of quality in DE/e-learning is covered in this book.

Details of the 16 cases can be found in: Jung, I.S., Wong, T.M., & Belawati, T. (Eds.). (in process). Quality assurance in distance education and e-learning: Best practices, challenges, solutions, and lessons from Asia. India, New Delhi: Sage Publications India Pvt Ltd.

Asian QA Model for DE
proposed by IDRC QA project team

The aforementioned studies have revealed that there exist social and cultural diversities in QA concepts, systems and activities. Different countries and institutions in Asia have developed and applied their own QA model for DE based on their social environment and cultural values. There are therefore variations in the QA policies, criteria and methodologies they propose. Furthermore, they are at different stages of educational and technological development and adopting new and innovative approaches to teaching and learning.

A general model of QA in DE cannot entirely apply in all countries in Asia, but previous studies, for example Jung (2004), reveal that most of the key elements do in fact apply in most countries even though some countries depart further than others from the general model. The open, flexible and distance nature of DE requires a QA system that pays attention to open access, flexible learning methods, course development, reliable infrastructure, interactive communications and support. This suggests that a common or general QA model can be used as a starting point from which to map differences and divergences, and to which each country adds own QA dimensions and elements, or modifies procedures and methods. These deviations will be determined by cultural and other attitudes toward DE and learning in general, the stage of technological development, the flexibility of the legislative system of QA and other practicalities.

We propose a general model that can be benchmarked or modified by a nation or a DE institution as a basis for adapting QA in DE to its own unique context and seeking quality DE provision and social recognition.

Quality assurance policy framework

A QA policy framework at the national level (Figure 1) should have the following features:

There should be no distinction between QA in DE and conventional education, so the QA framework for DE should be within the national QA framework for higher education and under the same QA regulatory authority.
DE providers and their internal and external auditors should be provided with QA/accreditation standards, processes and methods which reflect the unique features of DE in each country’s societal and cultural contexts. These standards for DE should be used in conjunction with the national QA standards for higher education.
Learners’ needs and perceptions of quality should be taken into account in developing and applying the QA standards for DE.
Any national QA policy framework should be aligned with regional and/or international QA frameworks considering the expansion of cross-border DE activities.

Figure 1: QA policy framework for DE at national level

be firmly linked to the national QA policy framework.
be linked with the regional and/or international QA policy frameworks if the institution provides or plans to provide cross-boarder education.
engage key stakeholders.
promote a culture of quality among all members of the institution.
coordinate QA activities of all units and members.
be based on internal QA standards which reflect national (and/or regional/international) QA standards for both higher education and DE.

Figure 2: QA policy framework for DE at institutional level

The QA standards for DE should reflect the various quality concerns of key stakeholders including DE providers, assessors, employers, graduates and learners.

Thus the QA standards proposed here are created based upon the findings of the aforementioned studies which identified quality concerns at national, institutional and personal levels. In addition, the following references published by QA agencies or professional bodies are used.

DEC Handbook on Assessment and Accreditation of Open & Distance Learning Institutions published by Distance Education Council (India) in 2009
AAOU Quality Assurance Statements of Best Practices published by Asian Association of Open Universities in 2010
Guidelines for Good Practices: Open and Distance Learning published by Malaysian Qualifications Agency in 2011
Quality Assurance Toolkit for Distance Higher Education Institutions and Programs published by Commonwealth of Learning in 2009, in collaboration with Distance Education Modernization Project (DEMP) of Sri Lankan Ministry of Higher Education
Teaching and Learning Management Standards in Distance Education published by Research Center of Distance Education in Beijing Normal University (China) in 2009
Cyber University Evaluation Handbook published by Korea Education and Research Information Service in 2008
ACODE Benchmarks published by Australasian Council on Open, Distance and E-Learning in 2010
Accreditation Handbook published by Distance Education and Training Council (USA) in 2011
Quality On the Line published by Institute for Higher Education Policy (USA) in 2000
ODLQC Benchmarks published by Open and Distance Learning Quality Council (UK) in 2005
European Association of Distance Teaching Universities (EADTU)’s E-xcellence and Quality Manual for E-learning in Higher Education
European Foundation for Quality in e-Learning (UNIQe)’s criteria
Quality Matters Rubric

In order to devise the QA standards, we first propose 18 QA criteria or areas across six domains as shown in Figure 3. Environmental, Pedagogical, and Supportive domains include 9 QA criteria in total which reflect key concerns of Asian distance learners. Institutional, Cyclic and Outcomes domains include 9 QA criteria in total which are major concerns of other stakeholders including DE providers, assessors, and other stakeholders.

Environmental domain

Infrastructure refers to the array of policies and measures that ensure the reliability and security of media/technology systems to develop and deliver DE, and the provision of educational resources.
Institutional QAMechanism refers to policy measures and activities by a DE institution with regard to the existence of QA standards and guidelines specifically for DE, and periodic internal and external evaluations.
Institutional Credibility refers to the status of national and international accreditation, institutional achievements, demonstration of leadership, and staff’s qualifications.

Pedagogical domain

Course Development refers to policies and guidelines that help ensure and maintain the quality of content and course, instructional design and course materials and resources development.

Teaching &Learning refers to all teaching and learning activities and online and physical resources provision. It also includes learning activities that promote teacher-learner and learner-learner interactions.

Evaluation &Assessment refers to policies and performance in regard to learning assessment, feedback, and various stakeholders’ evaluation.

Supportive domain

Faculty Support refers to policies and procedures for training, support during course development and delivery, and faculty welfare.

Student Support refers to policies and guidelines for technical, financial, psychological, social and administrative support, flexible payment and appeal mechanisms.

Information & Publicity refers to the provision of course-related and other logistical information and clear indication of requirements.

Institutional domain

Governance and administration refers to the institution’s system of governance and management to facilitate the successful accomplishment of its mission and goals.

Vision/Mission refers to the institution’s vision, mission statements, goals and intended outcomes of DE.

Finance refers to resource acquisition, management, allocation and investment for DE.
Partnerships refers to an institution’s policy and effort to build an effective and collaborative network for DE within the institution and with other organizations.

Cyclic domain

Research refers to an institution’s policies and activities in regard to research in DE and utilizing the results for continuous improvement.
Continuous Quality Improvement refers to an institution’s policies and procedures with regard to promoting a QA culture, QA activities and continuous improvement in DE.

Outcomes domain

Learning Outcomes refers to specifications of what the DE learners should know and be able to do on completing each course, how to measure their learning, and what values DE should offer.
Return-on-Investment refers to specifications and assessments of what institutions and learners gain after from DE, in terms of learning outcomes, graduation rates, employment, lifelong learning, etc.
Contribution to Society refers to the social and economic contributions an institution and its graduates have made to society, including access and equity, gender empowerment, etc.

Figure 3: 18 QA criteria for DE across six QA domains

KPIs are defined as a set of both quantifiable and qualitative measures that a DE institution can use to gauge performance in terms of meeting its QA standards. But more specific evidences to measure these KPIs will be different across countries and institutions.

References

Social commentary Cackle