Quality Audit: What Everybody Should Know

Australian University Quality Agency (AUQA)

What is the Australian Universities Quality Agency?

AUQA was established by federal and state governments to (i) guarantee the quality of Australian universities, especially for international marketing (ii) quarantine the name ‘university’ for institutions like those now using the name (iii) establish processes to evaluate new entrants to the Australian university system (post-Greenwich University).

What does AUQA do?

AUQA monitors the Quality Assurance System each university is using. Each university is expected to state its goals clearly, and to provide evidence that is achieving them. JCU is developing its Quality Assurance System from its existing somewhat rudimentary strategic planning process during 2002-3.

For audit, the university submits a report on its Quality Assurance System called the Performance Portfolio (about 20,000 words, plus substantial support documentation including key policies and data about the University). An Audit Panel is appointed: an AUQA Audit Officer, three Australian university senior staff (usually DVC, PVC or Executive Dean level), an international academic of similar level, and a senior business leader. The Audit Panel checks the claims made in the documentation by interviewing about 250 staff of the Panel’s choosing over three days. The Audit Panel report is published by the AUQA Board.

What is Quality Assurance?

Quality assurance involves the use of evidence to check that goals are being achieved and that the evidence is used to reshape goals and practices to bring about improvement. A quality assurance audit checks that evidence is being used effectively. Questions posed to the institution are designed to check the quality of evidence use about the context, the inputs, the processes and the products and how the evidence is used. Consider each one in turn and the example questions:

Context : What evidence does the University use to know it is a good idea to

conduct a nursing program?
support the Chamber Music Festival?
provide resources for research into the Irukandji syndrome?

Inputs: What evidence does the University use to show that its

Code of Conduct for staff is an appropriate policy?
teaching staff have the skills appropriate to the range of teaching practices expected by the University?
research ethics policy meets nationally recognised standards?

Processes: What evidence does the University use to show that its

curriculum and teaching practices are acceptable to students?
curriculum and assessment practices meet national university standards?
research in teacher education is of national or international standard?
research practices improve the undergraduate curriculum?

Products: What evidence does the University use to show that its

graduates are achieving the generic skills described in Academic Board policy?
engineering graduates are as well qualified as those from bigger universities?
graduates are valued in the regional community?

One key task for the University is to improve and document the ways these kinds of evidence are used for improvement.

The Quality Audit

Why does the AUQA audit matter?

The Audit Panel’s public report on the University makes ‘commendations’ and ‘recommendations’ about the Quality Assurance System of the University. These are readily re-interpreted as commentary on the quality of the University. Quality Audit reports attract media attention. Any adverse report will have a very negative impact on the University in Australia, and on international marketing.

Whom does the Audit Panel interview?

Assume anyone within the University may be called for interview: If you cannot answer questions about the QA system as it applies in your job you will let the University down.

How long have I got to prepare?

JCU Quality Audit Timetable

Trial Audit as part of the training and preparation Sept 22-24, 2003

Performance Portfolio completion Dec 2003

Audit Panel Visit May 2004

What should I do to prepare?

All university staff:

Basically the task for all individuals is quite simple. Prepare to answer questions like these:

What are you trying to do in your work (referring to your unit’s goals)?
How do you know if you are successful? What evidence do you use?
How is the evidence used to bring about improvement in your work?

Staff in management roles and all academics

Staff in management roles will be expected to refer more explicitly to the strategic plans for their areas and the ways these relate to the goals of the University described by In the Third Millennium, the Council document that defines the nature and direction of the University. Managers and all academic staff should be able to describe:

the goals of their organisational unit and why they have them
the policies and legislation that are relevant and how they are used
the strategies and practices in place to achieve the stated goals
how they know whether they are achieving their goals (that is, what evidence is used)
who is responsible for

collecting and interpreting evidence,
acting on it (improvements) in practice and reshaping goals
informing (and getting information from stakeholders about what has happened.

Quality Assurance and JCU

What is the JCU Quality Assurance System?

The JCU Quality Assurance System is based upon the strategic planning process that has been developing for three years. The Quality Assurance System is simply the strategic plan structure with more direct attention to the (i) integration of relevant policies (ii) explicit use of feedback and evidence and (iii) action loops that show who is responsible for acting on the evidence and implementing improvements. The Quality Assurance System has several elements:

Strategic Plans

In the Third Millennium: Our Future and How We Get There

Broad strategic plans such as Internationalisation

Strategic Plans for Faculties and Divisions

Strategic Plans for Schools, Directorates, Research Centres and other organisational units

Individual Performance Planning and Review ….

Key Policies (and relevant Legislation)

Code of Conduct

Academic Promotions

Quality of Teaching and Learning

Teaching and Assessment Code of Practice

Research and Research Training Management Plan

Intellectual Property….

Faculty, School, Division, Directorate, Research Centre policies (and procedures).

Governance Structures

Council, Academic Board and its committees

Faculties and Divisions and their committees

Schools and Directorates and other organisational unit committees

Line management structures

Delegation structures

Review Practices

Course Review

Research and Research Training Management Plan review practices

Student Feedback about Subjects

Student Feedback about Teaching

Course Evaluation Questionnaire

Graduate Destination Survey

What are the important aspects of quality?

A university is a complex place, but it is possible to identify some key areas where we aspire to quality. Typical evidence useful to assure the quality of each area follows in the list.

Quality teaching and learning programs

External expert review of curriculum development and implementation, student ratings of subjects and teaching, benchmarking and cross-institutional comparisons of assessment standards

Quality research and research training

External peer review, publications in prestige journals, competitive research grants demand for research training places, ratings by students of supervisors and support services

Quality staff

Numbers of quality applicants , strong qualifications and scholarly activity before appointment, national and international recruitment, systematic staff development

Quality infrastructure for support

Benchmarking of library and IT provision and performance against other universities, student ratings of service, survey of staff and student satisfaction

Quality corporate conceptual support

External expert review of compliance with audit law and standards of governance, policy and procedure comprehensiveness, clarity, accessibility

Quality community relationships (service, input, advocacy)

Systematic expert evaluation of each practice identifying adequacy of purpose, the nature of interactions and their outcomes

Quality students

High entry scores, rates of participation and completion of elite students and students from under-represented groups

Quality graduates (achievement outcomes, employment …)

Rates of employment, graduate salaries, strong alumni associations

Quality critique

Active subject development and teaching teams, shared course materials development, strong collection and use of feedback from students, peer review, external accreditation.

What will be the foci of the JCU audit?

Teaching and Learning
Research and Research Training
Support Services for Students
Service to the Region and Community
Strategic Planning, Management and the Quality System; Council and Governance
Administration and Financial Management, Assets and Facilities Management, Human Resource Management.

The Performance Portfolio will describe the University’s own review of quality assurance in and its impact in each of these areas.

Key Terms

What are some key terms it would be useful to be able to use?

Strategic planning documents include reference to the following concepts though terminology may vary a little because of the devolved system and authentic discipline differences, persuasions and preferences. The left hand column of concepts would be useful, but knowing things closer to your own work is the most important.

Objectives: What are our existing aspirations, goals and objectives, as expressed in ‘Management Unit’ or ‘Organisational Unit’ publications?
Strategies: What are our existing key strategies for each objective as expressed in ‘Management Unit’ or ‘Organisational Unit’ publications? These should include continuing operating strategies as well as new initiatives.
Associated Documentation: What documentation, including policy, principles, legislation and guidelines, do we have that act as drivers to the objectives and strategies. Note also where documentation needs to be developed?
Documentation Deployment: How, where and to whom is this documentation disseminated?
Data Collected: What data do we collect to inform the development, practice and outcomes of each strategy. What level of data is available – broad and comprehensive, local, or not available?
Performance Indicators: What are the indicators used to evaluate the achievement and effectiveness of the objectives?
Key Agents: Who is responsible for acting upon the feedback from data collection and other quality assurance processes?
Reviews of Practices: What internal and external mechanisms do we have in place for reviewing our practices and processes?
Identified Deficiencies: What deficiencies were identified as a result of the last review/s of practices and processes?
Resultant Changes: What changes resulted from the review/s?
Improvement Potential: How can we improve on current actions, even those that are already effective?

Quality Assurance: What Everybody Should Know