Measurement tools outdated

Regional universities, with a very different mix of students, will be at a disadvantage when compared with metropolitan institutions writes Bill MacGillivray In December the Department of Education, Employment and Workplace Relations (DEEWR) released a set of discussion papers on the development of performance measurement instruments in higher education. This included an overview for designing such instruments and two more specific papers reviewing the Australian Graduate Survey (AGS) and addressing the assessment of generic skills. The release of these papers followed the government’s removal of performance funding for such measures from university compacts. Principles that should inform the development of such measures are included in each paper. The proposed principles are: fit for purpose, consistency, auditability, transparency and timeliness. I can confidently predict that there will be considerable debate about each of these, particularly “fit for purpose”, given the change in using the outcomes not for determining performance funding but rather for public information. But I would propose an addition to the principles, that of “equity” – information is relevant for the entire demographic and does not disadvantage any student cohort or any institution. The AGS was instituted in the 1970s when the typical undergraduate student at Australian universities was a school-leaver who studied full-time, on-campus and upon graduation either sought full-time employment or pursued further study. I know, I was one of them. A small minority of students gained employment directly from school and studied part-time, attending campus in the evening. Distance education was in its early days then. While this portrait of a university undergraduate is still true today in many metropolitan universities, it is not the case for others, particularly the regionals. At Southern Cross University, only 20 per cent of commencing undergraduates are school-leavers, and the average age of our undergraduates is 24 years. About 50 per cent of SCU undergraduates are in employment and study part-time; 30 per cent study by distance. And yet the survey instruments designed continue to favour the traditional student. For example, the Graduate Destination Survey (GDS) is one of the AGS instruments. The GDS measures the success of university graduates in moving from study into employment. The figure reported is the percentage of those in full-time employment compared with those seeking full-time employment. This immediately reduces the sample measured for regional institutions. Until last year, DEEWR reported only those graduates who had studied full-time. The Good Uni Guide goes further, only reporting measures for Australian graduates under 25 years of age working full-time in their first full-time job. No wonder the outcomes for regional universities are so poor in the GUG. A more equitable measure would be to ascertain the career improvement of a graduate. The discussion papers consider student surveys, both current and potentially future, against a student life cycle framework structured as pre-entry, undergraduate study and post-study. While some of the current instruments engage students at any time of their undergraduate study, such as the International Student Barometer (ISB), others such as the AUSSE (the Australian Survey of Student Engagement) measure the experience of first year and final year students. Both of the proposed new instruments, the University Experience Survey (UES) and the Collegiate Learning Assessment (CLA), are designed to ascertain performance measures from first and final year students. The tables provided in the overview discussion paper show that there is a first year, middle years and a final year. The difficulty is that for many degree courses this neat structure is no longer relevant. There are now a plethora of pathways for someone to gain admission to a university. The Australian Qualifications Framework (AQF) on which the TEQSA Qualifications Standards are based require a university to provide a minimum of eight units of advanced standing for someone admitted to a degree with a recognised diploma in a similar discipline, or 12 units for an advanced diploma or 16 units for an associate degree. What does the student life cycle look like under these scenarios? If first year is eliminated, then students arriving at universities through these pathways are excluded from participating in the UES and CLA. In addition, how will the difference in length of undergraduate courses be treated when assessing the improvement in skills? As described in the newly released report on its development, the areas of focus for the UES are proposed as learner engagement, teaching and support and educational development. Members of the Project Advisory Group suggested strongly that this list was too restrictive. Now that the government has removed performance-based funding as a response to these survey outcomes but rather indicated that results could be published on the My Uni website, the fit for purpose of the instrument now comes into question. Potential on-campus students will also wish to know about the quality of the facilities available for when they are not in a classroom: refectory, leisure facilities, wireless connection are to name just a few. Potential distance students will be interested in the ease of access to information, to learning management systems and to asynchronous support. As the report itself states, “As the UES survey was developed with the primary purpose of allocating performance-based funds, the instrument will need further development to ensure it collects data that is as useful as possible for informing potential students about where they might study”. In government circles at least, there seems to be a view that an improvement in generic skills over the course of study by an undergraduate student is a direct measure of their learning outcomes. I do not think many academics would subscribe to this view. There is no doubt that the measurement of learning outcomes is an on-going challenge but one that the ALTC undertook with some commitment. TEQSA is now developing a framework for teaching and learning standards but have admitted that this will take some time. While describing learning standards as the nature and levels of student attainment, the standards discussion paper indicated that these would include course and discipline-specific skills and knowledge as well as generic skills. It is therefore of considerable concern that the performance measurement discussion papers nominate the CLA as a proxy for measuring higher education learning outcomes while suggesting that the inclusion of discipline-specific measurements may be added at some point in the future. The CLA was designed as one tool to assist participating American colleges in attaining higher quality teaching through an internal continuous improvement process. Even though the performance of a college can be benchmarked against others by standardising its outcomes using the results of its student cohort in admission tests such as SAT, the instrument was not intended as a national performance measure. In 2011, SCU participated in the CLA rounds, albeit our first and final year students were assessed out of phase with the freshmen and seniors in the United States. We are currently awaiting the report on our students’ performance. The assessments are undertaken in a supervised environment and are approximately 90 minutes duration. The exercise was resource intensive given our multi-campus structure. We did not address the challenge of assessing students studying at a distance. With the removal of performance funding for teaching and learning, the impetus to introduce new instruments for measuring institutional performance is the imminent introduction of the My Uni website. The proposed use of the UES and CLA as performance measures to inform student choice questions the fitness of purpose of these instruments. In a sector of diverse institutions, universities such as SCU will be disadvantaged if these proposals are implemented. Bill MacGillivray is the deputy vice-chancellor at Southern Cross University and was a member of the UES Project Advisory group.