Australian Mathematics Literacy Perspectives: Assessment

This is a companion paper and should be read with the earlier one (White, 2017) that focused mainly on Australian governmental policies. In this paper, a more classroom and student focus will be adopted while summarising a few of the salient points of the first paper. A brief listing of five different types of mathematical literacy is provided. The definition used by the Program for International Student Assessment (PISA) is selected andsome brief remarks are provided on this program. PISA items are presented in the form of word problems. If a country such as Australia or Indonesia wants to improve their PISA results then it would demand that some class time is devoted to the development of the abilities of younger students to handle word problems before they turn fifteen and are subjected to PISA. To support classroom teachers, Newman (1977, 1983) defined five specific mathematical literacy levels as crucial to performance on mathematical wordproblems: reading, comprehension, transformation, process skills, and encoding. Newman's Error Analysis (NEA) provides a framework for considering the reasons that underlay the difficulties students experience with mathematical word problems and a process that assists teachers to determine where misunderstandings occur. NEA also provides directions for where teachers could target effective teaching strategies to overcome them. There is a brief discussion of strategies to assist students improve theirmathematical literacy.


Introduction
The word literacy has been attached to many fields of knowledge such as scientific, mathematical, financial, environmental, digital, visual, computer, economic, statistical, arts, technology, information, multimedia, functional, rhetorical, and critical literacy. Mathematical literacy has become important to many countries including Australia and Indonesia because of their membership of PISA (Program for International Student Assessment), which is the OECD's international program of assessment of reading, scientific and mathematical literacy (www.oecd.org/pisa).

Figure 1.
Cycles of PISA and the major and minor domains of assessment for each cycle (Thomson, De Bortoli, & Underwood, 2016, p. xv).
PISA aims to assess the ability of 15 year olds to use the knowledge and skills they have acquired at school on real world problems and challenges. It also uses questionnaires to gather data on students' attitudes to learning and the conditions of schooling. Since 2000, PISA has tested the mathematical literacy of 15 year old students from many countries around the world. There were 72 countries and economies that participated in PISA 2015, including 35 OECD countries and 37 partner countries or economies This paper is a companion to an earlier paper that concentrated at a governmental level on the Australian performance and experience (White, 2017 in press). In this paper, a more classroom and student focus will be adopted while summarising some of the salient points of the first paper. Jablonka (2003) attempted to define the term 'Mathematical literacy' as more than mathematical knowledge and included an individual's ability to use and apply the mathematical knowledge to a specific context. She categorized five different types of mathematical literacy.

Definition of Mathematical Literacy
Her list included: mathematical literacy for cultural identity, where mathematics is seen as a pan-cultural universal activity for representing and explaining the natural world and that all cultures have developed some kind of mathematical capital.
Another on Jablonka's (2003) list is mathematical literacy for social change where it becomes a process of developing a critical consciousness for re-interpreting parts of reality and pursuing different realities. This recognizes attempts to avoid privileging Western academic mathematical knowledge by incorporating ethno-mathematical and everyday knowledge into school mathematics.
Mathematical literacy for environmental awareness was another type which is a process for developing skills and attitudes for understanding the interrelationships between humans and the natural environment.
Mathematical literacy for evaluating mathematics was another type where all applications of mathematics are seen as value driven and thus the school curriculum requires criteria for evaluation.
And the type that will be used in this paper arises from mathematical literacy for developing human capital that looks at the world through mathematical eyes, emphasizing higher order thinking, real world problems, and underpinned by critical citizenship for participation in public and economic life. Thus PISA (2006) defines mathematical literacy as an .... individual's capacity to identify and understand the role that mathematics plays in the world, to make well-founded judgments, and to engage in mathematics in ways that meet the needs of that individual's current and future life as a constructive, concerned and reflective citizen (OECD, 2006, p. 72).
In order to assess students' use of mathematical literacy on real world problems, many of PISA's items begin with a description of a real life context and it is here that the difficulties arise for students in solving word problems. Figure 2 is a released item from PISA and provides an example.
You are making your own dressing for a salad. Here is a recipe for 100 millilitres (mL) of dressing. Salad oil: 60 mL Vinegar: 30 mL Soy sauce: 10 mL How many millilitres (mL) of salad oil do you need to make 150 mL of this dressing? Answer: ……………….. mL If a country such as Australia or Indonesia wants to improve their PISA results then it would seem that they should mandate that some class time be devoted to the development of the abilities of younger students to handle word problems before they turn fifteen and are subjected to the PISA assessment. Across many countries the performance of students on word problems has a poor history which will be discussed in the next section while briefly looking at a particular diagnostic tool for mathematical literacy called Newman's Error Analysis.

Real World Word Problems
The place and importance of mathematical word problems in the school curriculum have attracted diverse opinions. "Teachers seem not to like word problems. Many have asked me why these are used to 'trick' children in assessments" (Askew, 2003, p. 78). There are teachers who see their job as just teaching algorithmic or procedural proficiency and they would like to remove all words from mathematics tests.
Not surprisingly there are those who disagree and point to the centrality of language in the teaching and learning of mathematics (Clements & Ellerton, 1993). Others would also argue that a deeper level of mathematics is needed beyond procedural proficiency, and that a conceptual knowledge of mathematics is the goal (Carpenter & Lehrer, 1999). Language provides a vehicle for rich classroom discussions and assists teachers and students to appreciate the power of mathematics in making sense of their world.
... if the essence of mathematics is the setting up of and working with mathematical models, and if we treat word problems in such a way, then they might have a role to play in helping children better understand the process of mathematizing. And with the increasing mathematizing of the world (from national test scores to pension prospects), informed and critical citizens need to be aware that mathematizing is not something that arises from the world, but something that is done to the world. In a small way, working on word problems might help begin to develop this awareness (Askew, 2003, p. 85).
In arguing against the removal of word problems from the curriculum, there is evidence to suggest that a word problem is not necessarily more difficult than the corresponding arithmetic using the same mathematics skills (Clements, 1980). Data was gathered from interviews of 126 Year 6 Australian primary school students after the students completed a class test with question pairs of skill and corresponding word problems. For example: the test would contain a question 1 -¼, and later in the test a word problem where a cake is cut into four pieces and one piece is removed and the students were asked how much remained. There were a number of students who got the first question wrong but the word problem correct. The study found that the imagery evoked by the cake problem helped students, and where some other students felt they couldn't proceed with fraction sums presented in numeral form.
So while the language demands of the mathematics curriculum are important they also contribute to the difficulties experienced by the students and teachers. A teacher who wishes to address the mathematical literacy problems of a student requires a diagnostic instrument to isolate the problem(s). Newman (1977Newman ( , 1983) designed NEA as a simple mathematical literacy diagnostic procedure for students attempting to answer a mathematics word problem. She claimed that a student had to be able to pass over a number of successive hurdles or levels: Level 1 Reading (or Decoding), 2 Comprehension, 3 Transformation, 4 Process Skills, and 5 Encoding (see Table  1 for the interview prompts). Along the way, it was always possible to make careless errors and there were some students who gave incorrect answers because they were not motivated to answer to their level of ability. While there are many other procedures available, NEA became popular among teachers because it was simple and easy to use. In her initial study, Newman (1977) gave a test to 917 grade 6 pupils in 31 classes in 19 schools in Melbourne. She then interviewed 124 Grade 6 low achieving students who had made a total of 3002 errors and found that errors made by the students occurred with reading (13%), comprehension (22%) and transformation (12%). Thus, almost half the errors (47%) occurred before getting to the standard mathematical algorithms. Or to rephrase this, nearly half the students never got to demonstrate their mathematical process skills (26% errors) or encoding skills (2% errors). On top of this there were a high number of careless errors (25%). It becomes quite stunning if you add the non-mathematical errors (72%), where only just over a quarter of errors actually deal with mathematics performance. This result meant that teachers were better able to target individual difficulties rather than the often pointless strategy of giving all students more mathematical drill and practice. It demonstrated that students appear to struggle with both the literacy and mathematical demands of typical mathematical word problems.

Newman's Error Analysis (NEA)
Newman's research generated a large amount of evidence highlighting that far more children experienced difficulty with the semantic structures, the vocabulary, and the symbolism of mathematics than with the standard mathematics algorithms. In many Newman studies carried out in schools the proportion of errors first occurring at the Comprehension and Transformation' stages was large (Marinas & Clements, 1990;Ellerton & Clements, 1996;Singhatat, 1991). Studies regularly reported that approximately 70 per cent of errors made by struggling Year 7 Australian secondary students on typical mathematics questions were at the Comprehension or Transformation levels (Ellerton & Clarkson, 1996).
Researchers have also used the NEA on a full ability range of students and across a number of school years. For example, during the years 1977 til 1980, Clements (1980) used trained teachers and university students to conduct interviews of middle years students (Years 5 and 6 in primary school and Years 7 and 8 in secondary school) based on errors made on a 36 item test. The teacher data is shown in table 2.  Melbourne, 1977-79 (Clements, 1980 Grade 5 (n= 55) Grade 6 (n= 207) Grade 7 (n = 280) Error Category If we concentrate on the total of process and encoding errors, errors directly attributable to mathematics we have Year 5 (29%), Year 6 (34%) and year 7 (27%) hovering either side of 30 percent, which is very similar to Newman's data.
In 2007, NEA was included in the Australian state of New South Wales (NSW) Counting On program which was a large state wide initiative to address the needs of students who were struggling in the middle years because of a lack of understanding and proficiency with the early school mathematical skills and knowledge and this included word problems. The NSW Department of Education and Training (NSWDET).produced a range of resources to assist teachers. Below is an example of a classroom poster that was distributed. Teachers used the poster for individual students as well as in a whole class setting, by selecting individuals to read aloud either the problem or the prompts. Class discussions then involved the comprehension level concentrating upon the meaning of words and the meaning of the context. This process provided a scaffold for those struggling with one or more of the early levels. One of the word problems used in the NSW Counting On program is shown below and a series of videos were produced for teachers to show how to use the procedure with this problem.
Natalie paddled 402 km of the Murray River in her canoe over 6 days. She paddled the same distance each day. How far did Natalie paddle each day? Figure 4. A typical NSW Counting On Program word problem.
After demonstrating that they can read the problem, students would move onto the comprehension level. The ability to understand the context is very important at this level. Words such as canoe can be a challenge for children living in a desert region. The unfairness inherent in PISA items due to unfamiliar contexts is one source of criticism when used in comparisons across countries.
Of all the levels that teachers found difficult in assisting students was the transformation level. How do teachers teach what mathematics procedure should be used? The NSW Counting On program made use of what is known as 'tape diagrams' to assist teachers and students and will be briefly illustrated later in the paper.
Levels involving Process Skills and Encoding were the easiest for teachers because it was what they were trained to do. What became obvious from the 2007 NSW Counting On program data was that teachers saw another use for NEA beyond the diagnostic features. So comments such as the following were common: "One head teacher has adopted/adapted it to assist senior students in Stage 6 mathematics" (White, 2008, p. 12). In one school the NEA was changed to stand for "Nail Every Answer". Another adaptation of NEA saw teachers using the prompts as a general classroom problem solving approach. Given an unfamiliar problem, individual students would be asked to work through each prompt, and the teacher would assist the student with difficulties as they arose at each level. Students were also able to use the prompts to self-diagnose their own learning difficulties.
So once NEA has provided teachers with data about the difficulties the students are having, they are now able to make informed decisions about the strategies that effectively target the students' learning needs. The next section will briefly discuss a strategy for difficulties with the transformation level.

Tape Diagrams
The NSW Department of Education Curriculum Support Directorate developed teacher material involving the use of what are known as 'tape diagrams' (see figure 4) to assist teachers. The tape diagram is a central visual representation used extensively in the Japanese mathematics curriculum (Murata, 2008). For example Figure 4 shows the tape diagram for the problem: ' Paul went on a bike hike. He rode 402 km on his bicycle over 6 days. He rode the same distance each day. How far did Paul ride each day?' The use of tape diagrams by students experiencing difficulties at the transformation level has proved successful as it acts as an organising tool to keep track of all the data in the problem and focus upon the goal. So this concludes the brief discussion of NEA as a powerful mathematics literacy diagnostic assessment and pedagogical tool for word problem solving in the mathematics classrooms. In the next section I want to briefly summarise Australia's experience with PISA which can be found in greater detail in the earlier paper (White, 2017 in press).

Australia Experience of PISA and Mathematical Literacy
In 2000 Australian educators were feeling reasonably satisfied because Australia's 15 year old students performed very well on the mathematical literacy scales in terms of the knowledge and skills in PISA for 2000(OECD 2000, 2004. As well, the Trends in International Mathematics and Science Study (TIMSS) for 1994/5 and for 2002/03 revealed that Australian Year 8 students' were significantly higher than the international average in all content areas considered (Thomson & Fleming, 2004). And both PISA and TIMSS became components of the Australian national assessment program ( Figure 5). Figure 6. Australian national assessment program (Thomson, et al., 2013, p. 2).
However since then there has been a steady decline (see figure 6) which has been attributed to governmental interference by educational researchers and experts Sahlberg (2011). Figure 7. Australia's mathematical literacy performance on PISA (Thomson, De Bortoli, & Underwood, 2016, p. 21).
It is possible to summarise the directions of federal and state educational policies since the start of the new century that have accompanied the decline in Australia's international student performance.
 Increased politicisation of education across Australia;  Greater centralisation of the curriculum and adoption of business management models;  Major political parties unable to compromise and collaborate;  Greater funding of independent and private schools and failure to implement Gonski's needs-based funding of all schools;  Increase in the amount of testing and reporting of student performance on basic skills;  Increasing inequality and less emphasis on reducing educational disadvantage;  A loss of autonomy by teachers and schools and a greater administration load.
Australia has a history of following trends in education that have arisen in the United States of America and/or Great Britain in spite of mostly performing better than these two countries on international assessment studies such as TIMSS and PISA. There are lessons to be learned by other countries in observing the Australian experience and that is to avoid adopting policies and programs from countries that are performing worse than you.

Update on 2017 Paper
Since 2017 there has been some interesting developments at a governmental level that represent a correction to the issues listed in the 2017 paper (White, 2017). The former minister of education in the state of New South Wales (NSW) stated recently in a national paper: The NAPLAN (National Assesment Plan in Literacy and Numeracy) testing and reporting regime is clearly in its death throes. The federal Labor Party want to review it, the NSW Minister of Education is arguing for it to be scrapped, pretty much every principal and teacher hates it and more parents are seeing the negative consequences for their children and their schools (Piccoli, 2018, p.33).
The Finnish educator Sahlberg is now working for the NSW Department so the situation regarding national testing and reporting will undergo close scrutiny.
The former minister (Piccoli, 2018) made some predictions with national testing to continue but the reporting on the MySchool website would change to remove school comparisons, a major contributor to the negative effects. He stated that: The idea of of publishing results in order for competition to drive improvement has been discredited around the world and education systems are moving away from this approach (Piccoli, 2018, p.33). He suggested that any review of national testing should follow a number of principles such as: 1. Provision of useful feedback to parents, teachers and schools; 2. Publication of results on the MySchool website should be scrapped because of the negative impact upon students; 3. Requirement of a reliable way for government and policy makers to evaluate the performance of school systems. Sample testing, akin to PISA could be considered; 4. Rejection of competition between schools; 5. Involvement of teachers in the process of design, implementation, and systems of accountability.

Conclusion
At the heart of this paper is assessment and choosing the most beneficial type for the improvement of mathematical literacy. One could follow the path of Australia (now under review) and adopt a strategy of more national assessments which greatly increases teachers' administration load but is of little value to teachers in the teaching and learning cycle. The decline in performance is regarded as a failure of this approach and detrimental to teachers as the administrative demands associated with the test rob the teachers of time and effort which would be better focussed on the students.
One alternative strategy is to encourage the greater focus on word problems and greater use of diagnostic assessment by classroom teachers such as NEA. This on time assessment would be of far greater value to teachers in improving the mathematical literacy of their students and would be helpful to parents.
Is it really necessary for countries such as Indonesia to spend such vast amounts of money on PISA assessment items that are really designed for European students? Why not develop an Indonesian localised program such as SEAQiM is currently doing by constructing a regional assessment program for Southeast Asian called Mathematics Regional Wide Assessment (MaRWA)?