The correlation between teachers' and researchers' judgements of instructional level was calculated for the first three testing points using the formula: Agreements ¸ (agreements - disagreements) x 100. Overall there was a 62% agreement between the researchers and the teachers. This is considerably higher than the level of chance (11%) in which case teachers and researchers would have randomly identified instructional levels. Forty one percent of the mismatches consisted of the researcher judging the level to be higher than that identified by the teacher and 59% where the researcher deemed the instructional level to be lower. Mismatches between teacher judgement and the results of systematic assessment were reported in Ngä Kete Körero Framework research (1993-1995) where teachers were found to significantly over and underestimate instructional levels for children.

Inter-observer reliability was also calculated. This involved the two researchers administering Pükete Pänui Haere and in some instances täutu reta and whakamätautau kupu with their respective children (a number equating to 20% of the total number of children at that testing point) while the other simultaneously recorded and scored the child's responses. The researchers then independently decided upon an instructional level for each child immediately following the testing. A high agreement of 90% between the researchers was achieved.

Writing

A writing sample was collected from children under standard conditions at each testing point. The researcher read a story and initiated some discussion as a motivation for the writing. Texts were selected on the basis that they comprised a strong narrative storyline, were not likely to have been used as instructional texts with the children to maintain a novelty effect and were sourced from the same series (He Purapura )11 to ensure a consistency in publisher house style. Children were given a timed 5 minutes to draw a picture and a further timed 10 minutes to record a story unassisted.

Over 350 writing samples were collected from children during the research. These were used to develop a writing framework similar to Ngä Kete Körero Framework for reading which teachers have reported to have found useful (Bishop, Berryman, Richardson 2001). Comparable samples were grouped together then common characteristics identified for each set. These were then ordered to demonstrate progression in writing skill and tentative levels were then assigned to the sets.

Trialling was conducted by a variety of teachers in Mäori medium. They were asked to sort randomly selected writing samples into levels according to the characteristics or descriptors provided. The descriptors were revised and refined as required until acceptable levels of agreement (i.e. agreement within one level) were achieved. Four broad levels were identified.

The terminology used for describing the broad levels in reading in Mäori were assigned to the levels identified in writing. Several trials were required to ensure the descriptors discriminated effectively for level and the exercise highlighted the need for teachers to be carefully trained in the future use of descriptors to sort writing samples into levels.

Oral Language

Kï Mai, a story retelling procedure from Aromatawai Urunga Ä Kura/School Entry Assessment was used to assess children's oracy but was only administered with 35 children who were five years of age to five years two months at the time of testing. This is consistent with its use as a school entry assessment. Inter-observer agreement was calculated at 79% for 20% of the children assessed with Kï Mai using taped retellings. Only those criteria directly related to complexity of sentence structure and message were selected for analysis.

The Results

The following results reflect the actual performance of the children who participated in the research study. For the reading and writing assessments, data was collated according to length of time in immersion bands i.e. 0-3 months, 4-7 months, 8-11 months and 12-17 months.

Reading

The mean (average) and standard deviation were identified and used to construct a graph (Figure I). Performance could then be located by plotting children's instructional reading levels after the appropriate number of months in Mäori immersion.

A box plot (Figure II) was also constructed as an alternative way of locating reading performance.

Graph 1 Blueprint of reading achievement of children in total immersion in South Auckland and Waikato Schools in 2000 and 2001.
FIGURE I. Mean and variability of Ngä Kete Körero reading levels from 0-18 months of Mäori immersion.
Click on the image to enlarge.

Based on a normal distribution, the middle bands mark one standard deviation above and below the mean indicating the range of performance of approximately 68% of the children in the sample. The uppermost band indicates two standard deviations above the mean and the performance of approximately a further 16% of the children in the sample. These two bands capture the performance of approximately 84% of the children. The lower band captures the performance of the remaining 16% of children. The horizontal lines and corresponding numbers indicate the size of the sample for the respective time in immersion bands.