1 The Application of the Suffolk Reading Scale (2) on South African Learners Maria Ramaahlo 0402394d A research report submitted to the Faculty of Humanities, University of the Witwatersrand, Johannesburg, in partial fulfillment of the requirements for the degree of Master of Educational Psychology by Coursework and Research Report Johannesburg, November 2010 2 Declaration I hereby declare that this research is my own independent work, and had not been presented for any other degree at any academic institution, or published in any form. It is submitted in partial fulfillment of the requirements for the degree of Master of Educational Psychology by Coursework and Research Report at the University of the Witwatersrand, Johannesburg __________________ Maria Ramaahlo 3 Acknowledgements I wish to extend sincere thanks and grateful acknowledgement to the following: To my God, who has given me the opportunity, strength and ability to pursue studying and completing my dissertation. To my supervisor, Mr. Joseph Seabi, for all his expertise, guidance, and encouragement. Your patience as a supervisor was greatly appreciated. Thank you for trusting me to be part of this larger study and to conduct my research in this area. I hope that this dissertation will make a difference in the field of psychometrics. Thank you for walking the path of growth and professional development with me for the last two years. To the principals and learners of the selected schools in Kwa-Zulu Natal for their enthusiastic participation in this study. To Mr. Brandon Morgan and Ms. Juliana Van Staden for their statistical assistance and expertise. To Mr. Paul Court, for proof reading and language editing this dissertation. To my family who have given me the opportunity to pursue my studies. Thank you for your kindness and all your prayers. To the B Ed students, Dr. Goldshagg and Nicky, a special thank you for the many hours, checking, typing and transcribing all the test scripts- To spikey squares! 4 To my friends who have always encouraged me. Thank you for your kindness and all your prayers. To all those who have encouraged and supported me through this process, which made the process easier. 5 Table of Contents Declaration ................................................................................................................................................................ 2 Acknowledgements ................................................................................................................................................... 3 List of Tables ............................................................................................................................................................ 8 Abstract ..................................................................................................................................................................... 9 Chapter 1 ................................................................................................................................................................. 10 Introduction ............................................................................................................................................................. 10 1.1 Research Rationale ........................................................................................................................................ 11 Chapter 2 ................................................................................................................................................................. 13 Literature Review.................................................................................................................................................... 13 2.1 Function of Psychological Assessments ....................................................................................................... 13 2.2 Origins and History of Psychological Assessments ...................................................................................... 14 2.3 Controversies in Psychological Assessment ................................................................................................. 16 2.3.1 Test Bias ................................................................................................................................................. 17 2.4 Assessment: A South African Perspective .................................................................................................... 18 2.4.1 Language Bias ........................................................................................................................................ 20 2.4.2 Culture Bias ............................................................................................................................................ 23 2.4.3 Gender bias ............................................................................................................................................. 25 2.5 Assessing Reading Comprehension: Suffolk Reading Scale ........................................................................ 26 2.6 The translation and adaptation of assessment ............................................................................................... 28 2.7 Research Questions ....................................................................................................................................... 29 6 Chapter 3 ................................................................................................................................................................. 31 Methods................................................................................................................................................................... 31 3.1 Chapter Overview ......................................................................................................................................... 31 3.2 Context of the Study ...................................................................................................................................... 31 3.3 Research Design ............................................................................................................................................ 32 3.4 Hypotheses .................................................................................................................................................... 32 3.5 Participants .................................................................................................................................................... 33 3.6 Instruments .................................................................................................................................................... 36 3.6.1 Biographical Questionnaire .................................................................................................................... 36 3.6.2 Suffolk Reading Scale ............................................................................................................................ 36 3.7 Threats to Validity ......................................................................................................................................... 37 3.8 Procedure and Ethical Considerations........................................................................................................... 37 3.9 Data Analysis ................................................................................................................................................ 38 Chapter 4 ................................................................................................................................................................. 40 Results ..................................................................................................................................................................... 40 4.1 Overview of Chapter ..................................................................................................................................... 40 4.2 The psychometric properties of the SRS2 ..................................................................................................... 40 4.3 The reliability coefficient .............................................................................................................................. 41 4.4 Comparison between EFL and EAL participants on the SRS2 ..................................................................... 41 4.5 Comparison between male and female participants on the SRS2 ................................................................. 42 4.6 Comparison between EFL and EAL participants on individual items of the SRS2. .................................... 43 7 Chapter 5 ................................................................................................................................................................. 46 Discussion and Conclusion ..................................................................................................................................... 46 5.1 Introduction ................................................................................................................................................... 46 5.2 Reliability of the SRS2 .................................................................................................................................. 46 5.3. Performance between language groups on the SRS2 ................................................................................... 46 5.4 Performance between genders on the SRS2 .................................................................................................. 47 5.5 Performance between language groups on individual items of the SRS2..................................................... 48 5.6 Limitations .................................................................................................................................................... 51 5.7 Recommendations for future research........................................................................................................... 52 5.8 Conclusion ..................................................................................................................................................... 52 References: .......................................................................................................................................................... 54 Appendix ................................................................................................................................................................. 59 Appendix A: Biographical Questionnaire ........................................................................................................... 59 Appendix B: Consent letter to parents ................................................................................................................ 87 Appendix C: Consent letter to learners ............................................................................................................... 89 Suffolk Reading Scale 2 ...................................................................................................................................... 92 8 List of Tables Table 3.1 Gender distribution of participants Table 3.2 Racial group distribution of participants Table 3.3 Language distribution of participants Table 4.1 Descriptive statistics for the SRS2 Table 4.2 Cronbach coefficient for the SRS2 Table 4.3 t-test result for language Table 4.4 t-test results for gender Table 5.1 Chi Square Test: Significant differences in performance on individual items of the SRS2 9 Abstract The field of psychometrics in South Africa faces many challenges. Among these are, that practitioners in the field of psychology do not always have access to standardised assessments for the South African context. Imported assessments pose various biases to South Africa?s multilingual and multicultural situations, hence the need for test adaptation. The Suffolk Reading Scale (2) (SRS2) is an English proficiency assessment that measures reading comprehension. English, being the language of learning and teaching in most South African schools, makes the SRS2 a significant gain to practitioners and the field of psychometrics. Consequently, this research aimed to narrow the gaps in this area of knowledge and contribute to it by evaluating the applicability of the SRS2 as a measure of reading comprehension for South African learners. Primary school learners in Kwa-Zulu Natal were administered the SRS2. The purpose of the study was to ascertain the degree of reliability of the SRS2. Of the 338 participants, 51.9 percent (n=140) of the sample were female, and 48.1 percent (n=130) were male. Sixty-six percent (n=164) of the participants spoke English as an Additional Language (EAL), while thirty-four percent (n=83) spoke English as a first language (EFL). Results indicated a significant difference in performance between EFL and EAL learners, whereby EAL learners performed lower than EFL learners. However there was no difference in performance between female learners and their male counterparts on individual items of the SRS2. The SRS2 proved to have a suitable internal consistency; however questions of bias do arise. Keywords: Psychometrics, Reliability, Standardised Assessments, Reading Comprehension 10 Chapter 1 Introduction The need for the investigation of psychometric assessments to establish their validity and reliability in the new South African context is unquestionable (Foxcroft, 1997). The current use of assessments is being questioned not only by researchers and practitioners, but by government in their policies as well (Department of Education, 1997). Furthermore the National Commission of Special Needs Education and Training (NCSNET) and National Committee on Educational Support Service (NCESS) recommended that urgent attention be given to the re-evaluation of all standardised assessments. Hence the demand for research in the area of psychological and educational assessments, in addition to, the analysis of standardised assessments used for grading learners in the South African context. Despite a change from apartheid to democratic government, the changes in educational policies have shown limited improvement within the educational system and its formulated policies have not improved the position for many South African learners. Conversely, improvement of these policies can be made with the use of valid and reliable instruments that assess learners? performance (Kame?enui et al., 2006). Assessments are useful in evaluating learners? performance, as well as tailoring programs to an individual?s needs. Accordingly there is the need and space for the development of uniquely South African assessments. Very few measures of reading comprehension are available for use in South Africa (Karlsen & Gardner, 1986; Catto, 2010). South Africa is in need of reading comprehension assessments in order to support all learners. The average learner has by Grade Eight, mastered decoding skills but struggles with reading comprehension, particularly English Additional Language (EAL) learners (Pretorious, 2000). Hence there is an imperative need for reading comprehension assessments in South Africa. An important aspect involved in interpreting scores on reading comprehension, is their comparison to a normative sample so that 11 the practitioner can understand a learner?s ability relative to their age-related peers (Wolfaardt, 2001). South African learners are therefore put at a disadvantage when they are evaluated on assessments normed on western populations. Consequently, practitioners using measures to assess South African learners need to be cautious in interpreting scores relative to western norms. As a result, research that evaluates the reliability and validity of assessment instruments would prove valuable in assisting practitioners. 1.1 Research Rationale Access to assessments standardised for the South African context is limited, causing many challenges in the field of psychometrics for many practitioners (Foxcroft & Roodt, 2001). Many of the assessments used currently, like the Suffolk Reading Scale 2, have been imported. The issue with imported scales is that often they have not been standardised for South African contexts. South Africa being a multilingual and multicultural society, imported assessments have been specialised for first language English speakers from a western culture (Broom, 2001). Many assessments have language and scholastic content that is suited only for specific types of cultural groupings of particular educational backgrounds. Due to the apartheid regime many people in South Africa live far from the western ideal of urbanized lifestyles and well established schooling systems. There are differences in educational and socio-economic backgrounds. These remain a constant challenge, particularly in the field of psychological assessment, where it is known that these factors impact on the performance of some participants. Tests that are imported could yield results that are not reliable or valid because they are not standardised for a South African context. The Professional Board for Psychology of the Health Professionals Council of South Africa (HPCSA) and the constitution requires assessments that meet the psychometric properties of reliability and validity and, above all, to be culturally and linguistically fair to all people from different backgrounds (Foxcroft & Roodt, 2001). This study therefore attempts to investigate whether the SRS2 meets technical measuring standards and is not biased by means of statistical methods. 12 In summary, this chapter explored various issues relating to the use of western standardised assessments within multicultural/multilingual contexts, specifically the importance and effects of using these assessments within a South African context. Regarding the following chapters, in Chapter Two the literature review focuses on research pertaining to the history and trends in assessments. Various biases that arise from the use of imported assessments in the South African context are also presented. Moreover, an examination of test translation and test adaptation is provided as possible solutions to multicultural/multilingual assessing. Chapter Three provides a description of the methodology of the current study. This description includes details of the participants, as well as the statistical methods employed to analyse the obtained data. Chapter Four, presents the results derived from the data analysis, while Chapter Five discusses the results as well as limitations of this study. In addition recommendations for future research are made, and these are followed by a conclusion. 13 Chapter 2 Literature Review As noted earlier, in this chapter the effects of imported assessments in multilingual/multicultural context are critically examined. The historical context of assessments internationally and in South Africa is provided. The discussion subsequently explores test translation and test adaption as a way forward. 2.1 Function of Psychological Assessments Psychological assessments are usually associated with IQ assessments by the general public. These however, represent only a small proportion of the type of assessments available. Essentially, assessments are conceived as objective and standardised measures of a sample behaviour described with scores and categories (Anastasi, 1982; Gregory, 2007). Psychological assessments are used to provide information to guide individuals, groups and organizations to understand and make relevant and suitable decisions regarding psychological functioning, job placement and personality profiles. (Foxcroft & Roodt, 2005). Furthermore, assessments assist in identifying intervention and therapy needs, measuring the effects of an intervention programme, and in gathering research data to increase psychological knowledge based on human behaviour or to inform policy making (Foxcroft & Roodt, 2005). Assessments include many different procedures that are and can be used in psychological, occupational (industrial), and educational evaluations. Specific domains of function (i.e. intellectual ability, personality, and organizational climate) are sampled by assessment measures. These samples allow for inferences to be made about normal and dysfunctional behaviour or functioning (Foxcroft & Roodt, 2005). 14 Customarily, assessments are constructed or developed by an author, who initially develops or collects a large pool of items that are assumed to measure the characteristics of interest (Weiner, Freedheim, Graham, & Naglieri, 2003). Item pools are then selected with the aid of theory. Thereafter items of suspect quality (based on rational grounds) are discarded. These items are then administered to a tryout sample. With the aid of statistical procedures, items that seem to measure unintended characteristics or more than one characteristic are identified and thereafter either discarded or modified (Alderson, Clapham & Wall, 1995).The remaining items are then presented to a large but diverse group. This group is meant to be a representative of a larger population called a standardised sample or a norming sample. Hence the sample must reflect every important characteristic of the population who will use the final version of the assessment, to which raw scores will be obtained (Weiner et al., 2003). A number of biases emerge from low proportions of minorities in the norming sample of an assessment (Sharma, 1997). This seems to be the main issue for most assessments not just recently, but since the beginning of assessing. Thus a look into its history is essential. 2.2 Origins and History of Psychological Assessments A retrospective overview of psychological assessments will help gain an understanding of the factors that have shaped psychological assessments, particularly in South Africa. Furthermore the history of psychological assessment has abundant relevance to present-day practice (DuBois, 1970) Psychology only started to grow and prosper as a science since the development of the scientific method in the early 1930?s (Foxcroft & Roodt, 2005). Assessment measures then became well established during the Italian Renaissance after the translation of Huarte?s book the Tryal of Wits in 1698. It was only then that a discipline of assessment was proposed (McReynolds, 1986). It is believed that Huarte argued that people differ from each other with regard to certain talents and that a system needs to be developed to assist in determining specific patterns of abilities of different individuals and groups, so that they can be guided into appropriate interventions (McReynolds, 1986). 15 The twentieth century saw a shift in the field of assessments, towards the measurement of individual differences and thus saw the inclusion of a wider variety of measuring instruments such as cognitive and personality measures (Foxcroft & Roodt, 2005). The latter half of the twentieth century saw promising developments in the field of assessments whereby statistical methods aided the analysis of data obtained from measures to determine their relationship between variables, as well as to unearth the underlying dimensions being tapped by a measure (Hersen, 2004). The year 1904 saw the breakthrough in the development of modern psychological assessment (Aiken, 2002). When the French minister of Public Instruction was appointed to the French Commission, he found ways to provide appropriate educational interventions to intellectually challenged individuals, so that they could be provided with appropriate educational interventions, by means of assessments. Alfred Binet a French physician was one of the members of this French Commission. Whilst there, he developed the first measure that provided a comparatively practical and reliable way of measuring intelligence (Aiken, 2002). The infamous Binet Simon Scale developed in 1905, became the benchmark for future psychological assessments (Foxcroft & Roodt, 2005; Kaplan & Saccuzzo, 2008). However, normative samples were developed using a rather small representative sample, thus lacking adequate normative data. This scale was criticised for relying heavily on the verbal skills of the assessee and in the early years this scale was available only in French and English. Thus proving inappropriate with Non-English and Non-French speakers and hence the development of non-verbal measures (Foxcroft & Roodt, 2005). The period between World War I and World War II saw an escalation of the number of psychological measures being developed (Foxcroft & Roodt, 2001). Notably there was criticism at this time that queried the appropriateness of these assessments for illiterate individuals. The criticism of intelligence scales (to present day) is that they still are too dependent on language and verbal skills, thus reducing its applicability to many individuals In response to these critiques Weschler in 1937 included in his I.Q scale performance subtests 16 that require little or no verbal responses (Foxcroft & Roodt, 2005). Therefore it can be argued that the evaluation of the applicability of assessments has been an area of concern since its inception. The latter part of the twentieth century saw the rise of culture free assessments (Anastasi & Urbina, 1997). It was soon realized that assessments cannot be free of any cultural influences. Thereafter assessment developers focused more on ?culture reduced? assessments that aim to remove as much cultural bias as possible e.g. Raven?s Progressive Matrices. However most assessments were developed in the U.S.A and U.K and tended to be more appropriate for westernized English-speaking populations (Anastasi & Urbina, 1997). Due to globalization, the 1980?s and 1990?s saw a need for more culturally approved measures and thus a shift to cross-cultural test adaptation (Foxcroft & Roodt, 2005). It seems that this shift would be in support of assessments in South Africa. However, in this millennium, test bias is still widespread and a controversy is faced by many professionals dealing with assessments. 2.3 Controversies in Psychological Assessment According to Kriegler and Skuy, (1996, p. 113), the root word of assessment was derived from the Latin word ?assidere? which translates into ?sitting beside?. This implicates that the broader definition of assessment suggests imminence to the individual or group being assessed. Assessments are usually standardised measures for particular objectives such as the measurement of an individual?s performance on a task (Anastasi, 1982). According to Gregory (2007), assessments are standardised procedures used to measure particular aspects such as behaviour; the measures are then described using categories or scores. Owen (1998) argues that not only do assessments measure behaviours but it measures inter- and intra- individual differences between the sample i.e. differences between and within individuals. 17 Foxcroft and Roodt (2001) note that: ?A measure is usually designed in a certain context (society, culture) for a specific purpose, and the normative information used to interpret test performance is limited to the characteristics of the normative sample. Consequently, the appropriateness of an assessment measure for an individual, group, or organization, from another context, culture, or society cannot be assumed without an investigation into possible test bias and without strong consideration being given to adapting and re-norming the measure? (p. 6). This quotation highlights the fact that every assessment is context-bound and, therefore, is only appropriate for use with the population from which it came. Assessment designers do not always create assessment measures that can be applicable across cultures, which brings to question the issue of test bias. 2.3.1 Test Bias Among many of the controversies in psychological assessment, test bias is the most common. Research has found that test bias occurs when an assessment is culturally and gender biased, so as to discriminate unfairly against racial and ethnic minorities, women, and the poor (Gregory, 2007). Jensen (1980) argues that bias in testing occurs when there are systematic errors in the validity of assessment scores of a participant on the basis of group membership of that participant. Gregory (2007) asserts that test bias refers to objective statistical markers that investigate the patterns of assessment scores for relevant populations. Statistically biased assessments do not take into account social values in assessment usage. In the South African context, test bias pertains particularly to the content of an assessment, whereby the constructs being measured do not translate easily across different groups and sub-populations. The ideas being measured may have subjective meanings and do not mean the same to different people (Owen, 1998). 18 Cockcroft (2002; 2008) argues that language, religion, socio-economic status, age, and gender are variables associated to an individual, which may cause discrepancies in the comprehension of an assessment?s content. In addition, culture also presents a source of bias. Because many assessment are formulated based on western cultures and expectations, levels of performance in these evaluations will vary, based on cultural groupings (Eyesneck, 1986). Language, like culture, also presents an obvious source of bias. Every language does not equally translate to the next. Thus every concept presented in an assessment can be misunderstood or misinterpreted. It is unfortunate that pioneers of these largely ignore the impact of cultural and linguistic backgrounds (Gregory, 2007). The factors mentioned above are potential sources of bias, reinforcing the issue of the need to investigate test bias in relation to validating the utility of psychological measures. Language, culture and gender biases, and how they impinge on assessments are discussed further in the South African context. 2.4 Assessment: A South African Perspective Psychological assessment was introduced in South Africa through lineage of South Africa?s colonial heritage (Mauer, 2000). The classification, possession, control, and use of psychological assessments were strictly controlled by legislation. Furthermore, psychological assessments in South Africa were developed in an environment of unequal distribution of resources, based on racial groupings and rankings. Thus psychological assessments portrayed the racially segregated society of South Africa (Foxcroft & Roodt, 2005). South Africa is only 16 years into its democratic government; it is thus still experiencing the effects of the apartheid era. The apartheid regime affected the way in which assessments were carried out. Assessments were only normed for the White population of South Africa, yet they were administered across all populations for school placement, job interviews, and other fields (Stones, 2001). 19 In South Africa, the distorted use of psychological assessments was paralleled by the increase in racially-based differential access to educational opportunities. That, in turn, led to differing career opportunities (Stones, 2001). The use of assessments within South African schools is still not seen in a positive light. Since the first democratic government in 1994, school readiness testing and other assessments were banned by many provinces because it was seen as exclusionary and perpetuating discrimination (Foxcroft & Roodt, 2001). Although psychometric assessments contribute to identifying learning problems and perhaps possible education programme interventions, there is concern that these measures, which are often imported, are not cross culturally applicable to all learners (Foxcroft & Roodt, 2001). Therefore, measures that are currently in place in South Africa need to take into account the history and culture of the social and political systems of the apartheid regime, which has had an intruding effect into the democracy of South Africa. Psychometric assessments need to be investigated, so that their reliability and validity can be established and suited for a South African context, thus reducing the possibility of bias in them. The Health Professions Council of South Africa (HPCSA) is the sole statutory body responsible for inter alia classifying, registering and receiving of psychometric tests, questionnaires, instruments used for the determination of intellectual ability, aptitude and psycho-pathological functioning (Foxcroft, Patterson, Le Roux & Herbst, 2004). The Human Sciences Research Council (HSRC) has developed many of the assessments used today. The control and regulation of assessments developed by the HSRC is the responsibility of the HPCSA (Foxcroft et al., 2004). In earlier years individuals were not protected from discrimination that arises when assessing (Paterson & Uys, 2005). However with the new Constitution and the Labour Act of 1996, worker unions and individuals now have the support of legislation that disregards and forbids any form of discrimination in the workplace. This has been ensured within the assessment arena, by means of the Employment Equity Act No.55 of 1998 (Section 8) which refers to psychological testing and states that: 20 ?Psychological testing and other similar forms of assessment of an employee are prohibited unless the test or assessment is being used: a) Has been scientifically shown to be valid and reliable; b) Can be applied fairly to all employees; c) Is not biased against any employee or group.? (Government Gazette, 1998, 16). This act has many implications for those conducting assessments in South Africa because many of the measures that have been imported or developed locally have not been investigated for bias or been validated cross-culturally. Foxcroft, et al. (2004) conducted a study on issues regarding assessment in South Africa with a sample of 881 participants that included all registered: psychologists, counselors, psychometrists, assessment designers, psychology students, and interns. They were questioned about the needs of a practitioner conducting educational and psychological assessments. Results showed that for ?cross-culturally appropriate tests?, ?more training with regards to the adaptation, application, and interpretation of tests for cross-cultural purposes? were among the main necessities (Foxcroft et al., 2004, p. 36). As stated previously, one of the main issues regarding assessments in South Africa is their lack of applicability cross-culturally due to biases, of language, culture, and gender. 2.4.1 Language Bias A common critique for most assessments is that they usually are not language fair. In the South African context, language is one of the most important factors impacting on an examinees performance on an assessment (Nell, 1994). The Sapir-Whorf Hypothesis (Sternberg, 1996), or the linguistic relativity hypothesis states that people will develop and have an affinity to different cognitive styles and interpretations, based on the limitations imposed on them by the language they use to communicate 21 (Sternberg, 1996). Hence individuals taking such a test will be mentally restricted to the existing models procedures and contexts made permissible by their language. Linguistic relativity further explains that the understanding of assessment items become potentially unique to the language of an assesee (Sternberg, 1996). In South Africa, where there are eleven official languages, one can only imagine the constrictions imposed on those who do not have a command of the English language. Nine of these languages are considered to be African (Broom, 2001). Seventy-four percent of the population speaks an African language as their first language and just nine percent have English as their first language. Nevertheless Afrikaans and English have been given priority over other languages (Martin, 1997). During the apartheid regime, black learners were by law deprived of an education from an Afrikaans or English education system (Martin, 1997). Thereafter the Bantu Education Act of 1953 (No. 47) stated that all academic subjects be taught in Afrikaans and non-academic subjects being taught in the learner?s first language (Martin, 1997). Language policies aforementioned led to the Soweto Uprising in 1976, at which learners protested against educational oppression (Broom, 2001). These educational policies have now created scenarios whereby learners choose to learn in either their second or third language, therefore being taught in English as an additional language. Despite these issues many parents desire their children to be taught in English from the outset, due to the high status of English both locally and internationally (Martin, 1997). The Suffolk Reading Scale 2 (SRS2) was developed in the U.K; thus learners assessed by this instrument might be subject to linguistic bias if used in the South African multilingual context. It is obvious that the ability to read requires that an individual has language proficiency in the language of the text. Gregory (2007) argues that monolingual assessments do not cater for a bilingual test takers. As a result the bilingual assessment taker or the EAL assessment taker is at a disadvantage. This reiterates the fact that language barriers play a role in determining whether or not an assessment is appropriate for EAL learners (Foxcroft et al., 2004). 22 Cummins? (1991) studied bilingual speakers, and looked at the relationship between bilingualism and cognitive development. He argued that submerging learners in a second language at the expense of the first language does not necessarily promote second language proficiency (Cummins & Mulcahy, 1978) Cummins? (1981) theory on reading proposes the idea that there are two types of language proficiency namely, Basic Interpersonal Communication Skills (BICS), which are skills of listening and speaking that, are easily acquired from language backgrounds similar to English; and Cognitive Academic Language Proficiency (CALP) the ability to cope with academic demands in language. Non-native speakers of English, who may be fluent in day to day spoken English, do not always, have the academic language proficiency. Cummins (1998) goes on to argue that groups of bilingual and culturally diverse people experience underachievement due to issues of status and power associated with the English language. Cummins (1991) reported that psychologists neglect the fact of the difference between these two types of proficiency when assessing bilingual learners/ EAL learners. Although South Africa is a multilingual society, the language of learning and teaching in most schools is now English (Rooyen & Jordaan, 2009). Therefore one can question whether learners do possess the skills of BICS and CALP required to comprehend texts and work in the school setting. Although these learners appear to be fluent in English they may not have mastered the CALP skills. The use of an assessment would be the best indicator to demonstrate if school learners possess these skills. The SRS2 is a scale that assesses reading comprehension, but needs adaption in the South African context. As mentioned before, in South Africa particularly, factors that cause a barrier to assessments are cultural differences and language barriers. Furthermore a great variation exists between schools in South Africa. Schools that were previously designated for White learners (Model-C schools), whom have an educational advantage, are very different from previously disadvantaged schools under the Apartheid system (Staden & Howie, 2006). Consequently if an imported assessment, not standardised for the South African 23 population is used in South African schools, children from previously disadvantaged backgrounds may perform inadequately. As a result a high emphasis should be placed on the re-evaluation of the appropriateness of traditionally used psychometric assessment. Psychologists should take into account the issues of language, particularly the relationship between bilingualism and academic achievement and the difference between BICS and CALP. These issues impact directly on assessments, especially when based on English language. As with language bias, culture bias, impinges upon the fairness of psychological assessment. 2.4.2 Culture Bias The South African population consists predominantly of individuals of African descent. Nevertheless the enterprise of assessments was tailored by and for the white population of South Africa. Hence the suitability of existing assessments for the assessment of diverse populations should not be taken for granted (Gregory, 2007). This raises important questions particularly when assessment results are used to decide placement in jobs or more importantly studying in educational institutions. The assessment of persons with disparate cultural backgrounds existed since the midcentury (Anastasi, 1982). Regrettably, the pioneers of these assessments largely ignored the impact of cultural backgrounds on the assessment process. For this reason, Gregory (2007) contends that practitioners need to acknowledge the cultural background of examinees and take into account how it may impact on the entire assessment process. Traditionally, cross-cultural assessments would attempt to rule out parameters along which cultures vary. For example speed (motor speed on assessments) was seen as culture parameter. Some cultures do not attach value to rapid performance and hence have a slower tempo of daily life (Anastasi, 1982). This plays an important role with subtests that have a speed component to them where examinees may find it difficult to perform optimally. 24 While acknowledging the influences of cross cultural assessment, it is also important not to be stereotypical and over generalize across cultures (Gregory, 2007). Cultures are unique, practitioners cannot assume that the Sesotho culture is the same as the isiZulu culture and hence the same principles apply. No two cultures are the same. Moore (1986) conducted a study on family socialization and performance on IQ assessments, across traditions and cultures. Her study revealed the relevance of cultural backgrounds for understanding assessment performance. Most importantly her study showed that there are culturally based differences in response style on an assessment. This response style could mask the underlying competence of examinees. In South Africa, there are many assessments imported from Western countries (Foxcroft et al., 2004). These assessments are not suitable for the South African population given our diverse cultural groups. Scores would either be significantly lower or higher due to various factors. Using these imported assessments affects the validity and reliability of these assessments because they have not been standardised for the South African context. Conversely, while many researchers and practitioners acknowledge the cultural bias of imported assessments, Shuttleworth- Jordan (1996) argues that we should not abandon all assessments standardised on western populations and attempt to develop new culturally relevant assessments. She states that there are ?erroneous exaggeration of cultural effects, which fails to take into account the acculturation process? (p.96). In South Africa people are at different stages of urbanization, westernization, and literacy. Consequently U.K pioneered scales like the Weschler range, could perhaps be applicable to certain populations in South Africa. For instance individuals who attend former Model-C schools, are more likely to be assessed on an assessment battery that includes the Weschler scales, as they have been given or have rich cultural opportunities that include access to libraries, television and newspapers, as compared to those who have attend schools located in disadvantaged communities where very little cultural opportunities of learning exist. 25 2.4.3 Gender bias Differences in performance on an assessment are influenced by more than just language and culture. Research shows that gender differences also influence assessment performance (Jensen, 1980; Neisser et al., 1996). As a result there is the possibility of gender bias. Findings of research based on gender and assessment performance indicate that on certain tasks there were large and consistent performance differences that existed between genders. Differences that exist have been attributed to factors such as socialization of girls and boys, confidence levels, intrinsic ways that males and females acquire and process information (Neisser et al., 1996, Pallier, 2003). For example females have a distinct advantage when performing verbal tasks (e.g. tasks that assess verbal fluency and synonym generation) while males do better on performance tasks (e.g. visuo-spatial tasks), and mathematical reasoning. Differences in gender extend beyond physical and biological characteristics. There could perhaps be reading preferences between boys and girls, whereby girls could engage more in activities that involve reading, or reading for enjoyment and are thus better at reading, as opposed to their male counterparts. Chiu and McBride (2006) conducted a study to assess gender and reading comprehension. The sample consisted of fifteen year olds from 43 different countries, in which girls prevailed in reading comprehension in every country. Girls were found to be more adequate readers than their male counterparts. Chiu and McBride (2006) argue that when looking at gender differences in reading comprehension, the following should be taken into account: influences of peer culture of reading for enjoyment; number of books available to read at the household of each child; and the socio-economic status of the school the child attends. Rutter et al. (2004) aimed to generate new empirical data through four epidemiological studies on the nature and extent of gender differences in reading difficulties. Summaries of previous research findings show that such difficulties are more prevalent in males than in females. The findings of this U.K-U.S.A study are aligned with previous research findings and hence reading difficulties are more prevalent in males. Taking these findings into account, the possibility exists that gender could influence scores on the SRS2 depending 26 on the gender of the participant, whereby females could perform better than their male counterparts. Neale, (1989) argues that bias could be incorporated by means of test content in terms of relevance and interest to no specific sex. 2.5 Assessing Reading Comprehension: Suffolk Reading Scale ?Reading is a process that requires thought. It is one activity, through which the child?s cognitive development can be furthered? (Pumfrey, 1977, p.2). Furthermore reading can be defined as ?the ability to comprehend the thoughts and feelings of another mind via the medium of text? (Pumfrey & Elliot, 1990) Reading is a multifaceted skill involving letter-word identification, literacy and comprehension skills. These skills are acquired during childhood and form part of the developmental process (Pumfrey, 1977). Pretorius and Naud? (2002) investigated the preparedness of South African children before entering formal education. This study portrayed the poor reading ability of South African learners aged five to seven years living in informal settlements. Pretorius and Naud? (2002) contend that these children exhibited inadequate literacy skills, poor sentence construction, poor sentence syntax, and inadequate sound development, and knowledge of the alphabet. Therefore with regards to the sample chosen for this study it is expected that learners may struggle with reading items of the SRS2. Orasanu (1986) argues that comprehension is the purpose of reading. Comprehension refers to the notion of understanding a simple phrase that comprises of letters, words, and sentences, which encompass a meaning (Orasanu, 1986). In order to understand the meaning of a particular text, one needs to have an established recognition of familiar words in titles, headings, and phrases. Recognition of these words are influenced by ?our expectations that certain words will occur, based on knowledge on our knowledge of language, communication, and what we have already read? (Orasanu, 1986, p.2). The decoding of words 27 makes comprehension attainable. A learner will create a meaning based on the text, as well as his or her knowledge about the content, language and structure of the text (Orasanu, 1986). Reading comprehension refers to the ability of reading with understanding (Blachowicz & Ogle, 2008). Armbruster, Lehr and Osborn (2001) argue that the reading process involves phonemic awareness (ability to notice and work with individual speech sounds in words), phonics (relationship between letters of a written language and sounds of a spoken language), vocabulary (words we need to know for effective communication), fluency (ability to read text accurately and quickly), and comprehension. These processes cater for effective proficiency in these skills which improves one?s comprehension of texts. Comprehension is the rationale behind reading (Armbruster, Lehr & Osborn, 2001). An assortment of basic language and cognitive skills is imperative for reading comprehension. Familiarity of word meanings could be related to reading comprehension. However restricted vocabulary does not necessarily limit comprehension (Cain & Oakhill, 2006). Factors that do not inhibit reading comprehension are texts that may be linguistically biased. All texts are in a particular language, be it English, German, or Spanish. As a result, if the reader does not have a good command of the language of the assessment, these language barriers could impact on how the reader comprehends the assessment he/she is reading. Alongside language barriers, difficulties in reading can also hinder reading comprehension. There are various ways of assessing reading comprehension. The SRS2 is one such scale that assesses reading comprehension. The SRS2 consists of eighty-six unsystematic multiple choice questions with no consistent theme. This format is known as a cloze task. Cloze tasks consist of sentences where a single word has been omitted and a replacement word has to be selected from a choice of three to five options (Cain & Oakhill, 2006). The advantage of cloze tasks is that they can be administered to groups of children. The SRS2 is highly dependent on an individual?s word reading skill and as a result, results would have to be interpreted accordingly (Cain & Oakhill, 2006). However one should take into account that although scores on the SRS2 are dependent on word reading skills, there are other crucial factors that can influence scores on 28 this scale, namely language, culture, and gender. Thus as aforementioned, language, cultural and gender biases could influence scores on the SRS2. 2.6 The translation and adaptation of assessment The unique challenges of cross-cultural assessments have become a sensitive issue due to specific concerns regarding the use of standardised assessments cross-culturally (Chang, 2001). During the 1990?s a substantial amount of time was given to the translation and adaptation of assessments and devices in the field of psychometrics (Weiner, Freedheim, Graham, & Naglieri, 2003). However there were problems with the test translation and adaptation in cross-national assessment. The International Testing Commission then developed guidelines over a period of time for assessment (Weiner, et al., 2003). This was seen as a major accomplishment for assessment and cross-cultural psychology. Despite this achievement there is still the misconception that test translation and test adaptation refers to the same process. Test translation can be defined as translating an assessment from one language to another without changing the meaning of the original assessment itself. Test adaptation is a slightly more complicated process. Test adaptation involves making an assessment applicable to the contexts it will be used in ?while using the same language? (Kanjee, 2001, p. 87), hence retaining the meaning of the assessment. Hambleton (1996) is in favour of test adaptation, because he argues test translation could over simplify the assessment and as a result compromise the validity of the assessment. Whereas with the use of test adaptation, only a few words, examples, and contexts would change in order to make the assessment more applicable for a different context in which it may have been standardised (Kanjee, 2001). In spite of this, poor test adaptation could possibly lead to a significant change in scores (Hambleton, 1996). Some constructs do not have meaning to the context it?s being applied, particularly in terms of different cultural contexts. For example, the SRS2 has constructs relating to the seasons, however from a Northern 29 Hemispheric perspective whereby winter comes in December and summer in July; for those living in the Southern Hemisphere it is a dissimilar scenario. As explained above test adaptation has its advantages and disadvantages. A particular advantage of it particularly for the South African context is that test adaptation conserves time and expenses. Another advantage is that the researcher can compare pre-existing data with newly acquired data and as a result allow for international and national cross-cultural studies (Chang, 2001). Many researchers argue that the advantage of test translation is that it leads to increase fairness in assessing, because individuals can be assessed in the language of their choice (Chang, 2001). Shuttleworth- Jordan (1996) reviewed Hemp?s memory assessment translated in isiXhosa for isiXhosa speaking children they found that some of the words were unfamiliar for Eastern Cape isiXhosa-speaking children living in an urban area. Despite the translation being informed by clinical observation, Shuttleworth-Jordan (1996) argues that the English version of this assessment is more relevant to the rural isiXhosa speaking population. In summary test translation and test adaptation may be the step forward towards fair assessment in the South African context; however test adaptation should take into account the context and value of not just the South African population as a whole, but each of its diverse cultures and contexts. One may argue that this is impossible, seeing that different cultures are at varying degrees of globalization or westernization. However these efforts need to be made so that assessments can be applicable to the general South African population. 2.7 Research Questions 1. Is there a high degree of reliability of the SRS2 when used in the South African context for primary school learners? 2. Is there a statistically significant difference between English Additional Language (EAL) and English First Language (EFL) learners on the SRS2? 3. Is there a statistically significant difference between male and female learners on the SRS2? 30 4. Is there a statistically significant difference between EAL and EFL learners on individual items of the SRS2? 31 Chapter 3 Methods 3.1 Chapter Overview Research design can be defined as a central component of scientific inquiry in order to obtain empirical data (Heppner, Kivlighan, & Wampold, 2008). A weak research design increases the chances of bias, distortion and random error to occur, hence the research design and method is the most fundamental aspect of a study. This chapter commences with an overview of the context of this study and the research design. Subsequently the research hypotheses are stated, followed by the research method (discussing participants sampled in the study, description of questionnaires as well the data analyses utilized). This chapter concludes with the ethical considerations and procedures followed in this study. 3.2 Context of the Study The current research formed part of a larger study, The Road and Aircraft Noise Exposure on Children?s Cognition and Health (RANCH-SA), which is broadly concerned with the effects of noise pollution on information processing. The RANCH-SA study aims to assess the effect of aircraft and road traffic noise on the way information is processed, retained, and recalled. Attention, memory and reading are factors involved in cognitive development at a primary schooling age of between 5-11 years (Stansfeld et al., 2005). The current research focuses only on reading and assesses whether or not the instrument used to measure reading comprehension is a reliable measure for South African primary school learners in Kwa-Zulu Natal. Reading comprehension was assessed by means of the Suffolk Reading Scale compiled by Fred Hagley in 1987. 32 3.3 Research Design This study employed a quantitative research paradigm, utilising a cross- sectional survey design. Quantitative research is concerned with explaining phenomena by collecting numerical data, analysed by means of statistical methods (Mujis, 2004). Within the quantitative paradigm there are two types of statistical techniques employed, namely descriptive and inferential statistics (Dyer, 1995). Descriptive statistics allows for the description and analysis of a given group without drawing any conclusions or inferences about a larger group. The goal therefore of descriptive statistics, is to summarise data into useful information in order to determine the general trend of the data. Inferential statistics aids in reaching conclusions, whereby it allows for the testing of hypotheses that are not candidly available. Consequently an inference is derived from the data set (Dyer, 1995). This study used descriptive statistics to describe the sample in terms of gender, race and language, whilst inferential statistics were used in testing the hypotheses of this study. Survey Design or survey research refers to the administration of questionnaires to participants to research a specific variable, in this case reading comprehension (Goodwin, 2008). To achieve this objective a single set of cross-sectional measurements was employed. The typical weakness of non experimental research, i.e. the lack of ability to establish causation (Dyer, 2006), does not have profound implications in this specific study, which aims to determine whether bias could be established empirically on the basis of SRS2 performance. 3.4 Hypotheses Hypotheses deal with claims or statements about some characteristic of a population by looking at expected results to be obtained from a research inquiry (Dyer, 1995; Kozak, Kozak, Staudhammer & Watts, 2007). Hypotheses tested in this research are: 33 (Ho1): The SRS2 is not a reliable instrument in the South African context for primary school learners. (Ha1): The SRS2 is a reliable instrument in the South African context for primary school learners. (Ho2): There is no difference in performance between EFL and EAL learners on the SRS2. (Ha2): There is a difference in performance between EFL and EAL learners on the SRS2. (Ho3): There is no difference in performance between male and female learners on the SRS2. (Ha3): There is a difference in performance between male and female learners on the SRS2. (Ho4): There is no difference in performance between EFL and EAL learners on individual items of the SRS2. (Ha4): There is a difference in performance between EFL and EAL learners on individual items of the SRS 3.5 Participants The sample consisted of 338 primary school aged (9-11) children with a mean age of 10.6. Of the 338 participants, 51.9 percent (n=140) of the samples were female, and 48.1 percent (n=130) were male. Of the four racial groups, 172 (68%) were Black, 23 (9.1%) Coloured, 57 (22.5%) Asian, and 1 (0.4%) White. The participants were from primary public schools in the Kwa-Zulu Natal province. With regards to sampling, the entire population may have little or no chance of selections as usually a more convenient, and/or prominent sample is chosen. Such sampling is termed non-probability sampling (Foreman, 1991). This sampling method is considered biased as there is a risk of selecting a biased sample. In spite of this disadvantage, the advantage of non-probability sampling is that it removes the requirement of having to specify the entire population, as in random sampling methods (Dyer, 1995). Non-probability sampling was therefore used, as there was no random selection of the sample. 34 Within non-probability sampling there are various forms of sampling methods, the most common being purposive sampling, as this method enables the researcher to select specific participants who will provide the most extensive information about the phenomenon being measured (Burns & Grove, 2005). For purposive sampling, participants are selected on the basis of their representation of certain characteristics that are considered relevant to the study. Although this method does not make use of a random selection process, it allows for the attainment of a sample that represents the characteristics of a target population rather than just the larger population (Stommel & Wills, 2004). Therefore the use of non-probability sampling, and purposive sampling in particular was employed in this study, as the required participants were primary school learners, attending schools within the vicinity of airports. In order to reduce bias utilising a non-probability sampling technique, two strategies were employed (Grarvetter & Forzano, 2008). Firstly, an attempt was made to obtain a representative sample. Thereafter, the description of the sample was intended to be clear and unambiguous. The researcher ensured that questionnaires were distributed to learners? representative of the larger population. Table 3.1 presents the gender distribution of participants in this study. Table 3.1 Gender Distribution of Participants Gender N % Male 130 48.1 Female 140 51.9 Missing Data 68 20.1 The participants in this study consisted of 51.9 percent (n=140) females, and 48.1 percent (n=130) males. Table 3.2 summarises the racial distribution of the participants in this study. 35 Table 3.2 Racial Group Distribution of Participants Race N % Black 172 68 Coloured 23 9.1 Asian 57 22.5 White 1 0.4 Missing Data 85 25.1 Of the four racial groups, 68 percent (n=172) were Black, 9.1 percent (n=23) Coloured, 22.5 percent (n=57) Asian, and 0.4 percent (n=1) White. Table 3.3 summarises the language distribution of the participants in this study. Table 3.3 Language distribution of participants Language N % English First Language (EFL) 83 66 English Additional Language (EAL) 164 34 Missing Data 91 26.9 36 In order to explore the reliability of the SRS2, participants from different language backgrounds were included in the sample. Approximately 66 percent (n=164) of the participants spoke English as an additional language, while 34 percent (n=83) spoke English as a first language. 3.6 Instruments One psychometric instrument was employed namely the Suffolk Reading Scale (2) (Hagley, 2002). A biographical questionnaire was also included to obtain the necessary biographical data for the study. In this section both the questionnaire and scale are discussed. 3.6.1 Biographical Questionnaire The biographical questionnaire required information pertaining to the participants? age, gender, ethnic group, grade, language, and age at which English was introduced. This information allowed for the selection of a representative sample, as it was possible to ascertain whether the sample was biased or unbiased in favour of certain groups of learners (See Appendix: A). 3.6.2 Suffolk Reading Scale The Suffolk Reading Comprehension Scale (2) is a standardised (United Kingdom) reading comprehension test that consists of 86 multiple choice sentence completion questions, each question having five potential answers (Oakhill, 1997). The multiple choice sentence completion formats allows for wide- scale measurement of reading progress and group administration (Hagley, 2002). This scale has three comparable levels according to an age group in the primary school age range. Each level has two parallel forms for easy testing arrangements. A study that looked at the psychological assessment of reading was conducted in the U.K with 38 625 children to yield reliability and validity estimates. The test- retest parallel form and internal consistency estimates were high (r = 0.88) indicating the scale is stable from one test to another (internal consistency). The teacher?s estimate of children?s reading 37 ability was used to assess the validity of the test in the primary range. The correlations between teachers? estimates (r = 0.85) and performance of the Suffolk Reading Scale were high (Oakhill, 1997) (See Appendix D). 3.7 Threats to Validity Within the administration process, the SRS2 was one of the latter tests to be administered (according to the RANCH-SA protocol). As a result boredom and fatigue could have been encountered by participants during testing. Consequently questionnaires were not completed resulting in a negative scoring effect (i.e. lower scores on the scale). However it should be noted that Cain and Oakhill (2006) argue that cloze tasks do tend to get progressively difficult. Hence this could be a plausible reason for incomplete scripts. 3.8 Procedure and Ethical Considerations Ethical clearance has been granted (Protocol number: 2008ECE94). Ethical procedures adhered to were as follows: The proposal for this study was submitted to the Faculty of Humanities Post-Graduate Committee and the Department of Education (Kwa-Zulu Natal division). The ethical considerations were approved by the respective committees. Thereafter (prior to the research being conducted at the schools) the headmasters of the respective schools received letters, entailing details of the study and requisition of the headmasters? consent to perform research at their respective schools. As the participants were all minors, parental consent had to be acquired by means of a consent form (See Appendix B). The researcher ensured that only learners with a signed consent form were allowed to partake in the study. Learners were invited to partake in the study by means of a letter which also requested their assent (See Appendix C). The questionnaires used in this research were administered during normal school times, as per permission from the headmaster and educators. The administration was performed by the researcher under supervision by a registered psychologist. The participants were informed about the nature of 38 the study, as well as that participation was voluntary. Questions were asked about a participant?s age, race and grade, his/her name for coding and demographic purposes. While anonymity could not be ensured as personally identifying information was asked, confidentiality was strictly upheld. Nonetheless, the participants were promised that anonymity would be maintained in the reporting of the results. The participants were informed that the research results would be made available in electronic format at the University of Witwatersrand library. 3.9 Data Analysis Based on the research problem, the general purpose of the study was to determine the applicability of the SRS2 on South African primary school learners in Kwa-Zulu Natal. The data analyses in this study were performed by using the Statistical Package for Social Sciences (SPSS). The data set being quantitative in nature was evaluated using descriptive statistics, reliability coefficients, independent sample t-tests and chi- square tests. Since all the variables were normally distributed and the conditions of homogeneity of variance were fulfilled, parametric analyses were conducted. Descriptive (mean scores and standard deviations) and inferential statistics were utilised in the analysis of the data set. The frequencies for the variables under study, namely gender, race, language were computed. The reliability of the SRS2 was ascertained by means of a Cronbach?s alpha coefficient. Used as a measure of internal consistency reliability, Cronbach?s alpha is able to ascertain the reliability of measures that have multiple items that are not dichotomous (Urbina, 2004). Internal consistency reliability is calculated from the number of items in the test and the average inter-item correlations. Consequently, the reliability co-efficient of the SRS2 can be interpreted as the proportion of variance in the observed test scores, by participants as accounted for by the covariance in the true scores. The reliability coefficient was reviewed at a confidence level of ninety-five percent. The Independent Sample T-test is a statistical technique that was used to determine the significant difference between home language (EAL and EFL) and gender (male and female) on the dependent variable, 39 reading comprehension. Thereafter, homogeneity of variance was established using Levene?s test for equality of variance, and this parametric assumption was met. The Yates Chi-Square test (Yates? continuity of correction) is a statistical technique used when testing for independence in a contingency table (Yates, 1934). The difference in performance on individual items of SRS2 by EAL and EFL learners was observed using Yates continuity of correction in a 2x2 contingency table. It should be noted that the use and effect of Yates? correction is to prevent overestimation of statistical significance of small data (Yates, 1934). 40 Chapter 4 Results 4.1 Overview of Chapter The aim of this study was to investigate the psychometric properties of the SRS2 in primary school learners in the South African context. The psychometric properties of the SRS2 were determined by Cronbach?s alpha. The results of the reliability are presented followed by reporting results of the hypotheses postulated in Chapter two. 4.2 The psychometric properties of the SRS2 This study aimed to determine the psychometric properties of the SRS2, as this instrument has not been previously used in the South African context (see section 1.7). The results of the reliability analysis are presented and followed by an interpretation of the independent sample t-test and chi-square test of independence. The means and standard deviations of the SRS2 were determined. The descriptive statistics for the SRS2 are presented in table 4.1. The highest score on the SRS2 was 70 (M=43.21; SD=13.85), while the lowest score was 0 (M=43.21; SD=13.85). Table 4.1 Descriptive statistics for the SRS2 Mean SD Minimum Maximum N of items SRS2 35.27 11.299 0 70 86 41 4.3 The reliability coefficient The soundness of a psychometric measure is in part, based on its ability to produce similar results under differing conditions - reliability (Martin, Volkmar & Lewis, 2007). According to Thorndike (2005) this can be expressed in one of two ways namely ?standard error of measurement? i.e. the maintained ranking position of every examinee, and reliability coefficient (Thorndike, 2005, p. 112). Alpha values equating to, or greater than .70 indicate good reliability (Nunnnaly & Bernstein, 1994). The Cronbach alpha coefficient for the SRS2 was 0.93. Table 4.2 Cronbach coefficient for the SRS2 Cronbach's Alpha Cronbach's Alpha Based on Standardised Items Number of Items 0.932 0.935 86 4.4 Comparison between EFL and EAL participants on the SRS2 As previously stated an independent-samples t-test was conducted to compare SRS2 scores for EAL and EFL participants. There was a statistically significant difference in scores between EFL participants (M= 41.42, SD=12.109) and EAL participants (M=33.97, SD=9.8); t (240) =5.116, p<0.04. The magnitude of the differences in the means (mean difference=7.445, 95% CI:-1.45-4.57) was moderate (eta squared=0.09). 42 Table 4.3 t-test results for language EFL EAL Mean 41.42 33.97 SD 12.1 9.8 Observations 240 240 df 239 t Stat 5.11 4.5 Comparison between male and female participants on the SRS2 An independent-samples t-test was also conducted to compare SRS scores for males and females. There was no significant difference in scores between males (M=34.92, SD=11.69) and females (M=36.46, SD=11.06); t (263) = -1.1, p>0.26. The magnitude of the differences in the means (mean difference = -1.54, 95% CI: 1.2-1.3) was very small (eta squared= 0.004). Therefore the null hypothesis is not rejected, thus there is no difference in performance between male and female participants. Table 4.4 t-test results for gender Males Females Mean 34.92 36.46 SD 11.69 11.06 Observations 263 263 df 262 t Stat -1.1 43 4.6 Comparison between EFL and EAL participants on individual items of the SRS2. The fourth research question required that the individual items of the SRS2 be analysed for the purpose of determining differences in each group?s performance for each item. The results of the Yates? correction for continuity (Yates? chi-square test) indicated that the EAL subjects performed significantly lower than the EFL participants on eighty percent of the items on the SRS2. At the ? = 0.05 level of significance, table 3.3 indicates that for items 6, 22, 23, 25, 27, 28, 30, 31, 32, 33, 34, 35, 36, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 52, 54, 61, 62, 64, and 66, there is a significant difference in performance between language proficiency and performance on the SRS2. Table: 4.3 Chi Square Test: Significant differences in performance on individual items of the SRS2 Item # df N ?2 p Effect Size SRS6 1 238 8.39 0.02 -0.163 SRS22 1 239 7.17 0.007 -0.183 SRS23 1 238 7.78 0.005 -0.19 SRS25 1 239 5.34 0.021 -0.16 SRS27 1 234 15.41 0.000 -0.266 SRS28 1 236 8.7 0.003 -0.202 44 SRS30 1 233 3.97 0.046 -0.14 SRS31 1 237 10.28 0.001 -0.217 SRS32 1 232 8.73 0.003 -0.203 SRS33 1 228 4.001 0.045 -0.142 SRS34 1 234 10.57 0.001 -0.222 SRS35 1 225 10.68 0.001 -0.229 SRS36 1 224 15.75 0.000 -0.275 SRS38 1 228 6.37 0.012 -0.177 SRS39 1 226 13.54 0.000 -0.254 SRS40 1 227 4.33 0.037 -0.148 SRS42 1 217 7.52 0.006 -195 SRS43 1 206 8.28 0.004 -2.11 SRS45 1 206 13.41 0.000 -0.265 SRS47 1 198 4.49 0.034 -0.161 SRS48 1 192 6.89 0.009 -0.201 SRS49 1 189 9.23 0.002 -0.234 45 SRS52 1 168 11.37 0.001 -0.273 SRS54 1 168 7.46 0.006 -0.226 SRS61 1 133 4.01 0.045 -0.195 SRS62 1 130 5.38 0.02 -0.222 SRS64 1 120 5.91 0.015 -0.25 SRS66 1 114 5.52 0.019 -0.246 SRS67 1 113 5.78 0.016 -0.248 46 Chapter 5 Discussion and Conclusion 5.1 Introduction The aim of this study was to explore the reliability of the SRS2 and to determine its applicability to South African primary school learners. This chapter discusses the implications of the results presented in chapter three. The findings are discussed in the context of the literature review. Thereafter the limitations and research implications of the study are presented and directions for future research are subsequently reviewed. This is followed by the conclusion. 5.2 Reliability of the SRS2 Hypothesis one was concerned with the psychometric properties of the SRS2 in the South African context in relation to the reliability of this instrument when used in the UK context for RANCH-UK . The psychometric properties of this scale have not been explored before in South Africa. The Cronbach alpha coefficient for the SRS2 in this study was 0.93. As such, the Cronbach alpha coefficient was considered satisfactory. Nunally (1978) argued that an alpha above 0.7 indicates that the test is a reliable measure for the group on which it was measured. Therefore the SRS2 was found to be a generally reliable instrument, used with primary school learners in the Kwa-Zulu Natal province in South Africa. This finding is consistent with findings of the SRS2 in U.K. in a study by Oakhill (1997), which revealed correlation estimates of above 0.7 (r =0.85) 5.3. Performance between language groups on the SRS2 Hypothesis two was concerned with the difference in performance between EFL and EAL learners on the SRS2. The results revealed a significant difference between the performance of EFL and EAL learners in favour of the EFL learners. This confirms the first hypothesis, which postulates that EAL learners would 47 perform significantly below EFL learners on the SRS2. These results support those of Lathy (2006), who found that EAL learners are faced with barriers to learning and thus perform at a lower level as opposed to EFL learners. Furthermore the literature review exemplifies the assumption that the performance of EAL learners are expected to have a lower performance rates on English Language proficiency assessments as a result of their language development crisis and English being the language of learning and teaching. These findings also support Cummins? (1991) argument that subtractive bilingualism is a contributing factor to lower performance on assessments, whereby EAL learners will perform significantly below their EFL counterparts. These findings suggest that as with the South African population, many children who speak English as an additional language have a tendency to underachieve in English literacy, especially in their primary years (Broom, 2001). These difficulties are often attributed to, as well as exacerbated by, low levels of English language fluency as they enter the education system (Hutchinson, Whiteley, Smith, & Connors, 2003). When entering the South African schooling system, EAL learners are not formally taught English. Their ability to be proficient in English is dependent on previous exposure to the language. EAL learners in this study may have performed lower than their EFL counterparts, because English is not their first language thus their understanding of English is limited as they were only exposed to it at school. The SRS2 being English based may have limited the ability of EAL learners to perform well on the SRS. Thus the SRS2 being English based, EAL learners are bound to perform poorer than their EFL counterparts. 5.4 Performance between genders on the SRS2 Hypothesis three was concerned with testing whether there is a significant difference in performance between male and female learners on the SRS2. The results demonstrated that there was no significant difference between the performance of male and female learners on the SRS2 (p>.005). These findings were inconsistent with those of Chiu and McBride (2006). They found that a gendered interest in reading was one 48 of the variables that accounted for gender differences in reading comprehension, i.e. girls have a liking for activities involving reading and thus perform better on reading tasks. Furthermore they found that girls were more adequate readers than their male counterparts. Females have a distinct advantage when performing verbal tasks (e.g. tasks that assess verbal fluency and synonym generation) while males do better on performance tasks. Research has shown that reading difficulties are more common in males than females (Rutter, et al., 2004). The findings of Rutter, et al. (2004) are not aligned with this research finding and hence it cannot be assumed if reading difficulties are more prevalent in boys than girls. This contests the second hypothesis, which postulates that there is a difference in performance between male and female learners on the SRS2. Whilst interest in reading may be gender bound, the participants of this study may have had limited exposure to reading materials as they live in disadvantaged communities with minimum resources. Consequently this may have limited the ability of both the males and females to improve their reading comprehension skills and subsequently perform well on the SRS2. 5.5 Performance between language groups on individual items of the SRS2 Hypotheses four was concerned with the difference in performance between EFL and EAL learners on individual items of the SRS2. EAL subjects performed significantly lower than the EFL participants on 69 out of a total of 86 items. Furthermore, at the ? = 0.05 level of significance, results indicated that there was a significant difference in performance between language proficiency and performance on the SRS2, whereby EFL learners surpassed the performance of EAL learners On close examination of these items, it was indicated that from item 30 onwards, a wider knowledge of English literacy is required. Research shows that reading comprehension is reliant on an assortment of basic language and cognitive skills. Familiarity of word meanings could be related to reading comprehension 49 (Cain & Oakhill, 2006). EAL learners may not be familiar with the words used from item thirty onwards as the words become increasingly difficult. Reading comprehension is affected or limited by texts that may be linguistically biased. As a result, an EAL learner may not have a good command of the English language used in the SRS2. These language barriers such as a lack of command of the English language or proficiency in English could impact on how the EAL learner comprehends the tests he/she is reading. As discussed in the literature review, Cummins (1991) argument of subtractive bilingualism explains the lowered performance of the EAL learners. These EAL learners are being taught in their second if not third or fourth language. Their English skills are possibly at a minimal level thus their understanding of an assessment standardised in the UK that is English based, is compromised by their limited command of the English language. It must be noted that the structure of the SRS2 is questionable, as the items raises issues of cultural bias within the South African context. Table 5.1 indicates items that require knowledge of the UK context. Items 21, 27, 37 and 58 are specific to the U.K context, whereby July falls in the summer and February falls in winter- in South Africa, it is a contrary situation. Furthermore, in South Africa footballers are known as soccer players. Item 58 refers to the political system of the U.K, whereby U.K is run by a monarchy, while South Africa?s political system is that of a democracy. Therefore, South African learners whose reading comprehension is assessed by the SRS2 may by biased by these items. Thus most learners have scored lower for these specific items as they may have answered from the context in which they live 50 Table 5.1 Item content of the SRS2 Item # Item Content 21 July comes in the summer. 27 Footballers often wear striped shirts. 37 February comes in the winter. 58 There was much loyalty to the prince who was first in line to accede to the throne. In addition, the structure of the SRS2 is that of multiple choices whereby children have to choose from an option of five answers for each 86 questions. Kanjee (2001) contends that South African students are accustomed to essay type questions while those in the UK and USA are more familiar with multiple choice questions. As a result the structure of the SRS2 could affect performance levels of South African students taking the test. Traditionally, cross-cultural assessments would attempt to rule out parameters along which cultures vary (Anastasi, 1982). The SRS2 has a speed component to it whereby participants are given twenty minutes to complete the assessment. Anastasi (1982) noted that motor speed on assessments was seen as culture parameter as some cultures do not attach value to rapid performance and hence have a slower tempo of daily life. Thus for this study some of the learners? performance on this assessments may have been hampered due to the time limit imposed on them. The speed component may have made it difficult for them to perform optimally. The former discussion thus gives evidence for the applicability of the SRS2 in the South African context despite there being a number of items that are considered to contain inapt content. Taking into account the fact that the SRS2 is in English, it is presumed that EFL learners will perform better than their 51 EAL counterparts as English is the first language of EFL learners and the second if not the third language of EAL learners. Thus due to language bias, EAL learners could not perform optimally. Seeing that reading comprehension is a verbal task, it is expected that females will perform better than males on the SRS2. Furthermore research illustrates males are more likely to have reading difficulties thus impacting comprehension. However due to limited resources and living in disadvantaged communities neither males nor females have access to reading materials thus limiting their ability to improve their reading comprehension skills. Despite language experience, gender or culture, the SRS2 is a reliable measure of reading comprehension for South African primary school learners, subject to amendments previously mentioned. 5.6 Limitations All studies are subject to limitations. This section addresses the limitations of the current study. Due to the limitations, caution must be used in generalising the results of the present study to the general population. This study used a non-probability purposive sampling technique. It was for the argument that EAL learners will perform lower than EFL learners, due the SRS2 being in English. Although this argument is apt, given the fact that English is on most cases the second, third or even fourth language of EAL learners, access exposure and formal training with English reading material play an important role. As a result language experience cannot be the only variable affecting scores on the SRS2. This study would have benefited from an additional investigation with a standardised assessment of reading comprehension. The results pertaining to the applicability of the SRS2 within the South African context could have been enhanced with an alternate assessment of reading comprehension that measures the same constructs as the SRS2. A comparison between results could have been conducted and as a result clarifies how consistently or not, the SRS2 measures reading comprehension in the South Africa. 52 5.7 Recommendations for future research This study aimed to contribute to the field of knowledge within the psychometric realm. Future research should embark upon a larger scale in order to standardise the Suffolk Reading Scale as a standardised South African assessment of reading comprehension. An extensive quantitative analysis will prove useful when analysing the SRS2. A distractor analysis to identify incorrect responses by various groups (EFL, EAL, male, female, different and racial groupings) would assist in detecting which items may contribute to content bias. Examining the links between the error types made by different groups would provide extensive knowledge to the field of psychometrics (Carpenter, Just & Shell, 1990). 5.8 Conclusion The unique challenges of cross-cultural testing have become a sensitive issue due to specific concerns regarding the use of standardised assessments cross-culturally. Often the argument is made that westernised assessments be disregarded in the South African context. Shuttelworth-Jordon (1996) proposed that practitioners should not hastily abandon western assessments. The findings of this research indicate that the SRS2 is a reliable measure of reading comprehension for South African learners despite it being an assessment standardised on the U.K population. It should be noted that the reliability of the SRS2 is largely dependent on adaptations made that are more relevant to the South African population. However one cannot ignore the fact that certain groups may be discriminated against and put at a disadvantage, due to the linguistic, cultural and gender bias, the assessment poses. Therefore these factors need to be taken into account when assessing learners with imported tests. Test adaption of the SRS2 is crucial, particularly for the South African context as it conserves time and expenses. The researcher can compare pre-existing data with newly acquired data and as a result allow for international and national cross-cultural studies. The applicability of the SRS2 is not absolute; however 53 with test adaptation the degree of applicability of the SRS2 as a measure of reading comprehension in the South African context could increase satisfactorily, if adapted. 54 References: Aiken, L.R. (2002). Psychological testing and assessment. (11th ed.). Massachusetts: Allyn and Bacon, Inc. Alderson, J.C., Clapham, C. & Wall, D.(1995). Language test construction and evaluation. Cambridge: Cambridge University Press. Anastasi, A. (1982). Psychological testing (5th ed.). New York: MacMillan Publishing Company Anastasi, A. & Urbina, S. (1997). Psychological testing. (7th ed.). New Jersey: Prentice-Hall. Armbruster, B.B., Lehr, F. & Osborn, J. (2001). Put reading first: The research building blocks for teaching children to read, kindergarten through grade 3. Jessup: National Institute for Literacy. Blachowicz, C. & Ogle, D. (2008). Reading comprehension: Strategies for independent learners. (2nd ed.) New York: Guilford Press. Broom, Y.M. (2001). Becoming literate in a multilingual society: Learning to read in South African schools. Unpublished Doctoral Thesis. Johannesburg: University of the Witwatersrand. Burns, N. & Grove, S.K. (2005). The practice of nursing research: conduct, critique, and utilization. (5th ed.). Missouri: Elsevier Saunders. Cain, K. & Oakhill, J. (2006). Assessment matters: Issues in the measurement of reading comprehension. British Journal of Psychology. 76, 697-708. Carpenter, P.A., Just, M.A. & Shell, P. (1990). What one intelligence test measures: A theoretical account of the Raven Progressive Matrices Test. Psychological Review, 97, 404-431 Catto, D.D. (2010) Evaluating the Standford Diagnostic Reading Test as a measure of Reading Comprhension and Vocabulary for South African Grade Eight Learners. Unpublished Masters Thesis. Johannesburg: University of the Witwatersrand. Chang, C.Y. (2001). Cross-cultural assessment: A call for test adaptation. Unpublished paper: Clemson University. Chiu, M.M. & McBride, C. (2006). Gender, context and reading: A comparison of students in 43 countries. Scientific Studies of Reading, 10, 331-362. Cockcroft, K. (2002). Intellectual development. In D. Hook, J. Watts and K. Cockcroft (Eds.), Developmental psychology, (pp. 200-217). Cape Town: UCT Press. Cockcroft, K. (2008). Language development. In J. Watts, N. Duncan and K. Cockcroft (Eds.), Developmental psychology (2nd ed.), (pp. 378-395). Cape Town: UCT Press. Cummins, J. & Mulcahy, R. (1978). Simultaneous and successive processing and narrative speech. Unpublished paper: University of Alberta. Cummins, J. (1981). Empirical and theoretical underpinnings of bilingual education. Journal of Education, 163, 16-29 Cummins, J. (1991). Language development and academic learning. In L.M. Malav?, and G. Duquette (Eds.), Language, culture and cognition: A collection of studies in first and second language acquisition (pp. 161-175). Clevedon: Multilingual Matters 55 Cummins, J. (1998). Research, ethics and public discourse: The debate on bilingual education. Washington: Presentation at the National Conference of the American Association of Higher Education. Retrieved February 12, 2009 from http://ourworld.compuserve.com/homepages/JWCRAFORD/cummins2.htm Department of Education (1997). Quality education for all: Overcoming barriers to learning and development. Report of the National Commission on special needs in education and training (NCSNET), National Committee and educational support services (NCESS). Pretoria: Department of Education DuBois, P.H. (1970). A history of psychological testing. Boston: Allyn & Bacon. Dyer, C. (1995). Beginning research in psychology. A practical guide to research methods and statistics. Cambridge, MA: Blackwell Publishers. Dyer, C. (2006.) Research in psychology: A practical guide to methods and statistics. Oxford: Wiley-Blackwell. Eyesneck, H. (1986). Is intelligence? In R.J. Sternberg, & D.K. Detterman (Eds.), What is intelligence? Contemporary viewpoints on its nature and definition (pp. 69-72). Norwood: Ablex Publishing Corp. Foreman, E.K. (1991). Survey sampling techniques. New York: Marcell Dekker Foxcroft, C.D. (1997). Psychological testing in South Africa: Perspectives regarding ethical and fair practices. European Journal of Psychological Assessment, 13, 229-235 Foxcroft, C. & Roodt, G. (2001). An introduction to psychological assessment in the South African context. Oxford: Oxford University Press Foxcroft, C., Patterson, H., Le Roux, N., & Herbst, D. (2004). Psychological assessment in South Africa: A needs analysis. The test use patterns and needs of psychological assessment practitioners. Pretoria: Human Sciences Research Council. Foxcroft, C. & Roodt, G. (2005). An introduction to psychological assessment in South Africa (2nd ed.). Oxford: Oxford University Press Gravetter, F.J. & Forzano, L.B. (2008). Research methods for the behavioural sciences (3rd ed.). Belmont: Wadsworth/Thompson Learning Gregory, R.J. (2007). Psychological testing: History, principles, and applications. Illinois: Pearson Goodwin, C.J. (2008). Research in psychology. Methods and design (5th ed.).New Jersey: John Wiley Government Gazette. (1998) Employment Equity Act No.55 of 1998 (Section 8) Psychological testing and other similar assessments. Vol. 400, No. 19370. Hagley, F. (2002). Suffolk Reading Scale II: Teacher?s Guide. London: GL Assessment Hambleton, R. K. (1996). Adapting psychological tests: technical guidelines for improving practices. International Journal of Psychology, 31, 439 (Abstract) Hersen, M. (2004.) Comprehensive handbook of psychology. Vol. 4. New Jersey: John Wiley and Sons Heppner, P.P., Kivilighan, D.M., Jr. & Wampold, B.E. (2008). Research design in counselling (3rd ed.). Belmont: Wadsworth Publishing Company. 56 Hutchinson , J.M., Whiteley, H.E ., Smith, C . & Connors, L. (2003). Literacy development in primary school children with English as an additional language: A review. Journal of Research in Reading, 26, 19- 32. Jensen, A.R. (1980). Bias in mental testing. New York: The Free Press Kame?enui, E., Fuchs, L., Francis, D., Good III, R., O?Connor, R., Simmons, D., Tindal, G., Torgesen, J. (2006). The adequacy of tools for assessing for assessing reading competence: A framework and review. Educational Researcher, 35, 3-11 Kanjee, A. (2001). Cross-cultural test adaptation and translation. In Foxcroft, C. and Roodt, G. (Eds). An Introduction to psychological assessment in the South African context. Cape Town: Oxford University Press. Kaplan, R.M. & Saccuzzo, D.P. (2008). Psychological testing: Principals, applications, and issues. Monterey, CA.: Brooks/Cole. Karlsen, B. & Gardner, E.F. (1986). Standford Diagnostic Reading Test: Directions for administering. Brown level. (3rd ed.). New York: The Psychological Corporation. Kriegler, S.M. & Skuy, M. (1996). Perspectives on psychological assessment in South African schools. In P. Engelbrecht, S.M. Kriegler, & M.I. Booysen, (Eds.), Perspectives on learning difficulties: International concerns and South African realities. Pretoria: Van Schaik. Kozak, A., Kozak, R., Staudhammer, C.H. & Watts, S. (2008). Introductory probability and statistics: Application for forestry and natural sciences. Cambridge: Cambridge University Press. Lathy, H.L. (2006). An assessment of reading in First Language (L1) and Second Language (L2) learners who experience barriers to learning. Unpublished Masters Research Report. Johannesburg: University of the Witwatersrand. Martin, D. (1997). Towards a new multilingual language policy in education in South Africa: Different approaches to meet different needs. Educational Review, 49, 129 ? 139 Martin, A., Volkmar, F.R., & Lewis, M. (2007). Lewis?s child and adolescent psychiatry: A comprehensive textbook (vol. 4). Philadelphia: Lippincott Williams & Wilkins Mauer, K.F. (2000) Psychological test use in South Africa. Retrieved October 17, 2009 from http://www.pai.org.za/Psychological%20test%20use%20in%South%20Africa.pdf McReynolds, P. (1986). History of assessment in clinical and educational settings. In R.O. Nelson and S.C. Hayes. (Eds.), Conceptual foundations of behavioural assessment. New York: The Guilford Press. Moore E.G.J. (1986). Family socialization and the I.Q-test performance of traditionally and transracially adopted children. Developmental Psychology, 22, 317-326 Mujis, D. (2004). Doing quantitative research in education with SPSS. London: Sage Neale, M.D. (1989). Neale analysis of reading ability ? Revised. British Edition. Manual. Oxford: NFER ? Nelson Publishing Company. Neisser, U.,Boodoo, G., Bouchard Jr., T.J., Wade Boykin, A., Brody, N., Cesi, S.J., Halpern, D.F., Loehlin, J.C., Perloff, R., Sternberg, R.J & Urbina, S. (1996). Intelligence: Knowns and Unknowns. American Psychologist, 51, 77-101 57 Nell, V. (1994). Interpretation and misinterpretation of the South African Weschler-Bellevue Adult Intelligence Scale: A history and a prospectus. South African Journal of Psychology, 24, 100-109 Nunnally, JC. (1978). Psychometric theory. New York: McGraw-Hill. Nunnally, J.C. & Bernstein, I.H. (1994). Psychological theory (3rd ed.). New York: McGraw-Hill. Oakhill, J. (1997). Review of the Suffolk Reading Scale. In J. R. Beech & C. Singleton (Eds.), The. Psychological Assessment of Reading, (pp. 324). London: Routledge Orasanu, J. (1986). Reading comprehension: From research to practice. New Jersey: Laurence Erlbaum Associates Owen, K. (1998). The role of psychological tests in education in South Africa: issues, controversies and benefits. Pretoria: HSRC Pallier, G.(2003). Gender differences in the self-assessment of accuracy on cognitive tasks. Sex Roles, 48, 265-270. Paterson, H. & , Uys, K. (2005). Critical issues in psychological test use in the South African Workplace. South African Journal of Industrial Psychology. 31, 12-22. Pretorious, E.J. (2000). ?What they can?t read will hurt them: reading and academic achievement. Innovation?, 21. Editorial Committee, c/o University Library, University of Natal, 33-41. Pretorius, E. & Naud?, H. (2002). A culture transition: Poor reading and writing ability among children in South African townships. Early child development and care, 172, 439-449. Pumfrey, P.D. (1977). Measuring reading abilities: Concepts, sources and applications. Manchester: Department of Education. Pumfrey, P.D. & Elliot, C.D. (1990) Children?s difficulties in reading, spelling and writing: challenges and responses. London: The Falmer Press. Rooyen, V.D. & Jordaan, H (2009). An aspect of language for academic purposes in secondary education: Complex sentence comprehension by learners in an integrated Gauteng school. South African Journal of Education, 29, 2. Rutter, M., Caspi, A., Fergusson, D.M., Horwood, L.J., Goodman, R., Maughn, B., Moffitt, T.E., & Carroll, J. (2004). Gender differences in reading difficulties: Findings from four epidemiology studies. Journal of the American Medical Association, 291, 2007-2012 Sharma, N. (1997). Evaluating children in primary education. New Delhi: Discovery Publishing House. Shuttleworth-Jordan, A.B. (1996). On not reinventing the wheel: A clinical perspective on culturally relevant test usage in South Africa. South African Journal of Psychology, 26, 96 ? 102 Staden, V. S. & Howie, S.J. (2006). South African teacher profiles and emerging teacher factors: The picture painted by PIRLS centre for evaluation and assessment, University of Pretoria Stansfeld, S. A., Berglund, B., Clark, C., Lopez-Barrio, I., Fischer, P., Ohrstrom, E., Haines, M. M., Head, J., Hygge, S., van, K., I, & Berry, B. F. (2005). Aircraft and road traffic noise and children's cognition and health: a cross-national study', Lancet, 365, 1942-1949. 58 Sternberg, R.J. (1996). Cognitive psychology. Fort Worth: Harcourt Brace College Publishers Stommel, M., & Wills, C. E. (2004). Clinical research: Concepts and principles for advanced practice nurses. Philadelphia: Lippincott, Williams & Wilkins. Stones, C. R. (2001). Socio-political and psychological perspectives on South Africa. New York: Nova Science Publishers. Urbina, S. (2004). Essentials of psychological testing. Hoboken, N.J: John Wiley Thorndike, R.M. (2005). Measurement and evaluation in psychology and education (7th ed.). Boston: Pearson Inc. Wolfaardt, J.B. (2001). Basic Concepts. In C. Foxcroft, and G. Roodt (Eds), An introduction to psychological assessment in the South African context. Cape Town: Oxford University Press. Weiner, I.B., Freedheim, D.K., Graham, J.R., & Naglieri, J.A. (2003). Handbook of psychology: Assessment Psychology (vol. 10). Yates, F. (1934). Contingency table involving small numbers and the x2 test. Journal of the Royal Statistical Society, 1, 217-235. 59 Appendix Appendix A: Biographical Questionnaire Questionnaire for children Your answers are CONFIDENTIAL . They will NOT be seen by your parents, carers or teachers. Code___________________ Date____________________ Level A__________________ 60 Everyone?s views are very important to us so please try to complete this questionnaire without talking. If you need someone to help you put your hand up and a monitor will come over to you. RANCH team, 3.15, BMS Building, Queen Mary, University of London E1 4NS. Tel. 020 7882 7649 61 Who are you? 1. Are you a boy or a girl? ?1 Boy ?2 Girl 2. What languages are spoken in your home? *tick ALL languages that apply* ?1 English ?2 Bengali ?3 Hindi ?4 Urdu ?5 Gujerati ?6 Punjabi ?7 Tamil ?8 Any African language ?9 Other(s) (please write) _______________________________________ 3. What is the main language spoken in your home? *tick ONE language* ?1 English ?2 Bengali ?3 Hindi ?4 Urdu ?5 Gujerati ?6 Punjabi ?7 Tamil ?8 Any African language 62 ?9 Other(s) (please write) _______________________________________ 4. Has the school offered to give you free school meals? ?1Yes ?2 No ?3 Don?t know 63 Your health 5. In general, would you say your health is? ?1 Very good ?2 Good ?3 Fair ?4 Bad ?5 Very bad 6. In the last month how often have you had a headache? ?1 Never ?2 A few times ?3 Once a week ?4 A few times a week ?5 Every day 7. In the last month how often have you felt like you were going to be sick, throw up or vomit? ?1 Never ?2 A few times ?3 Once a week ?4 A few times a week ?5 Every day 8. In the last month how often have you had a tummy-ache? ?1 Never ?2 A few times ?3 Once a week ?4 A few times a week ?5 Every day 9. In the last month how often have you found it difficult to get to sleep? ?1 Never ?2 A few times ?3 Once a week ?4 A few times a week ?5 Every night 64 10. In the last month how often have you woken up during the night? ?1 Never ?2 A few times ?3 Once a week ?4 A few times a week ?5 Every night 11. People sometimes feel sleepy during the daytime? In the last month, during your daytime activities, how often have you had a problem with sleepiness (feeling sleepy, struggling to stay awake)? ?1 Never ?2 A few times ?3 Once a week ?4 A few times a week ?5 Every day 65 Spending your free time at home For these questions I need you to think about your home and how you feel when you spend your free time there. The first few questions are about indoors, the others are about the outdoors. Listen to each question before choosing your answer? 12. When you are at home and indoors is it calm? ?1 Never ?2 Sometimes ?3 Often ?4 Always 13. When you are at home and indoors is it quiet? ?1 Never ?2 Sometimes ?3 Often ?4 Always 14. When you are at home and indoors is it noisy? ?1 Never ?2 Sometimes ?3 Often ?4 Always 15. When you are at home and indoors is it disturbing? ?1 Never ?2 Sometimes ?3 Often ?4 Always 16. When you are at home and outdoors is it calm? 66 ?1 Never ?2 Sometimes ?3 Often ?4 Always 17. When you are at home and outdoors is it quiet? ?1 Never ?2 Sometimes ?3 Often ?4 Always 18. When you are at home and outdoors is it relaxing? ?1 Never ?2 Sometimes ?3 Often ?4 Always 19. When you are at home and outdoors is it noisy? ?1 Never ?2 Sometimes ?3 Often ?4 Always 67 What do you hear at school? These questions are about the sounds you hear when you are at school?please follow them as I read them aloud then choose your answer by putting a tick in the box. Remember to tick only one box for each question. 20. Do you hear noise from road traffic when at school ? ?1 Never ?2 Sometimes ?3 Often ?4 Always 21. How often are you annoyed by noise from road traffic when you are at school? ?1 Never ?2 Sometimes ?3 Often ?4 Always 22. Do you hear noise from aircraft when at school? ?1 Never ?2 Sometimes ?3 Often ?4 Always 23. How often are you annoyed by noise from aircraft when you are at school? ?1 Never ?2 Sometimes ?3 Often ?4 Always 68 What do you hear at home? The questions on this page are about the sounds you hear when you are at home?listen to me read them out before you choose your answer. 24. Do you hear noise from road traffic when at home? ?1 Never ?2 Sometimes ?3 Often ?4 Always 25. How often are you annoyed by noise from road traffic when you are at home? ?1 Never ?2 Sometimes ?3 Often ?4 Always 26. Do you hear noise from aircraft when at home? ?1Never ?2 Sometimes ?3 Often ?4 Always 69 27. How often are you annoyed by noise from aircraft when you are at home? ?1 Never ?2 Sometimes ?3 Often ?4 Always Your parents and carers Remember that everything you write in this questionnaire is confidential ? your parents, carers and teachers will never see what you have written. These three sentences are about your parents and carers. Listen carefully as I read them aloud and then choose an answer. Tick one box for each question. 28. If I have a problem at school my parents/carers are ready to help ?1 Never ?2 Sometimes ?3 Often ?4 Always 29. My parents/carers are willing to come to school and talk to teachers ?1 Never ?2 Sometimes ?3 Often ?4 Always 70 30. My parents/carers encourage me to do well at school ?1 Never ?2 Sometimes ?3 Often ?4 Always 71 Emotional feelings 31. During the past month how often have you felt worried, miserable, irritable or sad? *To show your answer circle the number at the end of the row* None of the time 1 A little of the time 2 Some of the time 3 Most of the time 4 All of the time 5 School work 32. During the past month you were in school, how did you do? *To show your answer circle the number at the end of the row* I did very well 1 I did as well as I could 2 I could have done a little better 3 I could have done much better 4 I did poorly 5 School sounds Please think about the sounds you hear when you are at school? 33. In general, how annoyed are you by sounds you hear when you are at school? ?1 not at all ?2 a little ?3 quite a bit ?4 very much ?5 extremely 34. Thinking about the last year, when you are at school, how much does noise from road traffic bother, disturb or annoy you? ?1 not at all ?2 a little ?3 quite a bit ?4 very much ?5 extremely 35. Thinking about the last year, when you are at school, how much does noise from aircraft bother, disturb or annoy you? ?1 not at all ?2 a little ?3 quite a bit ?4 very much ?5 extremely 36. Thinking about the last year, when you are at school, how much does noise from other children bother, disturb or annoy you? ?1 not at all ?2 a little ?3 quite a bit ?4 very much ?5 extremely 37. Would you like your school to be quieter? ?1 No ?2 A little quieter ?3 Quite a lot quieter ?4 Don?t know Home sounds Please think about the sounds you hear when you are at home 38. In general, how annoyed are you by sounds you hear when you are at home? ?1 not at all ?2 a little ?3 quite a bit ?4 very much ?5 extremely 39. Thinking about the last year, when you are at home, how much does noise from road traffic bother, disturb or annoy you? ?1 not at all ?2 a little ?3 quite a bit ?4 very much ?5 extremely 40. Thinking about the last year, when you are at home, how much does noise from aircraft bother, disturb or annoy you ? ?1 not at all ?2 a little ?3 quite a bit ?4 very much ?5 extremely 41. Would you like your home to be quieter? ?1 No ?2 A little quieter ?3 Quite a lot quieter ?4 Don?t know School activities 42. Do you find that noise from road traffic at school, disturbs or interferes with playing outdoors? ?1 Never ?2 Sometimes ?3 Often ?4 Always 43. Do you find that noise from road traffic at school, disturbs or interferes with working in a group? ?1 Never ?2 Sometimes ?3 Often ?4 Always 44. Do you find that noise from road traffic at school, disturbs or interferes with quiet work by yourself? ?1 Never ?2 Sometimes ?3 Often ?4 Always 45. Do you find that noise from road traffic at school, disturbs or interferes with listening to your teacher? ?1 Never ?2 Sometimes ?3 Often ?4 Always 46. Do you find that noise from aircraft at school, disturbs or interferes with playing outdoors? ?1 Never ?2 Sometimes ?3 Often ?4 Always 47. Do you find that noise from aircraft at school, disturbs or interferes with working in a group? ?1 Never ?2 Sometimes ?3 Often ?4 Always 48. Do you find that noise from aircraft at school, disturbs or interferes with quiet work by yourself? ?1 Never ?2 Sometimes ?3 Often ?4 Always 49. Do you find that noise from aircraft at school, disturbs or interferes with listening to your teacher? ?1 Never ?2 Sometimes ?3 Often ?4 Always Home activities 50. Do you find that noise from road traffic at home, disturbs or interferes with playing outdoors? ?1 Never ?2 Sometimes ?3 Often ?4 Always 51. Do you find that noise from road traffic at home, disturbs or interferes with listening to TV, radio or music? ?1 Never ?2 Sometimes ?3 Often ?4 Always 52. Do you find that noise from road traffic at home, disturbs or interferes with talking? ?1 Never ?2 Sometimes ?3 Often ?4 Always 53. Do you find that noise from road traffic at home, disturbs or interferes with reading or doing homework? ?1 Never ?2 Sometimes ?3 Often ?4 Always 54. Do you find that noise from aircraft at home, disturbs or interferes with playing outdoors? ?1 Never ?2 Sometimes ?3 Often ?4 Always 55. Do you find that noise from aircraft at home, disturbs or interferes with listening to TV, radio or music? ?1 Never ?2 Sometimes ?3 Often ?4 Always 56. Do you find that noise from aircraft at home, disturbs or interferes with talking? ?1 Never ?2 Sometimes ?3 Often ?4 Always 57. Do you find that noise from aircraft at home, disturbs or interferes with reading or doing homework? ?1 Never ?2 Sometimes ?3 Often ?4 Always Dealing with noise at school 58. When at school, how often do you deal with noise by covering your ears? ?1 Never ?2 Sometimes ?3 Often ?4 Always 59. When at school, how often do you deal with noise by carrying on with your work? ?1 Never ?2 Sometimes ?3 Often ?4 Always 60. When at school, how often do you deal with noise by switching off (tuning out)? ?1 Never ?2 Sometimes ?3 Often ?4 Always 61. When at school, how often do you deal with noise by waiting for it to finish? ?1 Never ?2 Sometimes ?3 Often ?4 Always Dealing with noise at home 62. When at home, how often do you deal with noise by closing windows? ?1 Never ?2 Sometimes ?3 Often ?4 Always 63. When at home, how often do you deal with noise by staying indoors? ?1 Never ?2 Sometimes ?3 Often ?4 Always 64. When at home, how often do you deal with noise by moving to a quieter room? ?1 Never ?2 Sometimes ?3 Often ?4 Always 65. When at home, how often do you deal with noise by turning up the TV, radio or walkman? ?1 Never ?2 Sometimes ?3 Often ?4 Always That?s it?well done! Thank you for completing this questionnaire Appendix B: Consent letter to parents Dear Legal Guardian My name is Maria Ramaahlo and I am conducting research for the purposes of obtaining a Degree in Masters of Educational Psychology at the University of the Witwatersrand. The title of my research is ?Application of the Suffolk Reading Scale II to primary school learners in the South African context?.My area of focus is that of psychometric tests and to establish their validity and reliability in the new South African context. My research discusses controversies pertaining to testing particularly statistical bias and how this affects the Suffolk Reading Scale in the South African context. I would like to invite your child to participate in this study. Participation in this research will entail completing demographic questionnaire, student questionnaire and the Suffolk Reading Scale II. Participation is voluntary, and your child will not be advantaged or disadvantaged in any way for choosing to complete or not complete the questionnaire. Questions are asked about your child?s age, race and grade. The completed questionnaire pack will not be seen by any person in the school at any time, and will only be processed by myself, and my supervisor. Your child?s responses will only be looked at in relation to all other responses. He/she may choose to refuse to answer any questions she/he would prefer not to, and she/he may choose to withdraw from the study at any point. All information collected will be treated confidentially. There are no direct risks or benefits attached to participating in this study. If you consent for your child to participate in the study, she/he will be asked to complete the questionnaire pack as carefully and honestly as possible. The questionnaire pack consists of a student questionnaire, the Suffolk Reading Scale II, as well as other cognitive activities of memory and attention. Completing the activities would be done during one school morning before first break (8:00-10:00). Once she/he has answered the questions, these will be collected immediately and placed in a sealed envelope. Your consent for child?s participation in this study would be greatly appreciated. This research will contribute both to a larger body of knowledge on assessment in the South African context. Please do not hesitate to contact me or my research supervisor should you require further information. A feedback letter will be provided to the school once I have analyzed my results. Should your child experience any distress after participating in the study free helpful contact numbers of counseling organizations will be provided to her/him. A feedback letter will be provided to the school and yourself once I have analyzed my results Note that because participation is anonymous and confidential I will not be able to disclose information about whether your child?s reading comprehension scores. Kind Regards Maria Ramaahlo Contact details Joseph Seabi (research supervisor) - (011) 717-8331 Maria Ramaahlo (researcher)- 0832689463 Parental consent from I __________________________________ give consent for my child to partake in the study on _______________________________________________. I understand that: Participation is this study is voluntary. That my child may refuse to answer any questions he/she would prefer not to. My child can withdraw from the study at any time. No information that may identify my child will be included in the research report, and my child?s responses will remain confidential There are no direct risks or benefits attached to participation Signed: ____________________________ Date: ____________ Appendix C: Consent letter to learners Hello My name is Maria Ramaahlo and I am conducting research for the purposes of obtaining a Degree in Masters of Educational Psychology at the University of the Witwatersrand. The title of my research is ?Applicability of the Suffolk Reading Scale II to primary school learners in the South African context?.My area of focus is that of psychometric tests and to establish their reliability in the new South African context. My research discusses arguments in testing, especially how unfair these tests can be and how this affects the Suffolk Reading Scale in the South African context. I am inviting you to take part in this study. To take part in this research one has to complete demographic questionnaire telling me about your gender, race grade and age, student questionnaire and the Suffolk Reading Scale II. Participation is voluntary, and you will not be advantaged or disadvantaged in any way for choosing to complete or not complete the questionnaire. Questions are asked about your age, race and grade. The completed questionnaire pack will not be seen by any person in your school at any time, and will only be handled by myself, and my supervisor. Your answers will only be looked at in relation to all other answers. You may choose to refuse to answer any questions you would prefer not to, and you may choose to pull out from the study at any point. All information collected will be treated confidentially. It should be noted that there are no direct risks or benefits attached to participating in this study. If you choose to participate in the study, you will be asked to complete the questionnaire pack as carefully and honestly as possible. Once you have answered the questions, these will be collected immediately and placed in a sealed envelope. Your participation in this study would be greatly appreciated. This research will contribute both to a larger body of knowledge on assessment in the South African context. Please do not hesitate to contact me or my research supervisor should you want more information. A feedback letter will be provided to your school once I have analyzed my results. Should you experience any worries after participating in the study free helpful contact numbers of counseling organizations will be provided to you. A feedback letter will be provided to the school and you once I have analyzed my results Note that because participation is unkown and confidential I will not be able to disclose information about your reading comprehension scores. Kind Regards Maria Ramaahlo Contact details Joseph Seabi (research supervisor) ? (011)717-8331 Maria Ramaahlo- 0832689463 Participant Assent Form I __________________________________ assent to partake in the study on _______________. I understand that: Participation is this study is voluntary. That I may refuse to answer any questions I would prefer not to. I can withdraw from the study at any time. No information that may identify me will be included in the research report, and my responses will remain confidential There are no direct risks or benefits attached to participation Signed: ____________________________ Date: ____________ Suffolk Reading Scale 2