LESOTHO JUNIOR SECONDARY SCIENCE TEACHERS? PERCEPTIONS AND USE OF PAST EXAMINATION PAPERS IN TEACHING SOPHIA M. MAJARA Research report submitted to the Faculty of Humanities, School of Education of the University of the Witwatersrand, Johannesburg in partial fulfillment of the degree of Master of Education by coursework and research report. ii ABSTRACT Literature has shown that assessment has various purposes in education, ranging from establishing the starting point, to monitoring and evaluation, while the role of assessment in supporting learning is yet to be explored. In this study I explore the influence of assessment on teaching in the Lesotho junior certificate science curriculum. I achieve this by exploring the nature tools of summative assessment, teachers? views on these tools and how teachers engage the tools once they have served the summative assessment purpose. In exploring the nature of tools of assessment I conducted a documentary analysis of junior certificate science papers written over three years, and obtained teachers? views on the papers and their uses through a questionnaire and a focus group interview. The study is predominantly qualitative and interpretive and is informed by three theories; curriculum theory; which explains the cyclic relationship of the curriculum elements, Bloom?s taxonomy which guides the drawing of educational objectives and construction of tools of assessment and social theories explaining reflection on and understanding of one?s social practices. Analysis of the past examination papers showed that they are relatively valid for the syllabus for which they are designed, though there are some discrepancies. Teachers find the questions in the examination papers appropriate for use in teaching, though they need to be simplified at times. The question papers also assess the understanding of science in the context of life around the learner. The uses to which teachers put past examination papers range from planning to testing and the most common uses are teaching and testing. According to the findings, past examination papers do have an influence on what teachers select for teaching and how they approach what they have selected. Further recommendations that I can make on the basis of this study are; that teachers can derive themes for action research from which they can build pedagogical content knowledge for various topics. Finally, they can also be used as a source of a variety of tasks for learners. Keywords: Examination Junior Secondary Science Past Papers Perceptions Teachers Teaching Use iii DECLARATION I declare that this research report is my own unaided work. It is being submitted for the degree of master of Education by coursework at the University of the Witwatersrand, Johannesburg. It has not been submitted before for any other examination or in any other university. ______________________________ Sophia M. Majara This day ____ of ____________, _________ iv DEDICATION This work is dedicated to memory of my parents and a brother: Ntate Nthoant`so Majara 1923 ? 1984 `Me` `Malikonelo Majara 1931 ? 1970 Khaitseli; Mofihli Majara 1962 ? 2008 v ACKNOWLEDGEMENTS My sincere gratitude goes to the following: My supervisor; Professor Marissa Rollnick, for her patience and guidance in ensuring quality in this piece of work. Mr. Luke Sinwell; for editing the final document of this work. The Lesotho government; for sponsoring my studies and allowing me to collect data from the nation?s secondary schools. The Heads of Schools, Heads of Science Departments and the science teachers who willingly participated in this study. My sibling brothers; for keeping a watchful eye over my property and children while I worked on my studies in South Africa. All my friends and classmates from all over Africa; for their encouragement and support throughout our studies. My daughter and son; Palesa and Thato for affording me time to work on my studies. ?Me` Sophia Fosa; for taking a good care of Palesa and Thato while I worked on my studies. Above all; my thanks and praise go to the Lord, our GOD who empowered all of us to achieve more than this work. vi LIST OF CONTENTS Title i Abstract ii Declaration iii Dedication iv Acknowledgements v List of contents vi List of abbreviations x List of appendices xi List of figures xi List of tables xii CHAPTER ONE: INTRODUCTION 1 1.1 Introduction 1 1.2 Statement of the problem 2 1.3 Importance of the study 3 1.4 Rationale for the study 3 1.5 Aims of the study 4 1.6 Background to the study 4 1.7 Research questions 5 1.8 Outline of the research report 6 CHAPTER TWO: LITERATURE REVIEW AND THEORETICAL FRAMEWORK 7 2.1 Introduction 7 2.2 Theoretical framework 7 2.2.1 Curriculum theory 7 2.2.2 Social theories involved in curriculum operation 10 vii 2.2.2.1 Theory of communities of social practice 11 2.2.2.2 Critical social theory 13 2.2.3 Bloom taxonomy of classification of educational objectives 14 2.3 Assessment and curriculum 15 2.3.1 What is assessment? 16 2.3.2 Purposes of assessment 16 2.3.3 Requirement of assessment 18 2.3.4 Functions of assessment 24 2.3.5 Other influences of assessment on teaching and learning 27 2.4 Science education in the Lesotho secondary schools 31 2.5 Conclusion 31 CHAPTER THREE: METHODOLOGY 33 3.1 Introduction 33 3.2 Research design 33 3.3 Instruments 34 3.3.1 Focus group interview 35 3.3.2 Questionnaire 36 3.3.3 Document analysis 37 3.4 Sample 38 3.5 Data collection 39 3.6 Rigour 40 3.7 Validation of the findings 44 3.8 Data analysis 44 3.9 Ethical consideration 46 3.10 Limitations to the study 46 3.11 Conclusion 47 viii CHAPTER FOUR: ANALYSIS OF QUESTION PAPERS 48 4.1 Introduction 48 4.2 Examination papers analysed 48 4.3 Aspects for which question papers were analysed 49 4.3.1 Content validity of the examination papers 50 4.3.2 Cognitive levels of questions and skills assessed in the question papers 52 4.3.2.1 Information dimension 53 4.3.2.2 Skills dimension 53 4.3.2.3 Description of levels of classification and examples of questions at each level 53 4.3.3 Other aspects which emerged from the analysis of question papers 59 4.4 Discussion 60 4.5 Conclusion 62 CHAPTER FIVE: INFORMATION FROM THE TEACHERS 64 5.1 Introduction 64 5.2 Questionnaire 64 5.2.1 Demographic information of teachers involved in the study 65 5.2.2 Use of past examination papers 67 5.2.2.1 Current ways of using past examination papers 68 5.2.2.2 Other ways in which past examination papers can be use 69 5.2.2.3 Measures taken in cases of poor performance in questions from past examination papers 70 5.2.2.4 Ways of adapting question from the past examination papers for use in teaching 71 5.2.2.5 Effects of questions from past examination papers on ix teachers? practice 72 5.2.2.6 Additional information to the study 74 5.3 Focus group interview 75 5.3.1 Nature of questions from JC science past examination papers 75 5.3.1.1 Aspects assessed by JC science examination papers 76 5.3.1.2 Coverage of JC science question papers over the syllabus 77 5.3.1.3 Cognitive levels of questions in JC science examination papers 78 5.3.1.4 Ways in which teachers involve past examination papers in teaching 80 5.3.1.5 Effects of past examination papers on the teaching practice 82 5.3.2 Other issues which emerged from the focus group interview 84 5.3.2.1 Technicalities of construction of question papers 84 5.3.2.2 New trends in JC science examinations 85 5.4 Discussions 86 CHAPTER SIX: CONCLUSIONS AND RECOMMENDATIONS 92 6.1 Introduction 92 6.2 Nature and of past JC science examination papers 94 6.3 Uses of past examination papers 96 6.4 Effects of past examination papers on teachers practice 97 6.5 Way forward with past examination papers 99 6.6 Critical reflections on the study 102 6.7 Recommendations for further research 103 6.8 Conclusions 104 REFERENCES 105 APPENDICES 111 x LIST OF ABBREVIATIONS CAPITAL: Classroom Assessment Project to Improve Teaching and Learning ECOL: Examinations Council of Lesotho JC: Junior Certificate NCDC: National Curriculum Development Centre SOLO: Structure of Observed Learning Outcomes UNESCO: United Nations Educational, Scientific and Cultural Organization xi LIST OF APPENDICES Appendix A: A letter of the request to undertake research within Lesotho secondary schools 111 Appendix B: A letter of permission to pursue research within Lesotho secondary schools 112 Appendix C: Ethics Clearance 113 Appendix D: A letter to the head of the school 114 Appendix E: Consent form for the head of the school 115 Appendix F: A letter to the junior certificate science teachers 116 Appendix G: Consent form for teachers 117 Appendix H: Questionnaire 118 Appendix I: Interview schedule for focus group interview 122 Appendix J: Sample questions from the past examination papers 123 Appendix K: Demographic information of the teachers involved in the study 124 Appendix L: Example of a transcriptions of the focus group interview 125 Appendix M: Analysis of questions in the junior certificate science examination papers 129 Appendix N: Aims and objectives of Lesotho junior certificate science 138 Appendix O: Junior certificate science paper I, 2006 143 Appendix P: Junior certificate science paper II, 2006 156 LIST OF FIGURES Figure 2.1: Curriculum Cycle 21 Figure 2.2: Learning Cycle 22   xii LIST OF TABLES Table 2.1: Summary of Bloom?s taxonomy of classification of instructional objectives 15 Table 2.2: Modes and levels of Selection of observed learning objectives 21 Table 4.1: Summary of the main features of Junior Certificate Science examination papers 49 Table 4.2: Content validity of examination papers: Biology Section 51 Table 4.3: Content validity of examination papers: Chemistry Section 51 Table 4.4: Content validity of examination papers: Physics Section 52 Table 4.5: Scheme used in classifying items in the examination papers 54 Table 4.6: Distribution of marks over different skills and cognitive levels 58 Table 4.7: Distribution of marks over different cognitive levels (alone) 59 Table 5.1: Ages of teachers involved in the study 65 Table 5.2: Highest qualification held by teachers involved in the study 66 Table 5.3: Majors / subjects of specialisation that teachers hold 66 Table 5.4: Teaching experiences of teachers involved in the study 67 Table 5.5: Ways in which teachers use the past JC science examination papers 68 Table 5.6: Ways in which past examination papers can be used further in the teaching and learning of science 69 Table 5.7: Ways in which teachers respond to poor performance in questions from the past examination papers 70 Table 5.8: Ways in which teachers adapt questions from past examination papers 71 Table 5.9: Ways in which teachers feel questions from past examination papers have influenced their practice 73 2 1 CHAPTER ONE INTRODUCTION 1.1 Introduction Science is of paramount importance in enabling humans to explore and understand the world around them. The need to master and apply science and technology is indispensable to the process of modernisation and development of economics (Lewin, 1992). Curriculum designers have thus seen to it that science and its applications have a special place in the curriculum. In the case of Lesotho, science is a compulsory subject at the primary and earlier part of secondary school level. It is one of the four subjects which constitute the core curriculum together with English, Mathematics and Sesotho. Despite its acknowledged importance by curriculum designers and policy makers in general, its teaching and learning seems to face many challenges throughout the system. The increase in the numbers of students who enrol for science subjects at senior secondary school level and teachers who offer it are not as substantial as it is the case with other subjects. The numbers of students who sat for Biology and Chemistry/Physics combination in Lesotho at senior secondary level fluctuated between 1750 and 2902 and between 1276 and 1887 respectively over the years 1994 to 2000 as compared to 3683 to 6121 that sat for English Language over the same period (Molapo, 2003). The problem is not only confined to Lesotho; in his investigation among British children on why students get bored with science, Delpech (2000) pointed out a few factors which might contribute to the problem. Some of them being the perception that science subjects are generally harder than non-science subjects and that science related careers do not appear to be as financially rewarding as other careers. Performance in the science subjects also seem to be surrounded by problems; in the primary school leaving examination of Lesotho, the pass rate for Sesotho ranged between 56% and 60% in the years 2003 to 2006, while for science it ranged between 43% and 54% over the same period (Examination Council of Lesotho, 2006). This study attempts to take a close look at what is actually happening at the point of interaction between assessment and teaching of science at the junior secondary level in the Lesotho educational system. 2 1.2 Statement of the problem This study investigates the nature of junior certificate science final examination papers and their utilisation in teaching. These are established through getting information that relates to their status and use from the perspective of the science teachers, which will in turn enable inferences to be made about the possible influence of junior certificate science examinations on teachers? practices. The above-stated concerns and the amounts of money spent on a public junior certificate examination would make one to expect that such examination would serve a much wider purpose than regulating entry into senior secondary level. Swain (2000) shows that costs, time, impact on students and teachers as well as benefits to the society and hence consequential validity are all socio economic factors linked to examinations, so the intentions behind them should be clarified in order for their data to be used more effectively for future students. Assessment should advance the objectives of the curriculum, especially in science (Naidoo and Savage, 2004). Phoenix (2000) shows that one purpose of assessment should be shaping performance of learners and educators; as such their results can be used to evaluate courses, institutions and educational process. Junior certificate examination does not facilitate placement into a wider range of fields of training. Most fields of training available in Lesotho require a minimum of senior secondary certificate. In the case of science, junior certificate level can be described as the crossroads (Delors, 1998) determining a learner?s chance of pursuing it since it is no longer compulsory thereafter. It is against this background that this study is designed to look into the relationship between implementation of junior certificate science curriculum and its summative assessment; especially the effect of summative assessment on implementation. This study determines the nature (contents and structure) of junior certificate science examinations, teachers? views on them and how they (teachers) utilise questions from the examinations in their practice. This relationship is determined through analysis of question papers, establishing teachers? perceptions on them and ways in which teachers utilise the papers in their practices. The results would show whether teachers derive any feedback from the papers which in turn influence their practice. 3 1.3 Importance of the study The study tries to establish what is actually taking place at systemic level by seeking the perceptions and practices of teachers relating to Junior Certificate (JC) science past examination papers. The study will enable understanding on reconciliation between teaching and summative assessment in the case of JC science. The results may reveal the relationship between teaching and summative assessment. Some analysis is conducted on tools of summative assessment of JC science curriculum to determine their content and structure. The results may be useful to science teachers themselves and other stakeholders (policy makers and teacher trainers) of JC science curriculum. They may stimulate critical reflection on the part of teachers and hence action to redress the situation. On the other hand the results may also serve as a blueprint for possible inservice and preservice programmes on JC science assessment. 1.4 Rationale for the study Teaching and learning constitute a critical part of the entire curriculum operation, so by establishing a relationship between teaching/learning and summative assessment the study may determine if there is any feedback from summative assessment into teaching/learning. It is not only between teaching/learning and summative assessment where the relationship would be established. There is also a correlation between syllabus and examinations which would be determined implicitly or from the implementers? perspective. As it is the study may be said to be diagnostic in that it attempts to find out if the two relationships indicated above do exist and whether they have any bearing on the observed problems and the reality in the schools. However, it might ultimately end up emancipatory as it would stimulate reflection and hence action to change. Its diagnostic nature lies in the fact that it tries to understand the actual practice and its possible relationship with the observed problems. There was not much I could find on empirical studies relating to the use of past examination papers, other than their utilisation for revision and testing. There is very little I can retrieve from personal experience as a science teacher in schools. What we used to do as teachers was to take past examination papers and select questions relating to what we had just taught when setting quarterly tests or end of semester/year examinations. One other common use was giving learners 4 full papers to practice answering the questions. This used to happen after the students had written the third quarterly in September. It is against this background that this study has been conceived. 1.5 Aims of the study The study intends to look into the effect of summative assessment on teaching/learning in the case of JC science curriculum in Lesotho. This is achieved through establishing the nature of tools of assessment, teachers ?views on them, their utilisation in teaching and their possible impact on teachers? practice. 1.6 Background to the study The Lesotho mainstream school education is designed to take 12 years; seven years in the primary and five years in the secondary school. The secondary level has two exit points at which learners sit for nationally set examinations; i.e. after the first three years and at the end of the last two years. The common names assigned to the class levels are Standards in the primary school and Forms in the secondary school, so class levels are known as Standards 1 ? 7 in the primary school and Forms A ? E in the secondary school. Science is offered as a subject throughout the system and is compulsory for the first ten years of school system. The focus of this study is on teaching, learning and assessment in science in the first three years of secondary education which is also known as Junior Certificate. The curriculum at this level consists of a total of seven examinable subjects; the four compulsory ones (see page 1) indicated above and three others which may be chosen from social sciences, practical subjects and business education depending on the resources available at school. The JC science curriculum is designed in such a manner that general aims and specific objectives, teaching / learning methods and content to be covered are clearly specified so that the teacher is guided through. Teaching methods are suggested, but the teachers are still expected to be innovative so that they develop specified knowledge and a variety of skills in learners. The curriculum is founded on the acquisition and application of scientific knowledge and principles. Its mission statement is therefore: 5 ??to enable the learners to acquire attitudes, knowledge and skills in science and technology that would enhance permanent and functional literacy and numeracy for effective participation in social issues and activities? it is hoped that the learners will relate the science they learn through this curriculum to everyday life phenomena in their immediate environment and beyond?. (NCDC, 2002: 1) The syllabus (see Appendix M) is based on a total of seventeen aims, each of which is unpacked into specific objectives ranging from one to six in number. The majority of objectives reflect acquisition of a scientific knowledge, development of a cognitive or practical skill or development of a positive attitude towards a socially acceptable practice or value. They further reflect the acquisition of the ability to apply knowledge and skills in addressing the real issues in the immediate environment of the learner. According to the curriculum document teachers are expected to develop basic research, problem solving, scientific and critical thinking skills in learners (NCDC, 2002), which should be monitored and assessed continuously. The final examination which learners sit at the end of the programme is a pencil and paper examination consisting of two papers (see Appendices N and O for examples). Paper 1 consists of multiple choice items, carrying 45 marks while paper 2 is made up structured questions totalling 90 marks (see table 4.1). The appointment of the examiner, moderator and teams that prepare item banks are rest with the science panel and the examinations Council of Lesotho. These two papers are supposed to assess the attitudes, knowledge and skills indicated above. 1.7 Research questions The following three questions were framed about the Lesotho junior certificate science examination papers and guided the study: ? What do Lesotho Junior Certificate Science final examination papers assess? ? How do Lesotho Junior Certificate Science Teachers perceive JC science final examination papers? ? How do Lesotho Junior certificate Science teachers use Junior Certificate Science past examination papers in their teaching? 6 1.8 Outline of the research report Chapter one shows what the study is all about and describes the background in which it is located. Chapter two gives the theoretical framework within which the study is located, while at the same time it gives what literature has to say about the relationship between assessment and teaching/learning, especially in science. Chapter three describes the sample, the methods and analysis adopted in obtaining the data, ethical considerations made in collecting data. It further shows the limitations encountered when undertaking the study. Chapter four shows the information revealed by analysis of JC science examination papers for validity and cognitive levels of questions in them. Chapter five gives the views of science teachers on JC science papers and how they utilise such papers in their teaching. Chapter six consolidates the findings from the analysis of question papers and information from teachers. It concludes the study by highlighting further uses of past examination papers, critical reflections on the study and recommendations for further research on use of past examination papers. 7 CHAPTER TWO LITERATURE REVIEW AND THEORETICAL FRAMEWORK 2.1 Introduction This chapter gives the theoretical framework within which the study is located and reviews the literature in the field of assessment and implementation of curriculum. Instruction and formative assessment are both embedded in the teacher practice and therefore virtually inseparable (Black and Harrison, 2000). The chapter is divided into two main parts. The first part outlines the three theories which inform the study. The second part isolates aspects of curriculum implementation which show its relationship with assessment. The study is intended to explore this relationship, especially the influence of assessment on instruction as implementation is supposed to be embedded with assessment. 2.2 Theoretical Framework The study investigates the interplay between two elements of curriculum; implementation and assessment. It is therefore informed by theories which underpin both aspects. These theories include: ? Curriculum theory which shows the relationship between assessment and implementation of curriculum. ? Bloom?s taxonomy which provides details of construction of assessment tools. ? The two social theories; social critical theory and the theory of communities of social practices which underpin social practices and practitioners views on them. Each of these three theories is briefly described below. The manner in which each informs the study is also outlined. 2.2.1 Curriculum theory Different curriculum theorists use different terminologies to describe and explain the relationships among the why, the what, the how and the what results of curriculum 8 operation. Taylor and Richards (1987) differentiate between two curriculum theories which actually appear to be interwoven. These are rational curriculum theory; which they describe as being prescriptive and naturalistic curriculum theory which they describe as emerging from the prevailing conditions. According to rational curriculum theory, there is a clear specification of aims / targets of what a curriculum should strive for; the necessary conditions which would enable the attainment of the targets and measures to be employed in determining their attainment. On the other hand naturalistic curriculum theory shows that situational factors determine the overall operation of curriculum. The two theories actually come together in reality in that the specifications indicated under rational curriculum theory are derived from and aligned with the prevailing conditions so that the curriculum responds to the situation at hand. Rationalist curriculum theory provides guidelines of curriculum operation while naturalist curriculum theory defines the framework within which the curriculum should operate. Bernstein (1975) (in Taylor, Muller and Vingevold, 2003) shows that practices which optimise learning opportunities for all children do not just occur, but must be constructed element by element using a single guiding criterion. The construction takes into consideration the prevailing conditions. In locating the position of assessment in curriculum, Taylor et al. (2003) show that assessment completes the learning cycle in that it evaluates the work of learners, educators and institutions while at same time it explicates and exemplifies learning targets. Bernstein (1975) and Taylor and Richards (1987) describe curriculum operation as a system with inputs into it, processes and outputs from it into the environment. The environment reacts to the outputs by a feedback mechanism which influences the inputs into the system, telling it to carry on what it is doing or change it. Aims of the curriculum are some of the inputs into the process, while implementation encompasses processes and resources involved. Assessing the alignment of inputs and outputs is the feedback from output into the input of curriculum development processes. Bernstein (1975) and Porter (2006) break curriculum into three parts; intended, enacted and attained curricula all of which are continually influencing one another. Figure 2.1 illustrates the relationship of the three parts of curriculum. 9 Figure 2.1: Curriculum Cycle In showing the ideal intricate relationship which should exist between assessment and implementation at instructional level, and how they influence one another, Butler and McMunn (2006) advance a model depicting their integration, which they described as classroom assessment cycle. The model consists of four parts which are interwoven with classroom instruction. They are Clarification of learning targets, Gathering of assessment evidence, Making of inference on the basis of evidence gathered and Linking of assessment just made to instruction. Clarifying of learning targets means spelling out clearly what learners should acquire and understand. Gathering of evidence means employing a multiple of assessment strategies in order to determine the attainment of learning targets by the learners. Making inferences on the evidence gathered and determining what it means in as far as learning is concerned. Linking of assessment or inferences made on learning to instruction means re-planning and modifying instructional plans to meet the needs revealed by inferences and interpretations of the evidence gathered at the second stage. Enacted Curriculum Attained Curriculum Intended Curriculum 10 Figure 2.2: Learning Cycle (adapted from Butler and Mc Munn, 2006) As Taylor et al. (2003) indicate, assessment completes the curriculum cycle at all the levels by the yielding the attained curriculum and illuminating the strengths and weaknesses of reconciling planned and enacted curricula. It therefore acts as a feedback into the entire system involving interaction of planned and enacted curricula. Teachers are the ones implementing the curriculum so they handle the planned and enacted curricula to yield the attained curriculum. The feedback referred to in the theories is expressed in their perceptions and practices. The real feedback that assessment should ideally have is supposed to be reflected in the changes that teachers effect in their practice following exposure to examinations, as tools of assessment. The changes that teachers will be claiming to make in their practice following exposure to examinations will be interpreted in the light of this effect of assessment on curriculum implementation. 2.2.2 Social theories involved in curriculum operation The two social theories which have been taken to underpin teaching are the theory of communities of social practice and social critical theory. Teachers constitute a community of social practice by virtue of being co-implementers of curriculum. Their practice of facilitating learning is common to all of them regardless of the subject they teach. Use of the past examination papers is one such practice found within the activities Clarification of Learning targets Gathering of Assessment Evidence Making sense of Assessment Data Linking of Assessment to Instruction 11 of teachers. Teachers as a community of practice are also confronted with a number of factors some of which constrain their practice. The externally set examinations that learners sit and perceptions of teachers themselves all have an influence on teachers practice. The two theories through which teachers? practices are viewed are described below. 2.2.2.1 Communities of social practices According to Lave and Wenger (1991) a theory of social practice emphasises the relational interdependency of the agent and the world. The social practice shapes out of socially and culturally organised activities in which people are engaged. With the shaping out and delineating itself from other activities, its features such as language, resources, standard practices, constraints and concerns stand out and show its uniqueness (Bowen, 2005). The teachers of different subjects constitute smaller communities of practice within a bigger community of practice. In the different subject areas in which they teach, they face unique challenges which can be equated to constraints to their practice. They also have a variety of tools in the name of theories and tangible ones such as curriculum documents and tools of assessment. Past examination papers are some of the resources at the disposal of teachers. Their use by teachers is also a practice that teachers engage in. As practice, use of past examination papers can be examined and analysed in the light of social practices. The use of past examination papers for formative assessment and instructional purposes is the practice which is central to this study. This practice is examined in the light of its impact on teachers? overall practice. Embedded within the practice of science teachers as a community of practice is another idea with some explanatory power which pertains to assessment; it can help learning if it reveals information that can be used by both the teacher and learners to improve their practices (Black, 1998b). This theory also leads to another theory pertaining to challenges and constraints to teachers? practices. The implementation of curriculum as a set of social practices brings in several factors. These include decoding of the curriculum aims, translating them into instruction, as well as assessing their attainment by learners. As a practice, curriculum implementation also 12 employs a variety of resources. Concrete materials such as books, science equipment, and documents such as syllabi are some of the resources involved in the implementation of the science curriculum. The past examination papers, whose use is under investigation, constitute another viable resource for teaching and learning of science. There are also abstract resources such as language and ideas. Constraints and concerns that implementers of secondary science curriculum have are reflected in the interaction of the practice with other practices. One example is that of problems encountered by learners who decide to pursue science beyond junior secondary school into senior secondary school. They sometimes experience problems even though there is practically no gap in the intended curriculum as identified by Rollnick et al. (1998) in their study on gaps in education from a broader perspective in the Swaziland educational system, which has a similar structure to that of Lesotho. In trying to establish factors contributing to observed attitudes of learners towards science, Delpech (2002) noted that there is a feeling that science subjects are generally harder than non science subjects. All these are concerns that science teachers have, as an output of junior secondary science constitutes an input into the senior secondary science. One other concern that teachers of junior secondary science have is that of accountability to the public. The poor performance that is always reflected in the summative assessment of junior secondary science is another concern that science teachers have as a community of practice. Instruction that is embedded in formative assessment is one aspect that is central to curriculum implementation. It is one of the main practices of teachers. Within this interaction of instruction and formative assessment, there is this use of past examination papers. This study seeks to establish how this practice of using past examination papers is carried out, as a sub component of instruction/assessment endeavour. Views of teachers on JC science examination papers and uses to which such papers are put will be analysed and interpreted in the light of social practices. Such analysis will enable me to establish whether the practice of using past examination papers fits in with other aspects of the practice. 13 2.2.2.2 Critical (social) theory According to critical theorists human beings are continually confronted with forces or factors which restrict them in their practices in one way or another. Critical theorists describe these forces or factors as social constructs which can be untangled. By putting such factors into perspective and understanding them people can deal with them, break away from them and take control of their lives such that they change towards better relationships. Gibson (1986, 2) shows that ?? in identifying biases and distortions which prevent healthy personal and social growth and dealing with them, teachers can free themselves and their learners from those malforming constraints.? Though social critical theory has its roots in political history, it is applicable to contemporary education since education is surrounded by social inequalities and injustices. The degree of social control involved in the distribution of knowledge as Bernstein (1975) shows is an aspect which impacts on teachers practices and hence identities. Teaching and learning of science are hindered by teacher and learner factors such as learners? backgrounds, teachers? attitudes and interpretation of curriculum documents, expectations of the society and demands of public examinations,. These are only a few of the constraints to teachers? practice which can be put into perspective and be properly addressed. They are directly under the teachers? control, and the teacher can break away from them and improve their practice. By reflecting on one?s practice in the light of the contents of examination papers, a teacher can identify aspects of the practice which need some adjustment. According to Jacobs (2000), teachers can think for themselves and should therefore be given a chance to share their curriculum experiences so that they can work out modalities of improving their practice. In wide literature survey that they conducted on teachers attitudes, Jones and Carter (2007) also established that beliefs that teachers hold influence their perceptions and judgements which in turn affect their behaviour in classrooms. These beliefs held by teachers influence every aspect of their teaching; knowledge acquisition and interpretation, defining and selecting instructional tasks, interpreting course content and choices of assessment. According to critical theory these beliefs and their effects on teachers practice can be re examined and addressed for the better by the teachers themselves. By establishing teachers? views on the examination 14 papers and how they influence their practice, the influence of summative assessment on teaching can be determined. 2.2.3 Bloom?s taxonomy of classification of educational objectives The educational objectives articulated in the intended curriculum serve as a guide to implementation as well as assessment of the curriculum. They are supposed to be embedded in assessment standards. According to Fraser, Loubser and Van Rooy (1990), test items are derived and compiled from organised and clearly formulated teaching objectives. In the process of teaching and learning, learners are trained to handle and work with information to different levels of complexity. The ability of the learners to handle information at these different levels is one aspect which is normally assessed. This also implies that the objectives are of differing levels of complexity. One of the most famous schemes of classification of levels of handling information is that which was developed by Bloom, Krathwohl and Masia (1956) and later simplified by Bloom, Madaus and Hastings (1981). Bloom?s scheme gives a qualitative description of what the learner is expected to demonstrate following instruction or during assessment. The scheme is useful in formulating instructional objectives which will in turn inform instructional activities. It also provides a lens through which assessment tasks are viewed, as it is supposed to be embedded with themes constituting the learning content. Bloom?s scheme consists of six cognitive levels, namely; knowledge, application, comprehension, analysis, synthesis and evaluation which are supposed to be hierarchical. Table 2.1 gives a summary of these levels in their order of increasing complexity, starting with the lowest. The right hand column of the table gives an example of what is expected of the learner in an objective or test item at the level. At the level of knowledge, learners are expected to demonstrate knowledge to exactness and fine discrimination, while at comprehension level learners are expected to demonstrate understanding of a phenomenon, stating a proposition in their own words. In application, the learner is expected to apply the understood phenomenon to a new problem or situation successfully. The next levels have to do with breaking down information into components and building of concepts from sub concepts respectively. The cognitive levels and associated competences are summarised in table 2.1 below. 15 Table 2.1: Summary of levels of Blooms taxonomy of classification of instructional objectives, adapted from Biggs and Telfer, (1987) Level Expected competence 1 Knowledge Rote reproduction of the correct response 2 Comprehension Explaining response in student?s own words 3 Application Applying knowledge to a new situation 4 Analysis Isolating crucial components of the knowledge 5 Synthesis Recombining elements to yield new knowledge 6 Evaluation Applying higher order skills to test the worth of new knowledge Each of the levels of Bloom?s taxonomy gives the instructor a guide on what to develop in the learner in as far as the subject matter at hand is concerned. At the same time it provides a guide on how to assess the attainment of the learning initially specified in the objective(s). By applying Bloom?s taxonomy to items in the examination papers, their cognitive levels can be determined. With this classification, the cognitive skills that learners are expected to demonstrate in responding to the questions can be inferred. The scheme will therefore be used as a framework for classifying each of the items in the examination papers to be analysed. This means that items are going to be classified according to their cognitive levels. This exercise will lead to a qualitative description of the examination papers. The weight of each cognitive level will be determined. 2.3 Assessment and curriculum Components of curriculum showing the position of assessment have been shown under curriculum theory. This section will therefore establish the concept of assessment which will be used throughout the study, by showing its purposes and functions in the whole curriculum. 16 2.3.1 What is assessment? Student? assessment can be defined as a systematic collection of information about their learning and other variables associated with particular learning experience. It involves description of knowledge in at least two points; prior to their learning; which is known as preassessment to establish their background and upon completion of the instructional exercise; summative assessment to check for the attainment of the intended learning. Another aspect of assessment might be considered while learning is taking place is formative assessment (Tamir, 1998). Formative assessment forms a vital part of the entire teaching and learning endeavour and informs the instruction in a number of ways. It may be further complemented with another type of assessment that is designed to identify learning problems and their possible causes; diagnostic assessment. Formative assessment determines the direction that teaching should take. Black and Harrison (2000) describe formative assessment as embracing all the activities undertaken by the teacher and learners which can provide information that can be used as feedback to modify teaching and learning activities. Swain (2000) goes further to show that for assessment to function effectively, its results have to be used to adjust teaching and learning and hence the need to make teaching programmes more flexible. 2.3.2 Purposes of assessment Purposes of assessment are diverse and therefore call for a careful selection of the instrument and appropriate interpretation of the information it yields to serve each purpose (Black, 1998b). Purposes of assessment range from establishing the starting point for instruction to accountability in the whole process of teaching and learning. Assessment also serves as tool for monitoring the teaching and learning progress and diagnosing of observed learning problems. It can also be used to check for coherence of parts of the curriculum. According to Knutton (1994) and Tara (2007) there are three purposes of assessment; looking forward, looking backward and guiding action. Assessment actually acts as feedback into the teaching and learning. It should therefore result in reinforcement of observed good practices and remedial action where learning problems show up. 17 Formative use of the past examination papers can help in monitoring the learning as opposed to drilling for examinations. Items in the junior certificate science examination have been tried and tested for applicability and validity, so their use in a classroom situation makes them familiar to the learner (Sieborger, 2004). Once they have served their summative assessment purposes, they amount to what Linn and Miller (2005) describe as standardised achievement tests (batteries), which are tests designed to evaluate learning outcomes and daily progress. Information from the marked scripts of these standardised tests can be used for diagnostic, predictive and monitoring purposes. According to Perrone (1997), ideal assessment should enable the teachers to get closer to the students and allow teachers to have access to what students can do over time. It should therefore be flexibly organised so as to enable learners to reveal their understanding as opposed to coverage of the content learned. The purpose of assessment that is highlighted in this piece of work is the one that assists learning, as opposed to the one which only checks and evaluates the learning. It is the assessment which allows teachers to adjust their teaching to suit the needs of the learners. The assessment in this case will therefore inform subsequent teaching and learning. By revealing the understanding of learners, assessment serves another purpose of illuminating teaching and learning problems, and their possible sources. Bennett (2003) describes this dimension of formative assessment as being diagnostic. With this dimension of diagnosis, formative assessment can lead to modification in teaching and learning and hence facilitate improvement where weaknesses are revealed. Assessment at classroom level also serves an important function of monitoring the teaching and learning progress of both teachers and learners respectively. Lambert and Lines (2000) indicate that there is a potential for assessment in almost every learning task. By continually assessing learners formally, with scheduled tests and assignment; and informally, with incidental observations, teachers can pick up aspects of learners progress which need immediate attention and then make a quick adjustment on the spot (Bell, 2000, 2007). Such short assessment activities help the teacher to follow students? learning development closely. 18 The same notion was interrogated further by Black and Harrison (2000) who showed that formative assessment and its intricate relationship with instruction can play a productive role in curriculum operation. They showed that in implementing effective formative assessment, feedback to learners should not be an overall mark, but rather an identification of their own strengths and weaknesses together with some means of overcoming the learning difficulties identified. As a means of making formative assessment more effective they suggest a careful selection and construction of tasks used in formative assessment. Helping learners to appraise themselves and one another is one other worthwhile use of formative assessment. They indicate that such tasks should be justified in terms of learning aims while at the same time they are geared towards developing critical thinking. Assessment provides important information on students? growth as learners and informs curricular and pedagogical practice. It forms the basis for helping students to reflect on their own learning (Perrone, 1997). It can occupy different positions in the curriculum and therefore serve different purposes. It can be at the beginning, in the middle, or at the end of the programme depending on the purpose it is intended to serve. It has to be properly fitted wherever it is placed in order to serve the intended purpose. However, on the same notion of interrogating the concept of assessment deeper, Tara (2007) shows that treating formative assessment and summative assessment as two discrete processes can be misleading. According to her processes and functions of assessment, whether in summative or formative are the same; they both involve elicitation of information, interpretation of information and facilitation of further action. According to her, within every cycle of formative assessment (Bell, 2000, 2007 and Butler and McMunn, 2006) there is summative assessment embedded. 2.3.3 Requirements of assessment The purposes of assessment given above do not necessarily make up an exhaustive list. There may be other purposes which are applicable to the level of the school. The policy of assessment regarding nature and frequency administering formal assessment task affect classroom level but cannot be decided at the level of an individual teacher. One of the requirements of assessment whose decision rests with the teacher at subject level is 19 the nature of assessment tasks which are used for formative purposes. Assessment tools generally have to be reliable, valid and applicable (Fraser, et al., 1990, Black, 1998a and Porter, 2007). This implies that they should be so designed that they assess the intended competence within the context of theme for which they are meant. One of the requirements of assessment is the subject matter/content on which assessment tasks should be based. Porter (2006) breaks content into two aspects; theme and cognitive demand. The two aspects are derived from the planned curriculum and are supposed to be developed during enacted curriculum. They are used for determining the validity and skill/cognitive demand of the assessment tool. The attained curriculum which is embedded in assessment tools will determine the degree to which enacted curriculum has been valid and developed the skills and information stipulated in planned curriculum. The three parts of curriculum described are supposed to inform one another. Porter?s interpretation therefore calls for relevant and applicable assessment tasks. In the analysis conducted on examination papers for the International Labour Organisation, it was established that items which drew directly from life experience of learners; especially rural learners turned out to be scarce (Lewin, 1992). Assessment tasks should therefore be relevant in terms of content and context. Development of effective assessment tasks is as important as development of effective instructional strategies. It requires interactive research and development cycles, including students? interviews in which their responses are used as the basis for revision and refinement of assessment tasks. In the ideal, curriculum and assessment should be aligned both with each other and with specific, worthwhile learning goals (Stern and Ahlgren, 2002). However, the authors note that curriculum development drives the assessment development and that assessment is designed to align with actual content included in the teaching and learning material. As an exercise, assessment requires tasks which will allow for determination learners levels competences. As in other subjects, secondary science has instructional objectives of varying cognitive levels. Delpech (2002) and Morrison et al. (2005) suggest that according to the latest reforms in science education, learners should be assessed for scientific reasoning and understanding rather than on their discrete scientific knowledge. This implies that 20 learners? writing should be analysed for quality reasoning on top of scientific accuracy. Assessment tasks that are given to learners would have to facilitate application of scientific knowledge in authentic context for authentic purpose. Such tasks would have to allow for examination of process and product at the same time as well as allowing for assessment of higher order thinking skills. They therefore suggest the use of questions which would normally need learners to predict or infer on the basis of the given data. These suggestions have some implications on the structure of assessment tools. Assessment tasks which tap into higher order thinking have an advantage of assessing not only the subject matter, but thinking skills. Ability to deal with information at different levels of understanding was also interrogated by Biggs and Collis (1982), who came up with a five level scheme, which came to be known as Structure of the Observed Learning Outcomes (SOLO). The first three levels are concerned with building of the cognitive structure, while the last two are concerned with the ability to utilise the knowledge structures to different levels of sophistication. Pre-structural level, which is the lowest is characterised by possession of disconnected bits of information, which can be equated to information to be remembered. The next two levels; uni-structural and multi-structural are characterised by demonstration of organised bits of information and understanding of several components of information respectively. The last two levels; relational and extended abstraction are characterised by the ability to form reasonable connections between bits of information and extension of information beyond what is before the learner or what has been taught in class. The levels of Biggs scheme and their associated behaviours are summarised in table 2.2 below. 21 Table 2.2: Modes and levels of the SOLO taxonomy (Adapted from Biggs and Telfer, 1987) Mode Level Structure level (SOLO) Previous Pre structural: The task is engaged in, but the is either destructed or disturbed by irrelevant aspects belonging to the stage Target Uni-structural Multistructural Relational Learner focuses on relevant domain but picks up one aspect and works with it Learner picks up more and correct/relevant features but fails to integrate them Learner now integrates parts with each other so that a whole has a coherent meaning and structure Next Extended abstract Learner now generalises the structure to take in new and more abstract features representing higher modes of operation However this scheme appears to be only descriptive. It does not provide clear guidelines on what should be done to facilitate learning at these different levels (Collis and Telfer, 1987). It may also not be used to frame instructional objectives or assessment tasks. However, it may be more suitable for analysing student? responses. Bloom?s (1956) scheme which has been used as a theoretical framework was later adjusted by several authors to suit different contexts. They modified and used it for analysis of examination questions of different educational levels. These include Manyatsi (1996), Anderson (2005), Green and Rollnick (2007). The adjustments made by these latter authors made range from merging the levels to increasing the dimensions and rearranging the levels as shown by Anderson (2005). In working out the classification of items in the Swaziland equivalent of the Lesotho junior certificate science, Manyatsi (1996) used three levels; Recall, Knowledge with understanding and Handling of information. In recall, the item can be responded to by simply retrieving some relevant information from the memory of the candidate. In Knowledge with understanding, the item would require the candidate to respond by demonstrating a successful and relevant application of the information from their memory. The information needed may be a proposition, fact or concept. The third and 22 last level, which is referred to as handling of information, is higher than the first two. At this level, the learner is expected to demonstrate going beyond the information in front of them, by either inferring or predicting on the basis of it (information). At this last level, learners may also be expected to bring together more than one piece of information and then build a concept or solve a problem at hand. On the other hand, Green and Rollnick (2007) also used a scheme with three levels but use different terminology in describing the levels. They even went further to devise a classification scheme that can be used in classifying the items depending on how the item can be responded to. Theirs considered the learner?s prior interaction with the item or information in it. If the item can be responded to by simply reproducing information, which may not be fully, understood at times, it is assigned level I. On the other hand, if the item cannot be responded to by rote-learned information, but the learner has come across the concept earlier, and then it is assigned level II. Lastly if the item can neither be responded to by rote learned information nor has the learner come across it earlier, then it is assigned level III or above. Items falling under level three may sometimes require the learner to apply the learned information to a totally new situation. It may also involve reworking of the learned information and then generating a suitable solution to the problem at hand. To the authors, prior interaction with the item plays a critical role in determining its level. It can either place the item at a higher level if it has not been encountered before, or place an item at a lower level if the learner had seen it before. Anderson (2005) developed a scheme, which enables one to look at the instructional objective or a test item from two perspectives at the same time forming a matrix. The two perspectives from which an item or an instructional objective can be viewed are level of handling information and the level of mental activity in which the learner engages when working on the problem. This scheme appears to be a further development and reorganisation of Bloom (1956)?s taxonomy, which is, blended with Biggs and Collis (1982) scheme that focused on different levels of handling information. To Anderson (2005) knowledge held at any level according to Biggs and Collis (1982) in Anderson (2005) can be brought into play when employing any one of the cognitive levels (according to Bloom). 23 On the other hand Nitko and Brookhart (2007) identify eleven reasoning strategies which can be used to assess learning outcomes. They range from simple mentioning of facts or concepts to explaining of complex experimental or investigations activities. Some of them are directly applicable to the assessment of Junior Certificate science. They include comparisons where learner will be assessed for their ability to select comparable items as well as clearly and accurately pointing out similarities or differences, classification where the learner where the learner will be assessed for their ability to categorise objects accurately and for their ability to construct a support for or against a claim. A single item in the examination may therefore assess a number of aspects in it. According to Bell (2007) ?What is assessed?? is a critical question linking learning goals, curriculum and assessment strategies together. If what is assessed matches what was supposed to be taught, the learning goals are properly aligned with assessment strategies (Porter, 2006). Subject matter to be taught and learned is broad, so Bell gives a few examples of aspects that can be assessed in science; knowledge of scientific facts and principles, science processes, higher order thinking skills, problem solving skills and other skills needed to manipulate laboratory equipment are some, which science examinations can assess. Examination papers, as tools of summative assessment should be embedded with aspects of relevance to curriculum aims. Embretson and Yang (2006) therefore propose an Item Response Theory which focuses on the probability of how the correct answer to an item is related to the students? abilities and properties of the item itself. The parameters used in analysing the items are possibility of being guessed, level of difficulty and its power to discriminate between those who know and those who do not. They therefore advance two perspectives from which assessment items can be viewed; statistical modelling approach and objective measurement approach. The former entails determining the properties of the examination item from the performance of large population of learners on it. In the case of the latter, the properties and functions of the item are stipulated in advance and then built into the item before it is administered. The contents and structuring of examination items become vital in order for them to assess the theme and skills at the same time. Application of Anderson?s (2005) scheme which examines the item for two aspects simultaneously can facilitate making of inferences about knowledge and mental activity assessed in an item. These are the level to which a 24 learner engages with knowledge and cognitive activity involved in order to arrive at the appropriate answer to the item. Solano-Flores and Nelson-Barber (2001) expand on the issues of applicability and validity of assessment as indicated by Black (1998) and Porter (2007) and introduced another dimension which is cultural validity of science assessment. They define cultural validity as ?? the effectiveness with which a science assessment addresses the socio cultural influences that shape students thinking and the manner in which students make sense of science items and respond to them.? Socio cultural influences include all the sets of beliefs, communication patterns, teaching / learning styles and experiences that are inherent of the students? background. According to the authors, socio cultural environment has a great influence on the learners? thinking, so assessment tasks which are culturally valid show best what the learner is capable of doing with the science knowledge they have. This notion of cultural validity tallies with the nature of science learning which is situated, and therefore calls contextualisation of science assessment tasks. Assessment tasks should therefore be scientifically and culturally valid, which implies accuracy in as far as science concepts are concerned and relevance to the learners? environment respectively. 2.3.4 Functions of assessment The two subsections above have shown the purposes of assessment and what should go into the construction of tools of assessment. Clear articulation of purposes of assessment, construction of assessment tools and their appropriate administration will enable assessment to perform its functions in curriculum operation. Some of the functions of assessment which are over emphasised in literature include facilitating evaluation and informing subsequent action (Bell, 2000, 2007 and Tara, 2007). Interpretation of results released by assessment should inform future planning so that weaknesses revealed may be properly addressed (Butler and McMunn, 2006), to improve the overall performance of learners. At the same time it may be necessary to reinforce strong points identified in the operation of curriculum. 25 The same notion is advanced by Bloom et al. (1981) who show that feedback and correction, if used properly can adequately prepare learners for subsequent learning tasks (Lambert and Lines 2000), which can be very useful in science where content is spirally organised. Science is a subject which can be categorised as collection code with a vertical structure; which means it does not easily link up with other areas of knowledge and that its knowledge in it is organised in such a manner that mastery of one concept facilitates understanding of another higher level concept (Bernstein, 1975, 1995). Periodic assessment as part of teaching and learning in secondary science becomes necessary for effective teaching and learning. Bloom et al. go further to show that individualised corrective help should be provided to enable each student to learn important points they might have missed during instruction. This therefore highlights the importance of giving feedback during teaching and learning especially at individual level. The role that assessment can play in supporting the teaching and learning was also indicated by Black (1998b), who indicates that good formative assessment is not well developed in schools, though it can be a powerful tool for raising the standards of learning. He suggests that for it to serve the purpose of improving learning it should aim at diagnosing learning problems (Bennett, 2003), while at the same time it encourages peer and self appraisal among the learners. With effective implementation of formative assessment, the role of teacher becomes more coaching than drilling. Bell (2000) shows that assessment can be discursive, so with observations, discussions with learners and written work, teachers can pull out information that can be used for formative assessment, and hence facilitate improvement in the subsequent teaching and learning. The role of assessment in supporting learning was further highlighted by Lambert and Lines, (2000), Shepard (2000) and Brooks (2002). They all make reference to scaffolding which can be withdrawn once desired independent learning is attained. According to Brooks (2002), scaffolding assessment oscillates between teaching and traditional examination. Teacher monitors development of learning while at the same time, provides support where it is necessary. Where the learner shows desired competence the support can be withdrawn so that learning develops further on its own. The formative assessment that she suggests entails intervening (by the teacher) during the learning process to gather 26 feedback which will also be used to guide subsequent learning. In the same piece of work, she notes that summative assessment in the name of final examinations tend to dictate what is taught in classrooms; insufficient attention to assessment techniques which promote deep learning discourage development of higher order thinking skills such as critical reflection and speculation. At the same time, examinations focusing on low level aims of curriculum may encourage memorisation of isolated facts. Black and Harrison (2000) and Shepard (2000) even made suggestions on how assessment can be reworked further in order to improve learning. According to them, assessment should serve as a source of insight and help and therefore be moved to the middle of the teaching and learning process. It should form part of the normal classroom discourse pattern in which scaffolding and ongoing checks for understanding are embedded. In this manner it will even make teachers reflect on their practice and make necessary modifications in their teaching. She suggests a shift in the perspectives and beliefs that teachers have about assessment; they should see assessment as a tool for improving teaching and learning and not a device for separating those who know from those who do not. Examination papers and the level at which they are pitched can be ideal for the type of scaffolding indicated. Black and Harrison (2000) suggest that instruction and formative assessment cannot be divorced from one another. They further indicate that implementation of formative assessment leads to flexibility and responsiveness of the teaching and learning in the programme. Assessment and implementation are activities which need to be carefully selected and planned in order to culminate in success. In expanding on the role of assessment at classroom level, Lambert and Lines (2000) incorporated the notions of most commonly applied learning theories; behaviourism and constructivism. They showed that results of assessment; (demonstration of the behaviour) should be fed back to the learners so that learners can build a picture of their progress in terms of strengths and weaknesses and apparent steps to be taken to improve. At the same time with the results of assessment, learners can get to understand new concepts (Bloom, et al., 1981) or refine the old ones. They therefore suggest that the role of results of assessment should be expanded to feed forward and working out strategies of improvement. On the same note, (Black, 1998a) and Swain (2000) show that assessment 27 activity can help learning if it provides information that can be used as feedback by both teachers and their pupils in assessing themselves and each other in order to modify the activities in which they are engaged in. In the introduction of a book where they have constructed and compiled formative probes, Keeley et al. (2005) describe the use of probes as being to assess for learning and not assessing of the learning. This implies assessing to establish what conceptions learners have and restructuring the teaching such that it will move them away from such conceptions if they are not correct. The other alternative use of assessment is enabling the teacher to build on the conceptions of learners if they tally with what is universally accepted by the community of practice of science. Such assessment will also help the teacher to establish the starting point, for the teaching so that the teacher can start from where the learners are in terms of knowledge or skills development. This is one use to which the short questions from the past examination papers can be put. Such a strategy can form part of instruction and elicit learners? prior knowledge in a non threatening manner. 2.3.5 Other influences of assessment on teaching and learning Taylor and Richards (1987) and Lewin (1992) show that both style and contents of public examinations may influence school curriculum adversely or positively as some of the researchers will explain below. According to the authors named above, designers of public examinations make assessment to support learning and not determine it. The ideal situation is that curriculum goals; and not assessment criteria should define curriculum in action (Lewin 1992). He used a metaphor of a baton and choir to describe the observed relationship between examinations and the educational system in China and some of the developing countries. The observation was that examinations tend to dictate the teaching and learning in schools much more than goals of curriculum should. The intricate relationship between assessment and instruction which goes along with the influence of assessment on instruction was highlighted by Kempa (1986), Butler and McMunn (2006) and Jones and Carter (2007) though they focussed on different parts of assessment. The emphasis of Kempa (1986)?s work was on summative assessment while 28 that of the latter two was on formative assessment. They go further to show that teachers? beliefs about assessment greatly influence their practice. If for instance the teacher believes that the only way to motivate learners is through grading, very little formative assessment will be found in such a teacher?s classroom. Such beliefs directly bear on the teacher?s ability to implement new assessment and instructional strategies (Butler and McMunn, 2006). Teachers who are trapped in such beliefs can hardly reflect on their practices in the light of their learners? attainments. Varied uses of tasks are hardly implemented in such teachers? classrooms. A task can be an instructional strategy as well as an assessment method, according to Butler and McMunn (2006). Reflections on ones beliefs which inspire practice become vital if teachers are to change their practice for the better. In unpacking the role of summative assessment, Kempa (1986) isolates two areas where public examinations can have a considerable influence; which are content and orientation of the curriculum as well as teaching and learning procedures. He indicates that the effects of public examinations should be channelled in such a way that they becomes positive; i.e. the structure and content of examinations should be so devised that they are supportive of aims and intentions of educational programmes. Examinations should be curriculum led. Clear definition and articulation of educational aims and objectives of the course/curriculum should be the ones on which assessment and examinations are to focus. The neglect or little emphasis on application of concepts in the examinations does not allow for their treatment or development during the teaching. Methods of teaching and learning which are geared towards developing analytical and predictive thinking are not fully implemented. Rather, there is over directed and limited pupil activity. He further notes that even the practical work done tends to emphasise only skills which are to be assessed in the final examinations. One other positive effect of public examinations on instructional practices was made by Guskey (1994). He noted that assessment devises that tap into higher order thinking skills will elicit instructional practices that develop and emphasise higher order thinking skills. The added benefit of this orientation is that performance based assessment is likely to be an integral part of the instructional process rather than a separate after the fact check on 29 students? learning. This implies crafting in of summative assessment into formative assessment as also indicated by Tara (2007). Guskey further notes that the use of multiple choice items and standardised achievement tests tends to encourage teachers to skew their instruction to basic skills assessed in such tests, if they form part of the public examinations. However, following a wide literature search and critical examination of the interplay among policy, practice and research that he undertook, Black (2000) deduced that there was not much interaction between measurement/assessment and research on science education. This results in a situation whereby research on science education is making a little impact on measurement / assessment. The formative assessment which he feels has some positive effects on learning is not strongly developed as far as he is concerned. He shows that teachers are charged with the responsibility of guiding and assessing their students? work, so empowering them to improve their practice may imply gradually building in formative assessment into their practice. It is not only in learning where assessment can yield positive results. Assessment tasks and the results they yield can also help teachers to improve their practice. In identifying and analysing issues that emerge in science assessment, Gitomer and Duschl (1998) established that assessment can empower teachers to conduct a range of assessments from within inferences made. Assessment can also have consequences some of which are positive; providing assistance to learners in monitoring and improving their own learning, enabling teachers to monitor and improve their own teaching and shaping of curriculum, text books and in service programmes. According to the authors, worthwhile efforts of assessment are those that address the relationship between assessment and instruction, showing how assessment can be used to support and improved instructional practice. In assessing teachers? pedagogical content knowledge, Zohar and Schwartzer (2005) note that teachers were capable of teaching in such a manner that they developed higher order thinking skills to different degrees in different science subjects, but indicated that it took much of their time. The tendency is therefore to resort to a transmission model of teaching so as to cover the content specified in the curriculum. The problem with rushing to complete the syllabus is that it is achieved at the expense of deepening the 30 understanding of learners in some specified topics. What is intriguing is that public examinations do require higher order thinking and problem solving skills. This implies the need to develop the higher order thinking during the course of instruction. As the teacher develops deep learning of scientific concept, s/he has to keep an eye on how that kind of learning can be assessed. There has been too much emphasis on summative assessment for too long. Other important purposes that assessment should be serving have not come to the fore front of curriculum operation. Swain (2000) indicates that summative assessment has served political purposes for long and hence used for only accountability. Its use for improving teaching and learning are now being explored, so its purpose should be clarified. She points out several purposes that summative assessment can serve which include evaluation of programmes, evaluation of learners? performances and teachers? work in general. In the same piece of work she highlighted one important aspect of summative assessment, which is that ??summative data should lead almost automatically, to question analysis and hopefully to answers and provide the essential ingredient in the feedback loop of teaching, learning and assessment?? (p. 154). This implies that such analysis should provide information that will inform and enhance teaching of the future cohorts. It may not only be data pertaining to learning which should be used for feedback; even the structure of the summative assessment tools itself should stimulate reflection on the part of teachers. Evaluation and reflection on the part of the JC science teachers is the one which this study seeks to investigate. It is for the teachers to be able to detect themes assessed and cognitive levels to which questions have been pitched by looking at the examination questions qualitatively. These schemes are used by curriculum developers in designing the programmes of study. They are also used by the assessors of curriculum in evaluating the programmes of study. The alignment of programme design and its assessment is best detected and coordinated by the implementer; the teacher in the case junior certificate science curriculum. 31 2.4 Science Education in the Lesotho secondary schools It is only a few learners who pursue science to the senior secondary school level (see section 1.1). According to the figures given by the Ministry of Education and Training, (2005), it is an average of only 8% of the learners who enrol for Form A that reach their final year of secondary school level (Form E). Only half of this 8% enrols for science subjects (Molapo, 2003). These few Basotho learners who opt to pursue sciences at senior secondary still continue to show an unsatisfactory performance in them. The failure rate in Science and its sister, Mathematics were 35% and 70% respectively (ECOL, 2005) in Nyabanyaba (2008). Several scholars who took some time to look into the performance in science subjects attributed the poor performance to a number of factors such as; language, inadequate preparation for the examinations, employment of traditional methods of teaching and the quality of learners who are admitted into the secondary schools (Molapo, 2003). 2.5 Conclusion This study is intended to find out more about the views of the Lesotho Junior Certificate Science Teachers on the examination papers and how they utilsed such papers in their practice. This chapter therefore highlighted the theories informing the study; the position of assessment in curriculum, ideal nature of tools of assessment and theories explaining people?s dispositions on their practices. At the same time the chapter drew from different areas aspects of interpretations and involvement of assessment in curriculum and its possible impact on the teaching and learning practice. Literature does show that assessment tools and their contents have a wider role to play in supporting teaching/learning than just showing attained learning. The use of past examination papers can help teachers to explore and harness their potential in improving learning. The chapter also put under spotlight the effects of public examinations on the teaching and learning as documented in literature. From what the literature reveals, assessment and instruction cannot be divorced from one another. At the same time they are continually influencing one another. It is for the teachers to reconcile the two and manage their 32 reciprocal influence on one another to ensure effective learning of science. The system may therefore be manipulated such that the reciprocal influence yields positive results. The issues which relate to teaching, learning and assessment in secondary science that have been investigated from different perspectives shed light on aspects which motivated this study. Some of the issues relate to practices of learners while others relate to practices of the science teachers. The important role that assessment can play in improving the teaching and learning of science if properly crafted into instruction has been indicated by more than one author. From what the literature revealed, assessment and instruction cannot be divorced from one another. The last part of the chapter highlighted a few aspects relating to science education in the Lesotho secondary schools. The study will therefore explore the practice and perceptions of science teachers relating to past examination papers and their utilisation in teaching, while at the same time it establishes the nature of the examination papers. Aspects to be established in determining the nature and contents of examination papers are content and construct validity of the items. The next chapter describes the methodology adopted in undertaking the study. 33 CHAPTER THREE METHODOLOGY 3.1 Introduction This chapter describes the methodology adopted in undertaking the study. The study is aimed at looking into the effect of JC science examination papers on teachers? practices; and therefore a qualitative design has been chosen. This is achieved through establishing practices that relate to the operation of junior certificate science teachers. One of the practices in question is the use of summative assessment tools of for formative purposes. The study therefore documents aspects of the practice from the perspective of the practitioners and the meaning it has to them (Creswell, 1994 and Erickson, 1998). The other phenomenon under study is the effect of the summative assessment on curriculum implementation of JC science. This phenomenon was therefore investigated through establishing the views of the teachers on the questions in the final examination papers and their formative use once they have served their summative assessment purpose. The first part of the chapter therefore describes the research design, methods used in the study, size of the samples, the units of analysis involved in the study, instruments used for data collection and the methods of data analysis. The last subsection shows ethical considerations and limitations encountered when conducting the whole study. 3.2 Research Design The focus of the study is on abstract aspects which can only be inferred from observable practices and propositions that participants in the study advance. It explores the lived experiences and practices from the perspective of participants. It seeks to establish and understand the perspectives that participants use as their basis for their practices (Creswell, 1994, Hatch, 2002 and Opie, 2004) and is therefore exploratory. There are no manipulations carried out on the settings; they are studied as they are, so the study is embedded with elements of phenomenology and grounded theory (Creswell, 1994). The reality of the phenomenon under study can be bound to time and place, and therefore contextual and varied. The phenomenon under study which is the use of past examination papers as practice may exist and can only be estimated and not be precisely determined. 34 The practices may also change at any time. Different researchers may interpret it in a variety of ways so study is underpinned by constructivist paradigm of research (Hatch, 2002). The approach adopted in undertaking the study is predominantly qualitative though it has some elements of quantitative approaches. The study is designed in such a manner that it will bring out information pertaining to the practice of science teachers as it relates to the use of past examination papers. The time available may not allow for observation even though the interest is on practice which is a real phenomenon. Neither is the study concerned about how widespread the phenomenon is, so quantitative methods may not be viable. It therefore needs a small sample which will allow for examination of the phenomenon from more than one angle. The study will therefore be subjective and interpretive in that it can bring out different perspectives of the phenomenon. 3.3 Instruments The data were collected with the help of three different types of instruments; focus group interview, questionnaire and document analysis. The focus group interviews were conducted in three of the six schools selected. The questionnaire was distributed to all the junior certificate science teachers in the six schools. The analysis was conducted on the items in the question papers written over three years (see the details of the sample below). 3.3.1 Focus Group Interview A focus group interview is a conversation initiated by the researcher for the purpose of obtaining relevant information or gathering data through verbal interaction (Cohen and Manion, 1994). The group normally consists of people as few as four or as many as twelve. In this method of data collection the researcher engages in a discussion with a group of people who have a common interest. A focus group normally filters itself because it sometimes constitutes a community of practice. On the other hand the researcher may select its members from a bigger population simply because they have relevant information (McMillan and Schumacher, 2006). The researcher interviews them on the subject under study, in a group (Creswell, 1994). 35 Focus group interviews are useful for gaining information on how people think, or what their perceptions are on a certain event or experience. They generally help the researcher to gain more understanding of human experience (Hinds, 2003). They can be applied at a preliminary stage, in the middle or at the end of data collection activity (Stewart and Shamdasani, 1990), depending on the purpose that the researcher wants it to serve. They can also be used in the baseline study, to prepare for the main study or at the end of data collection to consolidate the data collected through some other methods. In the case of this study the focus group interview constituted one of the main methods of data collection, and it ran parallel to the other two methods (Creswell, 2003). One of the aspects that this study was seeking to find more about, related to the views, opinions and perceptions, so the focus group interview turned to be more ideal that a variety of them could be obtained from several people within a short time. The focus group in the case of this study consisted of junior certificate science teachers in the selected schools. The teachers could briefly recall and discuss aspects of JC science examinations among themselves during the interview. Focus group interviews offer the researcher some advantages; they allow the researcher to get information from more than one person within a short time, while eliciting open ended information from them (Puchta and Potter, 2004). In that regard they tend to be cost effective. They enable the researcher to interact directly with the participants, so that s/he can pick up some other non verbal responses which have some implications. Nodding and shaking of heads which are indicative of agreement or disagreement respectively are some of gestures that an interviewer can pick up during the interviews. They also tend to increase the validity of the information obtained, since the participants would be responding in their own words (Stewart and Shamdasani, 1990), and sometimes in their own language. With the focus group interviews, the researcher is able to go beyond factual information; into the why part of the subject under study. Further probing, clarification of unclear questions and making of follow up questions are all possible during a focus group interview (Cohen and Manion, 1994). A focus group interview generally offers an opportunity to learn further and make some new discoveries on the subject under study on the part of both the researcher and the participants. 36 As a method of data collection a focus group interview has some limitations which can be marshalled by employment of other methods of data collection. Some participants who are reserved may withhold relevant and vital information simply because they may not be free to talk in the presence of the researcher or other talkative fellow participants. To some participants the presence of the researcher may bias the responses (Creswell, 1994) in that the researcher may unconsciously provide some cues which will influence the responses. The researcher has to create an atmosphere which will allow members of the group to develop their own ideas, feelings, insights, expectations and attitudes so that they can say what they think with richness and spontaneity. Focus group interviews are therefore demanding in terms of interpersonal skills on the part of the researcher (Opie, 2004). It is not everybody who possesses such skills. In the case of this study, involvement of a questionnaire with relevant and open ended questions which allowed for extraction of such information was included as one of the methods of data collection. 3.3.2 Questionnaire This is a method of data collection in which participants are given questions to respond to in writing. It works well with literate respondents. The questions in a questionnaire may be close ended in which case the respondent is given a question and a number of options to choose from. They may also be open ended in which case the respondent is not restricted, but given an opportunity to respond to the questions by constructing and giving their own responses. The questionnaire revealed individual and uninfluenced opinions regarding the nature and use of the past examination papers. It allowed reserved participants to express their opinions freely. It is the information which is hopefully at every teacher?s finger tips. In the case of this study, the questionnaire complements the focus group interviews. It will also facilitate triangulation between it and the focus group interviews, thus increasing the validity of the information obtained. As a method of data collection, a questionnaire is economical in terms of time and money (Best and Kahn, 1994, Cohen and Manion, 1994 and Opie, 2004). It allows the researcher to reach out to more respondents than other methods. It can even be mailed electronically or through post if e-mail and postal addresses of target participants are known to the researcher. However electronically mailed questionnaires may not be ideal in all cases as 37 it is not everybody who has access to computers and internet. Mailed questionnaires also have a disadvantage of a poor response rate. Making of follow ups may not be easy in some cases. If the questionnaires are constructed in such a manner that the respondents remain anonymous, the reliability and validity of information they yield also tends to increase. The participants fill the questionnaire in their own time, sometimes uninfluenced by the presence of the researcher. The questionnaire has some limitations, which may require that it be supplemented with other methods of data collection. It can only collect factual information relating to the phenomena under study. It does not allow the researcher to go into the why part of the phenomenon (Opie, 2004). Most of the time a questionnaire is filled in the absence of the researcher, so the misunderstandings that it may present to the respondent may not be properly addressed; which may lower its validity or the information it will yield. In some cases, the researcher may miss out on vital and relevant information simply because the questionnaire was filled in a hurry (Cohen and Manion, 1994). In the case of this study, the questionnaire was complemented with a focus group interview which compensated for its shortcomings. 3.3.3 Document Analysis Documents constitute another source of information which is close to the primary one. They provide evidence of decisions and transactions carried out by the community of practice. They therefore have to be available and authentic (Preece, 1994). They may be private or public depending on the phenomenon under study. In this study, the documents which were analysed were junior certificate science final examination papers which were written over three years. In the case of this study, analysis of question papers constituted a secondary source of information, which would only be used to corroborate part of the information obtained from teachers. The primary source of information in this case is the teachers themselves. The main interest is on junior certificate science teachers? views on such papers. Documents offer the researcher some advantage in they constitute data which is already organised and compiled by the relevant persons. Information in them is already in printed 38 form, so the researcher does not have to go into the process of transcribing. However, documents may still be inadequate or inaccurate (Creswell, 1994), even though they may be coming from the relevant authorities. It is not every document from the relevant authorities which will have relevant information, so the researcher will have to be very selective and extract only what is related to the phenomena under study. The documents which were analysed were junior certificate science final examination papers. Each item in each of the three examinations was analysed in order to infer two aspects; the theme in which the item is located and the cognitive process through which the learner should go in order to get to the appropriate answer (Porter, 2006). 3.4 Sample The study involved two main units of analysis, namely; the junior certificate science teachers in the six selected school and the question papers of the national junior certificate science examinations written over three years. The schools were purposefully selected (Cohen and Manion, 1994) due to their conditions which were fairly similar. They were selected on the basis of their age and location. All of them were under thirty years old, so facilities such as laboratories and libraries were still in their developing stages. They were all located in the suburban area, away from facilities such as the national libraries. The teachers were also selected on the basis of the subject they teach, the level at which they teach (Freebody, 2003) and their willingness to participate. All of them were teachers involved in the teaching of science at JC level. They constituted what McMillan and Schumacher (2006) describe as rich sources of relevant data. The sample consisted of teachers who were directly involved with the teaching of junior certificate science. They were the only ones who could say something about the nature of junior certificate science examination and its impact on teaching of science. Their interaction with the junior certificate science past examination papers makes them more informed about the status of such papers. The manner in which they ustilised the past examination papers in their teaching once they have served their summative assessment purpose is embedded with the effect of such papers on the teaching; which is what the study is about. These teachers were identified in six secondary schools. All the teachers were requested to complete the questionnaire, while teachers in three of the six schools 39 were further interviewed as a group. Information from teachers was further complemented with analysis of junior certificate science papers which were written over three years. These papers constituted another sample of units of analysis. The two instruments; questionnaire and focus group interview schedule were derived from the research questions. Each of the three research question was unpacked into specific questions which would elicit relevant information from the participants and JC science papers (documents). These questions were compiled and organised into the two instruments. The nature of JC science papers, teachers? views on them and the manner in which they use them in teaching were all interwoven with the teachers? practice, so the instruments were organised in such a way that they would elicit the information from the participants. . 3.5 Data collection Information needed for the study involved ideas and interpretation of practices from the practitioners? (JC science teachers in this case) point of view. It could only be obtained from the teachers who were involved in the teaching of JC science. I therefore identified six secondary schools, given pseudonyms; Highlands High School, Foothills High school, Lowlands High school, Urban High School, Suburban High School and Peri urban High school. Having made the decision on whom collect data from, I consulted the authorities; Ministry of Education and Training and Heads of schools, Heads of Science Departments in the schools selected for permission to pursue data collection. In commencing with the data collection I employed a questionnaire and a focus group interview to get teachers? perceptions and use of past examination papers. I delivered questionnaires personally, but that was done during the end of year tests, when everybody was busy. There was no direct contact between me and the teachers in most schools. In almost all the schools, the questionnaires were taken by either the head of the school or the head of science department who would give them to the science teachers and collect them thereafter. In one of the schools where I only issued the 40 questionnaire the teachers were not available until January of the next year, so the return rate dropped from 100 % to 80% in that school. In the three schools where the focus group interview was to be conducted, the original plan was that it would be conducted on the day when the completed questionnaires were collected. The times set by the teachers coincided with either ad hoc meetings of the entire staff of the school or the times when more than half of the science teachers were officially out of the school. In Foothills High School, the time for the interview had to be rescheduled three times due to its conflicts with on official business. The only times which turned out to be suitable at the end of the day were either tea breaks or a lunch hours. Even the indicated breaks were problematic in terms of time available because some of the teachers had classes immediately after the break. The interviews were therefore done in a hurry, limiting the possibilities of deep probing. It was only in Highlands High School where the interview ran smoothly because the teachers scheduled it for lunch hour right away. The main purpose of the focus group interview was to views of teachers on JC science final examination papers in as much detail as I could within the time available. The questions regarding the status of JC science question papers were unpacked into specific questions which allowed teachers to respond in as much detail as they could (see Appendix H). The teachers did not have any questions papers with them so I provided them with a sample of items from the past examination papers which assessed different skills and science concepts in the context of familiar situations (see Appendix I). The teachers were asked to infer what the items assessed. 3.6 Rigour Rigour refers to the quality of the findings. It is inferred from the reliability of the instrument(s), the accuracy of the findings revealed by the instrument(s) and hence the validity of the data. This study is predominantly qualitative and hence subjective, so my biases as the researcher are also taken into consideration as they may affect the interpretations I will be making on the findings (Griffiths, 1998). Validity of the instrument is the degree to which it determines what it is supposed to determine. In the 41 case of qualitative studies, reliability of the instrument is inferred from the authenticity and trustworthiness of the results it releases. The validity of the two instruments; questionnaire and document analysis was ensured by piloting and peer validation respectively. The details of the piloting and validation are given below.This study is predominantly qualitative so the researcher is the main instrument (Leedy, 1993). The descriptions and interpretations made should be reflective of the reality on the ground. The reliability of the information released from the instruments lies not so much on it being replicable, but on the data it revealed and the interpretations by the researcher. The reliability of the information in this study was ensured in the design of the instruments. The three instruments were designed in such a manner that questions addressing the critical aspects of the study ran through all of them. On the other hand, validity refers to the relationship between the account given by the researcher and the reality on the ground. The quality of findings depends on the careful construction and interpretation of the instrument and data respectively. The coherence between instruments and insight into the phenomenon under study may form the basis for believability of findings and their interpretation (Creswell, 1994), and hence the validity of the findings. The trustworthiness of the results was established through member checking (Creswell, 1994) where in the case of the interview; the audio-recorded interview was given to one of the subject to listen to, in order to confirm that what was recorded was precisely what they said as participants. The findings from the interview were also triangulated with those of questionnaire and those of the question paper analysis to establish similarities or differences on questions which were common to both. In the case of this study, the questionnaire was first piloted among my fellow postgraduate students, who were experienced science teachers. They were first briefed on the aims of the study and the ways in which data was going to be collected, before being issued with the questionnaire. They completed the questionnaire and then made comments relating to estimated time needed to complete it, it relationship with the subject under study and its clarity to the potential participant. They suggested a number of adjustments all of which were incorporated before the actual implementation of the questionnaire. The suggestions included clear specification of the examination papers, 42 which were being referred to, focusing of some questions in it and also reorganising the questions so that open ended questions go to the end of the questionnaire. Validation of the scheme of analysis of questions was achieved through giving a sample of questions to three fellow postgraduate students who were experienced science teachers. Two of them were PhD students who had taught science in both secondary and tertiary institutions for some time. One was a Masters student who had a teaching experience of three years in the secondary schools, but was conversant with the educational setting from which the papers were selected. They were asked to comment on the viability of the scheme of classification. They were also asked to classify a provided sample of questions, using the scheme. The sample of questions provided had six multiple choice items and three structured questions all of which were systematically selected from the examination papers to be analysed. It had two multiple choice questions from biology, two from chemistry and two from physics and one structured question from each of the three disciplines. The scheme for analysis of question papers was also subjected to peer validation. It was given to three fellow students for critiquing together with a sample of questions that they had to classify using it. They were asked to analyse the scheme first and determine its viability for classifying the questions. They were asked for what they thought a learner had to go through when responding the questions. They were further given nine items; six multiple-choice items, two from each discipline and three structured items one from each discipline to classify using the scheme. The validators made suggestions all of which were incorporated in the construction of the scheme. The level of agreement with the researcher in classifying the questions appeared to be at sixty five percent and rose to eighty five percent following discussions with them. Only one person out of the three raised a number of questions concerning the scheme. However the questions were later answered after reading through the whole package which explained the scheme, its purpose and the sample questions. The three validators further raised a number of suggestions all of which were incorporated in working out the final scheme. Most of the suggestions were basically for making the scheme much more explicit. The suggestions included incorporation of the original schemes, clear 43 specification of the adjustments made on the original schemes. One of the suggestions was providing room for indicating all aspects assessed. If the item is assessing knowledge and skill in one, there should be room indication of both. In the use of the scheme for classification of examination items, the researcher turned out to be in agreement with at least two of the three over nine out of the fourteen items in the sample. It was only on two items where none of the persons validating the scheme agreed with the researcher. The interpretation that I made on the data and the responses that participants gave were potentially influenced by the biases that each of us held, some of which may threaten the validity of the results. As a researcher I had to guard against such biases. There were three sources of bias that I had to guard against. One source of possible bias that I had to guard against very strongly was the fact that I have been part of the system as a science teacher for more than ten years. The experience that I as a researcher bring into the study (Creswell, 1994) may influence my analysis. There was a possibility that my experience may make me blind to the changes that have taken place since I stopped working in the secondary schools. I might end up analysing more of my experience than the reality on the ground. One of the schools that I have chosen to work with is one in which I taught for eight years when I started teaching. I would therefore have to suspend my experience and be open to the changes which have since taken place. The other possible biases that I had to guard against was the knowledge that some teachers are always involved in the marking of the public examinations, and therefore were more informed about the nature of public examinations than others, while others are always involved in winter schools where the dominant practice is the use of past examination papers. Their experience might influence their responses. One aspect that may increase the validity of the information from teachers is my position as a researcher. I am a former secondary science teacher, but still part of the system in that I am in an institution where I can use the information from them to influence restructuring of courses in secondary science assessment in the pre service and in service teacher training. The teachers who participated in the study and the JC science examination papers analysed were all involved in JC science assessment and so increased the chances of 44 releasing valid information pertaining to JC science assessment and its probable influence on other aspects of JC science curriculum. 3.7 Validation of the findings This refers to the steps to be taken to ensure that the findings revealed by the instruments give the true picture of what is happening at the place where the study was undertaken. This implies the accuracy, trustworthiness and authenticity of the findings (Creswell, 1994). The two major steps that I took to ensure validity were member checking and peer debriefing. I gave the recordings to one of the members to listen to and confirm immediately after the interview and even asked whether they thought there was anything that they left out. In the second step I gave the overall findings from the three instruments to another colleague to judge their worth and believability using their own experience of the system. 3.8 Data Analysis Data collected for this study was in the form of completed questionnaires, audio recorded interviews between the researcher and junior certificate science teachers who constituted focus groups and tables showing classification of items in the question papers according cognitive levels and skills assessed. All these constituted what Erickson (1998) describe as resources for data or potential findings. Their organisation into usable data from which deductions can be made, entailed reworking them into descriptive statistics and themes (Blaxter et al. 1996, Hatch, 2002 and Creswell, 2003) relating to the views of teachers on examination papers and how they utilise them formative purposes. There were also tables showing percentages of marks carried by each skill and cognitive level out of the entire paper which needed to be put in the form of descriptions of the papers. Organisation of information from the analysis of question papers A determination of marks carried by items at each cognitive level was made so that a frequency count of items assessing each skill and falling under each cognitive level was worked out. Information from the analysis of question papers was presented in the form 45 of descriptive statistics (Blaxter, et al. 1996). Information from the analysis of JC science examination papers was organised into two parts; basically information relating to validity of examination questions to the syllabus and information relating to what questions assessed in terms of skills and cognitive levels. The weight of questions at each cognitive level and questions assessing specific skill was determined. Each was finally expressed as percentage of marks for the entire paper. Organisation of information from the questionnaire Information from the questionnaire was organised according to frequency counts of responses given by participants to each question. Fairly similar responses were clustered together. In the end the responses given for each question was expressed as percentage or proportion of the entire sample of teacher involved (Blaxter et al., 1996). The responses were further organised into possible themes relating to the views of teachers and their utilisation of past examination papers in their teaching. Organisation of information from focus group interview Organisation of information from the focus group interviews was transcribed and then categorised using typological analysis and inductive analysis. This meant that it was approached with tentative pre determined codes of classification which increased to accommodate new codes that emerged as the analysis went on (Hatch, 2002). The identified categories were then reorganised further into themes (Blaxter et al., 1996, Creswell, 2003) capturing the main aspects of the subject under study; the junior certificate science teachers? perception and uses of the past examination papers. The themes identified were then used to explain the observed situation in the junior certificate science assessment. The reality of the nature and impact of JC science examination papers on teachers practice was abstracted from their responses (Lemke, 1998). The descriptions of the papers that they gave and the ways of using the papers that they indicated gave me some clue to the effect of question papers on teachers? practice. 46 3.9 Ethical considerations The purpose of the study was articulated in writing and verbatim for the authorities and participants to understand (Creswell, 1994). In undertaking the study, I conformed to the requirements of Human Research Ethics Committee of the university. I also made it a point that I did no harm to others, self or the practice, be it mine or that of the participants as teachers. There were authorities whose permission I had to seek before commencing with data collection. They were the Ministry of Education and Training (of Lesotho), the Heads of schools and science teachers where I chose to collect data (see Appendices A and B). The Ministry of Education is overseer of all the official activities which take place in schools. Having obtained permission from the authorities, I proceeded to teachers whose permission I also had to seek and wait for them to agree to participate in the study. In commencing with the data collection I made it clear to the teachers that the information I was going to ask from them would be used purely for study purposes and nothing else. The information will be destroyed sometime after completion of the study. I also assured them of protection of individual identities and names of schools. I used pseudonyms to identify the names of the schools while I used codes to identify teachers. Individual identities of teachers and schools remained anonymous. The same codes have been used to identify teachers (see Appendix J) where excerpts from their responses were quoted. 3.10 Limitations to the study The time and financial resources available together with the research design only allowed me to work with a small sample of six schools which were relatively similar in terms of physical resources and location. The findings therefore present only part of the picture relating to the perceptions and use of the JC science past examination papers. The time available could not allow comparing the tasks that teachers gave with the past examination papers available. Neither could it allow extended stay in the schools to observe the practice. 47 3.11 Conclusion The methodology described above has attempted to explore the effect of JC science examinations on teachers? practice by establishing three aspects; nature of examination papers as tools of assessment, views of science teachers on them and ways in which the teachers use them for formative assessment purposes. JC science teachers? views on examination papers and the analysis I conducted on the papers revealed the status and the nature of the JC science examination papers. The results of the analysis I conducted on JC science examination papers are given in Chapter Four. The perceptions of teachers on the papers and the ways in which they use the papers are all described in Chapter Five. 48 CHAPTER FOUR ANALYSIS OF EXAMINATION PAPERS 4.1 Introduction This chapter addresses research the first research question one; ?What do Lesotho junior certificate science examination papers assess??. Bell (2007) shows that ?What is assessed?? is a critical question linking learning goals, assessment strategies and curriculum together. The contents of examination papers as tools of assessment need to be put under spotlight. It is from them and performance on them that the success or failure of implementation strategies can be determined. This chapter therefore gives the perspectives from which question papers have been viewed in order to determine what they assess. It is divided into four main parts. The first part describes the science examination papers that candidates write at the end junior certificate. The second part gives the aspects which were considered when the analysis was conducted; content, scope and cognitive demands that questions make on learners. The third part shows other important and prominent features of examinations which emerged apart from the intended analysis. The fourth part is a discussion of the findings and the implications that they are likely to have on the teaching and learning of junior certificate science. 4.2 Examination Papers analysed The papers which were available for analysis were those which were written in the years, 2003, 2005 and 2006. It was not possible to get a full set 2004 though I attempted. The study also commenced in 2007 before November, so the 2007 papers were not available. There were two papers written for each year. Table 4.1 below shows a summary of structure, contents and duration of each of the papers analysed. 49 Table 4.1: Summary of the main features of the Junior Certificate science papers Paper Type Content Marks Duration One Multiple Choice Biology Chemistry Physics 15 15 15 1 hour 15 minutes Two Structured Questions Biology Chemistry Physics Environment Scientific skills Technology 20 20 20 10 10 10 2 hours Total 135 All the papers are set on the syllabus which also serves as a guide for instructional practice. The division into the three disciplines which is seen on the examination papers is also reflected in the construction of the syllabus. For each discipline, there are specified topics which are supposed to be treated. Specifications of what learners should be able to do, demonstrate and know are clearly spelled out. There are also suggestions on the teaching and learning which guide the teacher while at the same time leave room for the teacher to be innovative in helping the learners to acquire the specified content. 4.3 Aspects for which question papers were analysed A brief look at the papers reveals a number of important aspects; validity, accessibility, and relevance to the syllabus. Only two aspects; validity and accessibility will be dealt with in detail in this chapter. A glance at the papers further shows that a great care has been taken to make the examinations accessible to all the learners in terms of language and context. Simple English has been used in constructing the questions. Britton and Schneider (2007) describe accessibility as one important aspect of national examinations which ensures that all candidates stand an equal chance of succeeding in them. The examples in diagrams and situations described in the background information of 50 questions are relatively familiar to students. The following subsection explores the extent of content validity of the examination papers. The papers were analysed for their validity to the syllabus and other aspects which they assessed. The items in the question papers were analysed for the cognitive demands made on candidates, skills and if possible, attitudes that students were assessed on. The analysis was done in conjunction with the syllabus to determine the themes of the syllabus from which they were sampled. The exercise enabled me to identify of themes which were repeated and those which were not assessed over the three years for which papers were available. This exercise also resulted in the determination of content validity of the papers themselves as well as alignment of the examinations with the syllabus (Porter, 2006). From the analysis I was able to have an idea of the structure and contents of the junior certificate examinations. The information to be obtained from the analysis of examination papers is more implicit as they were designed to serve other purposes than the study. It had to undergo further reorganisation in order to serve the purpose of the study. 4.3.1 Content Validity of the Examination Papers Black (1998a) describe content validity of the examinations as the degree to which questions match the content and the learning aims of the syllabus for which they are designed. This implies ensuring that areas of the syllabus have been fairly sampled within the boundaries stipulated. In the case of this study content validity of the examination papers was determined by juxtaposing the themes in the syllabus with the questions in the examination papers. The number of marks carried by each theme in each question paper was also determined. The examinations are organized in such a way that each discipline stands out, so the comparing of themes with the syllabus has been done per discipline for each of the three years. The distribution of marks over different themes has also been done and the summaries are given in tables 4.2, 4.3 and 4.4 below. 51 Table 4.2: Biology section Year 20033 2003 2005 2005 2006 2006 Theme Paper I (marks) Paper II (marks) Paper I (marks) Paper II (marks) Paper I (marks) Paper II (marks) Observing 1 - 1 - 2 8 Diversity of Organisms - - - - - - Cells 1 - 1 - 2 - Reproduction - 7 4 - 2 3 Nutrition 4 9 4 23 2 3 Breathing and Respiration 1 4 - - 2 - Excretion - - 1 - - - Transport systems 1 - 1 1 - - Locomotion and Support 1 - - - 1 - Responses - - 1 4 - - Diseases 2 4 1 4 3 2 Environment 4 6 1 10 1 14 Total 15 30 15 42* 15 30 Table 4.3: Chemistry section Year 2003 2003 2005 2005 2006 2006 Theme Paper I (marks) Paper II (marks) Paper I (marks) Paper II (marks) Paper I (marks) Paper II (marks) Experimental Techniques 3 - 2 4 1 1 Particulate nature of Matter 1 - 1 - 2 10 Atomic structure 3 5 3 4 2 3 Periodic table 1 10 1 3 2 3 Formation of compounds 7 10 8 1 5 13 Metals and non metals - 5 - 6 3 - Total 15 30 15 18* 15 30 52 Table 4.4: Physics section Year 2003 2003 2005 2005 2006 2006 Theme Paper I (marks) Paper II (marks) Paper I (marks) Paper II (marks) Paper I (marks) Paper II (marks) Pressure 3 16 2 4 2 5 Forces 1 - - 2 1 - Equilibrium 2 - 1 - - - Work, power, Energy and efficiency 3 - 1 3 3 - Elasticity of materials - - - - - - Simple machines - - - 2 2 - Electrostatics - - - - - - Current electricity 3 - 2 4 2 5 Magnetism - - 1 10 1 4 Thermal energy 2 14 2 4 1 13 Waves 1 - 5 3 3 3 Total 15 30 15 30 15 30 *A question carrying 10 marks and assessing scientific skill; interpretation of graph in section B is based on Biology (Animal nutrition). The other question carrying 2 marks in Section A taps more into Biology (Diseases) than chemistry of water. Tables 4.2 to 4.4 show some heavy and continuous testing of some topics while others are lightly or virtually not tested at all over the three years. Themes such as reproduction and nutrition in Biology section appear to be heavily tested while electrostatics and elasticity of materials in physics section have hardly been tested over the three years for which examination papers have been analysed. Content validity of examinations over the three years seems to have gone as far as the discipline level only. It seems to be lower at the level of specific themes within the disciplines. 4.3.2 Cognitive levels and skills analysis of individual examination items Individual items in each of the six papers were analysed for the cognitive demands they make on the learner or the skill level that they test. The scheme used in analysing the questions is a combination of schemes developed by Manyatsi (1996), Anderson (2005) and Green and Rollnick (2007). It has two dimensions; information dimension and skills dimension. The information dimension determines the cognitive level to which the learner engages with the information in responding to the question. On the other hand the 53 skills dimension determines the kind of skill that the learner will have to employ in order to come up with the appropriate response to the question. The two dimensions were not applied simultaneously to the question such that the questions were placed in a matrix as developed by Anderson (2005). Rather the two grids are placed parallel so that it can be easy to indicate if a question is reflective of the testing of both a skill and information in one. Items reflecting assessment of both information and skill in one have their marks indicated in the column of information, and an asterisk in the column of the skill reflected along the same row. Items in paper one carry one mark each, so their cognitive level(s) have been indicated with ticks in the columns of the appropriate level. Assessment of communication has not been easy to detect in paper one as candidates only select the appropriate answer. Each of the two dimensions has categories which are briefly described below. 4.3.2.1 Information dimension The framework used in determining the cognitive levels of the examination questions is an adaptation of Bloom?s (1956) taxonomy of instructional objectives, which was later simplified by Bloom et al. (1981). The original scheme had six levels; (recall, comprehension, application, analysis, synthesis and evaluation) which are assumed to be hierarchical.The adaptation made on the scheme for use in this study is that the last three levels (analysis, synthesis and evaluation) are condensed into one level to be described as higher order thinking. These are levels which are not common in time limited examinations of science (Green and Rollnick, 2007) as they require more time. The information dimension of the scheme will therefore have four levels; recall level, comprehension level, application level and higher order thinking. Each one of them is briefly described below. 4.3.2.2 Skills dimension This dimension will determine the skill (practical or cognitive) that the learner will have to employ in responding to the question. According to NCDC (2002), techniques employed in continuous assessment should take into consideration the knowledge and skills at different levels of cognition. The skills referred to here are basically what every 54 human being employs all the time in trying to understand the world around them. However, their implementation has to be informed by scientific concepts in the case of science education (Millar, 1989). The skills are observation, classification, measurement, use of time/space relations, communication, hypothesising, making operational definitions, prediction, inference, interpretation of data, identifying and controlling variables, experimenting and drawing conclusions (Unesco, 1980). In this framework, skills which are closely related have been grouped together, so that it will have only three categories to; identification, communication and higher order skills. An example of the grid showing the classification of items according to what they assess is shown in table 4.5 below  Table 4.5: A scheme used in classifying the items in the examination papers according to what they assess.  It em (20 03 pp 1) R ec a ll C o m pr eh en sio n A pp lic a tio n H ig he r o rd er th in ki n g Id en tif ic a tio n C o m m u n ic a tio n H ig he r o rd er sk ill s Q1  Q2  Q3  4.3.2.3 Description of levels of classification and examples of questions at each level Information dimension 1. Recall level: Questions falling under this category are those which require the learner to state a proposition or to name an object/situation needed in the question. This may mean recalling information which may sometimes not be fully understood. In most cases these questions at level involve what the learner has seen or experienced before. The 55 information needed in them is simply retrieved from the learner?s frame of reference. Such questions may require the learner to respond in a short statement or by giving one word. Sometimes questions in this category require the learner to name a process, object or event or ask the learner to label a provided diagram. A question: ?Name one household substance which is basic.? (1 mark) (2006, pp2 Sec A, Q6 (a). pg 7) requires the learner to retrieve the relevant information from his frame of reference. 2. Comprehension: The questions in this category require the learner to go beyond remembering and stating a piece of information. They need the learner to demonstrate a certain level of understanding the concept in a proposition or word. A question may require a learner to show understanding the information by using it in a familiar situation, like classifying. Describing and explaining are some of the abilities which may be required of the learner by questions at this level. A question: ?What is the importance of chloroplasts in a plant?? (2 marks) (2003, pp2 Sec A, Q1 (b). pg 2) requires the learner to demonstrate knowledge on what a chloroplast is, its function and understanding on how they benefit the entire plant. 3. Application: Questions in this category need the learner to show a much more advanced understanding of a piece of information than that needed in comprehension. A question at this level may require the learner to use a piece of information in a totally new situation. With the knowledge of properties of a substance, a learner may be expected to work out its behaviour when subjected to a certain situation. Questions at this level require the learner to show their understanding of a concept by successfully using it to solve a problem or interpret a situation. A question: ?Why should a clinical thermometer not be sterilised in boiling water?? (1mark) (2006: pp2 Sec A, Q10 (c). pg 11) 56 requires the learner to use their knowledge on features of a clinical thermometer and why the make it incompatible with boiling water. 4. Higher order thinking: Questions at this level may require the learner to work with more than one piece of information at the same time in responding to them. At times they may require the learner to isolate more than one aspect of a given issue, or bring together more than one aspect of a concept in working out the response. One example of a question at this level may need the learner to show the relationship between the structure and function of a naturally occurring object such as an organ or an artefact, such that its efficiency is increased. A question: ?How does cigarette smoking affect the alveolus??(2 marks) (2003 pp2, Sec A Q2 (c). pg 3) requires the learner to bring together several aspects such as structure, function and operation of an alveolus and cigarette smoking in order to show where the precise effect is exhibited. Skills dimension 1. First level: Identification: This level includes the first three skills; observation, classification and accurate naming of the object, factor or situation in question. Questions at this level may for instance require the learner to take a close look at data which may be in the form of a diagram or text and identify some certain aspects of it. The following is a typical example of a question requiring the learner to identify aspects of the data provided. In the following question some data is provided and the learner is asked to identify the subject in question. Which of the following is a male secondary sexual characteristic? A broadening of shoulders B menstruation C development of breasts D enlargement of hips (Q6, 2006; pp1, pg 3) 57 The working out of an answer does not require extensive reorganisation of information. 2. Second level: Communication: This level includes skills such as measuring, comparing, communication, use of time/space relationships and use of operational definitions. Questions at this level may require the learner to communicate the observation made or factor identified in text, diagram or figures (if it entails some measuring). An example can be a question requiring the learner to compare two objects or determine the size of one object. In a question below the learner is expected to translate the given information and represent it in another form. Atoms of lithium and sodium are represented respectively by the symbols: Draw a cross and dot diagram of an atom of lithium. (2 marks) (2003, pp2 Sec A Q4 (b). pg 5) 3. Third level: Higher order skills: This level includes all skills which need the learner to go beyond identifying and communicating information relating to the issue in question. They may need the learner to interpret the data provided. In some cases the questions may require the learner to make some predictions or inferences on the data in responding to them. The following extract of a question requires the learner to translate information from one form to another as well as deriving or making an estimation of the value of the factor in question. (b) The table shows the solubility of substance W at various temperatures. Solubility of W/g per 100g of water 5 7 10 19 24 37 46 Temperature 10 20 30 50 60 80 90 58 (i) On the graph, plot a solubility curve for substance W. (3) (ii) Join the points with a smooth curve. (1) (d) Use your graph to estimate the temperature at which W has a solubility of 17g/100g of water. (1) (2006: pp2, Sec B, Q2 pgs 14 ? 15) The scheme was restructured and reworked according to the suggestions advanced by the validators, and then implemented on the six examination papers which were available. Each of the item in the examination was classified according to its cognitive demand or and skill that it appeared to be assessing. Questions which were at the same cognitive level or assessing the same skill in each paper were put together so that the weight of each cognitive level and skill assessed were determined. The summaries of marks per level per paper are shown in tables 4.6 and 4.7 below. Table 4.6: Distribution of marks over different cognitive levels and skills Paper 1 Paper 2 Year 2003 2005 2006 2003 2005 2006 Cognitive Level Marks (%) Marks (%) Marks (%) Marks (%) Marks (%) Marks (%) Recall 14 (31) 18 (40) 17 (38) 23 (26) 20 (22) 26 (29) Comprehension 10 (22) 11 (24) 12 (27) 28 (31) 24 (27) 23 (26) Application 14 (31) 10 (22) 9 (20) 11 (12) 18 (20) 17 (19) Higher order skills 7 (16) 6 (13) 7 (15) 23 (26) 16 (18) 10 (11) Identification * * * 1 (1) 0 1 (1) Communication 4 (4) 10 (11) 10 (11) Higher order skill * * * 0 (0) 2 (2) 3 (3) Total 45 (100) 45 (100) 45 (100) 90 (100) 90 (100) 90 (100) * Assessment of skills is only implied in some questions (refer to Appendix L). In all the papers of the three years analysed, there is distribution of questions across all the different levels of cognitive domain. The lowest level which require the learner only 59 to recall relevant information, constitute an average of 24% of paper two while in paper one it is up 36%. Questions at comprehension and applications levels together constitute an average of 49% of paper one while it makes up 45% of paper two. Questions demanding higher levels of thinking make up an average of 15% of paper one while they constitute an average of 18% of paper two. Assessment of cognitive skills makes up the bulk of the paper. When considering Section A of paper two alone and working out relative weights of each cognitive level, it turns out that more than 60% of the questions are at comprehension and above. Table 4.7: Distribution of marks over different cognitive levels (alone) Cognitive level 2003 Marks (%) 2005 Marks (%) 2006 Marks (%) Recall 23 (27) 20 (28) 26 (34) Comprehension 28 (33) 24 (31) 23 (30) Application 11 (13) 18 (26) 17 (22) Higher order skills 23 (26) 16 (21) 10 (13) In paper two there is another section which is designed to assess practical and cognitive skills in the context of environment and technology. Questions in this section require the learners to describe and interpret the scenarios presented in the pictures or texts. In some cases questions require the learner to measure or construct representations on the paper. Others demand learners to interpolate some values in graphs, on the basis of the information provided. When considering questions in this section alone, and their weights, there is a gradual increase from 5% in 2003 to 15 % in 2006. 60 4.3.3 Other aspects which emerged from the analysis of question papers The questions appeared to assess more than factual knowledge of science and scientific skills. There are some questions which appeared to be open ended and requiring the learners to make their own informed judgement as well as advancing a justification for it. One example of such a question was the one asking the learner judge the worth of an argument raised in the background information of one question. The question reads: ?Mpho lives in block D where they use water system toilets. She believes that Mapere who is HIV positive will infect her because they share the same toilet.? ?What would be your argument on this matter?? (2 marks) (2003, pp2, Sec A Q1 (d) pg 12). This question gives the learner the liberty to agree or disagree with the argument raised, taking their own stand and giving an informed justification for it. It therefore needs the learner to demonstrate some logic in applying their understanding on HIV/AIDS transmission. One other aspect of the examination paper which became evident as the analysis was going on was an attempt of the question papers to assess the ability of the learners to reconcile science with society. Social and economic dimensions of science tend to stand out in a number of questions. Questions requiring the learners to interpret economic implications of harnessing properties of certain substances for economic reasons also tend to feature in some questions. A question such as ?In Lesotho, water is regarded as ?liquid gold? for the fact that Lesotho gets royalties from selling water to South Africa. Suggest two ways in which Lesotho could use its water to improve its economy.? (2 marks) moves the learner from listing and regurgitating properties of water into using those properties and discerning the socio economic benefits of harnessing such properties. 61 A question requiring the learners to show how the use of a thatched house can help cutting down the expenditure on fuel for warming the house is one example that assesses the ability of the learners to think beyond concepts of science and go into their socio economic effects. Such questions tested the ability of the learners to extend understood concepts of science into arenas outside science. They help to make the learners appreciate the role of science not just as an academic exercise but a socio economic endeavour. The moral dimension of science also featured in some questions. Questions requiring the learners to interpret results of poor waste management practices are also embedded with values such as respect for a clean and healthy environment and proper utilisation of resources. They also have elements of inducing the ability to discern socially acceptable practices and economically viable ones into learners. The three papers analysed also captured assessment of contemporary issues which are science related. These are issues about which every citizen has to be informed. Issues such as HIV/AIDS as an epidemic and the potential that water as natural resource has for the economy of Lesotho are some of the issues that are adequately addressed in the question papers. 4.4 Discussion Analysis of examination papers for content validity revealed that there is an even distribution of marks over all the papers in as far as the three disciplines are concerned. Each of the three disciplines weigh precisely one third of the six papers. Some discrepancies show up when one looks at the weight of specific themes within a discipline. There are some themes which have been heavily and repeatedly tested while there are some which have slightly or virtually not tested over the three years. Heavily tested themes include Reproduction, Nutrition, Diseases and Environment in biology. In Chemistry the heavy testing is observed on Atomic structure, Periodic table and Formation of compounds. In Physics the heavy testing is observed on themes like Current electricity, Pressure, Thermal energy and Waves. On the other hand themes such as Breathing, Locomotion, and Diversity of organisms in Biology, Elasticity of materials 62 and Electrostatics in Physics have been lightly tested over the three years for which the analysis was made. The analysis further revealed that questions are at different cognitive levels. About half of the marks of the two papers are at application and comprehension level. The other half of the marks of the paper is distributed over recall level, high order thinking and skills. There are several implications of this set up which can be pointed out; firstly, teachers are likely to leave out the lightly tested themes, which will in turn affect the learning of themes which require them (lightly tested themes) as prerequisite knowledge. Common phenomena or events which can be explained in the light of those topics will not be easy to interpret on the part of the learners. A phenomenon like lightning which is quite common in Lesotho needs some background knowledge of electrostatics which is lightly tested. The same applies for some infectious diseases which would require the learners to be grounded in the nature and characteristics of certain organisms. 4.5 Conclusion Question papers analyzed turned out to be comprehensive in what they assess; coverage of the syllabus and depth to which themes are supposed to be treated turned out to be quite significant according to the analysis made. Scientific skills, their applications in life and contextualization of science are aspects which question papers appeared to be assessing. According to Bell (2007) knowledge of scientific concepts, higher order science thinking and manipulation of laboratory equipment are some of the aspects which need to be developed during the teaching and learning of science and therefore should be assessed. Their assessment in the papers analyzed is evident. The way the question papers are structured tends to have some positive consequences on teaching and learning, one of which is influencing systemic reforms (Gitomer and Duschl, 1998), especially when they are used as feed forward (Brooks, 2000). Continuous linking of science and society in the teaching of science is one aspect that the papers analyzed induce. Practical work and interpretation of information which will allow for practicing of higher order skills outside examination (Green and Rollnick, 2007) is one aspect which is assessed by the papers 63 and therefore induced into subsequent teaching. Analysis of the papers revealed their validity to the syllabus. They also appeared to be assessing a wide range of science related aspects on top of factual information of science. The next chapter presents the views of the teachers on JC science papers and the ways in which they engage the papers in their practice. 64 CHAPTER FIVE INFORMATION FROM TEACHERS 5.1 Introduction This chapter addresses the last two research questions; ? How do Lesotho Junior Certificate Science Teachers perceive the Junior Certificate Science final examination papers? ? How do Lesotho Junior Certificate Teachers use the Junior Certificate Science past examination papers in their teaching? The information needed to answer these two questions relates to opinions, feelings and lived experiences of people, so it can only be obtained from purposefully selected people (McMillan and Schumacher, 2006). It could not be obtained through observation due to the limited time that was available for conducting the study. Neither could it be obtained from documents, since it was not systematically recorded by the relevant people. I therefore employed two instruments; focus group interview and a questionnaire to get the information. The chapter is organised into three main parts. The first part describes the information yielded by the questionnaire together with its interpretation. The second part describes the information yielded by the focus group interview as realist tales, which are in the form of extracts from teachers? narratives (Eisenhart, 2006). The third part integrates and discusses the findings from both the questionnaire and focus group interview. 5.2 Questionnaire The questionnaire (See Appendix G) was structured in such a way that it asked for the background information of teachers, their use of the past examination papers and their personal feelings towards the examination papers and the influence of the question papers on their practice. The questionnaire was delivered by me to all the six schools where heads of schools and science teachers agreed to participate in the study. A total of 22 JC science teachers completed the questionnaire. The questionnaire established three 65 aspects; demographic information of JC science teachers, use of past examination papers of science and teachers? assessment of JC final examination papers in general. The uses of past examination papers that the questionnaire established were current, foreseeable and potentially viable according to the science teachers. 5.2.1 Demographic information of teachers and their use of past examination papers There were 14 men and 8 women who took part in the study. The majority of teachers who participated in the study are mostly young and well qualified. Most of them are under the age of thirty and hold tertiary qualifications in science. Table 5.1 below give a summary of their ages; Table 5.1: Ages of teachers involved in the study Age Frequency 21 ? 30 13 31 ? 40 5 41 ? 50 4 The majority of teachers had a tertiary qualification with science as a subject of specialisation. Each teacher had two subjects of specialisation so they could handle at least one of the three disciplines of science for JC science with confidence. Tables 5.2 and 5.3 below give a summary of different qualifications and subject of specialisation held by the teachers. 66 Table 5.2: Highest qualifications held by teachers involved in the study Qualification Frequency Level of the qualification Post Graduate qualification (Bed Hons) 1 Post Graduate 1 First Degree: Pure science (with no education) Science with education 6 8 First Degree 14 3 year Diploma in Education 3 year Diploma in Agriculture 3 1 Diploma 4 2 year College Certificate in Law 1 Certificate 1 School Certificate (unqualified) 2 School Certificate 2 Table 5.3: Majors or subjects of specialisation that teachers have The total number of subjects comes to more than 22 as each teacher had more than one subject in their qualification. Teachers had varieties of combinations from Biology, Chemistry, Physics, General Science and other subjects. Major Subject Number of teachers with background in the subject General science 6 Biology 7 Chemistry 7 Physics 6 Mathematics 10 Geography 5 Others: (e.g. Law and Agriculture) 3 67 Eighty per cent of the teachers involved in the study have been exposed to science at tertiary level; they hold a tertiary qualification with science as a subject of specialisation. Out of the 67% of the teachers who hold a minimum of a first degree, half also have education as another qualification. All but one among the teachers who hold diplomas have a teaching qualification as the diploma is an initial teaching qualification. It is a Diploma in Agriculture, which has a strong background of science. Only 2 out of the 22 teachers hold either no tertiary qualification or lack a qualification in education. When it comes to teaching experience, most of the teachers have taught long enough to have seen at least two sets of final examination papers of JC science. Each of the six schools has at least one teacher with at least three years teaching experience. Teachers with three or more years teaching experience make up 59% of the entire sample. Table 5.4 shows the range of teaching experiences of the teachers involved in the study. Table 5.4: Teaching experiences of teachers involved in the study Teaching Experience Number of teachers Less than a year 3 1 ? 2+ years 6 3 ? 5 years 6 More than 5 years 7 Thus the majority of teachers have a minimum of a diploma and teaching experience of more than two years. With the adequate qualifications and teaching experiences of most of the teachers in the sample, their assessment and use of past examination papers which they advance is expected to be informed. They would be expected to have an idea of what science as a subject entails and the requirements of JC science. 5.2.2 Use of the past examination papers in teaching Almost all the teachers involved in the sample indicate that they do use past examination paper in their teaching, starting from different class levels. Some teachers start using past 68 examination papers of science as early as Form A while others only start using them at Form C level. All the teachers except one show that they modify the questions from past examination papers in a number of ways to suit the context or level at which they use them. In making a quick assessment of JC science examination papers, all the teachers except one found the questions appropriate for the level and the syllabus that they are supposed to be assessing. 5.2.2.1 Current ways of using past examination papers The teachers listed ways in which they used past examination papers. They interacted with papers at different parts of their teaching. Phases of instruction and specific uses to which teachers indicated they put past examination papers of junior certificate science are summarized in table 5.5 below. Table 5.5: Ways in which teachers use the past JC examination papers Phase of instruction Specific use Frequency Assessment Revision 4 Testing 14 18 Teaching Teaching 9 Class / group work 9 18 Planning Deriving concepts for teaching 2 Compiling notes 1 3 It turned out that all the teachers who took part in the study do use the past junior certificate science examination papers in a variety of ways. Ways in which the teachers use the past examination papers range from utilising the papers for planning to utilising them for testing. Each teacher was at liberty to give more than one way in which they utilised the past examination papers. The most common use to which papers are put appears to be testing followed by class work, which means that most of the teacher refer to the question papers after treating a theme or unit. Only two teachers indicated that they use the past examination papers to plan or compile notes so that they can work the concepts backwards from the concepts assessed in the examination. The rest of the uses imply interaction with the papers during or after teaching. However in general teachers use the papers equally for teaching and assessment purposes. 69 5.2.2.2 Other ways in which past examination papers can be used The teachers were further asked to suggest ways in which they feel past JC science examination papers could be used to improve the teaching and learning. There were not many suggestions advanced in response to this question. There was an average of one suggestion per teacher. The suggestions that the teachers gave are summarized in table 5.6 below. Table 5.6: Ways in which past examination papers can be used further in the teaching and learning of JC science Phase of involving past papers Specific use Frequency Assessment Practice of answering questions 8 Testing 3 Revision 1 Determining structure of questions 1 13 Teaching Deriving teaching approaches 3 Integrating questions into teaching 3 6 Planning Determining scope and depth concepts to be taught 3 3 The future methods of using question papers that teachers suggest turned out to be more inclined towards using the papers for testing, as opposed to the current ways of using them; uses for teaching balance with uses for testing. Most of the suggestions implied interacting with the papers after teaching has been done. Testing, revision and practice in answering the examination questions, on the part of the students still dominated the suggestions. Only less than 20 % of the suggestions made imply interacting with the question paper on the part of the teacher before teaching. The two big categories of uses of past examination papers; teaching and testing balanced in the current practice, and their use for planning is low in both. In the possible ways that teachers suggest for future, the inclination is towards using the past examination papers for testing. A few suggestions that advocate use of papers for teaching imply the realisation of the potential that they have for supporting and improving learning. Two of 70 the suggestions, for example indicated scrutinising of the papers by the teacher to determine coverage of the syllabus and depth of concepts to be taught. 5.2.2.3 Measures taken in cases of poor performance in questions from past examination papers All the teachers indicated that they used past examination papers with their students. They were therefore asked to give measures they took whenever they noticed that their learners had performed poorly in questions from the past papers. Responses to this part show that other teachers paused for a while and diagnosed the possible cause the poor performance; which implies some reflection on the part of the teachers. Some teachers simply demonstrate what should have been done in answering the questions. There was only one teacher who indicated she went as far back as the syllabus to find out if there is any mismatch between the syllabus and the questions. Each teacher had an opportunity of advancing more than one suggestion, so the suggestions exceed 22. All the ways in which teachers respond to a poor performance are summarized and listed in table 5.7 below. Table5.7: Ways in which teachers respond to poor performance in questions from past examination papers Measure(s) taken in case of poor performance Specific practice Frequency Reflection ? on the response of learners 2 ? on the syllabus versus questions 1 3 Discussions Oral discussions with learners 5 Group discussions among students 1 Revising questions with students 5 11 Moving forward Demonstrating anticipated algorithms 3 Offering further explanations 4 7 Offering opportunities for practice Training students on interpreting the questions 1 Adopting the style and resetting the questions 3 4 71 Responses given to this part of the questionnaire provide clues about the possible effects of the past examination papers on the teaching of some teachers. Only four out of the 22 suggestions imply reconciliation of the questions with the syllabus. The rest of the suggestions imply correcting the mistakes and moving forward. Some of the suggestions made include making the learners discuss the questions once more or demonstrating what should have been done in answering the questions. Real interaction with the questions on the part of the teacher before instruction is implied by any teacher. 5.2.2.4 Ways of adapting questions from the past examination papers for use The majority of the teachers indicated that they do adapt questions from the past examination papers for use in class. They were therefore requested to show the adaptations that they make on the questions. All the adaptations that teachers make on the questions appear are in table 5.8 below. Table 5.8: Ways of adapting questions from the past examination papers of JC science Adaptation(s) made on the questions Specific adjustments Frequency Making the question accessible to learners Paraphrasing and simplifying 1 Rephrasing to make the questions more specific 2 Correcting the poorly worded questions 1 4 Manipulating the surface structure of the questions Deleting labels and asking students to insert them 1 Adding more related sub questions 1 2 Manipulating the content of the questions Changing the whole question and retaining structure 4 Changing the question so that it fits the topic at hand 4 Adding more relevant sub questions to promote thinking 1 Changing values to make the question more difficult 1 10 72 Most of the teachers show that they do adapt the questions from the past examination papers for use. Only 18% of the teachers in the sample show that they do not adapt questions from the past examination papers, they give questions as they are to their learners. All the teachers who show that they adapt the questions have one thing in common; retaining the style of the question even if they change the content or the theme in the question. The changes made by the majority are those which make the questions to serve assessment purposes; additions of sub questions and at times simplifying the questions. The changes only make questions more accessible to the learners. Only two teachers showed that they make adjustments which will make the questions to serve instructional purpose. One indicated that she adds some other sub questions which can be answered on the basis of the same diagram or given background information. The other indicated that he changes the values so as to stimulate more thinking in students, even though he does that more in mathematics. None of the teachers use questions from the past examination papers as instructional tasks that help learners to learn on their own. The rest of the changes put forward by teachers show that they do not use questions for deepening the understanding of students on scientific concepts assessed. They mostly use the questions for assessing the learned content. 5.2.2.5 Effects of questions from past examination papers on teachers? practice Few teachers responded to this section of the questionnaire. This section required teachers to be more reflective on their practice and view it in the light of what is assessed in the final examination papers. Some of those who responded did not give specific answers, but nevertheless did indicate that examination questions and what they assess has some impact on their practice in a number of ways. A few responses which were given and the number of teachers who indicated them are shown in table 5.9 below. 73 Table 5.9: Ways in which teachers feel questions in the examination papers affect their practice Areas of teachers? practice affected Specific effects on teacher practice Frequency Teaching approaches They improve my teaching 1 They make me reflect on my teaching 1 I changed from spoon feeding to facilitated discovery 1 They make me pitch my preparation and teaching higher 3 6 Content for teaching They channel me into their contents 1 They make me teach up-to-date information 1 I sometimes teach according to them 2 4 General effects They give me direction 1 They enable me to help students to handle examination questions 2 3 None They have no effect on my teaching 1 1 The answers show that teachers respond to the questions by altering their practice. Five out of twenty two teachers indicated that they had to improve their preparations and teaching following exposure to questions. Only one teacher was specific enough to show that s/he had to change the approaches to teaching, so that the learners could do more of discovering than absorbing information from him/her. One teacher among those who responded to this part on effect(s) of questions on teaching indicated that he had to be on the outlook for current affairs that related to science. 74 5.2.2.6 Additional information for the study The last part of the questionnaire was more open ended than the other questions. It requested the science teachers to add to any subject which they felt was relevant to the study as a whole. Not many responded to this question. The four who responded to this question made suggestions and remarks which indicated that they understood and realised what the study was all about. They made relevant remarks and suggestions, although one of them (the first in the list below) is a big research study on its own. The second and the third relate to the question papers directly though they are beyond the scope of this study. The second was also highlighted by one of the teachers in the focus group interview. The last one also, does not relate to examination papers directly, but worth noting. The remarks, suggestions and further areas of research as given by teachers are listed below. Additional information to the study as given by the teachers ? Performance in junior certificate science in general. ? Questions in examination need to be in simple English ? Examinations are too practical oriented and therefore disadvantaging the under resourced schools ? Study materials need to cover more scientific aspects The responses of the teachers reveal that there is considerable use of the past examination papers of JC science for a variety of purposes. The teachers also adjust questions from the past examination papers in number of ways in order to suit their needs. Some of the adaptations that teachers make on the questions from the past examination papers and on their teaching reveal another a certain mismatch between the papers and the attainment of the students. The details of the mismatch are given in the discussion at the end of the chapter. 75 5.3 Focus Group Interviews Focus group interviews were conducted with Junior Certificate Science teachers in three out of the six schools in which teachers agreed to participate in the study. The focus group interviews were designed to get information relating to the in-depth assessment of the JC science papers from the perspective of the teachers. The interviews were conducted at different times with the teachers in the three schools. The questions which were posed to the teachers were the same in each case. The three interviews have not been treated as different cases which investigate one subject and reinforce one another (Borman et al., 2006), but rather as one big population consisting of three parts. Neither is the motive to compare them, nor to go quantitative in the study. They have been analysed together and regarded as another source of information, which complement the questionnaire and question paper analysis. Codes given to teachers in the focus group interview have been derived from the abbreviations of the pseudonyms given to schools (see page 38), e.g. FHS2 is teacher 2 in Foothills High School. Coding of data from the focus group interviews Organisation of data from the focus group interview was approached with three initial typologies which were based on parts of the information that I wanted to get (Hatch, 2002); what examinations of JC science assess, cognitive levels questions in the examination papers and the use to which teachers put past examination papers. Two other clusters emerged, as the organisation of data went on, so the scheme of organisation ended up with five clusters altogether, each with subdivisions. 5.3.1 Nature of question JC science papers and what they assess The first typology which was derived from the first research question as well the unfolding of interviews related to the general nature of the JC science examinations and the perceptions that teachers had on them. The subdivisions of the cluster appeared to be what the papers assess, coverage of the papers and cognitive levels of questions in the papers as seen by the teachers. 76 5.3.1.1 Aspects assessed by JC science papers The feeling that JC science papers assess only factual knowledge of science and skills featured in the responses given by the teachers from the three schools at the beginning of the interview but gradually faded. The teachers listed several aspects which the JC science papers assessed. One of the anticipated responses to this question would be knowledge of facts of science, ability to use the knowledge in solving the problems; hence the elaboration specifying what is likely to be assessed. In response to a question about what the papers assess, responses from the three schools came out as follows: Teacher 1: ?I think of late, final examination papers, they are really testing abilities and values, just a little is covered under knowledge.? (HHS1) Teachers 1and 2: ?Skills, Practical skills.? (LHS1 and LHS2) Teacher 1: ?Mostly, I would say mostly they assess ehm ..., I would say paper one assesses knowledge, but paper two has something to do with application.? (FHS2) Teachers did not have samples of the past examination papers, so I provided a sample of questions from the past examination papers of JC (see Appendix I). After looking through the sample of questions provided, teachers indicated that almost all multiple choice questions assessed factual knowledge of science, but in different contexts and disciplines. In classifying the questions according to what they assessed, the following responses came up: Teacher 2: ?The first question assesses knowledge in physics, the second question, I think it also assesses their knowledge, but in chemistry. The third, hmm ... also knowledge, but the last ... it assesses the awareness of the student; how he can relate the subject to everyday life,? (LHS2) 77 While one teacher in another school indicated that; Teacher 2: ?Yes, I think it assesses knowledge, basically if a student doesn?t know this energy, there is no way he can answer the question, therefore it is just recalling knowledge? (FHS1) A follow up answer by another teacher was: Teacher 3: ?Question three somehow needs some application, because it shows that Mpho cannot see well, then if a student knows the types of eye defects and what they really mean then he can apply his knowledge.? (FHS1)  The responses show that teachers do take time to assess the questions and try to think about what the examiner may have been assessing with them (questions). Teachers are also aware that questions are not of the same cognitive level in the examination. 5.3.1.2 Coverage of the question papers over the syllabus The second aspect that was initially anticipated and clearly isolated by the teachers was an aspect of coverage of the syllabus by the examinations. The teachers addressed this aspect of coverage at two levels; level of discipline and level of specific topics within the discipline. Teachers expressed awareness of the fact that papers of JC science were no longer biased towards any discipline, but rather balanced and have marks evenly distributed over the three disciplines which constitute the syllabus. However when it comes to topics within a discipline, there were some discrepancies noted. In responding to the question on whether questions in JC examination papers matched the objectives of the syllabus, teachers from Foothills High School said: Teacher 1: ?They do, they really do.? (FHS1). Interviewer: Why? Do you recall any question that you can say this one was just at the level of the syllabus, not above, not below? 78 Teacher 1: I am saying this because as teachers sometimes when we just look at or turn the question paper we may say this is beyond or under the syllabus, but when we sit down with other teachers and try to look closely into the questions you find that we don?t do the syllabus to the level we are supposed to; either we do a little or do more. I don?t have a specific example unfortunately.?(FHS1) While in Highlands High school the response to the same question was: Teacher 2: ?On my opinion, I think they match the objectives from the syllabus, but they are set up in such a way that it should not be something that can easily be observed, because they are not in the form of knowledge or recall type of questions; basically an application, so if one manages to observe or come across the application, then you will see that some of the questions are really covering the syllabus? (HHS2). In addressing the question on weaknesses of the examination papers, the following responses came up: Teacher 2: ?... You will find that there are some objectives or topics that appear in the syllabus, but sometimes they don?t even assess that ...? (LHS2). The above responses show that teachers do take time to check alignment of the syllabus and examination and note some discrepancies. They were able to show that coverage of the examinations over the syllabus was generally good, even though it can not be said to be perfect. The above response should have been probed further, to get the teachers? reason for describing limited sampling as weakness in the papers. 5.3.1.3 Cognitive level of questions in the examinations One aspect of the nature of papers which teachers brought out was the cognitive level of questions in the examination papers. Though teachers in some schools were inclined to take questions in the examinations to be at recall level, they gradually changed and began to show that there are some questions which really needed the learners to tap into higher levels of cognitive domain and apply their knowledge of science facts. In identifying the 79 strong points of the JC science examination papers, the following descriptions were advanced: Teacher 1: ?Actually, there has been an improvement, because compared to the previous way the question paper was set, we find this one well improved, because here the three disciplines are covered in the same percentage, the three disciplines that we have.? (FSH1). Teacher 2: ?Like some questions in paper two, may require them to explain in terms of experiments; so that requires learners to use their knowledge most of the time.?(LHS2). Teacher 1: ?From what I have seen, paper one questions are multiple choice. They require recalling; to recall knowledge but paper two questions, they also sometimes require students to think deeply.? (LHS1). Teacher 2: ?Most of the time they have got challenging elements. You?ll get a question that might come challenging you to try to get more information.?(LHS1). Teacher 4: ?Another strong point that I have observed is that science does not focus itself into Biology, Physics and Chemistry; there will be a question on technology, there will be a question on environment which I think is very, very meeting the needs of today? (FHS4) Teacher 4: ?Another strong point I have realised is that for example, paper one questions don?t require the learner too many things in one question; they don?t require the student to convert and calculate, ke hore (which means) at the same time.? (LHS4). The above responses which are picked from interviews with teachers in different schools are indicative of the fact that teachers are able to see beyond the science knowledge involved in the question. They are able to see into the aspects that are assessed in the context of the questions. The teachers were able to identify questions requiring scientific skills, application of scientific facts in responding, and questions requiring learners? ability to link science to life outside the classroom. HHS1 was able to unpack application 80 of the knowledge of the concept of energy. For example in describing what he thought JC science papers assessed, he indicated that: ?I think of late final examination papers; they are really testing abilities and values, just a little is covered under knowledge.? In a follow up on the same issue he said: ?I think of a question that once asked the Form C?s on energy conversions; the student is exposed to a question whereby somebody is standing below the tree and then the apple has fallen down to the ground, and is asked to describe the energy changes that take place when the apple falls down from the tree. So I think over there it is not a matter of what type of energy changes, but are they able to tell that when an object moves from a certain point to another, there are energy changes over there. They are not asked to define the forms of energy.? In a quick analysis and classification of a sample of questions (see Appendix I) I provided during the interview teachers were able to pull out at least two aspects which were assessed by each question. They successfully identified discipline and the theme from the question was picked, skill or mental activity to be employed in working out the response. FHS1 for instance managed to pull out application of knowledge which was embedded in a question on eye defects (see Appendix K). HHS1 also showed that questions had an element of ?? how what has been learned can be applied in a real life situation and hence the essence of doing that.? Assessment of applications of science concepts to life situation was implied in responses given by teachers in all the three schools. 5.3.1.4 Ways in which teachers use past examination papers in their practice The interviews conducted further revealed that there is substantial utilisation of the past examination papers by teachers in their practice. One area in which teachers use past examination papers is in testing. However the past examination papers are only a secondary source of questions. The primary sources of questions for testing are schemes of work that teachers prepare at the beginning of the terms and the instructional objectives that guided the lessons conducted. It is only where questions in the past 81 examination papers overlap with objectives or schemed content that teachers can utilise them (past examination papers). Teachers indicated a variety of ways in which they utilise the past examination papers in their practice. Others select questions and use them as they are, others select and modify the questions while others adopt the style in the examinations and frame the questions for the objectives they want to assess. The following extracts are some of the responses advanced by the teachers to a question on how they design the tasks for formative purposes. Teacher 2: ?Normally what I do is I check on the style of asking questions from those who are setting questions. I try to have my own questions in the same style as that one being used in the examinations papers, but at times I pick some from the question papers.? (HHS1) Another teacher in one school said; Teacher 3 ?... based on the objectives, I base myself on the objectives. That is where the weakness may be because when I look at the objectives I may not go to the depth of applying those or level where I teach them to apply those. When I look at the objective, I may say this is what the syllabus says ...? (FHS1) Teacher 1: ?We look at our schemes, what we have schemed for. For the first session examination, we look at what we have schemed for the first session. We may use the question papers, but use only questions related to those topics; what we have taught.?(LHS1) Teacher 2: ?We take past exam papers plus our own that we have set ourselves for internal examinations; I think our motive there is to give students practice of examinations and also broaden their horizons in terms of answering those questions.? (LHS1) According to the teachers, objectives initially stated before teaching remain the sole determinant of what teachers assess in their formative tests. In some cases teachers use the past examination papers to get the style used in constructing the questions, so that they can apply it to the content that they have covered. In some cases, teachers select the 82 questions relating to what they have taught and then modify them so that they can be understandable to the learners. In cases where the content has been covered to the high level (according to the teacher), questions from the past examination papers are given as they are, without any adjustment. There is therefore some substantial utilisation of the past examination in the preparation of assessment tasks by the teachers. 5.3.1.5 Effects of the past examination papers on the teaching of Junior Certificate science The focus group interview further showed that contents and structure of JC science examination papers does have an influence on practices of teachers. They affect what teachers select and teach. Different teachers from the three different schools expressed the ways in which they felt they have been influenced by the final examination papers. To some teachers, contents of examination papers have affected their approaches to the teaching of science, to others the effect has been on ways and means of going beyond what is stipulated in the syllabus. The following extracts show the ways in which teachers feel they have been influenced by JC science examination papers: A teacher who felt that question papers affected the level to which he treated themes gave the following response: Teacher 2: ?They do influence the way we teach, because in some cases you find that you would teach not knowing exactly where to end, but after looking the question paper you find that questions are asked to this, so I cover up to a certain level, which without looking at the question paper I would not. Looking at the syllabus alone I would not go to higher levels. After realising what questions are asked, I change teaching.?(HHS3) The above response shows the effect examination papers on both the depth of treating material and the manner of teaching. The same response was also indicated by more than one teacher in the questionnaire. A teacher who felt that the themes that she selects for scheming have been affected by the question papers gave the following response; 83 Teacher 1: ?... you would find that there are some objectives or topics that appear in the syllabus, but sometimes they don?t even assess that. So as teachers sometimes we end up not teaching those topics, because they don?t assess those topics ... they tend to influence the teaching.?(LHS2) The skewed sampling of question papers was also revealed by the analysis of question papers. There are some themes which have hardly been tested over the three years for which question papers were available. Another teacher who felt her selection what she teaches has been affected by what features in the examinations gave the following response; Teacher 2: ?... it depends on what type of a teacher, sometimes I am... maybe I am a lazy teacher, I just look at the way questions have asked, the questions that seem to appear more often and I just tell myself that this topic, I?m sure it is going to be there, then I stick there, I leave the other ones.? (FHS1) The science question papers have not only affected what the teachers teach. They have also affected how the teachers teach and assess. One of the teachers had the following response to give: Teacher 1: ?We teach and assess the students according to what we have seen in the in them (past examination papers)? (LHS3) While another one indicated that; Teacher 1: ?I try to have my own questions in the same style as that being used in the examination papers.?(HHS1) From what the teachers have indicated, access and use of the past examination papers have both productive and counterproductive influences on the teaching and learning of science. In-depth treatment of some topics is one good aspect that teachers have pointed out. HHS3 indicated that it was only after looking at the question paper that it became clear that there was a need to go deeper (than what the syllabus specified) in treating the topics. Among the counterproductive influences of JC science examination papers, there 84 is also narrowing of learners to the requirements of the examination in some topics, which deprives learners of some background to the other topics of senior secondary science, even for some which are treated later in the junior secondary science. 5.3.2 Other Issues which emerged from the focus group interviews There are two other important clusters of information which were not part of the three initial typologies that emerged. These relate to technicalities of construction of question papers and other new trends in the examinations of JC science. 5.3.2.1 Technicalities of construction of question papers There are two basic aspects that teachers indicated under construction of JC science examination papers. They relate to the specificity of questions in paper one and organisation of options in it. They described these as strong points of paper one. In highlighting the strong points of JC science papers one teacher indicated that: Teacher 2: ?Another strong point I have realised about them is that for example paper one questions don?t require too many things in one question. They don?t require the learner to convert and calculate, ke hore (which means) at the same time.?(FHS2) In the same response the same teacher highlighted the organisation of options in an increasing order (if they are numerical values) as one strong point of the examinations. He felt that this kind of arrangement minimises confusion on the part of the students. On the same issue of multiple choice questions one teacher indicated one of their weaknesses as having no key among the options provided. In his own words he said; Teacher 3: ?I think one weakness might be under multiple choice. One may find that some of the distracters, over there ... I mean there is no correct answer.?(HHS3) 85 The other aspect which relate to technicalities of construction of question papers may feature in to both papers one and two. This relates to language and structure of questions. One teacher in one school indicated the following: Teacher 2: ?I think what I have realised is that some questions are too long, they are wordy, rowdy; they are not so straight forward and they are sometimes confusing.?(LHS2) While the other one said; Teacher 1: ?In some questions students are given the diagrams, but the diagrams are not that clear to be related to the question.?(LHS1) One other general feature which applies to both paper two and paper one takes us back to the issue of cognitive levels. More than one teacher implied the realisation that questions are not all of the same cognitive level, though it appeared rather late in the interviews. One teacher indicated that: Teacher 4: ?I think one strong point is that sometimes ... they usually have a mixture of simple questions and some of them are ... ehmm ...., the levels of questions; some of them are high level questions. Some require the students to think and apply their knowledge most of the time.?(LHS4) 5.3.2.2 New trends in Junior Certificate science examinations The teachers who were interviewed isolated another aspect of the question papers which was not catered for in the initial clusters of information. This relates to the new trends that question JC science papers seem to have taken. The teachers, especially those with more than ten years teaching experience were able to pick the big difference between the previous mode of setting the science paper and the current one. The other trend which teachers identified as being noticeable is contextualisation of facts of science in the question papers. The questions are asked in such a way that they require the learner to reconcile science and life outside the classroom. The examinations also have the 86 questions which tap directly into the scientific and practical skills. The following extracts of teachers? responses express their awareness of the trends indicated above; In commenting on the general content of questions in the JC Science examination papers, the teachers indicated the following: Teacher 1: ?... science does not focus itself into biology, chemistry and physics; there will be a question on technology, there will be a question on environment ... which I think is meeting the needs of today.?(FHS1) Other teachers indicated that; ?... they require a student to see the importance of the topics in our real life situation; how what has been learned can be applied in real life situation and the importance of doing that.?(HHS1) ?... the awareness of the student; how he can relate the subject to everyday life.?(HHS1) One other aspect which teachers had observed and isolated is the repetition of questions in examinations which seemed to occur from year to year. In one school a teacher had perceived this repetition as a weakness which limited his students. The observation came up again in another school, but one teacher was able to explain its purpose outside the interview. He briefly indicated repetition as being determined by the national performance on it, as well as the number of years that a question has to spend in the usable item bank. 5.4 Discussion The questionnaire established the background surrounding the use and perceptions of teachers of JC science papers. This was achieved through establishing factors such as teachers? demographic information, manner of use of the past examination papers, class levels at which teachers start using the past examination papers and ways in which teachers adapt papers for use in their classes. The questionnaire was concluded by asking 87 for teachers? general assessment of the JC science papers. The focus group interview established the teachers? specific opinions on the question papers by asking them to be precise on what they feel papers assess, the nature of the papers, and how they use papers in their practice. It was concluded by asking teachers a question that required them to be more reflective on the question papers and their (JC science papers) precise effect on their practice. The teachers? demographic information revealed that the majority of them had a minimum of a diploma qualification and at least two years of teaching experience. With their tertiary qualifications which are coupled with science subjects as their specialisations and teaching experience they are presumably grounded in requirements of learning science at both levels (senior secondary and JC). These two aspects inform the assessment that they make on JC science papers and the use to which they put such papers. Information from questionnaire revealed that there is significant use of the past examination papers in the practice of JC science teachers. All the teachers indicated that they use past examination papers in their teaching. The views that teachers have on examination papers influence the manner in which they use such papers in their practice. The uses of past examination papers ranged from planning to assessing. This therefore implies utilisation of the papers at almost all stages of teachers? practice. The most common uses appeared to be assessing and teaching, though there were a few who used the papers for planning, which meant that most of the teachers interact with the papers before and after some teaching has been done. What also surfaced during the interview was that, teachers sometimes derived the style of constructing questions for tests from the past examination papers. There were only a few teachers who showed interaction with the papers before teaching. On teacher indicated (during the interview) that his teaching was once informed by the depth to which the concept had been treated in the question paper. This implies a slight mismatch between what the syllabus had specified and what the teacher observed in the examination paper. Two other teachers also indicated in the questionnaire that they utilised for planning and making notes. The spontaneous answer that almost all teachers gave in the questionnaire, on the question on the level of question papers was that they were appropriate. Even in the 88 interview, they were quick to say question papers matched the objectives of the syllabus. However, their responses to further questions on effects of question papers and adaptations they make on questions from the papers revealed a different story, as it will be shown below. Teachers indicated a number of adaptations that they make on questions from past examination papers. These adaptations included manipulations of content and structures of questions as well as language of the questions. One teacher indicated that ?? we modify the questions from the examination papers to make them a bit simpler for students.?, while two teachers indicated in the questionnaire that one adaptation they make to questions from examination papers is rephrasing to make them clear and specific. I presume that the act of simplifying is determined by the class level at which the teacher decides to use the questions. 10 out of the 22 teachers involved in the study showed that they used the past examination papers at Form A level, while 8 showed that they used such papers at Form B level. Adaptations made on the questions from the past examination papers are reflective of a higher level at which they are perceived to be. Most of the adaptations imply simplifying or bringing the questions down to the level of the learners. Only one response out of the thirteen adaptations suggested which implied vertical adjustment, meaning using the questions to move learners to a higher level of understanding the concept. Most of the adjustments are either lateral adjustment or vertical adjustment which brought the question down to the level of the learner. Among the types of lateral adjustments indicated, there was deleting of labels and making the learner insert them or changing the object in the background information, but still retaining the level of the question. In some cases the adjustments included changing the contents of the question and retaining the style or the level of the question. Reactions to question papers and effects of question papers on teachers? practices also revealed another perception which does tally with their appropriateness which was indicated earlier. Some of these were reflected in the measures taken in cases of poor performance in questions from past examination papers. The majority of responses to this part of the questionnaire implied going back and reviewing either the interpretation of the syllabus, explaining, or demonstrating to the learners what they should have done in 89 responding to the question. Responses that teachers advanced indicate offering further explanations which may mean re-teaching with some additions. The depth to which concepts had been treated may have not matched the level to which the question had been pitched. All these measures indicated acknowledging of some mismatch between the level of the questions and the content taught or approaches followed when teaching. Adaptations made on the questions and reactions to poor performance of learners in them are all indicative of the mismatch between the syllabus and the examination papers. Simplifying of questions before use and re-teaching in cases of poor performance on questions from past examination papers are all indicative of this mismatch. The responses given by the teachers showed that they were aware of a number of factors which should go into the construction of a question paper. In their responses, they were able to pick assessment of skills in the context of the themes of the subject. They also managed to show that questions are supposed to be at different cognitive levels. None of the teachers in the three schools had any question paper with them during the interview. All the same, they were able to recall aspects of the JC science examinations which were worth reinforcing and those that needed attention. The teachers indicated awareness of the ideal alignment which should exist between the syllabus and examination. They showed that it is there, although there are a few gaps which they indicated when it comes to coverage of the examination. Teachers? assessment of examination papers revealed acknowledging another aspect; which are some discrepancies in the content of examination papers. Repetition of some themes, omission of others in the examinations, ensuring balancing and different cognitive levels of questions are some of the features of JC science examination papers which have had some impact on teachers? practices. Teachers noted that there are some themes which are not assessed, so they end up not teaching such topics. To some teachers the impact has been on teaching approaches, while to others it has been on the selection of what should be taught. Responses such as ?they help me to pitch my teaching and preparation higher? and ?they help me to reflect on my teaching? are reflective of the positive influence of examination papers on teachers? practices. One teacher indicated 90 that following the exposure to examination papers, s/he decided to change from transmission model of teaching to facilitated discovery. The teachers were able to isolate some of these consequences. One aspect that teachers pointed out which is likely to have a negative impact on their practice is the coverage of examination papers, as indicated above. According to teachers, there is a significant repetition of some questions over years which goes together with omission of other themes in the examination. The likely influence of this aspect is that teachers may skew their planning and teaching towards the repeatedly assessed themes and leave out the lightly assessed themes altogether. It may however not be accurate to say that a theme is not assessed for some years and then leave it out, as it may be embedded in the background information of another topic which may be assessed at a higher level. Two themes which appeared to be lightly or virtually not assessed according to analysis are electrostatics and diversity of organisms are prerequisites for current electricity and infectious diseases respectively, which are two themes that are heavily tested. The perceptions and hence the use of past examination papers did not only reveal consequences that are likely to be negative. There are also positive consequences that teachers have implied in their responses. There two positive consequences of the examinations that can be pointed out are contextualisation of science and the depth to which topics seem to be treated in them. According to some teachers JC science examinations no longer focus purely on factual knowledge of science. They have been extended to include aspects of environment and technology. Teachers acknowledged this aspect as being worthwhile. Science is no longer as divorced from life around the students as it used to be. The positive influence of this aspect is that, teachers will teach science in such a manner that they keep on life outside the classroom and ensure reconciliation of the two. It will stimulate teachers to adopt constructivist approach to teaching. The learners will also learn science in the light of its concrete and potentially usable nature. The indication of the depth to which themes are to be treated is another positive influence of the examination papers on the practice of teachers. It was implied by more than one teacher, so questions in the JC science papers also served as a guide to the depth to which teachers may treat the concepts. 91 The cyclic nature of curriculum as explained by Bernstein (1975) and Porter (2006) is reflected in the operation of JC science curriculum. The content and structure of JC science examination papers does feed back into the implementation of its curriculum. According to what the teachers have indicated in the questionnaire and interview, they do reflect on their practice following exposure to question JC science question papers. They make adjustments on their practice in the light content and structure of question papers. The questions in the JC science papers are set in such away that they are at different cognitive levels when analysed in the light of Bloom?s taxonomy. This therefore stimulates the teachers to explore some means of helping the learners to attain higher order thinking skills. Use of the past JC science examination papers and the influence they bring about on teachers practice can also viewed as a relationship between the community of practice and its respective resource. JC science teachers, as part of a bigger community of practice engage in a variety of practices; planning, instructing and assessing. They are further influenced and sometimes constrained by several other factors, such as demands of the syllabus and examinations, expectations of society and their own perceptions towards these factors. In the process of critically analysing the examination papers and improving their practice in the light of the requirements of the examination papers the teachers can be described as working their way out of constraints imposed by examination papers. The next chapter concludes the study by capturing the findings of study as revealed by the three instruments; analysis of question paper, questionnaire and interview and critical reflections on the study as a whole. It further shows ways in which past examination can be utilised and recommendations for further research in classroom assessment. 92 CHAPTER SIX CONCLUSIONS AND RECOMMENDATIONS 6.1 Introduction This chapter brings together the findings of the study, discusses them and shows further uses to which past examination papers can be put as well as showing further research on the subject. This study investigated the effect of summative assessment on teaching in the Lesotho Junior Certificate science, so the research questions that the study attempted to answer were: ? What do Lesotho junior certificate science final examination papers assess? ? How do science teachers perceive the questions in Lesotho Junior certificate science papers? ? How do science teachers use past junior certificate science papers in their teaching? This relationship was determined through establishing the nature of JC science final examination papers, views of science teachers on them and the ways in which they use them (examination papers) in teaching. Data collection for the study therefore involved analysis of JC science examination papers themselves, interviewing teachers as a group, as well as asking them to complete a questionnaire. The interview and the questionnaire revealed part of teachers views and practices relating to JC science examination papers, while the analysis of question papers revealed what they actually assessed according to me. The interview provided me with an idea of what the teachers do as a department, while the questionnaire showed each individual teacher?s perception and ways of employing the past examination papers in practice. On the other hand the analysis of examination papers revealed the structure and content of the JC science examination papers themselves. The influence of the examination papers was therefore inferred from the views of teachers on them (the papers) and the changes they made in their teaching following exposure to examination papers. 93 From the sample and methods used in investigating the influence of assessment on implementation of JC science, the following revelations have surfaced from the study: ? Junior certificate science papers are quite comprehensive in what they assess; they assess scientific knowledge, scientific skills, applications of science in improving lives and managing the environment. They tap into higher levels cognitive domain. ? Junior certificate science teachers find questions in the papers valid and appropriate for the learners and the syllabus for which they are designed. They also find the questions in them challenging in terms of contextualization and depth to which they test scientific concepts. ? Teachers use questions from past examination papers in a variety of ways. They adapt and use them for teaching, testing and revising. They also derive the styles of constructing their own tests. In some cases the teachers apply questions as they are in testing. The analysis of question papers and outcome of the interview revealed that the question papers are broad in what they assess. They assess a variety of skills in the context of science concepts. They also tap into different levels of the cognitive domain. They are also valid for the syllabus, although there are some gaps in the alignment. Teachers find questions in the papers appropriate for the learners and the syllabus so they modify the papers and put them to a variety of uses. There was a tendency to take multiple choice questions as low level questions on the part of some teachers. What also surfaced was that structure and content of examination has some influence on the teachers practice. Teachers indicated some of the changes that they effected in their teaching following exposure to examination papers. The first part of the chapter considers the nature of junior certificate examination papers from the perspective of the teachers as well as per analysis as done by me. It further gives an interpretation of the uses to which teachers put the papers once they have served their role of summative assessment. The second part shows the implications of teachers? perceptions on questions from past examination papers their practice. It goes further to 94 reinforce some of the uses to which teachers put the past examination papers as well as highlighting other practices that can help to make the past examination papers more fruitful at classroom level. It is concluded by critical reflections on the study as a whole, recommendation for further research on the issue. 6.2 Nature of JC Science papers Analysis of examination papers revealed a number of aspects; items in the examination papers are generally valid for the syllabus although there are small gaps in the alignment, items are of different cognitive level and scientific knowledge and skills are assessed in the context of life around the learners. This issue of contextualisation was further indicated by more than one teacher in the interviews. There is some alignment (Porter, 2006) between the syllabus and items in the JC science final examination papers according to the teachers and analysis conducted on the papers. Teachers have indicated in both the questionnaire and interview that questions in the JC science final examination papers match the objectives of the syllabus. They were able to isolate a number of aspects assessed by the JC science examination papers. These included environmental education, technological literacy and scientific skills. They further showed that questions were of different cognitive levels. There is therefore some alignment of the planned and assessed curriculum according to the teachers as implementers of curriculum, though it is not total as discussion will show. Only one teacher found the questions easy for the students. The manipulations that teachers make on questions from the past papers questions show that the questions are slightly above what teachers feel learners can do. Most teachers feel the need to simplify the questions for classroom use. Analysis of the question papers showed that items in them are valid in terms of syllabus requirements and context. They assess scientific knowledge as specified in the syllabus in the context of life issues around the learners. However, validity for the syllabus revealed some problems in the alignment between the syllabus and examinations. There was over testing of some topics and under testing of others. This issue of skewed testing was also 95 picked up by some teachers who went further to show that they end up leaving out themes that are not frequently tested. Analysis of question papers showed that topics such as electrostatics under physics, particulate nature of matter under chemistry and diversity of organisms under biology are not strongly tested. Analysis also revealed distribution of questions over different cognitive levels. According to NCDC (2002), science teachers are expected to develop attitudes, knowledge and a variety of skills in learners during the teaching and learning of science. At the same time they are expected to monitor this development. In monitoring this development, teachers are also expected to take into consideration the different levels of cognition (Anderson, 2005). The papers analysed assess a wide range of aspects; scientific skills, science, applications of science to real life situations, and interpretation of environmental issues in the light science concepts. A quick assessment of the papers made by the teachers showed that questions are of different cognitive levels. They could not specify the proportions of marks at each cognitive, but they could recall that cognitive levels of questions differ. According to the analysis I conducted on the available question papers, approximately half of the marks carried by each of the question paper are at application and comprehension, while approximately 20 to 30% are at recall level and 20 to 25 % is at higher order thinking (see tables 4.3, 4.3 and 4.4). Despite the fact that teachers were able to pick up that questions are of different cognitive levels, some tended to slip into seeing multiple choice questions as low level questions in their own assessment. This happened despite there being many excellent examples of multiple choice items testing higher order skills (Lewin, 1992). Responses from teachers in two different schools implied that it was only in paper two where learners were supposed to think deeply in responding to the questions, implying that all multiple choice items were low level questions. In commenting on the cognitive levels of questions, responses from different schools indicated that the section that could be said to be cognitively demanding was Section B (of Paper 2), while another said, ?From what I have seen, paper one questions are just multiple choice. They just require recalling; to recall knowledge, but paper two questions; they require students to think deeply ?? Analysis of question papers and 96 responds from teachers also revealed a considerable degree of contextualization. Testing of the ability to relate science to life around the learner (Solano-Flores and Nelson- Barber, 2001) appears to be adequately addressed in the examination papers. Examples in the background information of questions are what each learner has at least heard about. The analysis of question papers did not reveal much about the assessment of attitudes. This may be due to the style of questions in them and the broad syllabus that may not allow for delving into assessment of one topic. The structure of questions papers and the scope of the syllabus from which they have to be sampled does not allow for comprehensive assessment of attitudes. Paper two questions which require free responses hardly go to the level of asking write a paragraph. The assessment of attitudes can only be done in a superficial manner. It was lightly implied in questions requiring learners to make judgments about certain practices portrayed in the some questions and their probable impact on the natural environment. 6.3 Uses of past examination papers Past examination papers constitute an important resource at the disposal of science teachers as a community of practice. Their use in practice can affect other practices to the detriment or betterment of the entire operation. They form part of the assessed curriculum which closes the curriculum cycle (Taylor et al., 2003) thus feeding back into the entire system (Bernstein, 1975 and Richards and Taylor, 1987). Teachers? responses on the uses of past examination papers were reflective of the influence of these papers on the enacted curriculum. From what the teachers have indicated, there is substantial utilisation of the past examination papers in science teachers? practice. The information revealed by the questionnaire showed that all the teachers in the sample use past examination papers in their teaching of science. They are used in a variety of ways from different class levels. Uses range from planning to testing. There is a balance between the use of papers for assessing and for teaching (see table 5.5). The use of past examination papers does not dominate their practice, so the danger of drilling for examinations or teaching to the 97 examination (Doran et al., 1994) with the past examination papers is not strongly implied. What is strongly implied is that teachers do take their time to carry out teaching that is informed by the syllabus objectives. Use of past examination papers only supplements the teaching and testing. Objectives from the syllabus inform the teaching and construction of formative tests. The times and ways in which teachers use the past examination papers leave room for giving feedback and hence improving where weaknesses show up before the actual summative assessment. Some teachers start using the past examination papers as early as Form A while others start using them only in Form B. 6.4 Effects of past examination papers on science teachers? practice JC science teachers, as a small community of practice responsible for implementation can be constrained by the demands imposed the planned and assessed curricula as well as expectations of parents and school administration (Doran et al., 1994). Parents expect their children to pass, while the planned curriculum requires development of critical thinking in learners. They therefore have to blend all these into their practice to meet all the expectations. Examination papers also have their effects on the practice of teachers. According to what teachers have indicated, JC science examination papers affect their practice what they teach and how they teach it. As a constituent of assessed curriculum, JC science final examination papers complete the learning cycle and exert an influence on planned and assessed curriculum. The influence that they exert is reflected in the views of teachers on them and the changes that they make in their practice. Effects of past examination papers on teachers? practice surfaced on three areas; content schemed for teaching, approaches to teaching and assessment strategies. The content and structure of JC science examination papers do have an effect on what teachers select for teaching. According to some of the teachers, there are some topics which they end up not teaching, following exposure to past examination papers. As teachers responses and analysis of examination papers have shown, the alignment between the planned and assessed curriculum was only considerable, but not perfect. There were a few discrepancies regarding sampling of subject matter which could be 98 noted. Analysis of question papers together with observation of some teachers showed that there were some themes which were heavily tested while others were lightly or virtually not tested at all. Heavy testing was noted on topics such as nutrition and current electricity. The negative effect of this aspect is inducing the teachers to leave lightly tested topics when scheming. The teachers could not give examples of themes which were not assessed in the limited time of the interview, but they could remember that examinations could not cover certain themes. LHS3 indicated that ?? you find that there are some topics or objectives that appear in the syllabus but sometimes they don?t even assess them.? Analysis of question papers also showed that there has been a little or virtually no testing on themes like Electrostatics in Physics and Diversity of organisms in Biology. The question papers of the three years analysed show some low content validity. It may however not be wise to be influenced by the observation to the point of leaving out the topics as they may be prerequisites for other topics which are heavily assessed. Electrostatics which seemed not to be tested is required for understanding of current electricity which is heavily tested over the three years. The same goes for diversity of organisms; features of an organism and its interaction with other organisms are closely related to its characteristics. The manner in which teachers perceive the questions in examination papers influences the ways in which they use the papers. The teachers showed that content and style of examination papers affect their practice. The effects of past examination papers that teachers mostly expressed related to content (depth and scope) and assessment. They indicated that they teach according to the syllabus but tend to put more emphasis on frequently assessed themes. HHS3 indicated that he was able to treat one topic in greater depth than the syllabus had specified following exposure to question papers, while one teacher indicated in the questionnaire that he realised that there was need to teach up-to- date information. All these generally imply adding more subject matter in some cases and eliminating some of the subject matter specified in the syllabus when planning lessons following exposure to JC examination papers. The other influence of examination papers that teachers expressed was on assessment. The teachers indicated that they modified and used questions from past examination 99 papers for constructing formative tests. The modifications that teachers made on questions from past examination papers range from simplifying to adding sub questions. In some instances teachers used questions as they were or adopted the style in making questions for other themes (see table 5.4). Teachers also showed that they tried to apply the style of asking questions that they observed in the question papers in constructing their formative tests. The effects of question papers on teaching methods were not strongly expressed. Only 4 out of the 15 responses given to the question on the ways in which examination papers affected teaching implied adjustments of teaching approaches. Three responses indicated that questions from examination papers induced pitching the teaching higher while one specified switching from teacher talk mode of teaching to facilitated discovery. The response was not elaborate, so I can only presume that facilitated discovery indicated implied more involvement of demonstrations, hands on learning and question and answer methods of teaching. 6.5 Way forward with past JC science examination papers Past examination papers constitute a potentially powerful resource for influencing the teaching and learning of JC science. Their use can be fitted within classroom assessment which Doran et al. (1994) show still need to be explored and researched deeply as it is complicated. It requires the teacher to measure and evaluate several variables in many learners within a short time. The teachers have to skilled in carrying out this classroom assessment for it to support learning. By critically examining their practice in the light of the demands imposed on them by the papers, teachers can work out some modalities of reconciling the two such that they improve their practice. Apart from their use in testing and teaching, they can also be used for deriving themes for action research. Through action research, teachers can identify internal factors that impede their teaching and break away from them. Factors such as attitudes towards practices and subject matter knowledge can be worked on and improved upon. External factors which affect one?s teaching can also be identified through collaboration and 100 sharing of information relating to teaching, assessment and available curriculum material. In their model of classroom assessment project to improve teaching and learning (CAPITAL), Atkin et al. (2005) show that assessment is tightly interwoven with what it means to be a teacher. They therefore advance CAPITAL; which focuses on students? learning by: establishing a starting point, collaboration within the school or department and reflective teaching, which lead into action research. From action research one can ultimately build some pedagogical content knowledge as explained by Shulman (1986). Systematised reflection on one?s teaching will enable one to bring together a number of factors such as subject matter knowledge, curriculum requirements for the theme, students? learning, and strategies of teaching. In determining the teachers? pedagogical content knowledge, Zohar and Schwrtzer, (2005) established that teachers were capable of teaching in such a manner that they developed higher order skills in learners; the overriding reason for the practices being the intention to complete the syllabus. The reason which appeared to be almost similar that was given by one of the teachers was identifying the frequently asked themes and put more emphasis on them. The problem linked to this notion of completing the syllabus is that it is sometimes achieved at the expense of deepening understanding of some concepts in the syllabus. This paradox still featured in the responses give by some of the teachers, even though they still showed that they were aware of what should happen in the classroom in order for the curriculum to be implemented properly. Past examination papers can be utilised in such a manner that the focus is on learners. Teachers can draw tasks that can be used for constructivist teaching from them. One of the viable ways of using past examination papers is drawing themes to be used in short term activities such as overnight homework, individual or group class work and practical work. These are teaching and learning activities which offer the teacher and the learner an opportunity for evaluation and immediate feedback. Hassard (2005) shows that while homework can be used for enabling the student to practice what has been presented or just started in class, they can also be used as a preparation for the material to be presented next. Competence reflected in the homework should therefore be evaluated so that students learning can be enhanced. Use of questions from past examination papers in preassessment tasks can also help the teacher to establish learners? prior knowledge. The 101 most effective strategies of preassessment according to Hassard include engaging learners in an activity, discussions, or conferences where learners can express their opinions visually and verbally. These activities offer time for immediate feedback and correction of visible misconceptions. They can therefore be used for facilitating development of higher order thinking skills. Tasks involving reading ahead can also be built using themes implied in the question paper. Butler and McMunn (2006) show that a task can be an instructional strategy as well as an assessment method. Questions from the past examination papers can be used to all serve these purposes and improve the learning of science. The use of past examination papers can improve the teaching and learning of science if used properly. Sieborger (2004) points out some of the positive and negative effects of placing the focus on mastering questions in the examination papers. Among the positive effects of their use are; it makes the examination structure familiar and hence remove fear for it (the examination), it allows for discovery of learning problems before the actual examinations assessment. Their overuse can also have some detrimental effects on teaching and learning of science as well. Some of the negative effects of narrowly focusing on answering examination questions include removing the supportive and instructional roles of formative assessment and placing undue emphasis on summative assessment. The learners may end up seeing assessment as an end in itself and not a preparation towards achieving an educational goal. One of the positive effects of use of past examination papers during the year which surfaced during the investigation was highlighted by teacher LHS2. What this teacher indicated was that the questions from the past examination papers did not dominate the tests set for learners. Such questions only serve as an additional source of questions. The main aim of drawing questions from past examination papers was to provide students with practice of examination and broadening the students? horizons in terms of answering those questions, which tallies with what Sieborger (2004) indicated as a positive aspect of exposing learners to past examination papers. 102 6.6 Critical Reflections on the Study The study has attempted to determine the effect tools of summative assessment on implementation in the JC science curriculum. This was done on a very small scale, using schools which were relatively similar in terms of resources and location. What surfaced was that the structure and contents of these tools of summative assessment mostly influenced what teachers selected for teaching and how they tested their students. The units of analysis and methodology chosen led me directly to where relevant empirical information pertaining to the effect of summative assessment on teaching lies. They provided me with the authentic answers to the research questions since they came from people and documents that are directly involved in the implementation and assessment of JC science. The science teachers are the ones who are involved in the implementation of JC science at classroom level while the items in the JC science final examination are the ones whose quality is to be used by the teachers and learners. The focus group interview and questionnaire enabled me to access science teachers? perceptions on question papers, practice relating to them, their effects on teaching practice while the analysis of question papers helped me to get the structure and content of JC science examination papers. However, the sample was rather small due to limited time and financial resources available. A bigger sample involving schools of different performance and from different locations might have given a broader picture of phenomenon. All the schools involved in the sample are from urban areas. Relation between the use of past examination papers and performance in the final examinations might have surfaced sharply if a diverse sample was employed. A bigger sample of examination papers involving papers of the past ten years would even go into the previous syllabus and hence structure and content of examinations. The papers which were available are all based on the same syllabus, so variations between the papers are not big. Comparison of the past examinations with the current ones that was indicated by some of the teachers in the interview would even be clearer. The degree of assessment of issues such as environmental education and science, technology and society that are addressed in the current papers would also be established in the old papers. 103 One other aspect which may have affected the depth of the findings is the fact that I relied only on teachers? testimonies. I did not have an opportunity to conduct the actual observations of the actual practice. Neither did I have a chance of comparing the teachers? formative tests with the past examination papers. A brief interview with the JC science examiner would also be worthwhile as s/he would remark on the trends observed in as far as performance in JC science is concerned. It was not easy to meet the officer. There are a few other limitations pertaining to time that can also be pointed out. The time available for data collection only appeared to be suitable to me, and not to the teachers. On the part of the teachers the time for data collection coincided with the end of the year and the beginning of the year, times in which teachers are occupied with other activities and duties with deadlines to meet. There was marking of internal examinations which had to be completed before embarking on marking of final examinations for other teachers. The beginning of the year also coincided with other administrative duties which did not give them enough time to engage in lengthy discussions with me. This led to another limitation; short time for focus group interview and hence inability to do sufficient probing. With more time I may have gone to the extent of looking through some of the formative tests that teachers had had prepared, though it would mean spending a long time in one school. The findings of the study, limited as they are have shed some light onto the relationship between summative assessment and implementation of JC science curriculum. They have shown that teachers carry out teaching that is informed by the syllabus while at the same time respond to the content and structure of examinations in their teaching. They further open up avenues for further research into the teaching and learning of JC science. The following areas can be explored further to uncover problems relating to the teaching and learning of JC science. 6.7 Recommendations for future research The study has revealed some of the influences that assessment has on implementation of JC science curriculum. More information could be uncovered that can be used for in- service training and policy formulation. The following avenues could be explored: 104 ? Research could be undertaken into teachers? pedagogical content knowledge. According to the demographic information of teachers who participated in the study, teachers have satisfactory subject matter knowledge. ? Action research as a viable option could be encouraged on the part of science teachers, so that there is exchange of expertise on the teaching of different themes of JC science by the teachers themselves. ? Comprehensive qualitative analysis on alignment may also be undertaken by following the teaching and learning of some selected topics from the objectives in the syllabus, through teaching to summative assessment in the final examination. ? Analysis of courses on teaching and assessment in science may also reveal some useful information that could assist improved implementation of science curriculum. ? Actual observation of teachers in their practice. ? Duplication of the study using schools of different codes (in Bernsteinian terms) may reveal more information that can be used for informing policy on assessment as well as in service and pre service training programmes on assessment. 6.8 Conclusions Assessment does have some influence on the teaching of JC science. From what the teachers have indicated, there are some changes that they effect in the selection of content for scheming, teaching approaches and assessment strategies following their interaction with JC science examination papers. Teachers take some time to look into the question papers qualitatively. They are able to pick up what should have been done in the teaching of science and what should be done in the teaching of the subsequent groups. The questions in JC science papers also have worthwhile aspects which need to be reinforced; these contextualization and assessment of higher order thinking skills. 105 References Anderson, L.W. (2005). Objectives, Evaluation and Improvement of Education. Studies Educational Evaluation. 31 (2005), pp 102 ? 113. Atkin, J. M., Coffey, J. E., Moorthy, S., Sato, M. and Thiebeault, M. (with foreword by Wheeler, G.) (2005). Designing Everyday Assessment in Science Classroom, New York, Teachers College Press. Bell, B. (2007). Classroom Assessment of science learning. In Abell, K.S. and Lenderman, N. G. (Eds.) Handbook of Research on Science Education: (pp. 965? 1006). Mahwah: Lawrence Erlbaum Associates, Publishers. Bell, B. (2000). Formative Assessment and Science Education: A model for Theorising. in Miller, R., Leach J. and Osborne, J. (Eds.). Improving Science Education: the contribution of research. Buckingham. Open University Press. Bennett, J. (2003). Teaching and Learning Science. London: Continuum. Best, J. W. and Kahn, J. V. (2003). Research in Education, 9th ed. Boston: Allyn and Bacon. Bernstein, B. (1975). Class, Codes and Control. Vol. 3; Towards a Theory of Educational Transmission, London: Routledge/Kegan Paul. Bernstein, B. (1995). Vertical and Horizontal Discourse. British Journal of Sociology of Education, 20 (2). pp 157 ? 173. Biggs, J. and Telfer, R. (1987). The Process of Learning. 2nd ed. Sydney. Prentice Hall. Black, P. (1998a). Assessment by Teachers and Improvement of Students? learning. In Fraser, B. J. And Tobin, K. G. (Eds.) International Handbook of Science Education: (pp. 811 ? 822). Dordrecht: Kluwer Academic Publishers. Black, P. (1998b). Testing: Foe or Friend? Theory and Practice of Assessment. London: Falmer Press. Black, P. (2000). Policy, Practice and Research: the Case of Testing and Assessment. In Millar, R., Leach, J. and Osborne, J. (Eds.) Improving Science Education: the contribution of research. (pp 327 ? 346). Buckingham: Open University Press. Black, P. and Harrison, C. (2000). Formative Assessment. In Monk, M. and Osborne, J. (Eds.) Good Practice in Science Teaching: What Research has to say. (pp. 25-40). Buckingham, Open University Press. Blaxter, L., Hughes, C. and Tight, M. (1996). How to Research. Buckingham: Open University Press. 106 Bloom, B. S., Madaus, G. F. and Hastings, J. T. (1981). Evaluation to improve learning, New York: McGraw- Hill Book Company. Borman, K. M., Clarke, C., Cotner, B. And Lee, R. (2006). Cross Case Analysis. In Green, J. L., Camilli, G., Elmore, B. P. (Eds.) Handbook of Complementary Methods in Educational Research, (pp 123 ? 139). Washington DC: American Educational Research Association / Mahwah, Lawrance Erlbaum Associates Inc. Bowen, G. M. (2005). Essential Similarities and Differences Between Classrooms and Scientific Communities: in Yerrick, R. and Wolff- Michael, R. (Eds.) Establishing Scientific Classroom Discourse Communities- Multiple Voices of Teaching and Learning Research. (109-134). Mahwah: Lawrence Erlbaum Associates, Publishers. Britton, E. D. And Schneider, S. A. (2007). Large Scale Assessment in Science education, in Abell, K. S. And Lenderman, N. G. (Eds.). Handbook of Research on Science Education. (pp 1007 ? 1040). Mahwah, Lawrence Erlbaum Associates, Publishers. Brooks, V. (2002). Assessment in Secondary Schools. Buckingham: Open University Press. Butler, S. M. and McMunn, N. D. (2006). Teachers Guide to Classroom Assessment; Understanding and using Assessment to improve students learning. San Francisco: Jossey Bass/ John Wiley and Sons Inc. Cohen, L. and Manion, L. (1994). Research Methods in Education, 4th ed. London: Routledge. Creswell, J. (1994). Research Design, Qualitative and Quantitative Approaches. Thousand Oaks: SAGE Publications. Creswell, J. (2003). Research Design, Qualitative, Quantitative, and Mixed Methods Approaches, 2nd ed. Thousand Oaks: SAGE Publications. Delpech, R. (2002). Why are School Students bored with Science? Editorial Comment. In Journal of Biological Education. 36 (4) pp 156 ? 157. Delors?, J. (1998). Learning, The treasures within: Report to the UNESCO of International Commission of Education for the twenty first century. Paris: UNESCO. Doran, R. L., Lawrenz, F. and Helgeson, S. (1994). Research on Assessment in Science. In Gabel, D. (Ed.) Handbook on Research on Science Teaching and Learning, (pp. 388?442). New York: National Science Teachers Association / Macmillan Publishing Company. Eisenhart, M. (2006). Representing Qualitative Data. In Green, J. L., Camilli, G., and Elmore, P. B. (Eds.) Handbook of Complementary Methods in Education Research. (pp. 567 ? 582). Washington D.C.: American Educational Research Association. 107 Embretson, S. and Yang, X. (2006). Item Response Theory. In Green, J. L., Camilli, G. and Elmore, P. B. (Eds.) Handbook of Complementary Methods in Education Research. Vol.2, Part 2, (pp. 385 ? 410). Washington, D.C.: American Educational Research Association. Erickson, F. (1998). Qualitative Research Methods for Science Education. In Fraser, B. J. and Tobin, K. G. (Eds.) International Handbook of Science Education, Vol. 2, Part 2. Dordretch: Kluwer Academic Publishers. Examinations council of Lesotho (2006). Primary School Leaving Examination; Pass list (for 2006 candidates), Maseru, ECOL. Examinations council of Lesotho (2003). Junior Certificate Examination, October 2003, Science Paper I and Paper II, Maseru, ECOL. Examinations council of Lesotho (2005). Junior Certificate Examination, October 2005, Science Paper I and Paper II, Maseru, ECOL. Examinations Council of Lesotho (2006). Junior Certificate Examination, October 2006, Science Paper I and Paper II. Maseru, ECOL. Fraser, W. J., Loubser, C.P and Van Rooy, M. P. (1990). Didactics for the undergraduate student. Durban: Butterworth. Freebody, P. (2003). Qualitative Research in Education; Interaction and Practice. London: SAGE Publications. Gibson, R. (1986). Critical Theory and Education. London: Hodder and Stoughton. Gitomer, D. H. and Duschl, R. A. (1998). Emerging Issues and Practices in Science Assessment in Fraser, B. J. And Tobin, K. G. (Eds.). International Handbook of Science Education, Vol. 2, Part 2, (pp. 791 ? 810). Dordrecht: Kluwer Academic Press. Green, G. And Rollnick, M. (2007). Let?s stop complaining about low cognitive levels of testing in time limited examinations: The case for Fair Testing Practices. South African Journal of Higher Education, 21(2), 255?266. Griffiths, M. (1998). Educational Research for Justice: Getting off the Fence. Buckingham: Open University Press. Guskey, R. T. (1994). What you assess may not be what you get. Educational Leadership, March, 1994, 51?54. Hatch, J.A. (2002). Doing Qualitative Research in Educational Settings. New York: SUNNY. Hassard, J. (2005). The Art of Science Teaching; Inquiry and Innovation in Middle and High school. New York: Oxford University Press. 108 Hinds, D. (2003). Research Instruments. In Wilkinson (Ed.). Researchers Tool Kit. London: Routledge/Falmer Press. Jacobs, M. (2000). Curriculum. In Lemmer, E. (Ed.). Contemporary Education: Global Issues and Trends. Sandton: Heinemann Higher and Further Education (PTY) Ltd. Jones, M. G. and Carter, G. (2007). Teachers? Attitudes and Beliefs. In Abell, K. S. and Lenderman, N. G. (Eds.) Handbook of Research on Science Education. (pp. 1067? 1104). Mahwah: Lawrence Erlbaum Associates Publishers. Keeley, P., Eberle, F. and Farrin, L. (2005). Uncovering Students? Ideas in Science Vol.1: 25 Formative Assessment Probes. Arlington: NSTA Press. Kempa, R. (1986). Assessment in Science. Cambridge: University of Cambridge. Knutton, S. (1994). Assessing Children?s learning. In Wellington, J. (Ed.) Secondary Science: Contemporary Issues and Practical Approaches. London: Routledge. Lambert, D. and Lines, D. (2000). Understanding Assessment: Purposes, Perceptions and Practice. London: Routledge / Falmer. Lemke, J. (1998). Analysing Verbal Data: Principles, Methods and Problems. In Fraser, B. J. and Tobin, K.G., International Handbook of Science Education, Vol. 2, Part 2, (pp. 1175?1189). Dordrecht: Kluwer Academic Publishers. Lewin, K. (1992). Science Education in Developing Countries: Issues and Perspectives for Planners. Paris: UNESCO. Leedy, P. (1993). Practical Research: Planning and Design, 5th ed. New York: MacMillan Publishing Company. Linn, R. L. and Miller, M. D. (2005). Measurement and assessment in Teaching, 9th ed. International Edition. New Jersey: Pearson/ Merrill Prentice Hall. Manyatsi, S. E. (1996). The Interface between Junior Secondary and Senior Secondary School Chemistry in Swaziland: An unpublished thesis submitted to the Faculty of Science, of the University of Witwatersrand in part fulfilment of the Degree of Doctor of Philosophy, Johannesburg, University of the Witwatersrand. McMillan, J. and Schumacher, S. (2006). Research in Education, Evidence ? Enquiry, 6th ed. Boston: Allyn and Bacon. Millar, R. (1989). ?What is scientific method? and can it be taught? in Wellington, J. (Ed.) Skills and Processes in Science Education. London: Routledge. Ministry of Education and Training (2005). Education Sector Strategic Plan, 2005 ? 2015. Ministry of Education and Training, Maseru. Molapo, L. (2003). Trends in performance and choice of science options at COSC examinations in Lesotho by schools candidates in 1994 to 2001. The Proceedings of the 11th Annual Conference of SAARMSTE, pp.174?179. 109 Morrison, A. J., McDuffie, R. and Akerson, L. V. (2005). Preservice Teachers? Development and Implementation of Science Performance Assessment Tasks. International Journal of Science and Mathematics Education, 3 (3) 379 ? 406. Naidoo, P. and Savage, M. (2004). Assessment and Examinations, in AFCLIST, School Science in Africa, Learning to teach, teaching to learn. (pp.12 ? 13) Lansdowne: Juta Gariep. National Curriculum Development Centre (2002). Science; Junior Secondary Syllabus, Maseru: Ministry of Education and Training (of Lesotho). Nitko, A. J. Brookhart, S. M. (2007). Educational Assessment of Students, 5th ed. Upper Saddle River: Pearson/Merrill Prentice Hall. Nyabanyaba, T. (2008). The illusive goal relevance and quality in Lesotho?s education system Nyabanyaba, T. (2008). The illusive goal relevance and quality in Lesotho?s education system: A brief macro-level analysis. Proceedings of the 16th SAARAMSTE conference, pp. 539 ? 546. Opie, C. (2004). What is Educational Research? in Opie, C. (Ed.). Doing Educational Research. London: SAGE publications. Perrone, V. (1997). Towards an Education of consequence: Connecting Assessment, Teaching and Learning. In Goodwin, A. L. (Ed.) Assessment for equity and Inclusion; Embracing all our Children. (pp. 305?315). New York: Routledge. Phoenix, D. A. (2000). Time to reassess the role of assessment. Editorial comment. Journal of Biological Education, 35 (1), 3?4. Preece, R. (1994). Starting Research: An Introduction to Academic Research and Dissertation Writing, London: Cassell. Porter, A. C. (2006). Curriculum Assessment. In Green, J. L., Camilli, G. and Elmore, P. B. (Eds.) Handbook of Complementary Methods in Education Research. (pp. 141? 159). Mahwah: Lawrence Erlbaum Associates, Inc. Puchta, C. and Potter, J. (2003). Focus Group Practice, London: SAGE Publication. Rollnick, M., Manyatsi, S., Lubben, F. and Bradley, J. (1998). A model for Studying Gaps in Education: A Swaziland Case Study in the Learning of Science, in International Journal of Educational Development. 18 (6), 453 ? 465. Shepard, L. A. (2000). The Role of Assessment in Learning Culture, in Educational Researcher, 29 (7), 4 ? 14. Shulman, L. D. (1986). Those who understand: Knowledge Growth in Teaching. Educational Researcher?., 4?14. Sieborger, R. (in collaboration with Macintosh) (2004). Transforming Assessment; A Guide for South African Teachers, 2nd ed. Cape Town: Juta. 110 Solano-Flores, G. and Nelson-Barber, S. (2001). On the Cultural Validity of Science Assessments. Journal of Research in science Teaching, 38 (5), 553?571. Stern, L. and Ahlgren, A. (2002). Analysis of students? Assessment in Middle School Curriculum Materials: Aiming precisely at Bench marks and standards. Journal of Research in Science Teaching. 39 (9), 88?910. Steward, D. W. and Shamdasani, P. N. (1990). Focus Groups: Theory and Practice, Newbury Park: SAGE Publications. Swain, J. (2000). Summative Assessment. in Monk, M. and Osborne, J. (Eds.) Good Practice in Science teaching; What Research has to say. Buckingham: Open University Press. Tamir, P. (1998). Assessment and Evaluation in Science: Opportunities to Learn and Outcomes. In Fraser, B. J. and Tobin, K. G. (Eds.) International Handbook of Science Education, Part Two, Volume 2, (pp. 761?780). Dordrecht: Kluwer Academic Publishers. Tara, M. (2007). Machination of Assessment; Metaphores, Myths and Realities. Pedagogy, Culture and Society. 15 (1), 55 ? 69. Taylor, N., Muller, J. and Vingevold, P. (2003). Getting Schools Working: Research and Systemic Reform in South African Schools, Cape Town: Pearson. Taylor, P. H. and Richards, C.M. (1987). An Introduction to Curriculum Studies, 2nd ed. Berkshire: NFER ? NELSON. Zohar, A. and Schwartzer, (2005). Teachers? Pedagogical Knowledge in the Context of Teaching Higher-Order Thinking. International Journal of Science Education, 27 (13), 1621?1638. 111 APPENDIX A 112 APPENDIX B A letter of permission to pursue research 113 APPENDIX C Ethics Clearance 114 APPENDIX D Letter to the Head of the School University of the Witwatersrand School of Education Student Name: Sophia M. Majara Student Number: 0514575F Supervisor: Prof. M. Rollnick Re: Request to Collect Data from the Teachers The Head of the School My name is Sophia Majara. I am based at the Lesotho College of Education, in the faculty of Sciences, under the Department of Pure Sciences. At the moment I am pursuing my studies at the University of the Witwatersrand, of which a research project is a component. The area I am going to look into is the teaching and learning of junior certificate science. My focus is the use of the tools of summative assessment for formative purposes by the teachers. The topic of my study is ?Lesotho Junior Secondary Science Teachers? Perceptions and Use of the Past Examination Papers in Teaching?. In the process of collecting data, I will be holding some discussions with the teachers of junior certificate science as well as asking them to complete a questionnaire. Attached to this letter of request is a letter from the Ministry of Education and Training (of Lesotho), granting me the permission to pursue the data collection. I undertake to maintain anonymity and confidentiality of the names of the teachers and the school. The information will be used for research purposes only and nobody will have any access to it except the supervisor under special conditions. A summary of the findings will be available to your teachers once the research report has been completed. I also wish to indicate that participation in this research activity on the part of your school is voluntary, and so withdrawal from it is possible at any time. If you allow your teachers to assist me in this respect, please sign the enclosed consent form for me. Should you require further clarification, feel free to use the following numbers to contact me. ------------------------------ Sophia Majara Contacts: 00266-58909200 0027-781937832 115 APPENDIX E Consent Form for the Head of the School I ??????????????. Agree that the researcher, Sophia Majara collects part of the data she needs from my school. I have read and understood the contents of the information letter, which I am presently keeping. I give consent to the following (Please circle the option to indicate your selection). (i) Interviewing of the science teachers Yes No (ii) Completion of the questionnaire by the science teachers Yes No ????????? ????????? Head of the School Date ????????? ????????? Researcher Date ????????? ????????? Witness Date 116 APPENDIX F Letter to the junior certificate science teachers University of the Witwatersrand School of Education Student Name: Sophia Majara Student Number: 0514575F Supervisor: Prof. M. Rollnick Re: Request to collect data from the JC science teachers; Dear Teachers My name is Sophia Majara. I am an education student at the University of the Witwatersrand. As part of my studies, I am undertaking some research. The area I am looking into is the teaching and learning of science at junior certificate level. My focus is on the use of tools of summative assessment for formative use by the teachers. The head of your school has granted me permission to get some information from you. I will hold some discussions during which I will ask you to answer a questionnaire. The discussions will be about the perceptions and use of the past examination papers of junior certificate science for teaching and learning. If you agree, I will request you to sign the attached consent form. Please note that participation in this activity is voluntary. Withdrawal from it is possible at any stage. I undertake to protect your identity. Your name and contact details will be kept in a separate file, for verification of information at a later stage. The information you will provide will be used purely for research purposes. It may also be shared with other members of the science education community in conferences or journals. In cases where such happens, you will be referred to by a pseudonym. Should you need any further information, feel free to use any of the following contacts: South Africa: 0027 - 781937832 Lesotho: 00266 - 58909200 E-mail: smajaram@yahoo.com 117 APPENDIX G Consent form for teachers Researcher: S. Majara Supervisor: Prof. M. Rollnick I __________________________________ agree to participate in the research activity undertaken by Sophia Majara. The details and purpose of the research have explained to me. I give consent to the following:- Focus Group Interview Yes No Completion of the questionnaire Yes No Audio recording of the interview Yes No Possible future use of the information Yes No Signatures _______________________ _____________________ Teacher Date _______________________ _____________________ Researcher Date _______________________ _____________________ Witness Date 118 APPENDIX H Questionnaire My name is Sophia Majara. I am an education student at the University of the Witwatersrand. As part of my study programme I am conducting research in which I would like to find more about the formative use of the past examination papers for the teaching and learning science at Junior Certificate level. May I therefore request you to furnish me with the information by completing the attached questionnaire. You do not need to write your name. You may only write a pseudonym that only you will recognise. Thanking you in advance. 119 1. Personal Details Please indicate your response by making a tick in the appropriate box. 1.1 Gender: Male Female 1.2 Age: 18 ? 20 21 ? 30 31 ? 40 41 ? 50 51 - 65 For questions 1.3 to 2.7, complete with a short statement or phrase. 1.3 Highest educational qualification held: ________________________________________________________________________ 1.4 Major(s) / Subject(s) of specialisation: 120 ________________________________________________________________________ 1.5 Teaching Experience: ________________________________________________________________________ 2. Use of the past examination papers: 2.1 Do you use past examination papers in your teaching? ________________________________________________________________________ If so at what level (e g. Form A, B or C) do you start using them? ________________________________________________________________________ 2.2 List all the ways in which you use them: ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 2.3 In what other ways can the past examination papers be used to improve the teaching and learning of junior certificate science? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 2.4 If you use the past examination papers in test or examinations; and find that performance of learners in questions from them is not satisfactory, what measures do you normally take? ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ ________________________________________________________________________ 121 2.5 Do you ever adapt or change the questions from the past examination papers for your teaching? ________________________________________________________________________ If so, how? ________________________________________________________________________ ________________________________________________________________________ 2.6 How is your teaching affected by the type of questions you find in the junior certificate examination papers of science? ________________________________________________________________________ ________________________________________________________________________ 2.7 How do rate the questions in the junior certificate science examinations (e g. are they easy, difficult or appropriate)? ________________________________________________________________________ 2.8 Is there anything you wish to add to this topic/ study? ________________________________________________________________________ ________________________________________________________________________ THANK YOU APPENDIX I 122 Interview Schedule for Focus Group Interview Question One From your own experience, what (abilities, knowledge or values) do final examination papers of Junior Certificate assess? Please take five minutes to discuss what you have observed in the papers. Probing questions: Give examples of questions that you remember which required that What implications do such questions have on the teaching and learning? If you were to classify the following questions according to what they assess, what would you say they assess? (Teachers are issued with appendix I). Question Two Do questions in the junior certificate science examinations match the syllabus objectives? Probing questions: Why? What can you say about their cognitive demands on learner? What can you say about their relevance to the syllabus? Question Three Have you identified any weaknesses in the questions of the junior certificate science examinations? Probing question: Could you please list them. Question Four Have you spotted any strong points of the junior certificate science examination? Probing question: Could you please list them. Question Five Is your teaching influenced by the nature of questions in the junior certificate science examinations? Probing question: If so how? Question Six How do you design your own formative tests in the junior certificate science? APPENDIX J 123 Sample Questions (For question one of the focus group interview schedule) Q1. Brakes are used for stopping a car. Into which form of energy is kinetic energy converted into (during the braking)? A heat energy B light energy C potential energy D sound energy Q2. Which of the following is a compound? A air B carbon C oxygen D steam Q3. Suppose Mpho, a student could only see well on the chalkboard when sitting in the front row. (i) Which eye defect does she have? __________________________________________________________________ (ii) What type of lens could be suitable for her spectacles? __________________________________________________________________ Q4. Substances that we come across in everyday life can be acidic, basic or neutral. (i) Name one household substance which is basic. __________________________________________________________________ APPENDIX K 124 Summary of Demographic and Personal information of all teachers involved in the study Co de G en de r A ge Qu al ifi ca tio n s M ajo r 1 M ajo r 2 Te ac hi n g Ex pe rie n ce U se o f p as t E x am Pa pe rs Fi rs t L ev el o f U sin g Th em A da pt at io n R at in g F1 M 21 - 30 Dip Ed Mathematics Physics 1 Y Form B Y A F2 M 21 - 30 Bsc Biology Chemistry 1.3 Y Form B Y A F3 M 31 - 40 Bsc Physics Geography 0.5 Y Form A Y A F4 F 41 - 50 Bsc Ed Biology Geography 17 Y Form A Y A H1 M 21 - 30 Bsc Physics Mathematics 0.5 Y Form B Y A H2 M 41 - 50 Bsc Ed Biology Chemistry 25 Y Form C Y A H3 M 31 - 40 Bsc Ed Mathematics Physics 8 Y Form B Y A L1 M 21 - 30 Bsc Biology Chemistry 1 Y Form B Y A L2 M 21 - 30 IGCSE Mathematics Science 2 y Form B Y A L3 M 21 - 30 COSC Mathematics Science 1.5 Y Form B Y A L4 M 21 - 30 Dip Ed Mathematics Science 1 y Form B Y A U1 M 21 - 30 Bsc Mathematics Physics 0.4 Y Form A Y E U2 F 41 - 50 Bsc Ed Chemistry Geography 17 y Form C Y A U3 F 21 - 30 Bsc Agric Agriculture Science 4 Y Form A Y A U4 F 31 - 40 Bsc Ed Biology Chemistry 4 Y Form B Y A U5 M 21 - 30 Dip Ed Mathematics Physics 3 Y Form B Y A S1 F 31 - 40 Bed Hons Biology Chemistry 12 Y Form B N A S2 F 21 - 30 Bsc Ed Mathematics Science 3 Y Form A Y A S3 F 41 - 50 Bsc Ed Chemistry Mathematics 18 Y Form A N A S4 M 31 - 40 Bsc Biology Geography 6 Y Form A Y A P1 F 21 - 30 Dip Agric Science Agric 3 Y Form A N A P2 M 21 - 30 Cert in Law Law Geography 0.5 Y Form B N A APPENDIX L 125 Example of transcription of the Focus Group Interview with Junior Certificate science teachers Lowlands High School Interviewer: From your experience as science teachers, what do final examination papers of Junior Certificate science assess? Teacher 1: Skills. Teacher 2: Practical skills Interviewer: Practical skills ? (in anticipation for more); ? now if you were to classify questions according to what they assess, what would you say the following questions assess? Teacher 2: The first question assesses knowledge in physics, the second question, I think it also assesses their knowledge but in chemistry. The third, hmm ? also knowledge but the last question, it assesses the awareness of the student; how he can relate the subject to everyday life. Interviewer: From what you have seen, would you say the questions in the final examination papers of JC science match the syllabus objectives? Teacher 3: Yes, I think they do, as far as I am concerned, because, some of the questions are not covered in the exam, in the syllabus, but most of the time they try to match the syllabus, and exam in some topics, they are not covered in the examination. Interviewer: What can you say about the cognitive demands they make on learners? that is during the examination? Teachers 1, 2 and 4: Can?t you repeat that question. Interviewer: I am asking about the cognitive demands that they make on learners; do they require the learners to think deeply or just to recall information, or to apply the information. Teacher 3: Most of the time, they require the learners just to remember the information that they have been taught. They don?t require learners to actually think deeply about any anything else. Teacher 1: From what I have seen, the paper one questions, they are just multiple choice. They just require recalling; to recall the knowledge, but paper 2 questions, they also sometimes require the students to think deeply. 126 Teacher 4: Yes to think deeply or to try and apply that knowledge that they have to solve those problems. Like some questions in paper two may require them to explain in terms of experiments; so that requires learners to use their knowledge most of the time. Teacher 3: I would say that paper two questions also needs learner to apply everyday knowledge, because the questions are based on everyday life. Teacher 2: Environmental questions, technical methods. Teacher 3: Environmental questions. Interviewer: Have you identified any weaknesses in the questions themselves? It can be in the construction or content, any weakness that you have spotted. Teacher 1: Yes, I think what I realised is that some questions are too long, they are wordy rowdy, they are not so straight forward and sometimes they are confusing. Teacher 2: Other times you find that in multiple choice, the answers of multiple choice are very close. That is those answers, can even confuse the learner to the extent that he will ? Teacher 1: In some questions, students are given the diagrams but the diagrams are not that clear to be related with the question / concept. Interviewer: What about the strong points in the question papers, something good about them? Teacher 4: I think the strong point is that sometimes, they usually have a mixture of simple questions; and some of them are ? ehmm ?, the levels of the questions, some are high level. Some require the learner to think and to apply the knowledge, most of the time. Teacher 2: Another strong point I have realised about them is that, for example the paper one questions, the questions don?t require the learner too many things in one question, for example they don?t require the student to convert and calculate ke hore, at the same time. Also most of the time the answers are put in the correct order; like the smallest option will be option A, the bigger one, option B and increase in that order, so they don?t confuse the student. Interviewer: Have they had any influence on your teaching? Teachers: What? the question papers? Interviewer: The question papers, yes. 127 Teacher 2: Yes, I think they have, because most of the time we use the previous examinations, and we teach the students according to how we have seen those questions being constructed. We teach according to the construction of the question papers, and we teach our students according to what we have seen, according to how we have seen the questions. Teacher 3: We teach and assess students according to what we have seen in them. Teacher 4: Most of the time they have got challenging elements. You?ll get a question sometimes that come might come challenging you to try to get more information. Teacher 3: They also give students the skills to attempt high level questions, how to manage time, how to look at the marks; so that, ke hore (which means) the student will know that if only one mark is required, I am not expected to express myself that much. Teacher 2: And again they influence the teaching, because as you study the papers, sometimes you realise that as ntate has just said, you find that there are some or objectives or topics that appear in the syllabus but sometimes they don?t even assess that. So as teachers sometimes we end up not teaching those topics because they don?t assess those topics, they tend to influence the teaching. Interviewer: finally, how do you design your own tests; internal or formative tests Teacher 2: We take past examination papers plus our own that we have set, ourselves for internal examinations, and I think there our motive is to give students practice of the examinations, and also to broaden their horizon in terms of answering those questions. Teacher 3: And we usually classify our questions into three categories, we look at the paper and here we are talking about science, three disciplines of science, because when we are testing, we cover three areas so you look at the questions when we are setting, they are distributed evenly. ?. Teacher 2: And we look at our schemes, what we have schemed for. For first session exam we look at what we have schemed for the first session. We may use the question papers, but use questions related to those topics, what we have just taught. Teacher4: Yes, sometimes we try to balance the topics so at sometimes we might be teaching physics, biology or chemistry so when we looking at those questions, we try to balance so that the other questions may not concentrate on physics alone or chemistry but test does not contain physics alone or chemistry alone, so we try to balance. Teacher 1: As we are balancing, we try to look at the types of questions. It should not be only one type of questions which need recalling only. We try to include all those aspects. 128 Teacher 3: I would also add that sometimes as a lack of English, we modify the questions from the question papers to make them a bit simpler for the students. Interviewer: Thank you. 129 APPENDIX M Cognitive levels of questions and skills they assess 2003 Paper One 2003 Paper Two Qu es tio n R ec al l Co m pr eh en sio n A pp lic at io n H ig he r o rd er th in ki n g Id en tif ic at io n Co m m u n ic at io n H ig he r o rd er sk ill s Qu es tio n R ec al l Co m pr eh en sio n A pp lic at io n H ig he r o rd er th in ki n g Id en tif ic at io n Co m m u n ic at io n H ig he r o rd er sk ill s 1  1(a) 3 * 2  (b) 2 3  (c) 2 4  (d) 2 5  2(a) 2 6  (b) 1 7  (c) 2 8  3(a) 1 9  (b) 1 10  (c) 2 11  (d) 2 12  4(a)i 1 13  ii 1 14  (b)i 2 * 15  ii 1 16  5(a) 1 17  (b) 2 18  (c) 2 19  (d) 1 20  (e) 1 21  (f) 2 130 22  6(a) 1 23  (b) 1 24  (c) 1 25  7(a)i 1 26  ii 1 27  8(a) 1 * 28  (b) 1 29  (c) 1 30  (d) 2 31  (e) 1 32  9(a) 2 33  (b)i 1 34  ii 2 35  (c) 1 36  10(a) 1 37  (b) 2 38  (c) 2 39  (d) 3 40  1(a) 2 41  (b) 2 42  (c) 2 43  (d)*1 2 44  (e) 2 45  2(a) 2 (b)i 2 ii 2* iii * 2* 131 (c)i 1 ii 1 3(a) 2 (b) 2 (c) 1 (d) 2 (e) 3 *1 Open ended question 132 2005 Paper One 2005 Paper two R ec Co m p A pp H o t Id Co m H o t R ec Co m p A pp H o t Id Co m H o s 1  1(a)i 2 2  (ii) 2 3  (b)i 1 4  ii 1 5  2(a) 1 * 6  7  (b) 1 8  (c) 1 9  (d)i 1 10  ii 1 11  iii 1 12  3(a) 1 13  (b) 1 14  (c)1 1 15  2 1 16  (d) 2 17  4(a) 2 18  (b) 1 19  (c) 2 20  (d) 1 21  (e) 2 22  2 23  5(a)i 1 24  ii 1 133 25  iii 1 * 26  iv 2 27  (b) 1 28  (c) 2 29  (d)i 1 30  ii 1 31  6(a)i 1 32  ii 1 33  (b)i 2 34  Ii 1 35  7i 1 36  Ii 1 37  Iii 1 38  Iv 1 39  8(a)i 1 40  Ii 1 41  (b)i 2 42  ii 2 43  9(i) 1 44  (ii) 1 45  10(a) 1 (b)i 1 Ii 1 1(a) 2 (b) 2 (c) 2 134 (d)i 2 ii 2 2(a)1 1 2 1 (b)1 1 2 1 (c)1 1 2 1 (d)i 1 ii 1 2 3(a)i 2 ii 2 iii 2 (b)i 2 ii 1 iii 1 135 2006 Paper One 2006 Paper Two R ec Co m A pp H o t Id Co m H o s R ec Co m A pp H o t Id Co m H o s 1  1(a) 2 2  (b) 1 3  * (c) 1 4  (d) 1 5  * (e) 1 6  (f)i * 1 7  ii * 1 8  2(a) 1 9  Ii 1 10  Iii 2 11  (b) 1 12  * (c) 2 13  (d) 2 14  3(a) 1 15  * (b) 2 16  (c) 1 17  4(a) 1 18  (b)i 1 19  Ii 1 20  (c) 2 21  (d)i 1 22  ii 1 23  5(a) 2 24  (b) 1 136 25  (c) 1 26  (d) 1 27  6(a) 1 28  (b) 1 29  (c) 1 30  (d) 2 31  (e) 1 32  * (f) 2 33  (g) 1 34 . * 7(a) 1 35  (b) 2 36  (c) 1 37  8(a) 1 38  (b) 1 39  (c) 1 40  (d) 1 41  (e) 1 42  9(a) 1 43  (b)i 1 44  ii 1 45  iii 1 10(a) 1 (b) 1 (c) 1 11(a) 2 (b) 1 137 1(a)1 * 1 2 1 (b)1 * * 1 2 1 (c) 1 (d) 2 (e) 1 (f)i 1 Ii 1 2(a) 1 (b)i 3 ii 1 (c) 1 (d) 1 1 (e) 1 1 3(a) 1 (b) 2 (c) 2 (d) 1 (e) 1 (f) 1 (g)i 1 Ii 1 138 APPENDIX N 139 140 141 142 143 APPENDIX O JUNIOR CERTIFICATE SCIENCE PAPER II, 2006 144 145 146 147 148 149 150 151 152 153 154 155 156 APPENDIX P JUNIOR CERTIFICATE SCIENCE PAPER II, 2006 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171