Formative Evaluation of OLPC Project Nepal: A Summary

This is a summary of the findings of a formative evaluation carried out by Mr. Uttam Sharma, a doctoral student at at the Department of Applied Economics at the University of Minnesota. The evaluation was carried out for OLE Nepal’s internal purpose. The results are expected to help learn about the shortcomings of the current test phase of the OLPC project so that necessary fine tuning can be done before next year’s larger deployment in more districts. Mr. Sharma can be reached at sharm061@umn.edu.  

Formative Evaluation of OLPC Project Nepal: A Summary

June-August 2008

Background:

The Nepal government’s Department of Education (DoE) and the Open Learning Exchange Nepal (OLE Nepal) had selected Bishwamitra Ganesh Secondary School and Bashuki Lower Secondary Schools, both in Lalitpur district of Nepal, as test schools for the One Laptop per Child (OLPC) initiative in Nepal. The laptops were distributed to all 135 students in grade 2 and 6 in those two schools on April 25, 2008. The students were also allowed to take laptops home. However, charging of the laptops could only be done in school. The implementation of the initiative has focused on four areas: digital content development, teacher preparation, network and power infrastructure development and government capacity building.

Study design and method

This study was initiated after about two months of program implementation. The objective of the present study is formative evaluation which assesses on-going project activities. This evaluation focuses on how the initiative is being implemented in the two test schools and looks at how the implementation can be altered or improved.

The study period was between mid-June to mid-August 2008. The first week was spent preparing survey questionnaires. The first round of field visit was conducted between June 22 and 27. The first three days were spent in Bishwamitra Secondary School while the last three days in Bashuki Lower Secondary School. The 2^nd round of field visit was conducted between July 27 and August31, 2008.

This study is based on the information collected from surveys with teachers, head-teachers, students in grade 6 and their family members, some School Management Team members, school records, the school censuses from the two test schools as well as detailed discussions with OLE Nepal officials and postings in the OLE Nepal blog (https://blog.olenepal.org/). One-on-one meetings with teachers were also conducted in addition to filling out the survey questionnaires.

The questionnaires were designed to elicit as much information about how the initiative is being run. Because of time constraints, we tried surveying only those students who roll numbers were odd (or only even numbered in other cases). All the regular teachers who were present in both the schools were surveyed. In total, 31 students from grade 2, 45 students from grade 6, 19 teachers that includes 2 head teachers, and 20 households were surveyed for the purpose. We also had discussions with many OLE Nepal officials, which aided us in putting our observations in proper context.

Teacher training

The teacher training was divided in two parts: residential training and in-school training. A four day residential training was conducted from March 29 to April 1, 2008 and four-day in-school training followed after a couple of weeks in between and was completed on May 2, 2008. The residential training included teachers from both the schools together and was held in Panauti, in the outskirts of Kathmandu Valley (35 kilometers from Kathmandu), while the in-school trainings were separately conducted in their respective schools. 10 teachers from Bashuki and 11 from Bishwamitra participated in the residential training.

One of the goals of the training was to make teachers comfortable to integrate laptop use in regular teaching. When the teachers were asked whether the complete training was sufficient for enabling them to properly integrate laptop in the regular classroom instruction process, 50 % of the teachers’ response was positive. This response underscores the need to have refresher training on a regular basis as most of the teachers are using computer for the first time.

Residential training

Teachers found the following aspect of the residential training useful:

• Encouraging peer-learning (learning from their colleagues rather than simply from the instructor)
• Information about how to use laptop for instructional purposes to make teaching more
• effective
• Sessions on how to make lesson plans
• An opportunity to meet colleagues and learn from them

Among those who said there were parts of the residential training that needed to be reduced or removed next time, all of them cited that a lot of time was spent on going over theoretical parts, which limited time spent on practical aspects on using laptop.

With regards to whether the materials in the residential portion of the training were adequate, we obtained conflicting answers from the two schools. While the overwhelming majority of the teachers in Bishwamitra (89%) thought that materials were adequate, less than one-third (29%) of teachers in Bashuki said that it was adequate. Among those who said it was inadequate, they wanted the focus to be more on practical rather than on theoretical aspects. Some teachers wanted more examples on how to teach using laptops. Though the majority of teachers at Bashuki found the materials inadequate, more than two-third of the teachers there (71%) agreed that the residential training was delivered in an understandable manner that was easy to follow. For Bishwamitra, the corresponding number was 89%.

When the teachers were asked whether it was a good idea to run residential and in-school trainings separately, almost everyone (15 out of 16) said that it was a good idea and wanted it to be that way in future as well.

In-school training

Teachers found many aspects of the in-school training useful such as:

• Sessions on class management, making lesson plans, and other tips on teaching
• Highlight the fact that the laptops should only be used as a tool
• Clearly pointing out the mistakes when teaching so that they could work on their mistakes and improve
• The feedback on their presentations

To the question what aspects in the lesson planning practice they found particularly useful, they mentioned:

• Encouraging the habit of incorporating lesson plans in their teachings
• Effective time management: how best to allocate time
• Teaching about how to present materials in the classroom
• Learning when and how to use laptop in the class

All 17 teachers found it useful to review lesson plans together in a group and to attend laptop classes conducted by colleagues and observe the class.

Classroom teaching learning process

Teaching learning digital materials

All the teachers said that the materials in the laptops were, in general, consistent with the interests of the students in their classes. However, when asked whether the materials in the laptops, in general, were consistent with the level or capacity of the students in their classes, the answer was rather mixed. Teachers were generally satisfied with the grade 2 contents, but had diverse views on grade 6. The teachers found many initially uploaded activities for grade 6 either too easy or too difficult. To the question about whether the materials in the laptops are able to meet the aims and objectives of your class as prescribed in the curriculum, the teachers feel that the recent digital contents are more relevant in this front.

The teachers mentioned that they generally spent one period to go over one activity. This means that if the expectation is that the teachers use laptops for three days a week, then there should be about three activities prepared for each week.

The teachers also mentioned that there were not sufficient exercises, especially for grade 6 students to work on. Since the teachers are so used to following the chapters in the book, the teachers suggested that the teaching materials in the laptops be sequenced according to the chapters in the textbooks. This way, they will not have to spend a lot of time figuring out which activity corresponds to the relevant chapter in the book.

The teachers also suggested that there be multiple choice question answers and practice questions. Some teachers proposed having different levels in the activities so that students develop a competitive spirit and try harder to go to a higher level. This could motivate them to study harder.

Classroom teaching learning

It is important to understand how classroom teaching learning has been affected by laptop use in classrooms. All the 17 teachers in both the schools feel that use of computers has helped their teaching. All (relevant) teachers feel that their lectures are now more organized and that it is easier to teach students new concepts. They also believe that the use of laptop based instruction has enabled students to learn concepts more easily. They are also of the opinion that laptop based instruction has made it easier to give students more practice exercises. In addition, they also feel that the use of laptop based instruction has made the classes more interactive. All the teachers are unanimous in saying that the use of laptop based instruction has increased the students’ interest in their studies.

In Biswamitra, teachers said that they generally use laptops 2 to 3 days a week while the number, according to the teachers, is at least 3 days in Bashuki. When they use the laptops, they mostly used the laptops between ¼ and ½ of the period. The grade 6 student survey, which was taken independently, also largely confirms this assertion. The mode in Bishwamitra was 3 days while it was 4 in Bashuki.

Students in grade 6 said that teachers mostly used laptops around the middle of the class. When some of the students forget to bring their laptops, the teachers ask those students to look at the laptop of the student who is sitting next to her.

The students in Bishwamitra generally say that the teachers give them homework every time the laptop is used in the classroom while those in Bashuki said that they are sometimes given homework. According to the teachers, those homework are mostly of review type to be done on laptops on a regular basis. The teachers ask students to do certain activity in the laptop for that day. The teachers, at times, look at the record in the computers to see whether the student has at least opened that activity at home or not, as one could find those details in the laptop.

When the students were asked how learning has been affected with the introduction of laptops in their classrooms, more than 95% of grade 6 students find learning using laptops easier,

As to why it is easier, the most common response was that you can do the exercise or activities as many times as you want. The second most common one was that these activities were very enjoyable to learn from. The students also appreciated the fact that you could do these activities at ones own pace. All these options were cited by more than 50% of the students.

When the students were asked whether other family members also use the laptops, more than eighty percent said yes. Siblings are the ones who use the laptop the most, followed by father and mother. This information is also confirmed with data from household survey where many siblings said that they use the laptop regularly.

One question that might reveal a lot about how the students are using the laptop was about what the students did the most in the laptop. In Bishwamitra, 68% of the grade 6 students (15) said they mostly use the laptop for educational activities while 23% (5) said to take pictures. The number for Bashuki was 32% for laptop activities while 42% was to take pictures.

When the grade 6 students were asked how effectively they are able to use digital activities in the laptop, most of them said they can use it well. 83% in Bishwamitra said that they can use it well while 17% said they can use it somewhat. In Bashuki, the corresponding figures were 75 % and 25 % respectively. No one said they cannot use the digital materials even though it was one of the options given.

Some teachers feel that the introduction of laptop has made managing the classes more difficult. Many teachers feel that the noise level has substantially increased. This has to do with the fact that students have difficulty controlling their excitements. The teachers hope that the noise level will subside once the students get used to using laptops. The teachers also admit that they are, at times, having difficulty managing time because they are not used to integrate laptop use in classrooms.

Class preparation

When teachers were asked “Compared to the effort you had to put in the teaching learning process before receiving the laptops, how much effort do you put in now?”, the overwhelming majority of the teachers feel that they put in more effort now.

Some of the reasons cited by the teachers on why the effort has increased: