Formative Evaluation of OLPC Project Nepal: A Summary

This is a summary of the findings of a formative evaluation carried out by Mr. Uttam Sharma, a doctoral student at at the Department of Applied Economics at the University of Minnesota. The evaluation was carried out for OLE Nepal’s internal purpose. The results are expected to help learn about the shortcomings of the current test phase of the OLPC project so that necessary fine tuning can be done before next year’s larger deployment in more districts. Mr. Sharma can be reached at sharm061@umn.edu.  

Formative Evaluation of OLPC Project Nepal: A Summary

June-August 2008

Background:

The Nepal government’s Department of Education (DoE) and the Open Learning Exchange Nepal (OLE Nepal) had selected Bishwamitra Ganesh Secondary School and Bashuki Lower Secondary Schools, both in Lalitpur district of Nepal, as test schools for the One Laptop per Child (OLPC) initiative in Nepal. The laptops were distributed to all 135 students in grade 2 and 6 in those two schools on April 25, 2008. The students were also allowed to take laptops home. However, charging of the laptops could only be done in school. The implementation of the initiative has focused on four areas: digital content development, teacher preparation, network and power infrastructure development and government capacity building.

Study design and method

This study was initiated after about two months of program implementation. The objective of the present study is formative evaluation which assesses on-going project activities. This evaluation focuses on how the initiative is being implemented in the two test schools and looks at how the implementation can be altered or improved.

The study period was between mid-June to mid-August 2008. The first week was spent preparing survey questionnaires. The first round of field visit was conducted between June 22 and 27. The first three days were spent in Bishwamitra Secondary School while the last three days in Bashuki Lower Secondary School. The 2nd round of field visit was conducted between July 27 and August31, 2008.

This study is based on the information collected from surveys with teachers, head-teachers, students in grade 6 and their family members, some School Management Team members, school records, the school censuses from the two test schools as well as detailed discussions with OLE Nepal officials and postings in the OLE Nepal blog (https://blog.olenepal.org/). One-on-one meetings with teachers were also conducted in addition to filling out the survey questionnaires.

The questionnaires were designed to elicit as much information about how the initiative is being run. Because of time constraints, we tried surveying only those students who roll numbers were odd (or only even numbered in other cases). All the regular teachers who were present in both the schools were surveyed. In total, 31 students from grade 2, 45 students from grade 6, 19 teachers that includes 2 head teachers, and 20 households were surveyed for the purpose. We also had discussions with many OLE Nepal officials, which aided us in putting our observations in proper context.

Teacher training

The teacher training was divided in two parts: residential training and in-school training. A four day residential training was conducted from March 29 to April 1, 2008 and four-day in-school training followed after a couple of weeks in between and was completed on May 2, 2008. The residential training included teachers from both the schools together and was held in Panauti, in the outskirts of Kathmandu Valley (35 kilometers from Kathmandu), while the in-school trainings were separately conducted in their respective schools. 10 teachers from Bashuki and 11 from Bishwamitra participated in the residential training.

One of the goals of the training was to make teachers comfortable to integrate laptop use in regular teaching. When the teachers were asked whether the complete training was sufficient for enabling them to properly integrate laptop in the regular classroom instruction process, 50 % of the teachers’ response was positive. This response underscores the need to have refresher training on a regular basis as most of the teachers are using computer for the first time.

Residential training

Teachers found the following aspect of the residential training useful:

  • Encouraging peer-learning (learning from their colleagues rather than simply from the instructor)
  • Information about how to use laptop for instructional purposes to make teaching more
  • effective
  • Sessions on how to make lesson plans
  • An opportunity to meet colleagues and learn from them

Among those who said there were parts of the residential training that needed to be reduced or removed next time, all of them cited that a lot of time was spent on going over theoretical parts, which limited time spent on practical aspects on using laptop.

With regards to whether the materials in the residential portion of the training were adequate, we obtained conflicting answers from the two schools. While the overwhelming majority of the teachers in Bishwamitra (89%) thought that materials were adequate, less than one-third (29%) of teachers in Bashuki said that it was adequate. Among those who said it was inadequate, they wanted the focus to be more on practical rather than on theoretical aspects. Some teachers wanted more examples on how to teach using laptops. Though the majority of teachers at Bashuki found the materials inadequate, more than two-third of the teachers there (71%) agreed that the residential training was delivered in an understandable manner that was easy to follow. For Bishwamitra, the corresponding number was 89%.

When the teachers were asked whether it was a good idea to run residential and in-school trainings separately, almost everyone (15 out of 16) said that it was a good idea and wanted it to be that way in future as well.

In-school training

Teachers found many aspects of the in-school training useful such as:

  • Sessions on class management, making lesson plans, and other tips on teaching
  • Highlight the fact that the laptops should only be used as a tool
  • Clearly pointing out the mistakes when teaching so that they could work on their mistakes and improve
  • The feedback on their presentations

To the question what aspects in the lesson planning practice they found particularly useful, they mentioned:

  • Encouraging the habit of incorporating lesson plans in their teachings
  • Effective time management: how best to allocate time
  • Teaching about how to present materials in the classroom
  • Learning when and how to use laptop in the class

All 17 teachers found it useful to review lesson plans together in a group and to attend laptop classes conducted by colleagues and observe the class.

Classroom teaching learning process

Teaching learning digital materials

All the teachers said that the materials in the laptops were, in general, consistent with the interests of the students in their classes. However, when asked whether the materials in the laptops, in general, were consistent with the level or capacity of the students in their classes, the answer was rather mixed. Teachers were generally satisfied with the grade 2 contents, but had diverse views on grade 6. The teachers found many initially uploaded activities for grade 6 either too easy or too difficult. To the question about whether the materials in the laptops are able to meet the aims and objectives of your class as prescribed in the curriculum, the teachers feel that the recent digital contents are more relevant in this front.

The teachers mentioned that they generally spent one period to go over one activity. This means that if the expectation is that the teachers use laptops for three days a week, then there should be about three activities prepared for each week.

The teachers also mentioned that there were not sufficient exercises, especially for grade 6 students to work on. Since the teachers are so used to following the chapters in the book, the teachers suggested that the teaching materials in the laptops be sequenced according to the chapters in the textbooks. This way, they will not have to spend a lot of time figuring out which activity corresponds to the relevant chapter in the book.

The teachers also suggested that there be multiple choice question answers and practice questions. Some teachers proposed having different levels in the activities so that students develop a competitive spirit and try harder to go to a higher level. This could motivate them to study harder.

Classroom teaching learning

It is important to understand how classroom teaching learning has been affected by laptop use in classrooms. All the 17 teachers in both the schools feel that use of computers has helped their teaching. All (relevant) teachers feel that their lectures are now more organized and that it is easier to teach students new concepts. They also believe that the use of laptop based instruction has enabled students to learn concepts more easily. They are also of the opinion that laptop based instruction has made it easier to give students more practice exercises. In addition, they also feel that the use of laptop based instruction has made the classes more interactive. All the teachers are unanimous in saying that the use of laptop based instruction has increased the students’ interest in their studies.

In Biswamitra, teachers said that they generally use laptops 2 to 3 days a week while the number, according to the teachers, is at least 3 days in Bashuki. When they use the laptops, they mostly used the laptops between ¼ and ½ of the period. The grade 6 student survey, which was taken independently, also largely confirms this assertion. The mode in Bishwamitra was 3 days while it was 4 in Bashuki.

Students in grade 6 said that teachers mostly used laptops around the middle of the class. When some of the students forget to bring their laptops, the teachers ask those students to look at the laptop of the student who is sitting next to her.

The students in Bishwamitra generally say that the teachers give them homework every time the laptop is used in the classroom while those in Bashuki said that they are sometimes given homework. According to the teachers, those homework are mostly of review type to be done on laptops on a regular basis. The teachers ask students to do certain activity in the laptop for that day. The teachers, at times, look at the record in the computers to see whether the student has at least opened that activity at home or not, as one could find those details in the laptop.

When the students were asked how learning has been affected with the introduction of laptops in their classrooms, more than 95% of grade 6 students find learning using laptops easier,

As to why it is easier, the most common response was that you can do the exercise or activities as many times as you want. The second most common one was that these activities were very enjoyable to learn from. The students also appreciated the fact that you could do these activities at ones own pace. All these options were cited by more than 50% of the students.

When the students were asked whether other family members also use the laptops, more than eighty percent said yes. Siblings are the ones who use the laptop the most, followed by father and mother. This information is also confirmed with data from household survey where many siblings said that they use the laptop regularly.

One question that might reveal a lot about how the students are using the laptop was about what the students did the most in the laptop. In Bishwamitra, 68% of the grade 6 students (15) said they mostly use the laptop for educational activities while 23% (5) said to take pictures. The number for Bashuki was 32% for laptop activities while 42% was to take pictures.

When the grade 6 students were asked how effectively they are able to use digital activities in the laptop, most of them said they can use it well. 83% in Bishwamitra said that they can use it well while 17% said they can use it somewhat. In Bashuki, the corresponding figures were 75 % and 25 % respectively. No one said they cannot use the digital materials even though it was one of the options given.

Some teachers feel that the introduction of laptop has made managing the classes more difficult. Many teachers feel that the noise level has substantially increased. This has to do with the fact that students have difficulty controlling their excitements. The teachers hope that the noise level will subside once the students get used to using laptops. The teachers also admit that they are, at times, having difficulty managing time because they are not used to integrate laptop use in classrooms.

Class preparation

When teachers were asked “Compared to the effort you had to put in the teaching learning process before receiving the laptops, how much effort do you put in now?”, the overwhelming majority of the teachers feel that they put in more effort now.

Some of the reasons cited by the teachers on why the effort has increased:

  • Making lesson plans
  • Preparing how to incorporate laptop while teaching
  • Selecting which activities matches the subject matter being taught
  • Extra effort to bring down the noise level
  • Preparing for the security and management of the laptop in the classrooms.

Laptop and network issues

Almost all teachers (16 out of 17) find the laptop and its layout easy to use. They think it is very intuitive and also feel that the students should not have much difficulty. Most of the teachers say it takes about 1.5 hours to fully recharge the computer and that the charge lasts for about 3 hours when the computer is in use.

We had also asked the teachers, students and their family members whether they had any problem with different aspects of computer use. The biggest problem more than 50 % of the respondents mentioned was the jumpy cursor. The dust, the humid climate and the student’s sweaty hands as a result of running around might be contributing to the situation. The problem seems to be slightly more acute in Bashuki.

More than one-third of the teachers thought the sound quality needs to be improved. There either is some problem with the sound quality or that some teachers might have difficulty adjusting the sound. Since earphones are not available, difficulties arise for students and teachers when using sound related activities. They largely ask students to listen to the sound from the teacher’s laptop while teaching.

The school, students and their family members in general are carefully protecting the laptop. At home the students keep the laptop away from fire and water. Those who have a closet at home keep the laptop there when not in use. Other students securely keep the laptop in the bag. The family members are aware that the laptop should be stored securely.

Difference between the two test schools

There are reasons to believe that the two schools are different. For example, teachers in Bashuki say that the community members there do not value education as much. The standardized tests taken in the early days of laptop provision also shows that the test scores in Bishwamitra are much higher than that of Bashuki. In addition, the attendance of students in Bishwamitra is more regular than in Bashuki.

There are differences in responses to some of the critical questions in these two schools which could be because of the reasons identified above. So, it is very essential to identify the root causes behind these differences. One could argue that these differences are hinting at the fact that there might be certain enabling conditions that need to be in place to make sure that the program’s objective are fully met. It could also mean that it might take time in some places to fully benefit from the initiative while we could see tangible results in some schools right away. For the pilot study, the OLE Nepal might want to seriously consider whether all the schools in the pilot phase need to be treated similarly or whether some activities, like sensitizing the community about the merits of ICT- in-education, need to be conducted in certain areas beforehand.

Some Suggestions and Recommendations

E-library and internet access

I think OLE Nepal should try to upload all the books that are in the website to student laptops sooner rather than later. This serves at least two purposes. First, students who want to read those books could easily read them. Second, the users (teachers, students and their family members) might offer some suggestions that might help further improve the way e-library is accessed (e.g. layout of the books). Needless to say the parents and teachers are really looking forward to using the e-library.

Without internet access at home, convincing others that one laptop per child is preferred to, say, a well-planned computer laptop in the school might be tougher, especially when the cost difference between these two alternatives can be substantial.

Work load on teachers

Most teachers in both the test schools feel that their workload has increased significantly. This is a serious concern. If this trend continues, it is likely that they will revert, to a large extent, to traditional way of teacher and ignore the materials in the laptop once the initial enthusiasm of laptop fades away. If this happens, the effect of laptops with educational content on student learning might be minimal. Steps to further reduce workload and encourage laptop use in classrooms need to be devised carefully.

Some ways to decrease teacher’s workload include:

  • Better integration of textbook and materials in the laptop.
  • Include ‘sample’ lesson plans for at least those lectures that use activities developed by OLE Nepal which teacher’s can follow if they like.
  • Refresher trainings to the teachers on a regular basis where different ways to use laptops/digital materials and the concerns teachers have are addressed.

It is apparent that teachers don’t have much free time in the two test schools. Most of them have to teach for six periods (out of seven) in Bashuki. The case is not much different in Bishwamitra. Moreover, for the teachers’ at Bashuki, almost all the teachers there have to walk more than 2 hours a day to reach the school. All these factors coupled with extra effort in laptop use might tempt teachers to revert to the style they are most comfortable with.

Chargers

I strongly feel that OLE Nepal should take steps in allowing students to take chargers home so that the laptop can be used longer and that other family members or community members can also benefit. If the charging is only done at school, there is concern that students might not be able to use the laptop adequately at home once more activities are uploaded in the laptop. If OLE Nepal expects students and their family members to use the laptop for materials in the e-library at home, it makes more sense to allow students to take chargers home.

I think the OLE-Nepal should go ahead with the plans about providing 2 chargers per student with laptops, so the students can take one charger home and the other one can be permanently kept at School. This way, the students do not need to worry about carrying charger every day. The chances of losing or misplacing ones chargers are also minimized.

Books by Janak Shikshya Samagri (official books used in classrooms) in the laptop

One concern that was raised by teachers and family members in Biswamitra School was that the load the students carry is huge. It is particularly dangerous during the monsoon season. Many suggested having soft copies of the books used in the school included in the laptops. This would also be one of the immediate and tangible result of the laptop provision. Disruptions in classrooms to late delivery of books, which happened this academic year, would also be minimized.

Allowance and recognition for teachers

Though I consider providing regular compensation for extra work resulting from laptop use as highly impractical, I however think some arrangements should be made to compensate teachers when they attend programs like teacher trainings. In addition, OLE Nepal might also want to provide them some recognition (e.g. letters) that teachers, for example, could use for professions development.

Conclusion

Considering the fact that this is the first time such initiative is being implemented in Nepal, the test phase in the two schools in the outskirts of Kathmandu Valley is running very well. The teachers, family members and the SMC members all see great promise in this initiative to improve student learning. The head teachers in both the schools consider the pilot program as very effective and see great promise in reducing the disparity in educational quality between private and public schools.

Almost all the teachers and parents think that there is positive relationship between laptop provision and student learning. Though it is hard to quantify, most teachers and parents also feel that the students who have been provided laptops have become cleverer. With this student-centered approach, students are interacting with each other more often. It has made students more curious and they are eager to learn new things. It has also helped in developing co-operative spirit as students are willing to help each other learn new technology.