Home | Wiki | Blog | Contact Us | Contribute

Formative Evaluation of OLPC Project Nepal: A Summary

September 28th, 2008 By:Rabi Karmacharya · 13 Comments

This is a summary of the findings of a formative evaluation carried out by Mr. Uttam Sharma, a doctoral student at at the Department of Applied Economics at the University of Minnesota. The evaluation was carried out for OLE Nepal’s internal purpose. The results are expected to help learn about the shortcomings of the current test phase of the OLPC project so that necessary fine tuning can be done before next year’s larger deployment in more districts. Mr. Sharma can be reached at sharm061@umn.edu.  

Formative Evaluation of OLPC Project Nepal: A Summary

June-August 2008

Background:

The Nepal government’s Department of Education (DoE) and the Open Learning Exchange Nepal (OLE Nepal) had selected Bishwamitra Ganesh Secondary School and Bashuki Lower Secondary Schools, both in Lalitpur district of Nepal, as test schools for the One Laptop per Child (OLPC) initiative in Nepal. The laptops were distributed to all 135 students in grade 2 and 6 in those two schools on April 25, 2008. The students were also allowed to take laptops home. However, charging of the laptops could only be done in school. The implementation of the initiative has focused on four areas: digital content development, teacher preparation, network and power infrastructure development and government capacity building.

Study design and method

This study was initiated after about two months of program implementation. The objective of the present study is formative evaluation which assesses on-going project activities. This evaluation focuses on how the initiative is being implemented in the two test schools and looks at how the implementation can be altered or improved.

The study period was between mid-June to mid-August 2008. The first week was spent preparing survey questionnaires. The first round of field visit was conducted between June 22 and 27. The first three days were spent in Bishwamitra Secondary School while the last three days in Bashuki Lower Secondary School. The 2nd round of field visit was conducted between July 27 and August31, 2008.

This study is based on the information collected from surveys with teachers, head-teachers, students in grade 6 and their family members, some School Management Team members, school records, the school censuses from the two test schools as well as detailed discussions with OLE Nepal officials and postings in the OLE Nepal blog (http://blog.olenepal.org/). One-on-one meetings with teachers were also conducted in addition to filling out the survey questionnaires.

The questionnaires were designed to elicit as much information about how the initiative is being run. Because of time constraints, we tried surveying only those students who roll numbers were odd (or only even numbered in other cases). All the regular teachers who were present in both the schools were surveyed. In total, 31 students from grade 2, 45 students from grade 6, 19 teachers that includes 2 head teachers, and 20 households were surveyed for the purpose. We also had discussions with many OLE Nepal officials, which aided us in putting our observations in proper context.

Teacher training

The teacher training was divided in two parts: residential training and in-school training. A four day residential training was conducted from March 29 to April 1, 2008 and four-day in-school training followed after a couple of weeks in between and was completed on May 2, 2008. The residential training included teachers from both the schools together and was held in Panauti, in the outskirts of Kathmandu Valley (35 kilometers from Kathmandu), while the in-school trainings were separately conducted in their respective schools. 10 teachers from Bashuki and 11 from Bishwamitra participated in the residential training.

One of the goals of the training was to make teachers comfortable to integrate laptop use in regular teaching. When the teachers were asked whether the complete training was sufficient for enabling them to properly integrate laptop in the regular classroom instruction process, 50 % of the teachers’ response was positive. This response underscores the need to have refresher training on a regular basis as most of the teachers are using computer for the first time.

Residential training

Teachers found the following aspect of the residential training useful:

  • Encouraging peer-learning (learning from their colleagues rather than simply from the instructor)
  • Information about how to use laptop for instructional purposes to make teaching more
  • effective
  • Sessions on how to make lesson plans
  • An opportunity to meet colleagues and learn from them

Among those who said there were parts of the residential training that needed to be reduced or removed next time, all of them cited that a lot of time was spent on going over theoretical parts, which limited time spent on practical aspects on using laptop.

With regards to whether the materials in the residential portion of the training were adequate, we obtained conflicting answers from the two schools. While the overwhelming majority of the teachers in Bishwamitra (89%) thought that materials were adequate, less than one-third (29%) of teachers in Bashuki said that it was adequate. Among those who said it was inadequate, they wanted the focus to be more on practical rather than on theoretical aspects. Some teachers wanted more examples on how to teach using laptops. Though the majority of teachers at Bashuki found the materials inadequate, more than two-third of the teachers there (71%) agreed that the residential training was delivered in an understandable manner that was easy to follow. For Bishwamitra, the corresponding number was 89%.

When the teachers were asked whether it was a good idea to run residential and in-school trainings separately, almost everyone (15 out of 16) said that it was a good idea and wanted it to be that way in future as well.

In-school training

Teachers found many aspects of the in-school training useful such as:

  • Sessions on class management, making lesson plans, and other tips on teaching
  • Highlight the fact that the laptops should only be used as a tool
  • Clearly pointing out the mistakes when teaching so that they could work on their mistakes and improve
  • The feedback on their presentations

To the question what aspects in the lesson planning practice they found particularly useful, they mentioned:

  • Encouraging the habit of incorporating lesson plans in their teachings
  • Effective time management: how best to allocate time
  • Teaching about how to present materials in the classroom
  • Learning when and how to use laptop in the class

All 17 teachers found it useful to review lesson plans together in a group and to attend laptop classes conducted by colleagues and observe the class.

Classroom teaching learning process

Teaching learning digital materials

All the teachers said that the materials in the laptops were, in general, consistent with the interests of the students in their classes. However, when asked whether the materials in the laptops, in general, were consistent with the level or capacity of the students in their classes, the answer was rather mixed. Teachers were generally satisfied with the grade 2 contents, but had diverse views on grade 6. The teachers found many initially uploaded activities for grade 6 either too easy or too difficult. To the question about whether the materials in the laptops are able to meet the aims and objectives of your class as prescribed in the curriculum, the teachers feel that the recent digital contents are more relevant in this front.

The teachers mentioned that they generally spent one period to go over one activity. This means that if the expectation is that the teachers use laptops for three days a week, then there should be about three activities prepared for each week.

The teachers also mentioned that there were not sufficient exercises, especially for grade 6 students to work on. Since the teachers are so used to following the chapters in the book, the teachers suggested that the teaching materials in the laptops be sequenced according to the chapters in the textbooks. This way, they will not have to spend a lot of time figuring out which activity corresponds to the relevant chapter in the book.

The teachers also suggested that there be multiple choice question answers and practice questions. Some teachers proposed having different levels in the activities so that students develop a competitive spirit and try harder to go to a higher level. This could motivate them to study harder.

Classroom teaching learning

It is important to understand how classroom teaching learning has been affected by laptop use in classrooms. All the 17 teachers in both the schools feel that use of computers has helped their teaching. All (relevant) teachers feel that their lectures are now more organized and that it is easier to teach students new concepts. They also believe that the use of laptop based instruction has enabled students to learn concepts more easily. They are also of the opinion that laptop based instruction has made it easier to give students more practice exercises. In addition, they also feel that the use of laptop based instruction has made the classes more interactive. All the teachers are unanimous in saying that the use of laptop based instruction has increased the students’ interest in their studies.

In Biswamitra, teachers said that they generally use laptops 2 to 3 days a week while the number, according to the teachers, is at least 3 days in Bashuki. When they use the laptops, they mostly used the laptops between ¼ and ½ of the period. The grade 6 student survey, which was taken independently, also largely confirms this assertion. The mode in Bishwamitra was 3 days while it was 4 in Bashuki.

Students in grade 6 said that teachers mostly used laptops around the middle of the class. When some of the students forget to bring their laptops, the teachers ask those students to look at the laptop of the student who is sitting next to her.

The students in Bishwamitra generally say that the teachers give them homework every time the laptop is used in the classroom while those in Bashuki said that they are sometimes given homework. According to the teachers, those homework are mostly of review type to be done on laptops on a regular basis. The teachers ask students to do certain activity in the laptop for that day. The teachers, at times, look at the record in the computers to see whether the student has at least opened that activity at home or not, as one could find those details in the laptop.

When the students were asked how learning has been affected with the introduction of laptops in their classrooms, more than 95% of grade 6 students find learning using laptops easier,

As to why it is easier, the most common response was that you can do the exercise or activities as many times as you want. The second most common one was that these activities were very enjoyable to learn from. The students also appreciated the fact that you could do these activities at ones own pace. All these options were cited by more than 50% of the students.

When the students were asked whether other family members also use the laptops, more than eighty percent said yes. Siblings are the ones who use the laptop the most, followed by father and mother. This information is also confirmed with data from household survey where many siblings said that they use the laptop regularly.

One question that might reveal a lot about how the students are using the laptop was about what the students did the most in the laptop. In Bishwamitra, 68% of the grade 6 students (15) said they mostly use the laptop for educational activities while 23% (5) said to take pictures. The number for Bashuki was 32% for laptop activities while 42% was to take pictures.

When the grade 6 students were asked how effectively they are able to use digital activities in the laptop, most of them said they can use it well. 83% in Bishwamitra said that they can use it well while 17% said they can use it somewhat. In Bashuki, the corresponding figures were 75 % and 25 % respectively. No one said they cannot use the digital materials even though it was one of the options given.

Some teachers feel that the introduction of laptop has made managing the classes more difficult. Many teachers feel that the noise level has substantially increased. This has to do with the fact that students have difficulty controlling their excitements. The teachers hope that the noise level will subside once the students get used to using laptops. The teachers also admit that they are, at times, having difficulty managing time because they are not used to integrate laptop use in classrooms.

Class preparation

When teachers were asked “Compared to the effort you had to put in the teaching learning process before receiving the laptops, how much effort do you put in now?”, the overwhelming majority of the teachers feel that they put in more effort now.

Some of the reasons cited by the teachers on why the effort has increased:

  • Making lesson plans
  • Preparing how to incorporate laptop while teaching
  • Selecting which activities matches the subject matter being taught
  • Extra effort to bring down the noise level
  • Preparing for the security and management of the laptop in the classrooms.

Laptop and network issues

Almost all teachers (16 out of 17) find the laptop and its layout easy to use. They think it is very intuitive and also feel that the students should not have much difficulty. Most of the teachers say it takes about 1.5 hours to fully recharge the computer and that the charge lasts for about 3 hours when the computer is in use.

We had also asked the teachers, students and their family members whether they had any problem with different aspects of computer use. The biggest problem more than 50 % of the respondents mentioned was the jumpy cursor. The dust, the humid climate and the student’s sweaty hands as a result of running around might be contributing to the situation. The problem seems to be slightly more acute in Bashuki.

More than one-third of the teachers thought the sound quality needs to be improved. There either is some problem with the sound quality or that some teachers might have difficulty adjusting the sound. Since earphones are not available, difficulties arise for students and teachers when using sound related activities. They largely ask students to listen to the sound from the teacher’s laptop while teaching.

The school, students and their family members in general are carefully protecting the laptop. At home the students keep the laptop away from fire and water. Those who have a closet at home keep the laptop there when not in use. Other students securely keep the laptop in the bag. The family members are aware that the laptop should be stored securely.

Difference between the two test schools

There are reasons to believe that the two schools are different. For example, teachers in Bashuki say that the community members there do not value education as much. The standardized tests taken in the early days of laptop provision also shows that the test scores in Bishwamitra are much higher than that of Bashuki. In addition, the attendance of students in Bishwamitra is more regular than in Bashuki.

There are differences in responses to some of the critical questions in these two schools which could be because of the reasons identified above. So, it is very essential to identify the root causes behind these differences. One could argue that these differences are hinting at the fact that there might be certain enabling conditions that need to be in place to make sure that the program’s objective are fully met. It could also mean that it might take time in some places to fully benefit from the initiative while we could see tangible results in some schools right away. For the pilot study, the OLE Nepal might want to seriously consider whether all the schools in the pilot phase need to be treated similarly or whether some activities, like sensitizing the community about the merits of ICT- in-education, need to be conducted in certain areas beforehand.

Some Suggestions and Recommendations

E-library and internet access

I think OLE Nepal should try to upload all the books that are in the website to student laptops sooner rather than later. This serves at least two purposes. First, students who want to read those books could easily read them. Second, the users (teachers, students and their family members) might offer some suggestions that might help further improve the way e-library is accessed (e.g. layout of the books). Needless to say the parents and teachers are really looking forward to using the e-library.

Without internet access at home, convincing others that one laptop per child is preferred to, say, a well-planned computer laptop in the school might be tougher, especially when the cost difference between these two alternatives can be substantial.

Work load on teachers

Most teachers in both the test schools feel that their workload has increased significantly. This is a serious concern. If this trend continues, it is likely that they will revert, to a large extent, to traditional way of teacher and ignore the materials in the laptop once the initial enthusiasm of laptop fades away. If this happens, the effect of laptops with educational content on student learning might be minimal. Steps to further reduce workload and encourage laptop use in classrooms need to be devised carefully.

Some ways to decrease teacher’s workload include:

  • Better integration of textbook and materials in the laptop.
  • Include ‘sample’ lesson plans for at least those lectures that use activities developed by OLE Nepal which teacher’s can follow if they like.
  • Refresher trainings to the teachers on a regular basis where different ways to use laptops/digital materials and the concerns teachers have are addressed.

It is apparent that teachers don’t have much free time in the two test schools. Most of them have to teach for six periods (out of seven) in Bashuki. The case is not much different in Bishwamitra. Moreover, for the teachers’ at Bashuki, almost all the teachers there have to walk more than 2 hours a day to reach the school. All these factors coupled with extra effort in laptop use might tempt teachers to revert to the style they are most comfortable with.

Chargers

I strongly feel that OLE Nepal should take steps in allowing students to take chargers home so that the laptop can be used longer and that other family members or community members can also benefit. If the charging is only done at school, there is concern that students might not be able to use the laptop adequately at home once more activities are uploaded in the laptop. If OLE Nepal expects students and their family members to use the laptop for materials in the e-library at home, it makes more sense to allow students to take chargers home.

I think the OLE-Nepal should go ahead with the plans about providing 2 chargers per student with laptops, so the students can take one charger home and the other one can be permanently kept at School. This way, the students do not need to worry about carrying charger every day. The chances of losing or misplacing ones chargers are also minimized.

Books by Janak Shikshya Samagri (official books used in classrooms) in the laptop

One concern that was raised by teachers and family members in Biswamitra School was that the load the students carry is huge. It is particularly dangerous during the monsoon season. Many suggested having soft copies of the books used in the school included in the laptops. This would also be one of the immediate and tangible result of the laptop provision. Disruptions in classrooms to late delivery of books, which happened this academic year, would also be minimized.

Allowance and recognition for teachers

Though I consider providing regular compensation for extra work resulting from laptop use as highly impractical, I however think some arrangements should be made to compensate teachers when they attend programs like teacher trainings. In addition, OLE Nepal might also want to provide them some recognition (e.g. letters) that teachers, for example, could use for professions development.

Conclusion

Considering the fact that this is the first time such initiative is being implemented in Nepal, the test phase in the two schools in the outskirts of Kathmandu Valley is running very well. The teachers, family members and the SMC members all see great promise in this initiative to improve student learning. The head teachers in both the schools consider the pilot program as very effective and see great promise in reducing the disparity in educational quality between private and public schools.

Almost all the teachers and parents think that there is positive relationship between laptop provision and student learning. Though it is hard to quantify, most teachers and parents also feel that the students who have been provided laptops have become cleverer. With this student-centered approach, students are interacting with each other more often. It has made students more curious and they are eager to learn new things. It has also helped in developing co-operative spirit as students are willing to help each other learn new technology.

Tags: Uncategorized

13 responses so far ↓

  • 1 Joe // Sep 29, 2008 at 2:03 am

    thank you for this insight into the operation of these laptops in Nepal.

    One thought occurred to me that might help alleviate the extra burden on teachers – Is there a way for teachers to share lesson plans? A Wiki where teachers can share information? this could help reduce the effort required in preparing lesson plans and adapting the laptop contents to the curiculum.

    Such a site could also provide a forum where teachers who do put in the extra work on lesson plans can get some recognition for that work.

    This is probably obvious and you have it all in hand already. Good luck to you all.

    Joe

  • 2 sanobhai // Sep 30, 2008 at 10:10 pm

    8 days of training and the teachers are expected to start using these modern technologies in teaching? And effectively? The teachers would not only need to learn how to use laptops but also how to teach! People who do not emphasize the latter need to just visit some schools in rural areas. The teachers hardly show up and when they show up beat the heck out of students, make them memorize a few things and go away. It goes without saying that most teachers are not trained to be teachers. I would think it might be a more impactful for Uttam, Joe, and Rabi to devote some of their time solving that problem.

    I feel that this project is becoming more about laptops and less about education. I forgot again – it has always been that way. People never thought – ok, we need to improve education, how can we do that? It has always been: in the US (We can build cool things, although we don’t know the problems in education sector in the poor countries, we have visited the poor countries once or twice and we have teaching experience in the US, this tool might help) and in Nepal (We are getting laptops and maybe we can use it in school to somehow improve some things). Note that I am not maligning your good intent – what is disappointing to me is lack of thought on what the major problems are and then finally deciding what tools are necessary to tackle those problems.

    I think the best use of these laptops would be to read books on them. Timely book distribution, as Uttam notes, is a big problem.

  • 3 Tim Moody // Oct 7, 2008 at 2:35 am

    What steps would be needed to publish standard textbooks on the OLPC in terms of securing and migrating the material?

  • 4 Ian // Oct 7, 2008 at 4:29 am

    Sanobhai,

    I find you comments very offensive to the excellent work put in by the Nepal Team.
    They are one of the better prepared OLPC projects and are willingly sharing their experiences.

    Beating up the US may be a useful pass time for you, but I am into improving the lives of children.
    Thank you Neapl for your excellent and helpful work

  • 5 Bryan Berry // Oct 13, 2008 at 2:06 pm

    Tim, lots of steps but the key step is to get permission from the Dept. of Ed. to distribute digital copies of the textbooks. This is a time consuming project that we are in the middle of. Most of textbooks are in Pagemaker or Word documents. We have figured out a handy way to convert Pagemaker docs to PDF or odt yet.

  • 6 sanobhai // Oct 13, 2008 at 7:00 pm

    Ian,

    I think you should re-read my comments. I am pointing out the misplaced enthusiasm among the planners of this project. The implementers are doing a fine job. You probably confused the two.

    If you asked people in Nepal what they need to improve education, laptops would probably not appear as one of the top ten items on that list. That is my uneducated and non-expert guess. And by the way, you would not want to ask people in Nepal what they need only if you think they don’t know what is good for them, which is a polite way of saying they are stupid. That is a reality check on donor driven development projects.

    Bryan – is there any chance that the textbooks will be available to everyone for free or it will be an exclusive sort of license to OLPC? I think open content has negligible near-term impact in Nepal but I am curious. How often do you think the textbooks will change over the years?

  • 7 eMBee // Oct 30, 2008 at 12:42 am

    while i agree that educating teachers is most important, i believe that having the laptops helps the teachers to accept a change in their teaching style and motivates them to improve their teaching work.

    the alternative would be to send a group of people from school to school and just offer teacher training.

    would that really be more effective?

    with the laptops, the need to change the teaching becomes more obvious while at the same time the children get more opportunities for independant learning. that way everybody wins.

    greetings, eMBee.

    regard man as a mine rich in gems of inestimable value. education alone can cause it to reveal its treasures and allow mankind to benefit therefrom.

  • 8 Greg Smith // Nov 5, 2008 at 2:40 am

    Very helpful report! Thanks.

  • 9 Sarama // Jan 19, 2009 at 2:36 am

    amazing post))

  • 10 Ksana // Jan 20, 2009 at 4:34 am

    Good night, bloggers =)

  • 11 waris // May 17, 2009 at 1:25 pm

    Good Job Nepal
    Can you share your monitoring and evaluation questioners with us (OLPC Afghanistan)???

  • 12 Rabi Karmacharya // May 17, 2009 at 2:18 pm

    Waris. all our questionnaires are in Nepali language, so I am not sure if they’d be of much help to you. You can contact Uttam Sharma at sharm061@umn.edu if you want more insights into planning and structure of the evaluation. We are also preparing the final result of the evaluation which was completed earlier this month.

  • 13 sac a main longchamp soldes // Mar 22, 2013 at 10:45 pm

    I wanted to inform you for this great help!! I definitely enjoyed every little bit of it.

Leave a Comment