Notes from Nepal’s OLPC Deployments

Nepal’s OLPC deployments are now more than seven months old. I get a lot of questions about deployment issues and solutions. I hope the following information is helpful. First, a snapshot of our deployments:


OLE’s progress in Nepal
  • Deployments began at Vishwamitra Ganesh and Bashuki public schools on April 23rd, 2008.
  • 135 XO-1 laptops deployed to students in classes 2 and 6
  • 0 laptops stolen, lost, or otherwise missing. One laptop has been seriously damaged when the child who owned it cleaned it carefully with soap and water. Otherwise no laptops have been seriously damaged as a result of use.
  • 8 bad motherboards, 5 bad microphones, and 4 bad keyboards
  • Kids use the laptops in the classroom 1-2 periods per week
  • Most significant technical issue we have encountered is the “jumpy cursor” problem
  • Teachers, parents, and kids are quite happy with the project according to our surveys

In the Beginning: Teacher Training

We conducted four days of teacher training off-site and five days on-site in the classroom with both the students and teachers. A large portion of our teachers had never used a computer before but they learned very quickly. Their enthusiasm was amazing. Training during the off-site sessions formally ended at 5:30 pm but the teachers stayed in our training room each night until 11 pm, pounding away on the XO’s and asking endless questions.

While the teachers learned how to use the XO very quickly, I miscalculated how difficult certain actions would be for them. Specifically, it took them a while to learn “dragging and dropping” with the touchpad. Many of the best activities on the XO require serious dexterity with the touchpad such as TurtleArt, Etoys, and Scratch. For this reason we couldn’t cover these activities during training. I recommend starting teacher training with activities that do not require a lot of dexterity with the touchpad.

Keeping Kids and Laptops Safe

We were quite worried about the security of the laptops and the safety of the children carrying these expensive machines to and from school everyday. Vishwamitra and Bashuki are both in poor communities. So far our fears have proved unfounded. Not a single XO has been lost, stolen, or otherwise gone missing. I believe this is due to two factors.

  1. The children and their parents value the XO’s and protect them the accordingly.
  2. Crime in rural Nepal is low relative to other developing countries. In rural communities, everyone knows each other’s business.
Nepal olpc art
Limbu script on OLPC XO

Laptop Maintenance and Support

One of the great surprises is how little time our team has had to spend on XO maintenance and support. The XO-1 is a fantastic piece of hardware and I am convinced that anyone with a mechanical aptitude can fix 90% of the hardware problems that arise by swapping out the bad component with a good one.

The “test-all” command in the XO’s OpenFirmware is an indispensable tool. Just run “test-all” and the firmware will spit out a report indicating if any particular hardware component has failed. Unfortunately, you cannot access the firmware prompt without activating the XO with the developer key. For that reason, we requested developer keys for all of our XO’s and disabled the firmware security on each and every one.

We trained select teachers from both schools how to fix software and hardware problems. They can handle most of the hardware-related problems that come up. In my limited experience, it is feasible to train teachers how to fix hardware problems but more difficult to teach them how to fix software problems in the linux kernel or within Sugar. If a serious software problem comes up, our teachers simply reflash the stricken XO with a Nepal-specific software image that includes the standard activities.

Under Pressure: Keeping Up with the Curriculum

In OLPC-land we like to talk about lofty concepts such as constructionism, co-learning, collaboration, etc. Meanwhile, teachers at Bashuki and Vishwamitra have more pressing concerns. The Nepali system does not practice social promotion. Children have to pass year-end examinations to move on to the next grade. Nepali teachers are interested in constructionism, co-learning, and collaboration as long as they don’t hinder their students progress through the educational system. Our teachers are quite happy with the E-Paath suite of educational activities that OLE Nepal developed in accordance with the national curriculum. The real attraction of OLPC for teachers is that in class they can task students with a problem on the XO and then spend much of the period working with students that need help.

the future of nepal
XO laptops in the Nepali wild

The Missing Piece: The School Server

When we deployed the school server back in April, XS-163 was a very immature server configuration. The XS has come a long way since then under the leadership of XS architect Martin Langhoff. Then and now, the XS does not come with a content filter pre-configured. We had to spend a significant amount of time configuring Dansguardian so that it wouldn’t block a lot of good content and still block the bad stuff.

For example, “Dikshit” is a common surname in Nepal. Dansguardian will block access to news articles containing the name because it interprets the last four letters as profanity. The XS is still under heavy development and deployment teams need a significant amount of linux expertise to deploy a fully functional XS.

Top Requests from Teachers and Kids

  • Easier way to play music and video
  • A better E-Book reader
  • A lot more activities for learning English
  • All the Nepali textbooks in digital format
  • A comprehensive digital library with lots of Nepali-language reading materials
  • A Typing Tutor program for learning English and Nepali
  • Interactive learning activities that match the Nepali curriculum
  • A car racing game (the kids)

We will incorporate the lessons we have learned in these two deployment schools when we expand to 15-20 schools in 5-6 districts in April 2009. Hopefully, I will find more time to write about our experiences over the next several weeks but don’t count on it. I have been working on OLPC for more than 2.5 years and I have to say it has absolutely, positively been worth all the trouble.

Postscript: Pradosh Kharel has been working on a comprehensive Deployment Plan for Nepal. Check it out here.

Bryan Berry is the Technology Directory of OLE Nepal and deadbeat co-editor of OLPCNews.com. OLE Nepal is implementing Nepal’s OLPC deployments in partnership with Nepal’s Department of Education.

Deploying Volunteers

You may have read Rabi’s recent post about the formative evaluation of our test schools Vishwamitra Ganesh and Bashuki. We are proud of the progress we have made and are now starting to focus on expanding OLPC to more Nepali schools in spring 2009. Our current plan is to expand from one district to five districts (equivalent to a state of province) and roughly 10-15 schools. Technical support for our current two schools within driving distance of the office has not been particularly challenging or time-consuming. However, supporting schools in five distinct regions of Nepal poses a tremendous challenge.

We have found that teachers and children can do much of the day-to-day support and repairs once they have proper training. I had 3 days of quality time with teachers Neema of Bashuki and Manoj of Vishwamitra back April. They are now able to solve a large part of the technical problems that arise. We have decided that we need on-site technical support for at least the first couple weeks of each school but there is no way we can spread out our small staff across all those schools. We could contract engineers to do it but short-term contractors won’t be familiar with Sugar and the activities on the XO. Someone once suggested we could get foreign volunteers to the deployment, a good idea in theory but not in practice. Foreigners won’t speak the local language nor will they be familiar with the sociocultural environment. This limits their effectiveness considerably. Additionally, foreigners are easy marks for local bacteria.

There is an active and vibrant FOSS community in Nepal composed primarily of university students. Their university calendar coincides with that of primary schools so they will not be able to skip the first 2-4 weeks of classes to deploy OLPC.

So this is a tough situation. How can we scale out technical support for deployment? Well, with not a little bit of luck we have come upon a solution. There are several prestigious high schools in Kathmandu that encourage their students to take a year off between high school and university to work on social projects in Nepal. Pradosh Kharel approached us for an internship soon after he finished his exams for Buddhanilkantha School last June. He has made a big impact on our team in a very short period of time. He came in to our office knowing nothing about linux and has already learned the basics of shell scripting and system administration. On his encouragement, his classmate Bibek Kafle joined us to work on internationalizing our E-Paath activities. Last month, Avash Pandit and Aakash Gautam joined us part-time and they start full-time next month.

Out of this lucky accident, I have started to develop our deployment strategy. We need roughly two technical support personnel full-time in each district for the first 2 months and then 1 person for each district thereafter. That adds up to roughly ten deployment staff. These personnel don’t need to know how to create activities from scratch or be expert systems administrators but they need to know enough to fix 90% of the problems and that will arise and work together with our professional network engineers to solve the remaining 10%. The “Buddhanilkantha Boys” as I call them, are extremely eager to work in the field for this project. We believe that we can recruit 6 more deployment volunteers in January and our core group can train the new recruits.

So what do we do next summer when all of the Buddhanilkantha Boys leave Nepal for Reed, Swarthmore, Vassar, etc.? We will wish them the very best and ask for their help in recruiting from the next crop of graduates.

Ten well-intentioned and energetic volunteers simply can’t train and manage themselves with out guidance. Anyone in the open-source world knows that volunteer management is a special vocation unto itself. That is why we are extremely lucky that Tony Anderson joined us in September. Tony retired from 36 years in the software business ten years ago and has been traveling the world ever since. He found the OLPC project so compelling that he stopped his magical mystery tour in Kathmandu to work on it. Since his arrival, he has focused on the School Server and organizing our deployment volunteers. As a professional engineer and natural educator, he has excelled at training our volunteers and focusing their boundless energy into productive ends.

Our deployment volunteers will work very closely with new full-time hires to OLE Nepal, Sunil Shrestha and Prithak Sharma. Sunil joins us as network engineer and brings a great combination of passion, technical skills, and experience working in rural Nepal. He particularly impressed us during his interview when he demonstrated how he had read our entire blog, wiki entries, and website. Prithak Sharma is talented system administrator with a number of years experience working at Kathmandu ISP’s.

At the end of writing this post, I have come up with a clearer formulation of our deployment strategy that has three tiers:

Tier 1: Local technical support provided by talented teachers, children, and interested community members. These people receive their training from the Tier 2 team.

Tier 2: Deployment Volunteers, primarily recent high school graduates who have received several months of training.

Tier3: The OLE Nepal engineering team which includes network engineer Sunil Shrestha, sysadmin Prithak Sharma, Sulochan Acharya, and myself.

I want to close by lettting everyone know how impressed I am by our deployment volunteers. They work very hard, ask for nothing in return, and their passion is genuine. I remember the first time I asked Pradosh if he would mind spending a month in a village supporting OLPC. He responded immediately, “Of course I don’t mind! I will love it!”

One Heckuva Nice Guy

Bernie Innocenti arrived in Kathmandu a week ago and he has changed the OLE Nepal office in so many ways. For one thing, this guy knows linux inside and out, in so much more depth than anyone else on the team. He has already made a big impact on the sysadmin team by showing us how to set up LVM RAID 1 for our E-Pustakalaya (E-Library). He is teaching us about Sugar and helping us optimize our activity performance.

Sometimes, I get the impression that Bernie can do anything and knows everything about linux. Yesterday, I had a problem with Apache and I told Sulo “I was going to consult with the Oracle,” Bernie.

He not only is darn helpful but is a super nice guy. It truly is a pleasure to have him hear. The guys in the office are already trying to figure out how to match him with Miss Nepal 2007 so he will stay here longer. Word on the street is that she likes Linux and C :)

Check out E-Paath and E-Pustakalaya at redmine.olenepal.org

I finally got redmine up and running on hosted server. Please do me a favor and check it out.

For those who aren’t familiar w/ it, redmine is a fantastic project management tool very similar to Trac but much, much more user-friendly and easier to set up. We wanted to use OLPC’s Trac set up but found it did not meet our needs.
Please take a tour of http://redmine.olenepal.org using the guest account, password guest. Let me know what you think. We hope that redmine will make it easier for outside folks to contribute to E-Paath and E-Pustakalaya (E-Library) development. I am using the thin webserver rather than apache with mod_rails, which ate up all the memory on my virtual private server.

In a separate subject altogether, I have absolutely fallen in love w/ OpenVZ. It is a fantastic virtualization tool that really meets our office needs.

It’s Going to Be a Great Summer

The office is getting crowded, really crowded. On Monday, we will have a total 21 people in the office full-time. That doesn’t count volunteers that come in and out of the office. What do we do w/ 21 whole people?

  •  6 full-time activity developers  + Ties
  • 1 full-time curriculum expert – the super-creative, super-wacky Kamana Regmi
  • 2 full-time graphic designers – who develop the graphics for E-Paath (formerly E-Paati)
  • Saurav, our education director
  • Rajeev, who handles government relations which is 2 full-time jobs
  • Upaya, who handles all things administration including finance (he’s the guy who keeps this place functioning :) )
  • Rabi our exec director
  • Dev, our network engineer
  • Sulochan, sysadmin and E-Pustakalaya (E-Library) developer
  • Me
  • Mahabir Pun, In charge of network infrastructure and community relations.

Then there is Christine, Punyasheel, and Bipul who help out w/ activity design and teacher training.

We still need to hire a full-time network engineer and a full-time power engineer. These guys will be focused on building the network and power systems we will need to expand olpc throughout Nepal.

Amazing that w/ all these folks we are all still working our butts and I often feel like I am way behind. Implementation is a big job.

Prasoon Karmacharya will be here for the next month full-time as a volunteer to help w/ testing and all things IT. Deepak Shrestha of Subisu has been incredibly helpful w/ all things networking.

Doug Mayeux joined us this summer as our summer intern. I am really impressed w/ him. Very energetic and intelligent. He will be doing several things. Primarily though, relaying our particular needs for content and activities to the larger olpc community and getting his hands on all documentation and materials related to using EToys in the classroom. We want to use the EToys environment in a granular fashion to introduce individual concepts from math and science. To do this we need all the lesson plans we can get. For example, you can use Etoys to model gravity but what if you have no idea what gravity is? We need a lesson plan that introduces the high level concept and then guides the kids into working w/ it themselves using Etoys.

David Van Assche will joing us some time in July for a period of at least five months. He’s a Moodle expert and will help us build a custom Moodle setup for Nepal. We will happily relay all work and lessons we gather to the larger olpc community.

Last but not least, Bernie Innocenti will be joining us here in Kathmandu for the summer. I am extraordinarily excited about this. We really can use his help to make our learning activities collaborative and his expertise to better understand Sugar. We also want him to interact w/ Nepali kids and teachers so that he can make Sugar better meet their needs.

that’s all for now, I have to go back to my day (writing) job for olpcnews.com

Working on Internal Systems

I haven’t been nearly as active in the OLPC mailing lists discussions and software testing for the last weeks due to a 10-day illness and then some internal office systems stuff that I am still working on. About a day after the glorious start of the Bishwamitra and Bashuki schools, my lower lip started mysteriously swelling. It kept swelling and swelling despite my sweet coaxings and entreaties to shrink. It swelled so large that I couldn’t close my mouth and was extremely was painful. The infection also caused a nasty fever that kept me in bed. I was too tired to do any serious work, but not too tired to learn some super cool vim plugins! like MiniBufExplorer, python omnicomplete, and I finally learned to fully appreciate Visual Mode.

Once I got back to work, I had to attend to some internal systems issues that I had never attended to. After 7 months in business, we didn’t have a file server, print server, contact manager, shared calendar, nothing.  So I have been pretty consumed w/ these matters in addition to working on some touchpad issues that are dogging the kids at Bashuki.

At heart I am an IT Manager/sysadmin and probably always will be. I really enjoy this kind of work and don’t really like being a leader, proposal writer, or schmoozing. I am slowly trying to work my way out of OLE Nepal leadership. I always intended for this organization to be entirely led by Nepalis. When I was still a member of OLPC Nepal, I was one of the leaders their and not entirely comfortable w/ that role. I don’t pretend to know what’s best for Nepal. I have always seen my role to enable talented Nepali leaders to reach their goals. In the case of OLE Nepal, those leaders are Rabi Karmacharya, Saurav Dev Bhatta, Mahabir Pun, and Rajeev Adhikari. And they are extraordinarily talented. When Rabi, myself, and Mahabir started OLE Nepal, I was involved in every single funding proposal, every strategy, and every big decision. The other day, Rabi, Saurav, and Rajeev held a long-term strategy session.  I wasn’t involved at all and extremely happy not to be. Instead I created a Debian init script for Fedora-Commons, got the Redmine project management app, up and running, and learned how to use update-rc.d. I am infinitely more happy working on the latter type of problems than the former.

So sysadmin stuff:

Office Wiki

I have set up MoinMoin for our internal officewiki. The problem with Mediawiki is that it has no access controls. We need to document our hiring process but those not involved don’t need to see that information. I don’t want to make all internal the sysadmin documentation accessible to all staff. They don’t need to see what security settings I have on X server and probably don’t want to. MoinMoin does have access controls and it is pretty easy to backup.

Project Management:

I looked at Trac, installed it, and well . . . I don’t like it.  At all. It was a pain to configure and pretty esoteric. I set up Redmine and it was an absolute breeze and offers a lot more functionality. I was bit wary because it is aRuby on Rails app and I have heard that Ruby on Rails has stability issues. I haven’t really stress tested it yet, so we shall see. I am using Redmine w/ mod_rails and apache. Mod_rails requires the pre-fork version of Apache 2.2 . I don’t really understand the difference between prefork and mpm but I do understand that the Apache documentation is consistently confusing.

Redmine is great. Each project gets its own wiki, forums, roadmap, gantt charts, and calendar. We aren’t using it w/ Subversion or git yet.  We will see how that works.

I would like to use the OLPC Trac but it is reallly slow across our office Internet connection. We also intend to use redmine for our internal testing of apps. The folks doing the application testng will be fairly non-technical and I think they will have a quite a bit of trouble using Trac.  Also OLPC’s Trac only have milestones for the Sugar releases. I can’t create milestones for E-Paati releases.

Asmita and Priyata Bhatta, software testers/voices of E-Paati activities

They are so cute!

Also, we will use Redmine to manage sysadmin and office administration tasks. Examples of this are: buy me 3 adapters! My mouse doesn’t work! I need a ride to the pilot school on Monday together w/ the Secretary of Education, please arrange.

It is easy to lose track of this stuff even w/ a small office. But — the office is growing rapidly. We are hiring 5-6 more folks in the next couple months. That should put our total full-time staff up to  22 people. We also have a # of folks that want to volunteer on a regular basis, we have two interns right now. So all these people requires more structure! I am amazed that we have a lot of people compared to OLPC organization itself. Well, I can tell you we aren’t slacking. We’re here from 9 am – 7/8 pm. Rabi and I usually log 12-14 hour days, Om works for several hours before he even shows up to the office, I constantly harrass Ties to do something amazing w/ Squeak.

We have a lot of people because content development and implementation are extremely resource-intensive! We have 4 developers + 2 graphic designers + 1 curriculum designer working full-time on content development. We have 3 technical people working on systems administration + networking and will add two more sysadmin/network admins in the next two months. We will also hire a power engineer to work on sustainable power solutions for rural schools.

I digress, back to sysadmin stuff. I have set up zimbra for our shared calendar and shared contacts but I am wondering if we should just use a shared Google Calendar and Google Contacts if they exist. Zimbra is awesome but it requires its own server because it has specially customized version of Apache that won’t play well w/ other versions of Apache. So maybe it will be back to google calendar, which does have a plugin for redmine . . .

I still need to set up a Samba file and print server, document our internal office and school network, and source control for our E-Pustakalaya application, yadda yadda. But I am making progress :)

My next goal after Samba server will be to set up an LDAP server using EBox. It promises to be a user friendly tool for LDAP management. And I really need to learn LDAP but it needs to be easy for others in the office to manage, i.e. someone in addition to me like Sulochan or Dev.

A few more thoughts in this ramble. I am extremely impressed w/ Redmine and probably w/ Ruby on Rails in general. Redmine is less than a year old and it already has a ton more functionality and ease of use than Trac. I have looked at the code and it appears that redmine is able to easily reuse code from other rails apps rather than creating it from scratch while most of the code for Trac seems to have been created for Trac. Perhaps Django provides this same advantage.

I am working on the touchpad issue but sidetracked for this week so I could finally get some internal sysadmin stuff done, otherwise it may never get . Thanks to Mstone, sj, tomeu and others for looking at this problem.

Notes from first few days at Bashuki and Bishwamitra

From an e-mail written by Rabi Karmacharya

Here are few observations and suggestions from the first few days at Bashuki and Bishwamitra:

Hardware:

1. Cursor has been a nuisance, especially in Bashuki. Bryan is working to resolve this. A combination of dirt and moisture (sweat and finger licking by kids) could be the problem. This will seriously compromise our activities since we have relied heavily on drag-and-drop action. Bryan/Sulo
2. Battery: No report on battery malfunction from the kids yet. There was a problem with one teacher’s (Manoj) battery whereby the battery indicator was not working (both light and on screen). Bryan

Sugar:

1. Frame: We have ask OLPC to stop the frame from popping up when the cursor is moved to the sides or corners. It is really annoying for the kids, especially when they are having cursor problems. Bryan/Sulo
2. Volume Indicator: It would be nice to have a volume indicator that shows up when kids press the volume keys so that they can see what level they are at. Bryan/Sulo
3. Sleep mode: Teachers often ask the children to fold the laptops in the middle of a lesson when they need their attention directed towards the teachers. When the kids open the laptops, and press the START button to resume the activity, the XO hangs from time to time, forcing them to reboot the machine. This wastes a lot of valuable class time. I was able to replicate it once at the school. Bryan/Sulo
4. Would be nice to have the volume level display when you change the volume. Not critical

E-Paati:

1. Make the selected Class and Subject more distinct so that users know where they are. Right now it is difficult to tell. OM
2. Make a numbering scheme on the main E-Paati page so that it is easy for the teachers to instruct which activity to open. We should do this in line with our future plan to categorize the activities of each class/subject into weekly bundles. OM
3. Instead of the small “Loading” notice, we should put something bigger with animation while the kids wait the activities to be loaded. OM
4. The volume of the sound clips have to be consistent. Right now one sound clip is louder than another. OM
5. The pronunciations have to be more un-American. OM
6. The spelling has to be consistently British (Color is spelled in the American way on the E-Paati main page) OM
7. There is a spelling error in geometry memory game. (Obtuse is spelled Abtuse) OM
8. The blue triangle cannot be dragged in the Triangle game. OM
9. Game Over is two words, not one. OM
10. Go Shopping: There should be an error if the user clicks the wrong button, and not just when the wrong number is entered. OM
11. Trash Can appears in few activities: Quick Multiplication, How Many Sheep. OM

Long Term:

1. We have to include some mechanism by which the teachers can check the HW that kids do on the E-Paati activities.

I have assigned the change/bugs to the individual in RED. You can change the assignment to the another person where appropriate. But this should to into our internal management system. Bryan/OM

--

Rabi