Kiwix Deaddrop en Poza Rica

Como parte de la comunidad de Kiwix, me uní al primer equipo de difusión de Kiwix por medio de Deaddrops & Geocaching. Si siguen este blog, ya sabrán un poco de lo que es el proyecto Kiwix, y que permite leer el contenido de la enciclopedia Wikipedia sin necesidad de conexión a internet. El proyecto Kiwix […]

A young developer’s story of discovery, perseverance and gratitude

This post is a discovery report written by Jared Flores and slightly edited for publication. It’s part of a series of candid essays written by Google Code-in students, outlining their first steps as members of the Wikimedia technical community. You can write your own.


When I initially heard of the Google Code-In (GCI) challenge, I wasn’t exactly jumping out of my seat. I was a little apprehensive, since the GCI sample tasks used languages such as Java, C++, and Ruby. While I’ve had my share of experience with the languages, I felt my abilities were too limited to compete. Yet, I’ve always had a fiery passion for computer science, and this challenge presented another mountain to conquer. Thus, after having filtered through the hundreds of tasks, I took the first step as a Google Code-In student.

The first task I took on was to design a share button for the Kiwix Android app, an offline Wikipedia reader. Though Kiwix itself wasn’t a sponsoring organization for GCI, it still provided a branch of tasks under the Wikimedia umbrella. With five days on the clock, I researched vigorously and studied the documentation for Android’s share API.

After a few hours of coding, the task seemed to be complete. Reading through the compiler’s documentation, I downloaded all of the listed prerequisites, then launched the Kiwix autogen bash file. But even with all of the required libraries installed, Kiwix still refused to compile. Analyzing the error logs, I encountered permission errors, illegal characters, missing files, and mismatched dependencies. My frustration growing, I even booted Linux from an old installation DVD, and tried compiling there. I continued this crazy cycle of debugging until 2 am. I would have continued longer had my parents not demanded that I sleep. The next morning, I whipped up a quick breakfast, and then rushed directly to my PC. With my mind refreshed, I tried a variety of new approaches, finally reaching a point when Kiwix compiled.

With a newly-found confidence, I decided to continue pursuing more GCI tasks. Since I had thoroughly enjoyed the challenge presented by Kiwix, I initially wanted to hunt down more of their tasks. However, finding that there weren’t many left, I gained interest in Kiwix’s supporting organization: Wikimedia. I navigated to Wikimedia’s GCI information page and began familiarizing myself with the organization’s mission.

“We believe that knowledge should be free for every human being. We prioritize efforts that empower disadvantaged and underrepresented communities, and that help overcome barriers to participation. We believe in mass collaboration, diversity and consensus building to achieve our goals. Wikipedia has become the fifth most-visited site in the world, used by more than 400 million people every month in more than 270 languages.” – About Us: Wikimedia (GCI 2013)

Reading through the last sentence once more, I realized the amazing opportunities that were ahead of me. Whenever I needed to touch up on any given topic, Wikipedia was always one of the top results. Moreover, Wikipedia had become a source of entertainment for me and my friends. We always enjoyed hitting up a random article, then using the given links to find our way to Pokémon, Jesus, or maybe even Abraham Lincoln: Vampire Hunter.

Eager to begin, I chose video editing as my second task for Wikimedia. I began the long endeavor of watching, reviewing, and editing the two forty-five minute clips. Despite the lengthy videos, I was quite amused in seeing the technical difficulties that the Wikimedia team encountered during their Google Hangout. It was also comforting to put human faces behind the Wikimedia mentors of Google Code-In.

As with my first task, the work itself sped by quickly. But also similar to Kiwix, I encountered some difficulties with the “trivial” part of the task. I had never worked with the wiki interface before, so the wiki structure was somewhat foreign. I only had a vague idea of how to create a page. I also didn’t know where to upload files, nor did I know how to create subcategories. Nonetheless, after observing the instructions in Wikipedia’s documentation, I finally managed to upload the videos. Marking the task as complete, I scouted for my third GCI task.

Unbeknownst to me, my third task for Wikimedia would also prove to be the most challenging so far. Since this task required me to modify the code, I requested developer access. With the help of Wikimedia’s instructions, I registered myself as a developer, generated a private key to use with their servers, and proceeded to download the source code.

Though my experience with Git was quite basic, MediaWiki provided an easy to follow documentation, which aided greatly in my efforts to download their repository. As I waited for the download to complete, I quickly set up an Apache server for a testing environment. Configuring the MediaWiki files for my server, I began the installation. Fortunately, MediaWiki’s interface was quite intuitive; the installer performed flawlessly with minimal user input.

“Off to a good start,” I chuckled quietly to myself, a grin spreading across my face. And with that statement I tempted fate and my troubles had begun. Upon opening the code, I realized I couldn’t easily comprehend a single line. I had worked with PHP but the code was more advanced than what I had written before.

Running my fingers through my hair, I sighed in exasperation. I spent the next few hours analyzing the code, trying my best to decipher the functions. Suddenly, patterns began appearing and I began to recognize numerous amounts of functions. I started to tinker with different modules until the code slowly unraveled.

Finally formulating a solution, my fingers moved swiftly across the keyboard, implementing the code with ease. Confident that I had tested my code well, I followed the instructions written in the GCI’s task description, and uploaded my very first patch to Gerrit.

I was surprised at how simple the upload was. But what especially surprised me was the immediate feedback from the mentors. Within a few minutes of the upload, MediaWiki developers were already reviewing the patch, making suggestions for improvement.

Thankful for their helpful input, I worked to implement the changes they suggested. Adding the finishing touches, I was ready to upload another patch. However, I was unsure if I should upload to a new Gerrit, or if I should push to the same patch as before. Unclear about the step I should take, I made the rookie error of uploading to a new Gerrit commit.

My mistake quickly received a corrective response from Aude via the Gerrit comment system. While I initially felt embarrassed, I was also relieved that I didn’t have to work alone. In fact, I was thankful that the MediaWiki collaborators taught me how to do it right.

Checking out the link Aude had given me, I learned to squash the two commits together. However, when I tried to follow Aude’s instructions, I somehow managed to mix someone else’s code with my own. What’s even worse was I already pushed the changes to Gerrit, exposing my blunder publicly.

Had it been any normal day, I would’ve just been calm and tried my best to fix it. But it just so happened to be the Thanksgiving holiday (in the United States). I had to leave in a few minutes for a family dinner and I couldn’t bear the thought of leaving my patch in a broken state.

I felt about ready to scream. I abandoned my Gerrit patch, and navigated to the task page, ready to give up. But just as I was about to revoke my claim on the task, I remembered something Quim Gil had told another GCI student:

“They are not mistakes! Only versions that can be improved. Students learn in GCI, and all of us learn every day.”

Remembering this advice, I cleared my mind, ready to do whatever it would take, and learn while I was at it. And like an answer to my prayers, Hoo Man, another developer, posted a comment in Gerrit. He guided me through how I could return to my original patch and send my new improvements through. And more importantly, he motivated me to persevere.

I came into GCI as a passionate, yet undisciplined student. I’m thrilled that in joining this competition, the Wikimedia open source community has already helped me plant the seeds of discipline, perseverance, and collaboration. It’s no coincidence that my hardest task thus far was staged on Thanksgiving. Every year I express gratitude towards friends and family. But this year, Google Code-In and the Wikimedia community have made my gratitude list as well.

Jared Flores
2013 Google Code-in student


Read in this series:

Kiwix, Wikipedia un regalo navideño

Llegaron las fiestas decembrinas y es como todos los años es tiempo de dar, recibir y compartir. No encuentro mejor momento para escribir de Kiwix, que no ha dejado de desarrollar este excelente programa. Como siempre con versiones para Windows, Mac OS, GNU/Linux y ahora también lo puedes encontrar en Android. Así es Kiwix ya […]

Wikimania 2013

Cette semaine se déroule la conférence annuelle Wikimania. Après Francfort, Boston, Taipei, Alexandrie, Buenos Aires, Gdansk, Haifa et Washington, c’est Hong Kong qui va accueillir plus de 1000 personnes qui vont échanger sur les projets Wikimédia pendant cinq jours.
Logo de Wikimania 2013

 

Cette année encore, Wikimédia France sera bien représentée avec une dizaine de membres et salariés qui interviendront sur les sujets suivants :
  • Des idées développées par la Wikipédia en français pour impliquer de nouveaux contributeurs, de la rue au wiki (Résumé)
  • A tale of moaning – Les Wikipédiens sont-ils des français ou des zombies ? (Résumé)
  • Comment les entreprises peuvent-elles aider Wikimédia à atteindre ses objectifs stratégiques ? (Résumé)
  • Paid editing : un cadre pour permettre aux entreprises de participer (Résumé)
  • Afripédia : l’expérience du Mali (Résumé)
  • GLAM scientifiques (Résumé)
  • Diversification des sources de financement (Résumé)
  • Wikipédia en 2022 (Résumé)
  • Wiki Loves Monuments (Résumé)
Note : les conférences sont diffusées en direct sur http://new.livestream.com/socreclive/wikimania.

Carry the entirety of Wikipedia in your pocket with Kiwix for Android

An Android tablet displaying the Kiwix app showing a Wikipedia article

The Kiwix app empowers Android users to download and view large sets of Wikipedia content on their devices while offline.

Kiwix isn’t just Yet Another Wikipedia app. Once you install it on your Android device, you can actually use it to download the whole website, and access its millions of articles even when you’re offline (or when you don’t want to use your data plan). You can also download smaller (and custom) selections of pages, if you’re short on storage space.

Providing access to Wikipedia and other Wikimedia sites to as many people as possible is one of the Wikimedia movement’s core goals. While their regular desktop, online version is enough for most users, it is inadequate for many others.

This is the reason why, throughout the movement, Wikimedians are working to diversify and facilitate access to Wikipedia and its sister sites:

  • Kiwix is a software that allows users to browse full snapshots of Wikipedia (and numerous other resources) from a personal computer that isn’t connected to the internet.
  • The Wikipedia Mobile App allows smartphone users to browse Wikipedia on mobile devices, save articles and much more.
  • The Wikipedia Zero initiative allows cellphone users in Africa and Asia to access parts of Wikipedia without incurring data charges.
  • The Kiwix-plug empowers African students to access full snapshots of Wikipedia and Wikisource right from their campus, on their own laptop or phone.

Kiwix for Android is the latest innovation in this series: This app, available from the Google Play Store or from the Kiwix website, allows users of Android-powered devices to browse offline content from Wikipedia and its sister sites.

The market of Android-powered devices is exploding. It is not limited to mobile phones and tablets, but it also powers TV, appliances, “USB computers” and an increasing range of cheap computers.

In places where connectivity is a difficult (at least 30 countries on the sole African continent), the only way to access Wikipedia content is Kiwix Desktop, but it still requires a computer and electricity. Cheap Android devices might spread more quickly, and we really want to facilitate access to free content everywhere.

That said, Kiwix for Android can also be just as useful to Westerners who want to walk around with the entirety of the world’s largest encyclopedia in their pocket (if they have a big enough SD card), accessible at no cost or data fees.

Where do I get it?

It’s very easy to use:

  1. Download the app from the Google Play Store, and launch it;
  2. Click on the “Open” button, select a ZIM file from the list (from your device or SD card);
  3. That’s it! You’re already browsing offline content.

If you don’t already have a ZIM file, Kiwix leads you to its online repository so you can download one. You can also use any of the Books created on Wikipedia.

Being a mobile version of a feature-rich desktop software, the Android version is lightweight. It provides access to the most essential features: opening and reading a ZIM file, search with auto-completion on article titles, in-page search, random navigation, zoom in & out, and access to the mobile-friendly ZIM file catalog.

How does it work?

Kiwix for Android uses a native Android user interface in Java (which offers a nice look & feel) and is built on the Kiwix and libzim C++ code. We combine these tools using the Java Native Interface.

It’s very easy to improve the app. Just download the code from our code repository and read the COMPILE file. You don’t even need to compile the C++ code: it’s available as a pre-compiled file. You can directly start hacking the UI and easily rebuild the APK package file.

What’s next?

This is the beginning of Kiwix for mobile. This Android version was developed during a one-week hackathon sponsored by Wikimedia CH (Switzerland) and our focus was on providing a stable and comfortable experience.

Our strategy for mobile development is not set in stone. We have some ideas but are also expecting feedback from users. Give it a try, and tell us what you want: new features, new paradigm, new platform target, etc.: We’ll try to make it happen.

Beside the Kiwix app itself, we are also working to provide more content in the ZIM format, so that every Kiwix-enabled device can become a large library of free content.

About Kiwix

Kiwix is an offline content reader working with ZIM (OpenZIM) files. It works on Windows, Linux, Mac and now Android. Additionally, it is available as a Web Server solution and powers the Afripedia Plugs.

Renaud Gaudin, Kiwix Developer

Afripedia project increasing off-line access to Wikipedia in Africa

(This is a guest post from Adrienne Alix of Wikimedia France. You can read the original Afripedia post here and you can read the French version of this post here.)

The Afripedia project was initiated in late 2011, and engaged Wikimédia France, the Agence Universitaire de la Francophonie (AUF) and the Institut Français (IF). Kiwix also supplied technological support.

The partnership, which was signed in June 2012, started materializing in November 2012 in Abidjan, Côte d’Ivoire.

What is the project about?

The Afripedia project aims to enable significant off-line consultation of Wikipedia and also to train contributors in French-speaking African schools and universities.

Hardware for Afripedia: wireless router, computer, and USB key.

Hardware for Afripedia project.

Our starting point was the fact that, although African universities have begun to be well-equiped with computer hardware, their Internet connectivity is often poor, or at least too weak or irregular to allow frequent, natural use of Wikipedia. These issues and lack of Internet access at home are factors that prevent students and teachers from using Wikipedia as much as they might like.

It was important for us to combine the dissemination of Wikipedia with contribution trainings so that students and teachers could contribute to Wikipedia and enhance content about Africa, which is notably under-represented on Wikipedia (only 2 percent of contributors are from Africa, and most of them from North African countries). Thus, when quality Internet access comes, Wikipedia contributors will be ready.

Using Kiwix, which has been offering off-line consultation of Wikipedia for several years, we implemented a computer deployment project to broadcast Wikipedia through offline WiFi networks. This mechanism, in place in universities of West Africa and Central Africa, allows people from those universities to connect easily to the network and freely read Wikipedia without an Internet connection or desktop computer. A more detailed description is available on Wikipedia: projet Afripédia.

What have we done?

Afripédia training, Abidjan, 2012.

Afripédia training, Abidjan, 2012.

With the help of AUF, we hired about 15 people from French-speaking digital campuses in 11 African countries: Senegal, Benin, Ivory Coast, Burkina Faso, Chad, Central African Republic, Mauritania, Niger, Togo, Mali and Burundi. From 5-9 November 2012, training took place at Abidjan’s French speaking digital campus, located in the Université Félix Houphouët-Boigny in Cocody (Ivory Coast). Over five days, we taught how to use and install the Wikipedia offline consultation tools. We also taught Wikipedia editing, explained concepts such as free licences and introduced the various Wikimedia projects. The week ended with a public lecture available to students (around a hundred people attended), and a contribution workshop animated by the people trained during the week.

These few days of training showed the need for developing best practices, which where not necessarily identified before. For example, the importance of a future offline export of Wikisource at the next content update would be very useful for accessing classical texts for French-language litterature teaching. There was also interest in an offline Wikipedia implemented directly on university servers, so that people who do not have WiFi access can consult the encylopedia on the university intranet.

We paid particular attention to the project follow-up, including the personal investment of each trainer, who has to master Wikipedia and the Wikimedia projects in order to teach these concepts efficiently to the local communities.

How far have we gotten two months later?

This report comes some weeks after the training in Abidjan in order to take a step back after the initial enthusiasm.

Afripedia training in Niamey, Niger.

Afripedia training in Niamey, Niger.

The situation is different depending on the country. For example, in Burundi and Abidjan, strikes in the university heavily slowed the implementation of the project in November. Elsewhere, however, installations and trainings were a success. In other countries, for planning reasons, the project will begin in January.

  • For example in Chad, offline Wikipedia has been available since mid-November in the Campus Numérique francophone (CNF — French-speaking digital campus) of N’Djamena and trainings for the use of Wikipedia were organized by Francis Beninga Deouro, technical manager of CNF of N’Djamena.
  • The same in Mali, where Michel Namar, manager of the CNF of Bamako, is leading an intensive action around the project: deployed it on the server of the Bamako University as well as in the Ecole Normale Supérieure of Bamako, Wikipedia is available through WiFi thanks to the plug-computers. Deployments are currently being organized in Segou (in central Mali).
  • In Senegal, Stefano Mekoudi is having a strong presence promoting the project and sharing information, especially taking the opportunity given by the “Carrefour des Possibles” forum that was held in Dakar at the end of November 2012.
  • Benjamin Sia, the person responsible for trainings at the CNF of Ouagadougou, Burkina Faso, established a system and created posters in both universities of the city to inform students of this new opportunity to consult Wikipedia. Training for contributors is scheduled in January.
  • Thanks to Abdoul Kane’s work, the system has been installed on a server and at the IT department of the University of Nouakchott, Mauritania.
  • Offline Wikipedia is also available in Niamey (Niger), thanks to Mariama Abdoul-Moumouni in the CNF and the university.

The first statistics about offline page requests and activity on Wikipedia are expected by mid-february.

Each trainer who attended the training in Abidjan in November is expected to become a regular contributor and also develop the contribution within their own university. The number of contributions was closely tracked for two month and first results are very encouraging:

  • After Abidjan, 15 active contributors were identified thanks to the Afripedia project. By mid-January 58 contributors made at least a contribution to Wikipedia in addition to 71 new accounts created.
  • Within a month, 233,700 bytes of text were published on Wikipedia, with around 600 distinct contributions. More than 85 articles were created, and more than a hundred modified.
  • More than twenty pictures have been uploaded to Wikimedia Commons, illustrating mostly daily practices or architectural elements.
  • Very few articles have been deleted thanks to kind and watchful support provided by more experienced contributors who acted as “Afripedia sponsors.”
  • Contributions related to the project are available here: Projet Afripédia/Contenus
  • Informations can also be tracked on twitter: @Afripedia

Conclusion

One month after the first deployment, and a little more than one year after the beginning of the project, we are pleased to note that it seems to have fulfilled its objectives.

  • It meets a true need of accessing Wikipedia’s ressources online and offline, especially for some universities that do not have internet access—like those in Niger, for example.
  • The CNF staff trained in Abidjan got really involved in the project, with both dissemination and contributions. Their energy when they came back to their respective countries indicates that the Afripedia project can be efficiently led from the CNF in Central Africa and West Africa.
  • The training week in Abidjan brought out new ideas and improvements that we will try to implement shortly.
  • The project was well-covered by the French speaking press (technology and Africa focus) and the bloggers from the Ivory Coast. It was also circulated to countries where we weren’t represented, raising great interest. For example, the CEDESURK network in the Democratic Republic of Congo (helped by Wikimedia France and Kiwix), is installing an off-line Wikipedia on its servers in the universities of Kinshasa and Lubumbashi, in advance of full deployment in the eight universities of the country and implementation of a training program.

A second step is necessary to expand the project to the countries of French-speaking Africa that aren’t part of Afripedia yet. This will probably take place in Spring 2013. It will also be an opportunity to review the project in further detail with the people who were already very involved, so as to facilitate Wikipedia access and expand African contributions to the Wikimedia projects.

Adrienne Alix, Wikimedia France
(Translation from French by Wikinade, Adrienne, PierreSelim, Mariiwakura, Seb35, and Chamatou)

Kiwix como mecanismo de desarrollo humano

Recuerdo mi primer semestre en la Universidad de Oriente, vivía en un pueblo olvidado de la Isla de Margarita en Venezuela. Para ese entonces, nada había cambiado nada en al menos 30 años, carreteras de tierra y casas de la época de la Colonia, los servicios básicos son practicamente inexistentes.

Junto a unos amigos de estudio, no las arreglamos para montar nuestro primer centro de computación, la Internet estaba a años luz de llegar a ese pueblo, y en la Universidad solo podía ser accedida por una red interna a ciertas horas del día, de forma impredecible e increíblemente lenta.

Como solución temporal, que luego se convirtió en una solución de varios años, nos hicimos de varias copias de la enciclopedia local y la distribuimos digitalmente en unos diskets. Casi 20 años han pasado desde entonces, y no precisamente años luz,  la situación sigue siendo la misma, el acceso a Internet es desconocido para la mayoría.

Este problema podemos verlo en casi cada zona rural de Venezuela y, en general de muchos países de Sudamérica. Según cifras recientes 30% de la población mundial ya cuenta con acceso a Internet, y el crecimiento va aunado en gran medida por los teléfonos Inteligentes y dispositivos alternos que no son computadoras.

A pesar de este crecimiento exponencial del acceso a Internet, no ha sido de manera equitativa en todas las zonas del planeta, existen zonas en las que el Internet posiblemente no llegue en décadas como el caso de poblaciones rurales del Continente Africano.

Nuestra labor

Partiendo de esta necesidad, fue lo que me hizo introducirme en el proyecto Kiwix. Al principio colaboraba con alguna traducción, luego me involucré más y, en este momento, estoy junto con el capítulo de Wikimedia Venezuela, coordinando un enlace con el gobierno local para su introducción en las escuelas mediante el proyecto Canaima.

He contado con muchos colaboradores, en donde destaca Cesar Sevilla, un compañero que conocí en festivales de instalación y listas de correo de usuarios Linux, él ha logrado instalar Kiwix en cientos de centros de computación local.

Hemos atravesado laderas y cerros para llevar él Internet a las personas que habitan en estos lugares. Estoy hablando específicamente de la península de la Guajira en Venezuela. Poblaciones accedidas por carreteras de tierra en donde cada dos horas pasa un automovil solo para dejar alguna sorpresa en el camino, sitios olvidados por los rostros de los hombres, en donde a duras penas las personas pueden alimentarse a diario con desperdicios.

Kiwix 

Nada más gratificante que ver a los niños usando Kiwix, abriendo artículos de Wikipedia que muchos usuarios han escrito, cientos de miles de artículos y horas hombre de investigación, todo allí disponible para ellos. Si todos comenzáramos a hacer esto, pronto algo importante pasará.

Kiwix era el lector de wikipedia offline mejor concebido hasta ese momento y aún sigue siendo la mejor solución, en gran parte por el trabajo realizado por el usuario de wikipedia Kelson.

Cuando entiendes que la pobreza es un asunto más cultural que económico, comprendes el papel importante de Wikipedia, todo el conocimiento humano disponible para todos. Y si hablamos de Kiwix, considero que es la mejor herramienta que conozco para colocar el contenido de wikipedia disponible al grupo de personas que más lo necesitan, niños habitantes de zonas rurales y de difícil acceso de los países pobres.

Kiwix es el medio, es la herramienta para llevar el lema principal de Wikipedia, a todos, disponible para todos. Kiwix es el camino a la independencia en conocimiento.

Un premier bilan d’Afripédia

Le projet Afripédia a été initié à la fin de l’année 2011 entre Wikimédia France, l’Agence universitaire de la francophonie et l’Institut Français, avec le soutien et l’appui technologique de Kiwix.

Logo Afripédia

Logo Afripédia

Signé officiellement en juin 2012, il a pris une réalité concrète au début du mois de novembre 2012 lorsque le premier déploiement a été effectué à Abidjan, en Côte d’Ivoire.

En quoi consiste le projet ?

Afripédia est un projet qui a pour ambition de permettre une consultation hors-ligne massive de Wikipédia, et également une formation à la contribution, dans les universités et écoles d’Afrique francophone.

Nous sommes partis du constat que les universités africaines commençaient à être bien équipées en matériel informatique mais que la connexion internet était trop souvent de mauvaise qualité, trop faible ou trop irrégulière pour permettre une consultation fréquente et naturelle de Wikipédia : débit insuffisant, coûts élevés, absence de connexion à domicile pour la majorité de la population, ces facteurs font que les étudiants et les enseignants n’ont pas la possibilité de consulter Wikipédia autant qu’ils le souhaiteraient.
Pour autant, internet est tout de même présent (même si de qualité médiocre) dans une grande part des universités notamment par le biais des campus numériques francophones de l’Agence universitaire de la Francophonie, et il nous semblait important de coupler la diffusion de Wikipédia à des formations d’apprentissage à la contribution, afin que les étudiants et enseignants puissent, lorsqu’ils en ont la possibilité, contribuer à Wikipédia et notamment enrichir les contenus concernant l’Afrique, notoirement sous-représentée sur Wikipédia (à peine 2% des contributeurs contribuent depuis l’Afrique, et pour une grande part ils viennent du Maghreb). Ainsi, lorsqu’un accès plus facile à un internet de meilleure qualité deviendra chose courante, il y aura déjà des contributeurs à Wikipédia sur ces territoires.

Dispositif de diffusion hors-ligne Afripédia

Dispositif de diffusion hors-ligne Afripédia - Photo Marie-Lan Nguyen - CC-BY-SA 2.5

Avec l’aide de Kiwix, qui depuis plusieurs années propose une consultation hors-ligne de Wikipédia, nous avons donc mis en place un projet de déploiement de plugs computers diffusant Wikipédia par des réseaux wifi hors-ligne. Ce dispositif, installé dans les universités d’Afrique de l’Ouest et Afrique centrale, permet aux personnes fréquentant l’université de se connecter facilement sur le réseau et de consulter Wikipédia à leur guise, sans avoir besoin de connexion ni d’occuper un poste informatique fixe.

Le projet est décrit plus longuement sur Wikipédia : projet Afripédia.

Concrètement ça se passe comment ?

Nous avons sélectionné avec l’aide de l’AUF une quinzaine de personnes venant des campus numériques francophones de 12 pays d’Afrique francophone : Sénégal, Bénin, Côte d’Ivoire, Burkina Faso, Tchad, Centrafrique, Mauritanie, Niger, Togo, Mali et Burundi. Du 5 au 9 novembre 2012, une formation a eu lieu au campus numérique francophone d’Abidjan (Côte d’Ivoire), situé au sein de l’université Félix Houphouët-Boigny à Cocody. Le schéma choisi est celui de la « formation de formateurs », afin de favoriser l’essaimage et la diffusion du projet.

Pendant 5 jours nous avons à la fois formé à l’utilisation et au déploiement des solutions de consultation hors-ligne de Wikipédia, mais également pris le temps d’apprendre à contribuer sur Wikipédia, expliqué les licences libres, présenté les différents projets Wikimédia etc. La semaine de formation s’est terminée par une conférence grand public ouverte aux étudiants (une centaine de personnes présentes) et un atelier de contribution pour ces étudiants, animé par les personnes formées dans la semaine.

Salle informatique CNF Abidjan

Formation au CNF d'Abidjan, novembre 2012 - Photo Kongraoul - CC-BY-SA 3.0

Cette semaine de formation a été extrêmement riche de rencontres humaines, mais aussi d’enseignements de chacun à propos de Wikipédia, de la situation des différentes universités en Afrique francophone, et plus généralement sur la contribution africaine à la mise en ligne de contenus à destination de tous sur Wikipédia ou ailleurs.

Ces quelques jours de formation commune nous ont permis également de faire remonter des pratiques ou des besoins qui n’avaient pas forcément été identifiés auparavant, parmi lesquels l’importance d’ajouter un export hors-ligne de Wikisource à la prochaine mise à jour des contenus (très utile pour l’accès aux textes classiques pour l’enseignement de la littérature francophone à l’université), mais également l’intérêt d’une installation de Wikipédia hors-ligne directement sur les serveurs des universités afin que la consultation puisse également se faire par intranet pour les personnes ne disposant pas de port wifi sur leurs ordinateurs.

Un accent particulier a été mis sur le suivi du projet, la remontée de statistiques de consultation et de contribution, mais également sur l’investissement personnel de chaque formateur formé, qui doit apprendre lui-même à bien maîtriser Wikipédia et les projets Wikimédia pour pouvoir efficacement relayer ces savoir-faire localement. Les objectifs ont été définis en commun à la fin de la formation et font l’objet d’un suivi régulier.

Un mois après, où en est-on ?

Volontairement ce compte-rendu arrive quelques semaines après la formation à Abidjan, pour pouvoir prendre un peu de recul sur le projet, une fois l’enthousiasme du moment passé.

La situation est différente selon les pays : au Burundi et à Abidjan notamment, des grèves à l’université ont fortement ralenti le déploiement du projet durant le mois de novembre. Ailleurs, installation et formation battent leur plein. Dans d’autres pays encore, pour des questions de contraintes d’emploi du temps, le projet ne démarrera réellement qu’en janvier.

Formation Afripédia à l'université de Ndjamena

Formation Afripédia à l'université de Ndjamena - Photo Francis Beninga Deouro - CC-BY-SA

Par exemple au Tchad, Wikipédia hors-ligne est accessible depuis mi-novembre au Campus numérique francophone de Ndjamena et des formations à l’utilisation de Wikipédia ont été organisées par Francis Beninga Deouro, responsable technique du CNF de Ndjamena.

De même au Mali, Michel Namar, responsable du CNF de Bamako, mène une action intensive sur le projet : déployé sur serveur à l’université de Bamako ainsi qu’à l’école normale supérieure, Wikipédia est également accessible par wifi grâce au plug computer. Des déploiements sont en cours d’organisation à Ségou (centre du Mali).

Michel a également mis en place des ateliers de contribution à Wikipédia tous les vendredis matin au Campus de Bamako.

Au Sénégal, Stéfano Amekoudi est très présent pour faire la promotion et l’information sur le projet, notamment lors du forum Carrefour des Possibles à Dakar à la fin de novembre 2012.

Benjamin Sia, responsable des formations du CNF de Ouagadougou au Burkina Faso, a installé le dispositif et créé des affiches dans les deux universités de la ville pour informer les étudiants de cette nouvelle possibilité de consulter Wikipédia. Des formations à la contribution sont prévues pour janvier.

En Mauritanie, le dispositif est installé sur serveur et au centre de ressources informatiques de l’université de Nouakchott grâce au travail d’Abdoul Kane.

Les premières statistiques de consultation de Wikipédia via les plugs hors-ligne sont attendues pour mi-février.

Chaque « formateur-formé » présent à Abidjan début novembre a également comme tâche non seulement de devenir un contributeur régulier, mais aussi de développer la contribution au sein de son université. Nous suivons donc la progression des contributions depuis un mois et le résultat est extrêmement encourageant :

  • en quittant Abidjan il y avait 15 contributeurs actifs identifiés comme contribuant grâce au projet Afripédia ; mi-décembre ils sont 44 contributeurs à avoir effectué au moins une contribution sur Wikipédia (et plus d’une soixantaine de comptes créés).
  • en un mois, 178 000 octets de texte ont été publiés sur Wikipédia, en près de 450 contributions distinctes. Plus de 60 articles ont été créés et plus d’une centaine modifiés.
  • plus d’une vingtaine de photos ont été téléversées sur Wikimédia Commons , illustrant notamment des pratiques quotidiennes ou des éléments architecturaux.
  • très peu d’articles supprimés, et un soutien attentif et bienveillant de la part de contributeurs anciens de Wikipédia, identifiés comme « Parrains Afripédia ».

Les contributions des participants au projet sont à suivre ici : Projet Afripédia/Contenus

On peut également suivre les informations sur le projet sur twitter : @Afripedia

Conclusion

Affiche Afripédia Bamako

Affiche Afripédia à Bamako

Un mois après le premier déploiement, un peu plus d’un an après le démarrage du projet, il est heureux de constater que celui-ci semble répondre à ses objectifs :

  • il répond à un vrai besoin d’accès aux ressources de Wikipédia en ligne et hors ligne, notamment pour certaines universités qui n’ont pas du tout accès à internet (au Niger par exemple)
  • les personnels des CNF formés à Abidjan se sont réellement approprié le projet, pour la diffusion comme pour la contribution, et leur dynamisme à leur retour dans leurs pays montre que le projet Afripédia peut être efficacement mené depuis les CNF d’Afrique centrale et de l’Ouest.
  • la semaine de formation à Abidjan a faire émerger de nouvelles idées, de nouvelles améliorations à apporter au dispositif, que nous essaierons de mettre en place rapidement.
  • le projet a été bien relayé par la presse francophone (technologique et africaniste) et les blogueurs ivoiriens. Mais il a également été diffusé dans les pays pour lesquels nous n’avons pas encore de présence, suscitant un intérêt marqué. Par exemple, le réseau CEDESURK en République démocratique du Congo, aidé par Wikimédia France et Kiwix, est en train de déployer Wikipédia hors-ligne sur ses serveurs dans les universités de Kinshasa et de Lubumbashi, avant de passer à un déploiement généralisé dans les 8 universités du pays et à un programme de formation à la contribution.

Un deuxième temps de déploiement est nécessaire pour élargir le projet aux pays d’Afrique francophone qui n’ont pas encore été intégrés dans Afripédia. Il aura probablement lieu au printemps 2013. Ce sera également l’occasion d’approfondir le projet avec ceux qui se sont déjà beaucoup investis, afin de développer encore l’accès à Wikipédia, et la contribution en provenance d’Afrique sur les projets Wikimédia.

Nginx or Apache, Mirrorbrain and Piwik

At Kiwix we were impacted by a long term issue: we were unable to give practical and accurate download statistics. Our traffic partly goes to our web site and the other part goes to a storage place where all the big files (ZIM, ZIP, ...) are published. Of course, we wanted to have as much details as possible about both of them. After hacking at little bit, this problem seems now to be fixed, here is how I did it.

Audience measurement

To measure Web audience, there are two approaches:

  • Insert a "plugin" (dynamic image or javascript code) on each web page. Every time a web page is loaded, it stores the page name, date and details about the visitors browser in a database. This is extremely efficient and there exist many free offers on the Web to do that; but this method has two disadvantages. First, if you use free commercial tools, like Google analytics or Xiti, you actually sell the privacy of your users. Second, this approach is inefficient to measure file downloads because no web page is involved. To avoid the first one, you can install a free solution like Piwik, which is pretty neat, on your own server; this will avoid informing a third part about your visitors. Unfortunately, you cannot overcome the second problem.
  • Configure your web server to log everything and parse afterwards the log. There is a free software to do that: AWstats. AWstats is pretty old and not practical at all. In addition, I failed to configure it so that it merges multiple requests from an IP on one file in only one record (big files are downloaded in chunks).

Our case

We have been using Piwik for many years for our web site and we are really satisfied with it: it's practical, quick enough and maintained. Our problem is focused on http://download.kiwix.org. Our case is even a little bit worse, because, in fact, we do not really host our files. download.kiwix.org acts as master redirecting requests to mirrors. It uses a solution called Mirrorbrain. In addition to its many weaknesses, AWstats is not able to deal correctly with HTTP 301 and 302 redirections.

We have decided to put everything in Piwik, because it is better to have everything in one tool and also because this is the best one to visualise logs. We have now for both sites:

Our solution

We have installed Piwik and inserted the mandatory piece of code on http://www.kiwix.org to track the visitors visiting our web site. This is easy and straight forward. Then we created in Piwik a second site for http://download.kiwix.org (one instance of Piwik can deal with statistics of many sites).

Later, we configured the Apache virtual host with Mirrorbrain to save logs on the hard disk and let it run for a few days.

Finally, we wrote a custom PHP script to upload the logs to Piwik, using a Piwik PHP class called PiwikTracker.php. Here are its features:

  • Parse all log files for a web site (also all .gz files of logrotate)
  • Able to start from scratch to upload all logs
  • Able to "follow" the log file an update in real time Piwik.
  • Count HTTP redirections (HTTP 301 and 302) errors as valid downloads (mandatory for Mirrorbrain)
  • Merge multiple similar requests to one (only one download for one content per IP in one month)
  • Avoid counting request for directories, md5 checksum, favicon, ...
  • Re-upload only new logs by two consecutive runs
  • Works with both Apache and Nginx
  • Configurable on the command line with arguments
  • Warning: it cannot report about the number of downloads completed.

This script is for our own purpose, but I think it's pretty simple to reuse it; it's pretty simple and you should not need to adapt it too much. Hope this was helpful!

#Kiwix - the interview

#Kiwix is the tool that allows you to read the content of a Wiki offline. It has been developed with Wikipedia in mind but is equally usable for Wikisource or Wikibooks. I am really happy to have interviewed Emmanuel who knows all the ins and outs of this wonderful piece of software.
Thanks,
     GerardM


What is Kiwix and what is it used for
Kiwix is a software that wants to enable people to read Web contents without internet connection. It's a reader which works with ZIM files containing all the content. It's used to access Wikipedia offline, by reading pre-packaged Wikipedia ZIM files. It's mainly used by people who want to have an encyclopedia, but are too poor to have access to the internet. It's also used,for example, by travelling people (plane, ship, train), prisoners and students at school. 

Can you tell us something about its popularity
We have users all over the world and the audience is increasing quickly: we have had 25.000 downloads of Kiwix in May

In how many languages is Kiwix supported 
Thank to the Translatewiki Web site andits community, Kiwix is localised in more than 80 languages. We also provide content (ZIM files), mainly Wikipedia, in around 25 languages. But we want to do more: thanks to a grant of WMCH soon we will offer ZIM files of Wikipedia in all languages

How do you support languages written in scripts like Malayalam, or Tibetan
Contents are Web contents and Kiwix itself is a sort of browser getting the Web pages from the ZIM file instead of the Web. So, we do not have special handling in Kiwix itself to render the contents. Everything should be well organised in the ZIM file, for example by using Web fonts. But, from the Kiwix fulltext search engine point of view, this is challenging. Natural languages have a lot of particularities. Kiwix uses the Xapian search engine and tries to integrate CLucene. We do our best with them to offer the smoothest user experience possible. 

Do you provide fonts with Kiwix for the languages that use these scripts
The Wikipedia ZIM files we are preparing still do not provide the Web fonts. Already for a few months, the integration of Web fonts has been a part of the Wikimedia projects, so we have to fix that ASAP, this is not a big challenge.

For some languages like Chinese and Serbian, we show the content in two scripts ... Can Kiwix do this as well ?
Kiwix does not provide any transliteration tool for now, but all the technology is already there in the soft. We use a powerful unicode library called ICU (http://site.icu-project.org/) which can do that. We
want to use it to allow users to do custom transliterationsC++ developer wanted there!

Kiwix uses the OpenZIM format ... can you tell us more about this format
The format is called ZIM. There is a volunteer driven project called openZIM  created a few years ago to specify the format and develop a standard library. The ZIM format allows to put millions of contents together, to compress a part of them, and add Metadata. In the end, you get only one file, which is, at the same time, extremely compressed and allows a constant and quick random access.

Nowadays, many publications are in the EPUB standard ... can Kiwix handle this as well
Kiwix is not able to deal with EPUB, but in the future it will. We think EPUB & ZIM format are complementary and we want Kiwix being able to perfectly deal with EPUB. Our plan is to integrate "Monocle" to do that. Also there, developers are wanted.

How do people find content available for Kiwix
Kiwix has its own content managerso you can download content from Kiwix itself. But you may also download the ZIM file from the Kiwix Web site (http://www.kiwix.org) or using the Mediawiki Collection extension.


In the future, we want to have a platform (something like Itunes) to offer  really easy to find and download contents (both ZIM and EPUB files). We have started a project in that perspective. We need your support! 

What is your biggest challenge at this moment in time with Kiwix
Building Kiwix-mobile for Android. We will have a first release in autumn. But we have many other projects running at the same time and others for which we still need volunteers.

Thank you
Emmanuel