The Effect of a Language Preference Question on Quiz Scores

PxD operates the MoA-INFO platform in collaboration with Kenya’s Ministry of Agriculture to provide free agricultural recommendations to farmers via SMS messages. To recruit farmers to the service, PxD worked with Safaricom (a large Kenyan phone company) to send SMS messages to owners of mobile phones in rural areas; the messages had a keyword in English or Swahili (“FARM” or “SHAMBA”) that farmers could use to register. Farmers who texted the keyword to the MoA-INFO shortcode were sent a registration survey. This experiment builds on a language prompt experiment “The Effect of a Language Prompt on Farmers’ Platform Engagement”, which tested whether offering users the option to switch languages affected registration, retention, and engagement with the MoA-INFO platform. That study found increased initial completion rates when users could choose their preferred language, but it did not find a sustained impact on engagement. This A/B test aimed to measure whether asking farmers about their preferred language affects their comprehension of the information in the registration survey content, as measured by their performance on a comprehension quiz.

Farmers who had been sent a registration survey were randomly assigned to the treatment group, who received an SMS asking them which language they preferred, or the control group, who did not receive the language preference question. Approximately one thousand farmers were randomly selected to receive an SMS invitation to take a quiz to test their knowledge about Fall armyworm (FAW). This randomization was stratified by treatment status if the user had been offered the opportunity to switch languages in the registration survey, and language of registration message (English or Swahili), resulting in four groups of equal size ~250. Users who were given the option to switch languages scored slightly higher on the quiz when scoring only the questions for which they provided a response. The treatment had a slightly negative effect on quiz scores calculated by the number of correct responses out of all questions, whether or not farmers provided a response. These results are not statistically significant and the findings do not provide supporting evidence that the opportunity to switch languages has a positive effect on the FAW comprehension quiz scores.