The Effect of an SMS Rating Survey on Platform Engagement and Practice Awareness
KEN -18 -1392Last modified on October 29th, 2025 at 12:03 pm
-
Abstract
PxD operates the MoA-INFO platform in collaboration with Kenya’s Ministry of Agriculture to provide free agricultural recommendations to farmers via SMS. Farmers are provided with four menu options: Pesticides, Managing Fall Armyworm (FAW), Detecting FAW, and FAW Origins and Lifecycle. Upon selecting a specific topic area, a farmer has a 25% chance of being asked to take a satisfaction survey via SMS. The survey asks the farmer to rate how useful the MoA-INFO platform is and if the farmer plans to follow the recommendations. Receiving the rating survey decreased the opt-out rate by 1.4 percentage points (pp) and decreased access to additional menu topics immediately following the survey by 6.1 pp. The pesticides menu had the highest average rating but the lowest likelihood of the recommendations being followed; the managing-FAW menu had the lowest average rating but the highest likelihood of the recommendations being followed.
-
Status
Completed
-
Start date
Q3 Jul 2018
-
End date
Q3 Aug 2018
-
Experiment Location
Kenya
-
Partner Organization
Kenya Ministry of Agriculture
-
Agricultural season
Short Rains
-
Experiment type
A/B test
-
Sample frame / target population
MoA-INFO Platform Users (farmers) who opted into a specific menu option
-
Sample size
12,164
-
Outcome type
Service engagement
-
Mode of data collection
PxD administrative data
-
Research question(s)/hypotheses
1.Which SMS menu topics receive the highest ratings for information usefulness (1–5 scale) and for the likelihood that farmers follow the recommendations?
2. Does receiving an SMS rating survey affect interactions with the platform in terms of accessing additional topics and opting-out)? -
Research theme
Communication technology
-
Research design notes
When users opted-in to a specific menu option of Pesticides, Managing FAW, Detecting FAW, or FAW Origins and Lifecycle, they were automatically enrolled in the experimental sample and had a 25% probability of being prompted to take a brief satisfaction survey. This survey asked the farmer two questions: (1) How useful is the information? (on a 1–5 scale, where 1 = not useful and 5 = very useful), and (2) Do you plan to follow the recommendations? (with response options of Yes, No, or Maybe).
To avoid survey fatigue and repeated exposure, a label was assigned to the user’s profile when a user completed the survey for a given topic, to prevent future survey prompts in the same topic. Users not selected into the survey prompt formed the control group, while those prompted formed the treatment group. Random assignment was implemented at the individual level upon topic selection.
-
Results
Of the 25% of farmers who randomly received an invitation to complete the satisfaction survey upon selecting a menu topic, 66% completed the survey. Receiving the survey decreased the opt-out rate by 1.4 pp (20%). On average, the platform was rated 3.93/5 by users, and the majority (58%) said they would follow the recommendations.
By topic, when asked if they planned to follow the recommendations, farmers who selected the "Managing FAW" topic responded yes at the highest rate (62%); those who selected "Pesticides" were the least likely to do so (51%). The pesticides menu option had the highest average rating (4.11), whereas "Managing FAW" had the lowest rating (3.85).
The trade-off is that farmers who received the SMS survey the first time they accessed the menu were less likely to access additional menu topics immediately following the survey, by 6.1 pp (15%). The majority of users (59.6%) accessed the menu only once.