Login to view full entry:
Adding Menu Replay if No Option is Selected (ATA Experiment 102)
ETH -17 -1247Last modified on December 19th, 2025 at 10:12 am
-
Abstract
PxD is partnering with Ethiopia’s Agricultural Transformation Agency (ATA) to help improve the effectiveness of their voice-based mobile-phone advisory service, the 8028 hotline, by conducting continuous iterations and experiments, as well as by making suggestions for improvements to and customization of the service. The service has millions of registered farmers and represents the first in Africa to be maintained by a government entity at such a large scale.
The 8028 hotline Interactive Voice Response (IVR) system uses phone key navigation. If a user is inactive for 10 seconds in any menu, then the system hangs up automatically. Users’ inactivity could be due to their difficulty in using the system (e.g., digital illiteracy) or because the options are read too quickly and the user has not yet decided which menu to access. We implemented an intervention to replay the menu if no option is selected in 10 seconds, for randomly selected farmers. We find that this feature significantly increased the probability of users accessing advisory content for the users affected by the treatment, meaning users assigned to the treatment group who did not make a menu selection within 10 seconds. -
Status
Completed
-
Start date
Q4 Nov 2017
-
Experiment Location
Ethiopia
-
Partner Organization
Ethiopian ATA
-
Agricultural season
_N/A
-
Experiment type
A/B test
-
Sample frame / target population
8028 hotline users
-
Sample size
160,000
-
Outcome type
Information access
-
Mode of data collection
Partner administrative data
-
Research question(s)
Does automatic replay of the IVR menu when no option is selected improve user engagement compared to when the menu is played only once?
-
Research theme
Communication technology
-
Research Design
8028 hotline users (n = 160,000) were randomly assigned to either the control group or the treatment group. If a user did not make a selection within 10 seconds of the menu options being played, then:
Control group: The system automatically hung up (status quo).
Treatment group: The menu was automatically replayed.We used administrative platform data to measure the user-engagement outcomes of language selection and content access.
The experiment was implemented for one month from November 29 to December 31, 2017.
Study publication forthcoming.
-
Results
Automatic replay of the menu led to modest and statistically significant increases in the proportion of all users in the treatment group who selected a language (1 percentage point, pp) and accessed content (1.5 pp) in the first call after the intervention was implemented (p < 0.01). These effects are concentrated in less than 10% of treatment group users who did not make a menu selection within 10 seconds (the other 90% made a menu selection). For this 10% of users affected by the treatment, the intervention during the first call led to a 16.7 pp and a 14.3 pp increase in language selection and content access, respectively. Over time with more calls, these effects decreased. Overall, the effect of the treatment is substantive and statistically significant on the users affected by the treatment, who show a 4 pp increase in content access.