not-registered Login to view full entry:

Adaptive Treatment Assignment of Outreach Enrollment Calls

INDIND -19 -3314

Last modified on January 27th, 2026 at 8:58 am

    Basic Information

  • Abstract
    PxD operates Ama Krushi, a free agriculture information service delivered over mobile phones, in collaboration with the State Government of Odisha Department of Agriculture, using a two-way Interactive Voice Response (IVR) platform with “outbound” push calls and an “inbound” hotline service. We designed an adaptive experiment—an approach that allows the experimental design to change based on accumulating data during the trial—to choose the best method from a variety of call methods for enrolling farmers in the Ama Krushi service. The aim was to decide which type of outreach call worked best—for example, whether to call in the morning or evening, and whether to send a reminder text message beforehand. Instead of testing all options equally, we ran the experiment in multiple waves. After each wave, we used the results to adjust how many farmers were assigned to each call strategy in the next wave. Strategies that looked more promising were tested more, while enough testing of alternatives was still kept in order to learn reliably. Over the 17 waves that involved about 10,000 phone numbers, the experiment gradually concentrated on the most effective strategies while continuing to compare close competitors. By the end, one approach—calling at 10AM with a text message sent an hour in advance—had about a 75% chance of being the best.
  • Status
    Completed
  • Start date
    Q2 Jun 2019
  • Experiment Location
    India / Odisha, India
  • Partner Organization
    Government of Odisha
  • Agricultural season
    _N/A

    • Research Design

  • Experiment type
    Other
  • Sample frame / target population
    Rice farmers on government lists of phone numbers not yet enrolled in Ama Krushi
  • Sample size
    10,000
  • Outcome type
    Service engagement
  • Mode of data collection
    PxD administrative data, Automated survey
  • Research question(s)
    1. Which type of outreach call works best to enroll farmers in the Ama Krushi service?
    2. Is adaptive treatment assignment a practical and effective method for identifying more effective design choices for PxD programs?
  • Research theme
    Message timing and frequency, Service design
  • Research Design

    We selected the sample of rice farmers in Odisha from a list of phone numbers provided by our government partner. We set aside a batch of 10,000 valid numbers that are not on the Indian “do not disturb” list, and randomly selected waves of 600 phone numbers for testing. We designed six treatments that combined automated voice calls in the morning (10AM) or evening (6:30PM) with text message alerts sent one hour or 24 hours ahead (or not at all).
     
    The outcome is a binary variable describing call completion: It equals 1 if the call recipient answered the five questions asked during the call to complete enrollment.
     
    Starting on June 3, 2019, a new experimental wave began every other day and was completed the following day for a total of 17 waves. For each call strategy, we tracked how many farmers successfully completed the enrollment call. We began by assuming all six strategies were equally likely to work, then updated our estimates as real data came in from each wave. Based on these updated success rates, we adjusted how many farmers would be assigned to each strategy in the next round—gradually shifting more farmers toward the strategies that were working best.
     
    For more information, see Kasy and Sautmann (2021).

    • Results

  • Results
    Calling farmers at 10AM after sending a text message one hour before was the most successful strategy. About 19% of farmers who received this type of call completed enrollment—giving this strategy about a 75% chance of being the best approach out of all six of the strategies that were tested. Because the experiment continuously learned which strategies worked better, it automatically assigned more farmers to successful approaches over time. Cumulatively, nearly 40% of farmers received the most successful type of call, whereas under 4% received the least successful call (at 6:30PM without a text message alert). The overall enrollment success rate was 18%, compared to an estimated 17.2% if we had assigned equal numbers of farmers to all six strategies throughout.

    This adaptive approach had two benefits. First, it produced strong evidence about which strategy was best, which helped PxD choose what to implement in ongoing service design. Second, even during the experiment itself, more farmers received better-performing call strategies, which slightly increased the overall success rate compared with a standard experiment that would have split calls evenly across all options.

You have successfully submitted or edited this experiment.

The PxD Experiment Registry is an online tool for documenting and tracking PxD's experimental research projects and outcomes. The registry includes experiments where we are deliberately testing something and measuring outcomes from simple A/B tests comparing different versions of a service to large scale impact evaluations and RCTs. The registry does not include general surveys, studies, or pilot projects that did not include an experimental component, for example our Covid-19 surveys. The various fields of the registry can be searched and filtered using set criteria, and the results of these searches are exportable.  Please contact info@precisiondev.org if you have any questions or publishing queries.

Contact Niek for assistance.