The PxD Experiment Registry documents our design experiments and impact evaluations—from simple A/B tests to large-scale randomized impact evaluations. It captures experiments we've conducted on our own services and with partners, measuring how specific service design changes affect outcomes and the overall impact of digital agriculture services. The Registry is searchable, filterable, and exportable, and is designed as an open resource to share learnings with others building digital agricultural services. For questions or publishing inquiries, contact info@precisiondev.org.

not-registered PxD staff internal login:

PaddyAI–Artificial Intelligence versus Human Engagement: A Comparison for IVR Advisories

INDIND -25 -3153

Last modified on March 6th, 2026 at 1:34 am

    Basic Information

  • Abstract
    PxD operates the Coffee Krishi Taranga (CKT) platform in collaboration with the Coffee Board of India to provide a voice-based advisory service for coffee farmers through a two-way Interactive Voice Response (IVR) system. We have developed an AI-powered assistant, PaddyAI, guided and controlled by agronomists, to generate deeply customized agricultural advisories on demand. At scale, the AI can rapidly produce the many customized, translated, and audio versions needed, a task that would be impossible for agronomists to do manually while maintaining quality and timeliness.

    This A/B test is the first in a series of service improvement experiments as PxD integrates AI-enabled IVR advisories through PaddyAI. The objectives of this A/B test are to understand (1) whether AI-generated advisory content has similar user engagement compared to human-generated advisory content, and (2) whether an AI voice has similar user engagement compared to a human voice delivering the content. Findings from this A/B test will help improve the generation of advisory and the text-to-speech processes of PaddyAI.
  • Status
    Ongoing
  • Start date
    Q4 Nov 2025
  • End date
    Q4 Dec 2025
  • Experiment Location
    India / Karnataka, India
  • Partner Organization
    Coffee Board of India
  • Agricultural season
    _N/A

    • Research Design

  • Experiment type
    A/B test
  • Sample frame / target population
    Coffee farmers in Karnataka
  • Sample size
    28,350
  • Outcome type
    Platform engagement
  • Mode of data collection
    PxD administrative data
  • Research question(s)
    1. Does AI-generated advisory content have similar levels of farmer engagement compared to human-generated advisory content?
    2. Does AI-voiced advisory have similar levels of farmer engagement compared to human-voiced advisory?
  • Research theme
    Agricultural management advice, Artificial intelligence (AI), Message narration, Service design
  • Research Design

    The A/B test will run for seven weeks, during which coffee farmers will receive one advisory per week based on the coffee crop calendar.

    Farmers will be randomly assigned with equal probability to one of three groups, stratified by district and gender. The groups are:

    • T0: human-generated instructional advisories delivered with a human voice (status quo).
    • T1: AI-generated instructional advisories delivered with an AI-generated voice.
    • T2: AI-generated instructional advisories delivered with a human voice.

    We will collect data for seven rounds (R1–R7). We will use PxD administrative platform data to assess user engagement, measured as:

    • Listening rates: The proportion of the advisory call that the farmer listens to.
    • Pick-up rates: The proportion of advisory calls that are picked up in Round 2 to Round 7 of calls.
    • 80% listening rates: The proportion of the advisory calls when farmers listen to at least 80% of the content.

    • Results

  • Results
    Farmers who received AI-generated advisories with an AI voice had 1.3 percentage points higher listening rate (or 2.2% higher) compared to farmers who received human-generated advisories. This difference is statistically significant (p<0.05).
    For the 80% advisory listening rate, farmers in the AI w/ AI audio arm had a 1.07 percentage points higher 80% listening rate (or 2.1% higher)
    While we see differences in average listening rates and the 80% listening rate at the advisory level, the 80% listening rate at the farmer level shows no differences. While AI-generated advisories, on average, have higher listening rates, the proportion of farmers who listen to 80% of the advisory duration is similar across the three arms.
    Even though we have promising results from the engagement metrics, “top of the funnel” behavioral metrics show a different picture.
    Advisory recall: farmers who received AI-generated advisories reported a 6.27% lower recall than those who received human-generated advisories.
    Advisory retention: farmers who received AI-generated advisories reported retaining 7.7% less information.
    Advisory comprehension: We do not see any difference across these groups
    Based on additional analysis, there seems to be a ‘novelty effect’ for AI-generated advisories (especially for advisories with an AI voice).

You have successfully submitted or edited this experiment.

Date Result Anticipated
Add reminder to Google Calendar to update your entry on: 13 Feb 2026

Contact us for assistance.