Login to view full entry:
The Order and Designation of the Prompts of a Rating Scale
INDIND -21 -1657Last modified on December 19th, 2025 at 10:41 am
-
Abstract
PxD operates Ama Krushi, a free agriculture information service delivered over mobile phones, in collaboration with the State Government of Odisha Department of Agriculture using a two-way Interactive Voice Response (IVR) platform with “outbound” push calls and an “inbound” hotline service. The outbound service is the primary way in which PxD provides farmers with timely, customized agricultural information. Farmers’ ratings of how useful they find the information provide important feedback for PxD to implement improvements to the service.
Different monitoring and evaluation efforts, such as polling surveys and rating scales, have yielded different results regarding the usefulness of the advisory. This variation may reflect the differences in the feedback channels. Of farmers who received advisory messages in a previous six-month period, 26% rated at least one message on a Likert scale ordered from 5 (very useful) to 1 (not useful), which was played at the end of the weekly push call. The average rating of the service during that six-month period was 4.2.
We conducted a test involving individual-level randomization of over 1 million paddy farmers. Different versions of an IVR-message rating scale were tested by comparing reversed-order and reversed-designation prompts to the default prompts. The results indicate that the ratings farmers provide are substantially influenced by the order the rating options are presented and the values assigned to them. -
Status
Completed
-
Start date
Q2 Apr 2021
-
Experiment Location
India / Odisha, India
-
Partner Organization
Government of Odisha
-
Agricultural season
Kharif
-
Experiment type
A/B test
-
Sample frame / target population
Ama Krushi farmers who primarily grow paddy and who (i) were receiving paddy-related advisory, (ii) had picked up at least one call in the six months preceding the test, (iii) were not located in RCT districts, and (iv) had not previously participated in A/B tests.
-
Sample size
1,097,661
-
Outcome type
Service engagement
-
Mode of data collection
Automated survey, PxD administrative data
-
Research question(s)
1. Are average farmer ratings determined by the sequence used for the ratings scale?
2. Do farmers press a number/select a rating on the basis of what they hear first? -
Research theme
Communication technology, Message framing
-
Research Design
Over 1 million paddy farmers were randomly assigned at the individual level to a control group and two treatment groups, with equal probabilities. Randomization was stratified by gender, smartphone ownership, and whether the farmer had actively rated an IVR message in the six months prior to the experiment. The three groups received the service rating prompt as follows:
Control group (T0): Received the standard Likert scale rating prompt ordered from 5 (very useful) to 1 (not useful).
Treatment 1 (T1): Received a reversed-order scale so that farmers heard “1—Not Useful” first.
Treatment 2 (T2): Received a reversed-designation scale, so that “5—Not Useful” was presented first, followed by the rest of the options.
The experiment ran for two weeks.
-
Results
Changing the design of the ratings scales caused large, significant differences in the mean rating. T2 of reversing the designation lowered the farmers’ rating by around 1.3 points on the scale of 1–5 compared to T0. T1 of reversing the order also caused a decrease, of around 0.1 points, in the mean rating compared to T0. Both results are statistically significant at the 1% level. This means that T0 farmers who received the standard Likert scale rating prompt from 5 to 1, where 5 means very useful, rated the service on average very useful (4.2) while T2 farmers who received a menu from 5 to 1, where 5 means not useful, rated the service below average (2.8). We do not know how much farmers noticed the changes in scales, so we cannot identify what is driving these effects.