How to develop and evaluate a PRO instrument - Pubrica
                     HOW TO DEVELOP 
EAVNADLUATE A PRO
INSTRUMENT
An Academic presentation by
Dr. Nancy Agens, Head, Technical 
Operations, Pubrica Group: 
wEmwawi.l:p  [email protected]
Today's Discussion
Outlin
What is 
e PROAsk the Doctor, Why Ask the 
Patient
What Information Do I Want from the Patient? – The PRO 
Instrument Designing a Valid PRO Instrument
Evaluating PRO 
Instrument
Conclusio
n
What is 
PRO?
PRO or Patient-reported outcome is a relatively novel method to evaluate the 
outcome of any treatment by clinician or researcher (in clinical trial) purely based on 
the inputs got from the patient alone.
The United States Food and Drug Administration (US-FDA) defines PRO as ‘any 
report of the patient’s health condition that comes directly from the patient, without 
interpretation of the patient’s response by a clinician or anyone else.’
Ask the Doctor, Why Ask the 
Patient?
Objective data regarding outcome can be given by the treating physician, by 
investigations and by the caregiver.
These are Observer-reported outcomes (OROs) and medicine has long relied on these 
for evidence-based management.
The concept of Health-related Quality of Life , a multi-dimensional approach that 
records patient’s perceived effect of the illness and treatment on quality of life in his/her 
physical, mental, emotional, economical, and social space.
PRO data is used to inform and guide patient-centered care, to assess risk and benefit 
from treatment, to decide on treatment protocols and to guide health policy makers.
What Information Do I Want 
fPraotmie ntht?e – The PRO Instrument
The PRO instrument is the means to collect the data related to the PRO concept, in 
the form of questionnaire and the information and documentation that supports its use.
In other words, PRO instrument is a questionnaire that compiles data related to the 
PRO concept of interest which can be summarized as a score.
Content validity is the extent to which an instrument measures the concept of 
interest that is desired from both the developer’s and user’s perspective.
Designing a Valid
PRO Instrument
Contd
..
Steps involved are: -
1. DEFINING THE CONCEPT
Concept should be clearly defined based on review and literature gap analysis of existing 
PRO instruments as well as taking inputs from scientific research services, developers and 
most importantly the patients or the users.
2. CONCEPTUAL FRAMEWORK
Breakdown the concept to sub-concepts and finally to items or the questions that will be 
included in the questionnaire.
3. END-POINT MODEL
Endpoint is the final measured outcome of the treatment that will be compared, and an 
endpoint model summarises all the endpoint based on OROs and PROs.
Contd
..
4. REVIEW OF PRELIMINARY INSTRUMENT
Literacy review is done to understand the proportion of people who may or
may not have the literacy level to take the questionnaire.
Translation review is done to ensure conceptual equivalence when
translated to another language.
Cognitive interview is the process where the user is made to take the
questionnaire by thinking aloud to understand the approach of the user to
the questions.
Contd
..
5. RELIABILITY TESTING
Measure to estimate reproducibility and consistent true estimation of outcomes 
under consideration. It is further described below.
6. VALIDATION
It is an ongoing never-ending process of improving the instrument by repeated 
feedback mechanism.
Evaluating PRO Instrument
As clinical validation data accumulates, a PRO instrument undergoes review and revision 
to make them robust. It includes the evaluating for the following: -
1. VALIDITY
is the extent to which the scores and their interpretations, supported by theoretical 
evidence, correlate with the proposed use of the instrument.
a. Content Validity- It is a measure of how much the instrument measures the 
concept of interest.
(i) Qualitative estimation – include items involved, their recall period, response
options, scoring pattern, method of conducting the questionnaire etc.
Contd
..
(ii) Quantitative estimation – e.g.
=> Item response theory (IRT) – A powerful tool to assess whether the items 
used measure the entire spectrum or continuum of the patient responses.
=> Rasch analysis - Only IRT tool to use the total of all the items in the PRO 
instrument making it a simple model.
b. Construct Validity- It is based on comparison with convergent and divergent 
groups with known validity and showing that the instrument under study conforms 
to similar interpretations.
c. Criterion Validity- Comparison with known criteria or known gold standard for
this concept.
Contd
..
2. RELIABILITY
a. Reproducibility or test-retest reliability- It is assessed under same conditions in 
same user in a close span of time to assess any variations in recording results that 
can be attributed to the instrument.
b. Internal consistency- The items should deliver same scores in similar situations 
consistently.
c. Floor and ceiling effect – i.e. the instrument’s ability to measure extremes of 
construct helps choose the proper instrument for the study.
d. Inter-interviewer reliability- there should not be significant difference in scores 
when interviewed by two different interviewers.
e. Variability – Instrument should have ability to detect changes when outcomes are 
different, else it is no good.
Conclusion
Adhering to good practices while designing and validating a PRO instrument 
is primary for it to be effective and receive regulatory approvals.
Readiness to continuously revisit and revise items is needed to do better justice 
to the concept of interest in the long run.
Contact Us
UNITED KINGDOM
+44-
1143520021
INDI
A+91-
9884350006
EMAIL
[email protected]