- Posts: 80
Wang, Chen, Usinger, and Reeve published a mode of administration study comparing phone interview and computer self-administration.
Qual Life Res. 2017 Nov;26(11):2973-2985. doi: 10.1007/s11136-017-1640-3.
To evaluate measurement invariance (phone interview vs computer self-administered survey) of 15 PROMIS measures responded by a population-based cohort of localized prostate cancer survivors.
Participants were part of the North Carolina Prostate Cancer Comparative Effectiveness and Survivorship Study. Out of the 952 men who took the phone interview at 24 months post-treatment, 401 of them also completed the same survey online using a home computer. Unidimensionality of the PROMIS measures was examined using single-factor confirmatory factor analysis (CFA) models. Measurement invariance testing was conducted using longitudinal CFA via a model comparison approach. For strongly or partially strongly invariant measures, changes in the latent factors and factor autocorrelations were also estimated and tested.
Six measures (sleep disturbance, sleep-related impairment, diarrhea, illness impact-negative, illness impact-positive, and global satisfaction with sex life) had locally dependent items, and therefore model modifications had to be made on these domains prior to measurement invariance testing. Overall, seven measures achieved strong invariance (all items had equal loadings and thresholds), and four measures achieved partial strong invariance (each measure had one item with unequal loadings and thresholds). Three measures (pain interference, interest in sexual activity, and global satisfaction with sex life) failed to establish configural invariance due to between-mode differences in factor patterns.
This study supports the use of phone-based live interviewers in lieu of PC-based assessment (when needed) for many of the PROMIS measures.
It is interesting that the Magnus et al. study found essentially no mode effect for telephone interviewer-administration than self-administration. That article cited the Hanmer et al. (2007) article that analyzed 4939 subjects from JCUSH, 23,006 from MEPS,
3844 from NHMS, and 3878 from USVEQ. Hanmer et al. found that telephone administration yielded more positive
HRQoL estimates than self-administration in older age (70 and older) groups. Other studies have also found similar results, e.g., Hays, Kim, Spritzer, Kaplan, Tally, Feeny, Liu, & Fryback, D. G. (2009). Effects of mode and order of administration on generic health-related quality of life scores. Value in Health, 12, 1035-1039:
Objective: We evaluate the effects of mode and order of administration on health-related quality of life (HRQOL) scores.
Method: We analyzed HRQOL data from the Clinical Outcomes and Measurement of Health Study (COMHS). In COMHS, we enrolled
patients with heart failure or cataracts at three sites (University of California, San Diego, University of California, Los Angeles, and University of Wisconsin). Patients completed self-administered HRQOL instruments at baseline and months 1 and 6 post-baseline, including the EuroQol (EQ-5D), Health Utilities Index (HUI), Quality of Well-Being Scale—selfadministered(QWB-SA), and the Short Form (SF)-36v2. At the 6 months follow-up, individuals were randomized to mail or telephone administration first, followed by the other mode of administration. We used repeated measures mixed effects models, adjusting for site, patient age, education,gender, and race.
Results: Included were 121 individuals entering a heart failure program and 326 individuals scheduled for cataract surgery who completed the survey by mail or phone at the 6-month follow-up. The majority of the sample was female (53%) and white (86%). About a quarter of the sample had high school education or less (26%). The average age was 66 (36–91 range). HRQOL scores were higher (more positive) for phone administration following mail administration. The largest differences in scores between phone and mail responses occurred for comparisons of telephone responses for those who were randomized to a mail survey first compared with mail responses for those randomized to a telephone survey first (i.e., mode effects for responses that were given on the second administration of the HRQOL measures). The QWB-SA was the only measure that did not display the pattern of mode effects. The biggest differences between modes
were 4 points on the SF-36v2 physical health and mental health component summary scores, 0.06 on the SF-6D, 0.03 on the QWB-SA, 0.08 on the EQ-5D, 0.04 on the HUI2, and 0.10 on the HUI3.
Conclusions: Telephone administration yields significantly more positive HRQOL scores for all of the generic HRQOL measures except for the QWB-SA. The magnitude of effects was clearly important, with some differences as large as a half-standard deviation. These findings confirm the importance of considering mode of administration when interpreting HRQOL scores.
In general, it is encouraged to use the same mode of administration in a research study. This will reduce the amount of noise (measurement error) from switching to one mode to another. For other PRO measures, there have been multiple studies comparing PC vs paper-based assessment and has found typically no significant differences. However, some studies have found moving from PC (or paper) to a phone-based interview, there tends to be some additional measurement error as taking a survey with a phone interviewer is a different experience than privately by PC (or paper). Social desirability may have an effect on phone interviews. It will also depend on the sensitivity of the subject being assessed (e.g., fatigue vs sexual functioning).
PROMIS investigators have also conducted mode of administration studies. For the adult measures, Bjorner et al (2014; 2014) conducted a study in 924 adults with COPD, depression, or rheumatoid arthritis. They found "no statistically or clinically significant differences in score levels or psychometric properties of Interactive-Voice Response, paper questionnaire, or personal digital assistant (PDA) administration compared to PC." PROMIS domains included were physical function, fatigue, and depression.
For the pediatric measures, Magnus et al (2016) compared PROMIS pediatric measures of depressive symptoms, fatigue, and mobility across PC and telephone interviewer for both children self-report and parent proxy-report. Participants included 377 children (ages 8-17) with a range of health conditions (e.g., allergies, asthma, ADHD, overweight, mental health) and 375 parents. "Overall, the correlations between scores obtained with the two modes of administration were high, approximately equal to reliability. And the differences between the means were sufficiently small that they would not affect overall interpretation of the level of scores relative to the population. So it should be reasonable to use either mode, uniformly, in any given study, or with stratification to correct for any mode effects that occur."
Bjorner JB, Rose M, Gandek B et al. Difference in method of administration did not significantly impact item response: an IRT-based analysis from the Patient-Reported Outcomes Measurement Information System (PROMIS) initiative. Quality of Life Research 2014;23:217-227.
Bjorner JB, Rose M, Gandek B, et al. Method of administration of PROMIS scales did not significantly impact score level, reliability, or validity. JCE 2014;67:108-113.
Magnus BE, Liu Y, He J et al. Mode effects between computer self-administration and telephone interviewer-administration of the PROMIS pediatric measures, self- and proxy report. Quality of Life Research 2016 (online first)1-12.