|
RCR TOOLKIT
<< Prev | Table of Contents | Next >>
When determining the goals of an RCR program, it is also important to consider how to evaluate whether these goals have been achieved. The RCR community has made significant strides in assessing the overall effectiveness of RCR education. However, attempts to determine a change in behavior – arguably the intended ultimate result of RCR training – have been difficult due to the multiple variables that impact behavior, the challenge of defining a control sample, and the question of whether an individual’s hypothetical behavior in a case study actually mirrors their behavior when faced with the situation in the real world (see for example Elliott & Stern 1996). Thus, the achievement of goals focusing on changes in knowledge or skills in RCR might be somewhat easier to assess than a change in attitude or behavior.
The most common approach to evaluating individual gains from a course or program is to use pre- and post-testing. This can assess an individual’s knowledge coming in to the program and then compare this to their knowledge of the same information upon completing the program. It can also assess to some extent attitudes and skills.
For a concise overview of considerations when evaluating a program, visit the ethics education repository research-ethics.net:
http://www.research-ethics.net/introduction/evaluation/
Common Evaluation Tools from the Literature
Defining Issues Test (DIT-2)
Rest, Narvaez, Thoma, &. Bebeau, 1999 “DIT2: Devising and testing a revised instrument of moral judgment.” Journal of Educational Psychology 91(4):644-659
The DIT-2 (version 2 of the Defining Issues Test) assesses moral judgment using a series of case studies to which subjects respond. Many programs have used the DIT-2 as a pre-test and post-test to gauge the degree to which an RCR program has impacted the moral reasoning of students.
To order the survey, visit The Center for the Study of Ethical Development website: http://www.centerforthestudyofethicaldevelopment.net
There you can purchase the exams and the Center will score them for you. They have a sliding fee scale depending on how many tests you order.
Ethical Decision Making Measure
Mumford, M., Davenport, L.D., Ryan, P.B., Connelly, S., Murphy, S.T., Hill, J.H., and Antes, A.L. 2006 “Validation of Ethical Decision Making Measures: Evidence for a New Set of Measures.” Ethics and Behavior 16(4):319-345.
These measures use a range of ethical situations and case studies to probe the respondent’s ethical decision making.
Baseline Knowledge Test
Heitman, E., Olsen, C.H., Anestidou, L., and Bulger, R.E. 2007 “New Graduate Students’ Baseline Knowledge of the Responsible Conduct of Research” Academic Medicine 82(9):838-845
This test provides an assessment of baseline knowledge of RCR topics targeting graduate students. While the test is designed for research purposes and encompasses all topic areas of RCR, a reduced set could be appropriate for a course assessment.
Views on Science-Technology-Society (VOSTS)
Aikenhead, G.S. and Ryan, A.G. 1992 “The Development of a New Instrument: “Views on Science-Technology-Society” (VOSTS)” Science Education 76(5): 477-491
The questionnaire probes attitudes about a scientist’s and engineer’s social responsibility.
It can be found on the author’s webpage at: http://www.usask.ca/education/people/aikenhead/vosts.pdf
Pre-Test and Post-Test Examples
Example attitude surveys for use during an RCR course
These surveys are more commonly used to engage participants during a course, but could form the basis for some pre- and post-test questions as well. Examples for all the RCR core areas can be found in Appendix I of the Scientific Integrity textbook companion website, scientificintegrity.net.
University of Washington Postdoc RCR Attitudes Survey
Assesses current attitudes and practices regarding professional development and responsible conduct of research program targeting postdocs
Download: UW RCR Attitudes Survey
Medical University of South Carolina International Postdoc RCR Survey
This survey was used as a pre-test to assess knowledge and attitudes towards RCR topics among international postdocs attending an orientation to responsible research practices and norms in the U.S.
Download: MUSC International RCR Presurvey
Florida State University Pre- and Post-tests on Graduate Student RCR Knowledge
Tests developed by Florida State University to test incoming and outgoing knowledge of RCR issues among graduate students. This was part of a program funded by the federal Office of Research Integrity and the Council of Graduate Schools:
Pre-test: http://www.cgsnet.org/portals/0/pdf/RCR_Pre-Test.pdf
Post-test: http://www.cgsnet.org/portals/0/pdf/RCR_Post-Test.pdf
Exit Survey: http://www.cgsnet.org/portals/0/pdf/RCR_ExitSurvey.pdf
Children's Hospital of Philadelphia Pre-Assessment and Course Evaluation
These surveys were developed by Children's Hospital of Philadelphia for use in their RCR courses for trainees. The Pre-Assessment is primarily a gauge of who the audience is and their general preparation. The course evaluation assesses the course usefulness and design.
CHOP RCR Presurvey
CHOP RCR Postsurvey
MD Anderson Cancer Center Postdoc Association Pre-Survey of RCR Training and Attitudes
A pre-survey conducted of MD Anderson's postdocs probing their past training, interests and attitudes with regard to RCR topics.
Download: RCR Presurvey MD Anderson
Oak Ridge Associated Universities Postdoc Baseline RCR Knowledge Pre-Test
Probes postdoc awareness of RCR issues prior to participating in training
Download: ORAU RCR Presurvey
University of North Carolina at Chapel Hill Postdoc Ethical Decision Making Survey
Examines the ethical decision making choices of postdocs and can be used to assess the impact of RCR training
Download: UNC Ethical Decision Making Survey
Course Evaluations
It is also important to do an evaluation of the program delivery to identify the relative success of the program format and potential improvements. Several examples of these are included below.
Examples of workshop evaluation forms:
Stanford; http://aic.stanford.edu/education/forms/wrkeval.pdf
Oregon State; http://osulibrary.oregonstate.edu/instruction/workshop.htm
North Carolina State: http://www2.chass.ncsu.edu/CWSP/fac_seminar/sem_eval.html
UCLA: http://www.library.ucla.edu/college/reference/email/seminar.htm
Other examples of evaluation questionnaires from HHMI Lab Management courses:
http://www.hhmi.org/resources/labmanagement/resources.html#evaluation
Selected References on Assessment of RCR Training Effectiveness
Elliott, D. & Stern, J. E. (1996). Evaluating teaching and students' learning of academic research ethics. Science and Engineering Ethics 2(3): 345-366.
In a study of the outcomes of a research integrity course, the authors find a null effect of the course on ethical behavior and suggest that evaluation should return to focusing on knowledge and skills.
Martinson, B.C., Anderson, M.S., and de Vries, R. (2005) “Scientists Behaving Badly.” Nature, 435, 737-738.
Study on self-reported questionable research practices, with comparison of late-, mid- and early-career scientists.
Anderson, M.S., Horn, A.S., Risbey, K.R., Ronning, E.A., De Vries, R., and Martinson, B.C. (2007) “What Do Mentoring and Training in the Responsible Conduct of Research Have To Do with Scientists’ Misbehavior? Findings from a National Survey of NIH-Funded Scientists.” Academic Medicine. 82(9): 853
A broad study of NIH PIs and trainees that finds that the RCR training method that seems to have the best result is mentoring.
Kalichman, M.W. and Plemmons, D.K. 2007 “Reported Goals for Responsible Conduct of Research Courses.” Academic Medicine 82(9):846-852
A survey of instructors of RCR courses for NIH training grant trainees finds a lack of consensus about stated goals of instruction.
Please complete a short feedback questionnaire on the toolkit and tell us how it might be improved!
|
