MSU HRPP Manual Section 3-3
Evaluation and Quality Improvement
Evaluation and quality improvement (QI) are complementary processes that assess and provide feedback regarding the functioning of the Michigan State University (MSU) Human Research Protection Program (HRPP). Evaluation processes collect information about the program and QI uses the information collected to implement changes in policies and procedures to improve the functioning of the program. The goal and mission of the QI program is to protect human research subjects through a quality HRPP that achieves and maintains compliance using efficient and effective processes. Periodic assessment and evaluation of the HRPP are necessary to measure and make systematic improvements to the program.
Evaluation and QI of the MSU HRPP is a shared responsibility and is implemented by multiple mechanisms to acquire and evaluate information regarding the compliance, quality, efficiency, and effectiveness of the HRPP. While each of these areas (i.e. compliance, quality, efficiency, and effectiveness) are described separately in this policy, they are highly dependent upon one another. For example, a review that is highly efficient but deviates seriously from Institutional Review Board (IRB) requirements would not be acceptable.
Stakeholder input assists the HRPP in conducting evaluation and QI activities and is encouraged (e.g. wide publication of HRPP contact information to obtain answers to questions, express concerns, and convey suggestions regarding the HRPP). See HRPP Manual 5-6 “Contact Information” for a description of how stakeholders can provide input.
The following sections provide an overview of evaluation and quality improvement activities.
Self-Evaluation by PI
PIs are responsible for the conduct and oversight of their research study, including oversight of personnel. See HRPP Manual 4-6, “Responsibilities of Investigators.” PIs are encouraged to utilize the “Site Visit” forms available on the HRPP website to perform self-evaluations of their research studies. For some studies, the Compliance office may request that the researchers complete a self-assessment and confirm once it has been completed.
IRB Office Record Reviews
IRB staff are responsible for assuring that IRB applications are processed in accordance with MSU HRPP policies and procedures and that proper documentation is maintained in the IRB record. The HRPP manager provides an internal check on approved IRB applications and may provide specific corrective actions (e.g. with a particular IRB administrator on a specific research study). The HRPP manager and/or designee reviews all federally funded full review research studies and may review a sample of other research studies. The HRPP manager may identify systematic improvements (e.g. identification of a recurrent issue) and address through changes in process, targeted education, suggested policy changes, and other corrective actions as necessary.
IRB administrator IIIs are responsible for generating IRB minutes for their respective meetings. The HRPP manager(s) and IRB chair or designee will review all IRB minutes and may provide specific feedback prior to IRB members’ review. The HRPP manager(s) may also identify areas of concern (e.g. identification of a recurrent issue) and address through changes in process, targeted education, suggested policy changes, and other corrective actions as necessary.
Evaluations
HRPP staff are evaluated on an annual basis following MSU Human Resource guidelines.
IRB chairs will evaluate IRB members on a regular basis. Input may be provided to the Office of Research and Innovation or their designee on the participation of IRB members. This input may include but is not limited to their attendance at scheduled meetings, the number and types of reviews conducted, the timeliness of reviews, participation in workshops, etc. IRB chairs will also evaluate vice-chairs on a regular basis based on the duties and responsibilities the vice-chair has been asked to perform (assist the IRB chair as necessary, perform the chair’s duties when the chair is not available).
IRB chairs will be asked on a regular basis to complete a self-evaluation form based on the duties and responsibilities of the IRB chair and will meet with the HRPP Director to discuss the results and feedback.
Feedback Surveys
Feedback surveys are provided to a sample of current investigators to collect information that will be used for QI in the IRB application and review process or in the research site visit process. The information collected will be reviewed by the HRPP director and managers to determine whether immediate action or targeted improvements are needed and to evaluate systematic improvements. Information will be shared with others as appropriate (e.g., assistant vice president for the Office of Regulatory Affairs (AVP ORA), IRB chairs, HRPP staff). The response rate will be periodically assessed to determine the effectiveness of the feedback survey.
Compliance Office Review of IRB Records
The Compliance staff may review a sample of IRB submissions. Such a review may include assessments of types of applications (e.g. initial, continuing review, modifications, closures) and also research studies that involve certain subject populations or types of actions (e.g. waivers, research involving patient care services, or items typically associated with a Current Procedural Terminology (CPT) and/or Healthcare Common Procedure Coding System (HCPCS) code, pregnant women, fetuses, or neonates, prisoners, children, individuals with diminished capacity, less than one year approval, engaged sites, previous noncompliance, engaged sites).
Research Compliance Reviews
Compliance office staff are responsible for conducting pre-reviews for the HRPP to assure compliance with applicable regulations including assessments for Clinical Research Billing Compliance, Food and Drug Administration (FDA) requirements for investigational new drugs and devices, humanitarian device exemptions, emergency use, registration with clinicaltrials.gov, the Health and Insurance Portability and Accountability Act (HIPAA), and other applicable regulations and guidance for all clinical research submitted to the Institutional Review Board (IRB). Compliance office staff are responsible for generating HRPP reports for their respective research compliance reviews.
Site Visits
The Compliance staff will evaluate research studies for compliance with applicable federal regulations (e.g., 45 CFR 46, 21 CFR 50, 21 CFR 56) and/or agency specific requirements, national Medicare regulations, state or local laws, and MSU policies and procedures as part of the site visit process. The review will encompass the IRB records. The Compliance staff will perform site visits for all full review initial applications with subjects enrolled and a sample of other research studies. For some studies, the PI may be required to conduct a self-assessment and confirm once it has been completed. Directed site visits may also be requested and can be limited to a particular area of concern or may encompass the entire research study. An expanded number of research studies may be reviewed as part of the site visit if the site visit findings indicate a need. Site visit findings are provided to the Principal Investigator and others as appropriate through HRPP reports. See HRPP Manual 8-10 “Site Visits” for detailed procedures.
TEACH
Teaching Education and Compliance Help (TEACH) is a collaborative, educational initiative designed to assist the PIs and their research team with regulatory compliance. As part of TEACH, Compliance staff meet with researcher(s) conducting FDA regulated research after a full board IRB application is initially approved but before research begins and will provide researcher(s) with regulatory information specific to the type of research study (e.g. FDA regulated, clinical research billing compliance, HIPAA). If the PI has multiple FDA regulated projects, the need for additional visits will be dependent upon on whether there are additional regulatory components not previously discussed (e.g. device study instead of a drug study). TEACH may also be performed for a sample of other research studies (e.g., National Institutes of Health / U. S. Health and Human Services clinical trials).
Fiscal, Budget, and Other HRPP Activities
The HRPP director provides information on an annual basis or more frequently as requested to the AVP ORA. This information will include an assessment and evaluation of physical space, staff resources, funding, and committee activities. The AVP ORA reviews the information in order to develop plans for the coming fiscal year and follows the overall university budget planning cycle. When funds are allocated, the AVP ORA reviews and allocates funds to its units, including HRPP. Special budgetary requests may be made as needed by the HRPP director to the AVP ORA during the current fiscal year after resource allocation.
Meetings with the Institutional Official (IO)
The IO is provided with regular updates on the HRPP. The AVP ORA, the HRPP director, HRPP managers, IRB chairs, legal counsel, and the IO meet typically quarterly for programmatic discussion and review of the HRPP. This may include review of workload issues such as the timeliness of IRB reviews, the number, types and complexity of IRB submissions and any perceived deficiencies, such as the number of IRBs or IRB members, necessary reviewer expertise, resources or HRPP staff allocations.
Internal QI Group
An internal QI group meets regularly to discuss areas such as programmatic questions, HRPP policies or procedures, metrics, site visit trends, and to discuss potential improvements. Representatives for this group may include the AVP ORA or their designee, HRPP director, HRPP managers, and IRB chairs.
Outreach
While evaluation of outreach activities occurs internally within other university units, the HRPP Director will collaborate with the internal QI group to evaluate outreach activities related to human research protection (see HRPP Manual 11-2, Outreach and Engagement). The evaluation will occur at least annually and changes will be made as appropriate.
Metrics
The HRPP director coordinates the generation, review, and distribution of metrics. Examples are provided below to illustrate the types of metrics generated for quality improvement activities.
Composition of the IRB. The volume of submissions, stratified by college and department, will be reviewed to determine representation and areas of expertise needed by the committee. Composition will also be considered by the IRB chair and recommended as needed (e.g. particular areas of expertise).
Turnaround time. Metrics are generated to provide the volume and average approval turnaround time by submission type (e.g., initial, continuing review, modification) stratified by committee and review level. The average turnaround times will be compared with the goals for turnaround. Turnaround time is defined as the difference between the date the application is approved and the date the application is submitted to the IRB and is complete (e.g. all necessary supporting documents). Metrics will be provided to the HRPP director, AVP ORA, IRB chairs, and the IO.
The goals for turnaround time are: (application is submitted and complete to approval)
-
45 CFR 46.118: 2 days [range 1-3 days]
-
Initial
-
Exempt: 8 days [range 7-9 days]
-
Expedited: 21 days [range 2-4 weeks]
-
Full Review: 60 days [range 8-12 weeks]
-
-
Continuing Review
-
Expedited: 10 days [range 1-2 weeks]
-
Full Review: 21 days [range 2-4 weeks]
-
-
Modifications
-
Expedited: 10 days [range 1-2 weeks]
-
Full Review: 21 days [range 2-4 weeks]
-
The HRPP manager may also review protocols that considerably deviate from the maximum turnaround time goals to determine the cause. Summary reports of these reviews will be provided to the HRPP director and as appropriate, the internal QI group to evaluate and determine whether system-wide improvements are needed. The HRPP manager will identify significant deviations and determine from the sample whether components should be targeted for QI, such as process changes, education, training, etc.
The goals for components of the IRB turnaround time are:
-
Staff:
-
Processing applications: 2 days [range 1-3 days]
-
Processing comments: 2 days [range 1-3 days]
-
-
Reviewers:
-
Reviews 7 days
-
Responding to comments: 2 days [range 1-3 days]
-
The turnaround goals of the HRPP will be evaluated minimally every year to determine if the goals should be modified.
Compliance
The goal of compliance is to assure that research studies are reviewed, approved, and conducted in accordance with federal, state, and local regulations, university policies, and ethical principles.
The measures of compliance may include but are not limited to:
-
Number of directed site visits
-
Type and frequency of serious and / or continuing noncompliance
Methods of compliance review are described in the section above and include reviews of internal IRB operations, including protocols and IRB meeting minutes, post approval monitoring visits, etc.
Efficiency
The goal of efficiency is to complete IRB reviews in a timely manner in compliance with federal, state, and local regulations, university policies, and ethical principles. Efficiency is typically calculated by the turnaround time for approval (i.e., from the date of a complete submission to the IRB office to the date of approval). Components of this time may be examined to determine where targeted process improvements may occur.
The measures of efficiency may include but are not limited to:
-
Number of submissions, type, etc.
-
Turnaround time (submission to notification of approval, including components of this total time)
Quality
The goal for quality is to maintain a high quality HRPP that protects human research subjects involved in research and continually seeks to improve its program.
The measures of quality may include but are not limited to:
-
Assessment of compliance mechanisms (e.g. reduction in noncompliance reduces the potential risk to human subjects)
-
Serious and/or continuing noncompliance
-
Investigator satisfaction with the overall program
-
Subject protection
-
Trained HRPP staff, IRB members, and researchers (e.g. assessment of the education and training program)
Effectiveness
The goal for effectiveness is to achieve an HRPP that protects human subjects in research while maintaining compliance with federal, state, and local regulations, university policies, and ethical principles and achieves its goals in a timely manner.
The measures of effectiveness may include but are not limited to:
-
Reduced instances of noncompliance
-
Reduced severity of noncompliance
-
Improved investigator satisfaction with the HRPP
-
Positive responses from subjects
-
Improved investigator, staff, member, and subject understanding of the HRPP
-
Greater human subject research protections (e.g. unanticipated problems, complaints)
Such indicators taken together will measure the effectiveness of the HRPP. The internal QI group will review the information gathered with various evaluation mechanisms and review in their totality to determine the effectiveness of the program.
This policy and procedure supersedes those previously drafted.
Approved By: Vice President of Research and Graduate Studies, 7-19-2011. Revision 1 approved by VP Research & Graduate Studies on 3-13-2013. Revision 2 approved by VP Research & Graduate Studies on 12-11-2015. Revision 3 approved by Assistant VP Regulatory Affairs on 12-11-2021. Revision 4 approved by Associate VP Research Regulatory Support on 7-23-2022.