MSU HRPP Manual Section 3-3

3-3, Evaluation and Quality Improvement

Evaluation and quality improvement (QI) are complementary processes that assess and provide feedback regarding the functioning of the Michigan State University (MSU) Human Research Protection Program (HRPP). Evaluation processes collect information about the program and QI uses the information collected to implement changes in policies and procedures to improve the functioning of the program. The goal and mission of the QI program is to protect human research subjects through a quality HRPP that achieves and maintains compliance using efficient and effective processes. Periodic assessment and evaluation of the HRPP are necessary to measure and make systematic improvements to the program.

Evaluation and QI of the MSU HRPP is a shared responsibility and is implemented by multiple mechanisms to acquire and evaluate information regarding the compliance, quality, efficiency, and effectiveness of the HRPP. While each of these areas (i.e. compliance, quality, efficiency, and effectiveness) are described separately in this policy, they are highly dependent upon one another. For example, a review that is highly efficient but deviates seriously from IRB requirements would not be acceptable.

Stakeholder input assists the HRPP in conducting evaluation and QI activities and is encouraged (e.g. wide publication of HRPP contact information to obtain answers to questions, express concerns, and convey suggestions regarding the HRPP). See HRPP Manual 5-6 “Contact Information” for a description of how stakeholders can provide input.

The following sections provide an overview of evaluation and quality improvement activities.

Self- Evaluation by PI
PIs are responsible for the conduct and oversight of their research study, including oversight of personnel. See HRPP Manual 4-6, “Responsibilities of Investigators.” PIs are encouraged to utilize the “Site Visit” forms available on the HRPP website to perform self-evaluations of their research studies.

IRB Office Record Reviews
IRB staff are responsible for assuring that IRB applications are processed in accordance with MSU HRPP policies and procedures and that proper documentation is maintained in the IRB record. The HRPP manager provides an internal check on approved IRB applications and may provide specific corrective actions (e.g. with a particular IRB administrator on a specific research study). The HRPP manager and/or designee reviews all federally funded full review research studies and may review a sample of other research studies. The HRPP manager may identify systematic improvements (e.g. identification of a recurrent issue) and address through changes in process, targeted education, suggested policy changes, and other corrective actions as necessary.

IRB administrator IIIs are responsible for generating IRB minutes for their respective meetings. The HRPP manager(s) and IRB chair or designee will review all IRB minutes and may provide specific feedback prior to IRB members’ review. The HRPP manager(s) may also identify areas of concern (e.g. identification of a recurrent issue) and address through changes in process, targeted education, suggested policy changes, and other corrective actions as necessary.

Evaluations
HRPP staff are evaluated on an annual basis following MSU Human Resource guidelines.

IRB chairs will evaluate IRB members on a regular basis. Input may be provided to the VPRGS or his/her designee on the participation of IRB members. This input may include but is not limited to their attendance at scheduled meetings, the number and types of reviews conducted, the timeliness of reviews, participation in workshops, etc.

Feedback Surveys
Feedback surveys are provided to a sample of current investigators to collect information that will be used for QI in the IRB application and review process or in the research site visit process. The information collected will be reviewed by the HRPP Director and managers to determine whether immediate action or targeted improvements are needed and by the internal QI group to evaluate systematic improvements. The response rate will be periodically assessed to determine the effectiveness of the feedback survey.

Compliance Office Review of IRB Records
The Compliance staff may review a sample of IRB transactions. Such a review may include assessments of types of applications (e.g. initial, renewals, revisions, closures) and also research studies that involve certain subject populations or types of actions (e.g. waivers, research involving patient care services, or items typically associated with a CPT and/or HCPCS code, pregnant women, fetuses, or neonates, prisoners, children, individuals with diminished capacity, less than one year approval, engaged sites, previous noncompliance, engaged sites).

Research Compliance Reviews
Compliance office staff are responsible for conducting pre-reviews for the HRPP to assure compliance with applicable regulations including assessments for Clinical Research Billing Compliance, Food and Drug Administration (FDA) requirements for investigational new drugs and devices, humanitarian device exemptions, emergency use, registration with clinicaltrials.gov, the Health and Insurance Portability and Accountability Act (HIPAA), and other applicable regulations and guidance for all clinical research submitted to the Institutional Review Board (IRB). Compliance office staff are responsible for generating HRPP reports for their respective research compliance reviews.

Site Visits
The Compliance staff will evaluate research studies for compliance with applicable federal regulations (e.g. 45 CFR 46, 21 CFR 50, 21 CFR 56) and/or agency specific requirements, national Medicare regulations, state or local laws, and MSU policies and procedures as part of the site visit process. The review will encompass the IRB records. The Compliance staff will perform site visits for all full review initial applications with subjects enrolled and a sample of other research studies. Directed site visits may also be requested and can be limited to a particular area of concern or may encompass the entire research study. An expanded number of research studies may be reviewed as part of the site visit if the site visit findings indicate a need.  Site visit findings are provided to the Principal Investigator and others as appropriate through HRPP reports. See HRPP Manual 8-10 “Site Visits” for detailed procedures.

TEACH
Teaching Education and Compliance Help (TEACH) is a collaborative, educational initiative designed to assist the PIs and their research team with regulatory compliance. As part of TEACH, Compliance staff meet with researcher(s) conducting FDA regulated research after a full board IRB application is initially approved but before research begins and will provide researcher(s) with regulatory information specific to the type of research study (e.g. FDA regulated, clinical research billing compliance, HIPAA). If the PI has multiple FDA regulated projects, the need for additional visits will be dependent upon on whether there are additional regulatory components not previously discussed (e.g. device study instead of a drug study). TEACH may also be performed for a sample of other research studies.

Annual Report
The HRPP director provides information on an annual basis or more frequently as requested to the assistant vice president for the Office of Regulatory Affairs (AVP ORA). This information will include an assessment and evaluation of physical space, staff resources, funding, and committee activities. The AVP ORA reviews the information in order to develop plans for the coming fiscal year and follows the overall university budget planning cycle. When funds are allocated, the AVP ORA reviews and allocates funds to its units, including HRPP. Special budgetary requests may be made as needed by the HRPP director to the AVP ORA during the current fiscal year after resource allocation.

Meetings with the Institutional Official (IO)
The IO is provided with regular updates on the HRPP. The AVP ORA, the HRPP director, HRPP managers, IRB chairs, legal counsel, and the IO meet typically quarterly for programmatic discussion and review of the HRPP. This may include review of workload issues such as the timeliness of IRB reviews, the number, types and complexity of IRB submissions and any perceived deficiencies, such as the number of IRBs or IRB members, necessary reviewer expertise, resources or HRPP staff allocations.

Internal QI Group
An internal QI group meets regularly to discuss areas such as programmatic questions, HRPP policies or procedures, metrics, site visit trends, and to discuss potential improvements. Representatives for this group may include the AVP ORA or their designee, HRPP director, HRPP managers, and IRB chairs.

Metrics
The HRPP director coordinates the generation, review, and distribution of metrics. Examples are provided below to illustrate the types of metrics generated for quality improvement activities.

Composition of the IRB. The volume of submissions, stratified by college and department, will be reviewed to determine representation and areas of expertise needed by the committee. Composition will also be considered by the IRB chair and recommended as needed (e.g. particular areas of expertise).

Turnaround time. Metrics are generated to provide the volume and average approval turnaround time by submission type (e.g. initial, renewal, revision) stratified by committee and review level. The average turnaround times will be compared with the goals for turnaround. Turnaround time is defined as the difference between the date the application is approved and the date the application is submitted to the IRB and is complete (e.g. all necessary supporting documents). Metrics will be provided to the HRPP director, AVP ORA, IRB chairs, and the IO.

The goals for turnaround time are: (application is submitted and complete to approval)

  • 45 CFR 46.118: 2 days [range 1-3 days]

  • Initial

    • Exempt: 8 days [range 7-9 days]

    • Expedited: 21 days [range 2-4 weeks]

    • Full Review: 28 days [range 3-5 weeks]

  • Renewals

    • Expedited: 10 days [range 1-2 weeks]

    • Full Review: 21 days [range 2-4 weeks]

  • Revisions

    • Expedited: 10 days [range 1-2 weeks]

    • Full Review: 21 days [range 2-4 weeks]

The HRPP manager may also review protocols that considerably deviate from the maximum turnaround time goals to determine the cause. Summary reports of these reviews will be provided to the HRPP director and as appropriate, the internal QI group to evaluate and determine whether system-wide improvements are needed. The HRPP manager will identify significant deviations and determine from the sample whether components should be targeted for QI, such as process changes, education, training, etc.

The goals for components of the IRB turnaround time are:

  • Staff:

    • Processing applications: 2 days [range 1-3 days]

    • Processing comments: 2 days [range 1-3 days]

  • Reviewers:

    • Reviews 7 days

    • Responding to comments: 2 days [range 1-3 days]

The turnaround goals of the HRPP will be evaluated minimally every year to determine if the goals should be modified.

Compliance
The goal of compliance is to assure that research studies are reviewed, approved, and conducted in accordance with federal, state, and local regulations, university policies, and ethical principles.

The measures of compliance may include but are not limited to:

  • Number of directed site visits

  • Type and frequency of serious and / or continuing noncompliance

Methods of compliance review are described in the section above and include reviews of internal IRB operations, including protocols and IRB meeting minutes, post approval monitoring visits, etc.

Efficiency
The goal of efficiency is to complete IRB reviews in a timely manner in compliance with federal, state, and local regulations, university policies, and ethical principles. Efficiency is typically calculated by the turnaround time for approval (i.e., from the date of a complete submission to the IRB office to the date of approval). Components of this time may be examined to determine where targeted process improvements may occur.

The measures of efficiency may include but are not limited to:

  • Number of submissions, type, etc.

  • Turnaround time (submission to notification of approval, including components of this total time)

Quality
The goal for quality is to maintain a high quality HRPP that protects human research subjects involved in research and continually seeks to improve its program.

The measures of quality may include but are not limited to:

  • Assessment of compliance mechanisms (e.g. reduction in noncompliance reduces the potential risk to human subjects)

  • Serious and/or continuing noncompliance

  • Investigator satisfaction with the overall program

  • Subject protection

  • Trained HRPP staff, IRB members, and researchers (e.g. assessment of the education and training program)

Effectiveness
The goal for effectiveness is to achieve an HRPP that protects human subjects in research while maintaining compliance with federal, state, and local regulations, university policies, and ethical principles and achieves its goals in a timely manner.

The measures of effectiveness may include but are not limited to:

  • Reduced instances of noncompliance

  • Reduced severity of noncompliance

  • Improved investigator satisfaction with the HRPP

  • Positive responses from subjects

  • Improved investigator, staff, member, and subject understanding of the HRPP

  • Greater human subject research protections (e.g. unanticipated problems, complaints)

Such indicators taken together will measure the effectiveness of the HRPP. The internal QI group will review the information gathered with various evaluation mechanisms and review in their totality to determine the effectiveness of the program.

This policy and procedure supersedes those previously drafted.

Approved By: Vice President of Research and Graduate Studies, 7-19-2011. Revision 1 approved by VP Research & Graduate Studies on 3-13-2013. Revision 2 approved by VP Research & Graduate Studies on 12-11-2015.

Related HRPP Manual Sections

Approved PDF Version

Please note that hyperlinks have been added to the web version of this document for ease of use. View approved PDF version of Section 3-3, Evaluation and Quality Improvement.