Risk-Based Monitoring is the future of clinical trials – but despite significant advances in recent years, some of the practicalities of implementation need to be further reinforced and supported.
That’s why CluePoints is launching a new, data-driven dashboard that will enable sponsors and CROs to benchmark projects both across protocols and portfolios but also against industry standards via the CluePoints community.
Francois Torche, CEO and Sylviane de Viron, Senior Research Data Analyst, explain what they have learnt from building the tool, and how it can help inform an industry gold standard for Risk-Based Quality Management (RBQM) and centralized monitoring (CM).
Risk-Based Monitoring: A Coming-of-Age Tale
RBQM adoption has been growing steadily since the ICH E6 (R2) guidelines reinforced that it could make studies more efficient and increase the chances of operational success back in 2016. Over the last year, however, implementation has accelerated greatly.
In the race to develop and test vaccines and treatments that counter the global threat of SARS-CoV-2 quickly and safely, sponsors have turned to RBQM and Centralized Monitoring in droves – at CluePoints alone, the number of studies conducted using our RBQM solutions increased by 50% in 2020.
The approach involves identifying, assessing, monitoring, and mitigating the risks that affect the quality or safety of the trial very early.
Working on the assumption that data from all sites should be largely equivalent, it uses comprehensive statistical analyses to detect sites (or patients, countries and regions) having atypical patterns of data. Those anomalies can be detected via a Data Quality Assessment which assesses all of the data collected in a trial or via Key Risk Indicators (KRIs) and Quality Tolerance Limits (QTLs) that assess specific risks of interest. Atypical data patterns are flagged as risk signals, allowing study teams to investigate and correct potential issues before they affect the integrity of a clinical trial.
Despite the increasingly wide-spread use of RBQM, which has been backed by the FDA, EMA and other regulatory bodies, there is still significant variability in how it is implemented. If we are to embrace the full potential of this powerful method of study issue detection, sponsors and CROs need a standard to work towards.
At CluePoints, we believe we already have the answers – all we need to do is extract them from the data and present them in an appropriate visual format.
The Story in the Metadata
For the last eight years, our industry leading RBQM platform has had a front row seat in the sector’s shift to Centralized Monitoring.
With our clients’ consent, we have been collecting the metadata – from KRIs, risk signals and actions – that provide a bird’s eye view of RBQM in action.
And with BEYOND, our business intelligence platform, we will soon be able to share these insights through a new metadata dashboard, allowing users to truly “focus on what matters”.
The metadata collates anonymized information from almost 57,000 risk signals which have been generated by more than 650 studies across nearly 120 different organizations.
Drilling down into the figures, our analysts have found that the average number of KRIs per study over the last eight years was 18.6. Unsurprisingly, this figure varied by study phase, with an average of 15.7 KRIs being monitored per phase one study, compared to 18.8 per phase three. There was also variation by therapeutic area. Cardiovascular studies, for example, had an average of 29 KRIs per study, whereas ophthalmology studies had an average of 12.
Interestingly, we found that the number of KRIs per study had reduced over the last 3 years (Figure 1). One potential explanation for that might be the growing experience of monitoring teams in selecting the right targeted indicators to optimize the process of risk review by study teams but further investigation would be needed to confirm this hypothesis.
Figure 1. Average Number of KRIs per study in the last 3 years
While almost 100% of clinical trials set at least one KRI analysis by site, fewer than 50% set a KRI analysis at the country level. This seems low knowing that 42.21% of the signals at the country level have been at least once in watch status (meaning that it reveals a potential risk for data quality and integrity and requires continued monitoring) during their life cycle.
The three most popular KRI categories used across trials, our analyses found, made up almost 70% of all the KRIs listed in a study’s KRI dashboard. The top category was safety, which includes non-serious and serious adverse event rate and in average 3.1 KRIs were created per study. Data quality was the second most frequent category in a clinical trial, with metrics such as visit to entry cycle time, auto query rate, and query response cycle time. For this category an average of 7.2 KRIs per study have been created. Next came compliance which includes metrics such as monitoring rates of missed assessments, protocol deviations, and off-schedule visits rate and in average 4.4 KRIs were created per study. It is also worth noting that one study in twelve also included KRIs related to efficacy measurements (Figure 2).
Figure 2. Overall Rate of Studies having at least 1 KRI analysis by KRI Category
While these results highlight how much KRIs are critical for assessing the major risk of a trial, it clearly shows that KRIs are only providing a partial view on the trial quality calling for a more comprehensive data quality assessment of the data. By contrast to KRIs that focus on pre-defined risks within a study, CluePoints Data Quality Assessment (DQA) is used to assess the quality of all data collected during a trial, including EDC data and all other types of data (eg. Central Laboratory data, ePRO or eCOAs data). Our metadata analysis of the signals collected since 2018, showed that 37.5% of the signals (meaning more than 21,000 signals) describing data anomalies have been triggered by a full DQA.
Informing the Future of RBQM
In the coming months, each organization using CluePoints will be able to access metadata dashboards, called ‘Usage Dashboards’ as part of our Business Intelligence offering, known as ‘BEYOND’. The usage dashboards will allow sponsors and CROs to monitor their own portfolio of studies and derive usage metrics across studies such as average number of KRIs, signal mitigation history or percentage of signals that are currently in Watch or Alert status for their own organization. They will be able to use this information to ensure study teams across their organization are compliant and, further, this will drive a standardized process across protocols and portfolios.
To give users a wider point of reference of the use of CluePoints platform across organizations, CluePoints is currently investigating how benchmark information across organizations could be integrated to the Usage dashboards of each specific organization. Clients will, for example, be able to compare their ‘signal to issue resolution time’ with the one across all organizations and all trials collected in the platform. They will be able to see if they have an above or below average number of critical signals or alerts in a study or a group of studies in comparison to all trials or for example within a certain therapeutic area.
Of course, it doesn’t begin and end at KRIs and signal and actions. The Usage Dashboard has opened a whole new world, and while the team has so far only looked at signals and actions, and KRIs, there is a wealth of knowledge still to be surfaced.
Our next step in this development is to analyze the risk assessment (RACT) metadata we hold to unlock insights into critical processes, critical data risks, and risk mitigation.
The resulting dashboard, which we hope to release later this year, will enable our clients to understand if there are studies or therapeutic areas that have intrinsically higher risk scores than others.
The goal is to offer a collection of Usage Dashboards that present a holistic, rather than a study-by-study, view of how RBQM is taking shape on the ground.
By facilitating sector-wide comparisons, we hope to provide Business Intelligence that guides individual company strategy while informing industry-wide best practice, and, ultimately, contribute to the creation of a gold standard framework for Risk-Based Monitoring and Centralized Monitoring.
