Please enable JavaScript.
Coggle requires JavaScript to display documents.
RELIABILITY AND VALIDITY OF RISK ASSESSMENT TOOLS FOR VIOLENT EXTREMISM -…
RELIABILITY AND VALIDITY OF RISK ASSESSMENT TOOLS FOR VIOLENT EXTREMISM
KEY MESSAGES
3)
No reviewed studies met the established threshold for a robust methodological design
.
The validity studies reviewed were postdictive (i.e., they looked back in time) and showed a lot of variation. They
did not provide evidence of true predictive validity.
Apart from the TRAP-18, no risk tool had more than two validation studies comprising primary data.
The TRAP-18 interrater agreement values came from multiple studies that
did not provide standard errors or percentages of agreement
.
The studies relied on
publicly available data that might be biased or influenced by political agendas
.
The studies had
very small samples
.
The authors' potential conflicts of interest
might have caused some
reporting bias
.
2) This review
cannot conclude that one tool is better than another.
Three studies found that the
TRAP-18 was fit for purpose and demonstrated good fit in real-world data
.
However,
triangulation of public data is insufficient to score the TRAP-18
yet
most TRAP-18 validation studies rely on such data
.
Construct validity
was
not established for the ERG22+
Some tools (e.g. IVP guidance) might not appropriately capture the risk dynamics of some groups
. There might also not be any major differences in item fit for 'lone actors' and those who are members of a group.
The
internal consistency
of the
IVP guidance
and of
many ERG22+ sub-scales
was
lackluster
and only one study found
moderate interrater agreement for Der Screener - Islamismus among users
with varying levels of expertise.
The
VERA
was assessed as
fit for purpose
with items
applying to both 'lone actors' and members of offline extremist groups
.
Experts often gave
similar scores
to the same tools
in research settings
, however this was
not always the case in real-world situations
For example,
practitioners trained on the ERG22+ showed unsatisfactory agreement
, which raises
concerns about its reliability in practice
.
1) Some widely used and publicized violent extremism tools (e.g., the
ERG22+, IVP guidance, MLG-V2, TRAP18 and VERA-2R) rely on evidence that is either very slim, non-existent or not published by governments and organizations.
Concerns have long been raised about the empirical validation of these tools and
the lack of data about their potential to generate false positives and cause iatrogenic effects
.
Many risk assessment tools rely on evidence that is now being challenged
(e.g., the fact that sociodemographic characteristics might hold less explanatory power than psychological and personality traits)
BLIND SPOTS
The
overrepresentation of male participants
is flagged however no reflection is offered about the likely overrepresentation of male researchers in the PVE field too and the masculinist bias that might come with it.
The article seems to present
the development of internal scales
as primarily driven by the absence of tool validation. Since it can be assumed that some of the SPJ tools were at some point "internal scales", it would be worth
acknowledging the value of developing internal scales.
It would also be worth
examining how these scales are developed and used, and what that reveals about the shortcomings of standard tools.
The lack of qualitative studies in the initial results could signal the positivist nature of risk assessment tools and their incompatibility with a qualitative approach. It might be worth
exploring the limits of quantitative analysis in the context of risk assessment tools.
The review refers to the
lack of prospective studies about risk assessment tools
but does not indicate
how long such studies should ideally last.
What longitudinal design could help account for life events that could act as tipping points in an individual's trajectory?
No language was actively excluded for the review, however,
the publication bias
(i.e., the hurdles that researchers might face when attempting to publish in languages other than English or western languages) and the
default settings in search engines
might lead to
primarily identifying results in specific languages
.
The article states that
psychological and personality traits have seemingly more weight than sociodemographic characteristics.
As such,
examining self-reported scales (e.g., ARIS, RWA, RFS) would theoretically be informative.
The article mentions the
Der Screener - Islamismus
without further context. It would be relevant tot explain
why such a specific tool exists
,
where that tool is primarily used
,
why there are seemingly no other tools that are specific to a single form of extremist violence
and
whether that tool could contribute to exacerbating islamophobia
.
KNOWLEDGE MOBILIZATION OUTPUTS
Short animation video
or
infographics
presenting the
main findings
of the review
Infographics
presenting an
overview of findings for every tool
included in the review
Training module(s)
offered to the research assistants who supported the systematic review
Training module(s)
about
how to navigate disagreements and power dynamics
when working on a systematic review
CPN could consider
developing training modules explaining how to use self-report scales (e.g. ARIS, RWA, RFS)
for practice and research.
OPPORTUNITIES FOR IMPROVEMENT
The identified overrepresentation of male participants could be used to
explore the connection between masculinities and extremist violence
in future studies, training or knowledge mobilization material.
Flagging "convenience outcomes" in studies (e.g. whether an attack was stopped) could inspire
some reflections about other relevant outcomes that could be used to evaluate the validity of risk assessment tools
.
The lack of prospective studies could be used to
brainstorm ethical research designs that could help test the predictive validity of tools
.
When studies done with
participants in prison settings
are included, it might be worth providing more information (e.g. type of facility, sentences received, etc.) to
deliberately rehumanize these participants
. It might also be worth
interrogating the participation bias at play
: e.g. an individual eligible for parole and someone who is not, might not have the same interest in participating in such studies.
It would be
relevant to explain why Germany was singled out as a context of interest for this review
. Along the same lines, it
would be appropriate to include positionality statements
for all team members, particularly those who joined the team at the request of Public Safety Canada.
It would be instructive to
indicate the specific approaches used to solve 'disagreements' between research assistants
. This would leave clear peer pressure and 'group think' did not lead more junior staff to agree with lead researchers.
It could be useful to
further describe the training offered to research assistants
(i.e., objectives, approaches, modalities).