Week 5

Demonstrating Rigor & Quality

Criteria for Evaluation

EPISTEMOLOGICAL CHOICES INFLUENCING CRITERIA

RELIABILITY, VALIDITY AND REALISM

TRUSTWORTHINESS AND SUBTLE REALISM

REFLEXIVITY, PRAGMATISM AND RELATIVISM

‘CONSIDERATIONS’ FOR EVALUATION

EVALUATION IN PRACTICE

coherence

systematic research conduct

convincing interpretation

account of researcher role.

Different paths

Ontology

way they view the nature of the social world

Epistemology

the way knowledge is constructed

influences which criteria

Position - Key questions

How are reality and truth represented

What - claim to be accessing when generating data

How stable and universal do they claim their interpretation

World view

view the data as ‘out there’, waiting to be garnered or elicited?

What roles - when interpreting data?

How - authors - competing explanations about the phenomena under investigation

How - own views, agendas and experiences feature in descriptions of their work

Interpretations as constructions, perhaps fashioned by yourself alone or jointly with research participants

participants’ beliefs and attitudes? or accounts and representations?

interpretation is the correct way of understanding the data // alternative explanations originating from the same data?

more in quantitative paradigm

phenomena under study exist independently of the researcher and the research endeavour

explore very specific questions

as self-explanatory, unproblematic and perhaps emerging from a review of the literature or prior research.

Criteria

Reliability

Validity

Generalisability

the degree to which findings can be deemed accurate and repeatable

the extent to which claims for the findings truly reflect the nature of the phenomena under study

the degree to which findings have explanatory power within other contexts and situations

asking another researcher independently to code a portion of the data and comparing coding frame

reference to an ‘expert’ group

enhanced by describing in detail the context and participants.

‘naturalist’ (Lincoln and Guba, 1985) or ‘critical realist’ (Cook and Campbell,1979).

underlying single reality that is to be explored,

various parties or players will have different views and explanations about what is happening.

criteria proposed by Lincoln and Guba (1985)

Trustworthyness

3 stratergies

Triangulation

different methods are used to elicit information about the same phenomenon

thick description

extensive details about both the context and the participants are included

audit trail

Researcher demonstrates how her work and thinking progressed throughout the project with the use of verifiable documents, such as a research diary, personal memos, dated computer files etc.

deny the existence of any external reality, claiming that nothing exists independently of the language and perspectives that bring phenomena into being (Edwards et al., 1995).

not to privilege their own interpretation over competing explanations,

instead presenting it as one of a number of possible representations.

reflexively ‘write themselves into’ their accounts.

Not like established criteria.

alternatives

reflexivity

conscious and systematic efforts to view the subject matter from different angles’;Alvesson, 2002

pragmatism

the extent to which knowledge generated is practically useful and helpful

matching

research aim

, view of research endeavour

researchers’ accounting for their role

epistemological approach adopted

depending on the methodology adopted

Eight “Big-Tent” Criteria for
Excellent Qualitative Research

Why Criteria and Why This Model?

Eight Criteria of Quality in Qualitative Research

Following, Playing, and Improvising

Unhelpful

Bochner (2000) argues that traditional empiricist criteria are unhelpful and even “silly

universal criteria are problematic, if not fruitless (Guba & Lincoln, 2005)

Schwandt (1996) argues - virtual cult

Criteria formed

(e.g., Bochner, 2000; Schwandt, 1996)

(Ellingson, 2008; Golafshani, 2003). - more flexible and contextually situated

Lather (1993) - Poststructuralist -

Reasons

Useful

Rules and guidelines help us learn, practice, and perfect

s helpful pedagogical launching pads across a variety of interpretive arts.

encourage dialogue with members of the scientific, experimental, and quantitative communities

frame our work, if desired, as systematic and structured

different areas

—narrative, phenomenological, grounded theory, ethnographic research, and case study research

Rich Rigor

Worthy Topic

Requisite variety : a concept borrowed from cybernetics, refers to the need for a tool or instrument to be at least as complex, flexible, and multifaceted as the phenomena being studied

face validity—which is concerned with whether a study appears, on its face, to be reasonable and appropriate (Golafshani, 2003)

evidence their due diligence, exercising appropriate time, effort, care, and thoroughness.

Means to achieve rigor

Enough Data

time spent

Appropriate context & samples

Practices

care and practice of data collection and analysis procedures.

transparency regarding the process of sorting, choosing, and organizing the data.

Sincerity

end goal can be achieved through self-reflexivity, vulnerability, honesty, transparency, and data auditing.

authenticity and genuineness

Self reflexivity

frank about their strengths and shortcomings

examine their impact on the scene and note others’ reactions to them.

self-reflexive commentary - subjective feeling and sense making

use of the first person voice

“how much self-reflexivity

show rather than tell self-reflexivity by weaving one’s reactions or reflexive considerations of self-as-instrument

Transparency

honesty about the research process.

Auditing

“a methodologically self-critical account of how the research was done”

“clear documentation of all research decisions and activities”

credit is given where due in terms of author order and acknowledgements to participants, funding sources, research assistants, and supportive colleagues.

Credibility

trustworthiness, verisimilitude, and plausibility of the research findings

achieved through practices including thick description, triangulation or crystallization, and multivocality and partiality.

Thick description

in-depth illustration that explicates culturally situated meanings (Geertz, 1973) and abundant concrete detail (Bochner, 2000)

show rather than tell.

Crystallization and triangulation.

Triangulation

if two or more sources of data, theoretical frameworks, types of data collected, or researchers converge on the same conclusion, then the conclusion is more credible

aimed to rid research of subjective bias. - Realism

Crystalisation

relates to the practice of using multiple data sources, researchers, and lenses—but is motivated by poststructural and performative assumptions—is crystallization

open up a more complex, in-depth, but still thoroughly partial, understanding of the issue.

Multivocality

Closely aligned with the notion of crystallization and showing rather than telling, is multivocality.

the verstehen practice of analyzing social action from the participants’ point of view.

the verstehen practice of analyzing social action from the participants’ point of view.

Member reflections

the verstehen practice of analyzing social action from the participants’ point of view.

allow for sharing and dialoguing with participants about the study’s findings, and providing opportunities for questions, critique, feedback, affirmation, and even collaboration

test of research findings as they are an opportunity for collaboration and reflexive elaboration.

resonance

refer research’s ability to meaningfully reverberate and affect an audience.

empathic validity

The potential of research to transform the emotional dispositions of people and promote greater mutual regard has been termed “empathic validity” by Dadds (2008)

2 practices

Aesthetic merit

text is presented in a beautiful, evocative, and artistic way

“interactive introspection”—a method in which researchers consciously self-examine how their felt emotions affect their private and social experiences.

Transferability and naturalistic generalizations.

potential to be valuable across a variety of contexts or situation

Transferability : readers feel as though the story of the research overlaps with their own situation and they intuitively transfer the research to their own action.

Naturalistic generalization

readers make choices based on their own intuitive understanding of the scene, rather than feeling as though the research report is instructing them what to do

Significant Contribution

Theoretically significant research

“intellectually implicative for the scholarly community” (Tracy, 1995, p. 210), extending, building, and critiquing disciplinary knowledge

examining how existing theory or concepts make sense in a new and different context.

Heuristic significance

Providing readers with suggestions for future research

Practically significant research

is the knowledge useful

Catalytic validity

political consciousness to cultural members to act

Tactical authenticity

Train participants to political action

Phronetic research

enable practical wisdom and space for transformation