Please enable JavaScript.
Coggle requires JavaScript to display documents.
Critiquing research + presentation - design, sampling, measures,…
Critiquing research + presentation - design, sampling, measures, background and analyses
-
Examples of myths
Myth 41 - There’s recently been a massive epidemic of infantile autism
WWW - identified the use of persuasive language, statistics and opinions presented as fact (celebrity endorsement, emotive language)
- Identified scientific research to illustrate points, including assessment of the assumption of the myth
Autism epidemic gathers 85,000 hits referring to an astonishing increase in autism diagnosis in children
- Blame towards vaccines containing ‘thimerosal’ which were administered before autistic symptoms
- Categorised as a disorder in which there are language deficits, stimming, stereotyped and ritualised activities, and low emotional bonding (in DSM 4)
- Affects 1 in 150 people, 657% increase from 1993-2003
Myth was proliferated by high-profile media spokespeople, Meet the Press episode about it and endorsement by celebrities with autistic children - also villainisation of scientific evidence in media
It is far more likely that loosening diagnostic criteria is the cause - it used to be that individuals had to meet all 6 requirements to be diagnosed in the DSM-III, but the DSM-IV only required 8 of the 16 characteristics to be met
- DSM-IV also contained 5 diagnoses of autism, compared to the two in the DSM-III
- This led to more diagnosis
Changes to educational policy around declaring disabled students also meant more reporting by school districts
- Also coupled with increased public awareness (detection bias) - heightened reporting resulting on how easy it was for it to be detected by the observer
Study in UK found no increase in diagnosis with the same criteria over a 6 year period in the same area
- Equally, a study in the US showed diagnostic substitution - autism diagnosis increased, but mental retardation and learning disability diagnosis fell at an equal rate
- Similar impact on language disorders as autism becomes more popularly diagnosed
No link between vaccines and autism - rates still climbed even when thimerosal was removed from the vaccines (hard to prove a negative, but substantial evidence of negligible link) This myth about vaccines also led to an increase in measles and some deaths due to it
Diagnosis is increasing, but there is no increase in its prevalence, it is just diagnosed more than ignored Similar myth regarding savants and autism -
- 10% of autistic individuals are savants, compared to 1% of non-autistic individuals - savants tend to have higher IQs, but this is not due to their autism
- Belief came from films that portrayed autistic individuals with this ‘ability’
- Types of savant - calendar calculation, exceptional music talents, specific fact memory
- However, misconception could have led to a misguided treatment known as facilitated communication - autism was viewed as a motor, not a mental disorder (normal people in abnormal bodies) and so savants evidence this fact
- One study suggested FC worked to support mute autistic individuals in forming complex sentences on computers, but many other studies have disproved this, as the facilitator would unknowingly influence what was typed
Myth 18 - Students learn best when teachers match their learning styles
- A wealth of support for learning styles prompted the widespread nature of this myth, and it also translated well into pop-psychological literature and quizzes - it ignores ability and motivation, and puts a positive spin on failing students as students who are not taught effectively
Ended up with 71 LS models due to commercial success (VAK model, activists/theorists/reflectors and pragmatists) and a range of testing inventories and questionnaires
- However, less than ¼ of the evidence base is peer reviewed, and many were not well-controlled studies
- Other research has pulled apart initial theories and disproven them, especially as there is no consensus on learning styles and instead many conflicting models
- No reliable and valid way to answer the question - the questions cannot be applied to all learning situations
- No evidence to support effectiveness of matching instructor and learner styles - the instructor instead does well by engaging all students in an interactive way
- It is also hard to train educators to adapt to all learning styles - advice is inconsistent, and studies are few and far between
- It also would lead to students enhancing their strengths and ignoring weaknesses, which would continue to decline - all students can choose any learning style at any time
-
Knowing design, measurement and analysis
Qualitative - inductive and bottom up approach - exploratory, used when little is known about the subject
- Theories come up from the data, rather than being imposed down
- Discourse analysis, conversational analysis, thematic analysis and narrative analysis
- Links between variables - all participant produced; tells us they believe it works, but it may not actually be causative
- Alternative explanations - may be third factors or placebo effects (confounds)
- Strengths - no hypotheses beforehand; allows them to come about after, enable deep study and can generate rich data in small samples
- Limitations - not generalisable beyond specific data set, truth is relative to the participant
Experimental design / trial -
- Randomisation - should be done by number generator and not by an order factor such as alphabetical organisation, entry into study or time of day
--> Done by third party to avoid bias and before telling the participant which arm they are in (avoids research bias in choosing interventions for those who benefit most) - more effective the larger the sample
--> Ensures only the intervention is making a difference
- Choice of control intervention - sometimes treatment as usual (TAU) designs if new treatment, but it can be hard to decide what the control should be for specific questions
- Blinding researches and participants where possible - double blinded trial, reducing social desirability
--> Not doable with obvious interventions
Analysed using Intention to Treat (ITT) or Explanatory Analysis (EA) also known as Proof of Principal (POP) - depends on denominator
- ITT - all participants offered intervention are included in analysis even if they did not do it or dropped out
--> Often leads to missing data, and have to carry through baseline data to follow up or using mean data to replace missing data
--> Used in clinical trials
- EA and POP - just include those who took part
- Always used in laboratory experiments - used before an ITT analysis
Links - separated and measured by researcher, and separated by time -
- Enables strong conclusion about causality without reverse causality
- Only independent variable has changed - more control over dependent
Alternatives - follow-ups introduce third factor issues of tim, but this is solved by a control group due to process of randomisation (each arm is matched) as they all match apart from one key variable (sampled first then split)Strengths - causal conclusions, many strengths in terms of reverse causality erasure and third factor protectionLimitations - many occasions where this is not an ethical design, and also not feasible due to the time needed to measure the impact of some factors (high attrition etc)
- Imperfect studies have to be used sometimes to give the best possible, ethical evidence
- Limited generalisability also - both to context and sample
- Researcher bias - Narrative reviews are subjective, constructed stories rather than neutral summaries
--> Choice about papers to include / exclude = bias
--> Often written after results to make study look perfect
- Apples and pears -
--> Systematic reviews try to be objective, but sometimes they mix studies with different questions, designs, samples and result, and this leads to different studies that cannot be compared well
- Premature reviews -
--> Overly strict inclusion criteria
--> Reviews and meta-analyses can give false confidence or end too early
Other elements of design - reverse causality, third factor problem and ecological validity
Longitudinal study:
- Links - measured separately, and can be put together
--> Main benefit of time is that it solves reverse causality as things in the future cannot change the past
- Alternatives - third factor problem remains as confounding variables can change over time - immediate third factor of time
- Strengths - enables data collection over time, to know long term impact of effectiveness, no reverse causality
- Limitations - third factor problem
Longitudinal study with control group:
- Links - solves problem of reverse causality
- Alternatives - issue of time is resolved by control group as both have had time, and so mechanism cannot be this - however, other differences can still explain findings
- Strengths - no reverse causality
- Limitations - third factor problem
Case-control study:
- Find a selection of participants with a specific case, then matches a control group without the case
- However, matching the groups is difficult to do without matching them on the variable you are studying
- Links - researcher-driven, but measured at the same time
- Alternatives - many are ruled out through matching, but third factor problem remains (and unexpected mechanisms can emerge - not always bad e.g. smoking and lung cancer)
- Strengths - useful when a variable is rare and enables comparisons, matching controls for third factors
- Limitations - does not solve reverse causality and cannot contain all third variables as this is not possible
Cross sectional study with comparison group:
- Comparisons can be made, using large sample
- Links between variables - researcher-driven, researchers can separate variables and measure them differently to see how they are linked
--> However, this does not create causality, and there is always a risk of reverse causality as they are measured simultaneously - only exception is when measuring a past variable and a current variable
--> Alternative explanations - third factors can still influence results, and confounds also; but, can also show as potential mechanisms that link the variables, and can actually be the real predictor variable
--> Strengths - enables large sample data, separate variables, and useful when variables are common, researcher-driven link
--> Limitations - reverse causality and third factor problems
Cross sectional - asking questions online or using a questionnaire - wider population than qualitative, therefore more generalisable
- Links - still asking participants for their links between variables
- Alternatives - many alternative explanations, placebo effects and third factors
- Strengths - larger sample
- Limitations - purely descriptive
Measurement
- Objective v subjective measurement
- Unidimensional v multidimensional measures
Stages:
- Stage 1 - Conceptualisation
- Stage 2 - Validity, construct, reliability
- Stage 3 - Operationalisation, validity, measurement tool
Measuring effectiveness -
- Internal reliability
- Test-retest reliability
- Split-half reliability
- Inter-rater reliability
- Intra-rater reliability
- Face validity
- Sampling or content validity
- Concurrent validity - which measure correlates with existing definitions of measures
- Incremental validity - is it better than what already exists
Data analysis:
- Quantitative data analysis - looking for difference, looking for associations, normative data
--> Descriptive data, statistical tests, the p-value, effect sizes
--> Confidence intervals
- Qualitative analysis - Thematic, conversational, discourse, grounded theory