Please enable JavaScript.
Coggle requires JavaScript to display documents.
Core elements of innovation fund design (Evaluation (What data needs to be…
Core elements of innovation fund design
Evaluation
How will the evaluation be disseminated?
Will it be published? Where?
Who needs to see if?
Keep track of the evaluation and monitoring centrally, to cross-reference data for impact assessment and policy development
In-house or evaluation partner?
Evaluation partner
From what stage?
Evaluation partner appointed at the same time as the grant administrator?
How much will this cost?
~10% (Trusted Relationships fund)
15-20% (EEF)
Same evaluation partner for all the projects under one fund for consistent and comparable results
Will the evaluation partner advise as the project is underway? (monitoring)
Will the implementation partner/department feel "beaten" with the evaluation?
How will the evaluation partner be appointed?
EU Commecial OJU process (Trusted Relationships fund)
How will the competition be advertised?
Crown Commercial Service Evaluation Supplier Framework pre-approved list only has big consultancies
Is the evaluation partner always given a contract rather than a grant?
In house
How do you avoid bias?
Extra staff/training/skills needed?
How much will this cost?
Who will challenge the evaluation plans?
Who is the evaluation for?
Practitioners? Which Practitioners? What should they do with it?
HMT? What should they do with it?
What type of evaluation?
What evidence standards will this adhere to?
Nesta evidence standards?
Dartington Social research ‘confidence framework’?
Quantitative?
Rarely used by Innovation funds
Problems: appropriate counterfactual, sample size, time frame
Qualitative?
Process evaluation (formative evaluation)
How and why was the intervention successful? Evaluation of implementation and delivery
Impact evaluation (summative evaluation)
Linked to outcomes for the target group
What are you comparing the intervention to?
What data needs to be collected?
Does data collection impact delivery or effectiveness?
Is it a direct or proxy measure?
Has an expert said this is an appropriate measure/collection method?
Use the same data/measures for projects in the same fund for easy comparison?
Ongoing evalutation/monitoring
Make monitoring requirements clear to implementors/contractors from the start
Give feedback to the provider
How much will this cost?
Dissemination of learning
Within the team?
Within the department?
What Works-style scale? (Public, published, easy to access)
across the sector
Administration
Contract or grant?
Competitive? (grant or contract)
SBRI
Process improvement
Change market offerings or consumer behaviour
Grow market
IP remains with contractor
Will you evaluate what happens to the product after the funding period?
Assessment of competition
What are the contractors asked? :
Specificity
Too specific: may not meet public sector equality duty considerations
Contractors will interpret the the call as broadly as possible
Capacity
Do they demonstrate they can deliver what's promised?
Can they record/measure outcomes and proxy measures the department wants to see?
How do they demonstrate VfM?
What about for interventions that don't do what they were meant to in the evaluation?
Can you check a contractor is not being double-funded for the same thing by this project and another government fund?
Will you provide feedback to unsuccessful applicants?
(Legal) restriction of applicant pool, eg specifying a geographical area
Who are the assessors?
Volunteers?
Paid?
Who finds them?
External body?
Department
Conflicts of interest?
Is it rigorous?
Where and when is the competition advertised?
Is there enough time for smaller contractors to bid?
How many stages are there?
Two stages
Initial (feasibility) funding, then selection of best interventions for development funding? (most common)
Short expression of interest, then full application by invitation?
Pre-stages
Is there the option to combine bids?
Is there a "warming up" phase to encourage collaboration between contractors?
Is there a pre-application consultation/workshop?
Interview stage to see if partners can work together?
One stage
Who has oversight?
Payment of money
(Usually) in arrears
Milestone based: contractor proves value at each stage
Money held by another body? eg UKRI
Timescale? Short term, long term?
What's the state of the market?
Highly competitive?
Can you buy the solution you need? (Procurement)
Small?
Is part of the innovation to grow the market? (SBRI)
Does it matter if the intervention is commercially successful after the SBRI funding is over? Will you evaluate this?
Non-competitive? (grant)
Is competition appropriate?
Does it avoid being State Aid?
Under de minimis (< €200,000 over 3 rolling years)
Under GBER
Under research, development and innovation?
Under procurement
Seek advice: State Aid Manual
Grants
Higher administrative burden than contracts
Less prescriptive
Used for LAs by HO Trusted Relationships Fund
But, a grant = gift/donation: how the grant is spent is based on trust
Clawback provision included under Grant Standards
Following Grant Standards makes it similar to a contract
If designed too stringently, they might become like contracts and procurement rules might apply
Check for duplication on Government Grant Information System (GGIS), via the Grants Hub
Contracts
Faster than a grant
Terms can't be changed, e.g. to make evaluation better
Can hold back some minor decisions for post-tender negotiations to avoid discriminating against small contractors, e.g. whether payment will be upfront or in arrears
Legally enforceable
Usually milestone based
Subject to VAT
Include clawback provision
Check for duplication via CaSIE system (from Crown Commercial Services)
In-house or external?
External, eg Innovate UK
How much will this cost? (up to 10%, Olly Clarke notes)
How much will the home department have to do?
How is it contracted out? Competition?
Crown Commercial Service ‘commercial partners’ procurement framework has pre-approved list
Alternatives to CSS, to get a larger selection?
How is the money given to the external body?
In house
Extra staff/training/skills needed?
Reasonable time line
When does the money need to be spent by to avoid underspend?
Align with timelines in target organisation e.g. schools
Be aware of long set up phase
sequencing, time for each phase - are there any external constraints to policy development or delivery?
Is it novel?
Novel funds require HMT clearance; may require primary legislation
Who can apply?
Partnerships or individual organisations?
A group of businesses with a Research and Technology Organisation (RTO) leading the project?
Businesses?
Third sector organisations?
Government departments/organisations and public bodies?
Research organisations?
Academic organisations?
Public sector research establishments?
Research council institutes?
Where is the money from?
One department?
Pooled fund?
Does the department have the legal right to make a grant?
How much money is there?
One project?
Lots of projects?
Theory of change
Scope
What is in scope?
What is out of scope?
Literature review to identify promising practises and the evidence base?
What are the different aspects of the problem?
Which aspects does the intervention address?
What is the fund's aim?
To build an evidence base?
For whom? What should they do with it?
how specific is the challenge?
broad policy problem/opportunity to explore/.
one clear question/challenge to answer?
want to catalyse innovation more than build evidence base- agnostic about specific activity/evidence
to deliver a particular policy priority - e.g. delivery of service (in the way grant funding would, but maybe evidence base not clear on how to fund/want to preserve local variation
to catalyse innovative activity in the sector?
what is the aim of the evaluation?
Proof-of-concept?
Pilot?
Implementation
In house?
Extra training/staff needed?
Implementation partner?
Can find implementation partner by separate contract (Trusted Relationships fund)
When and how will the implementation partner share learning?
Will you check the intervention partner and evaluation partner can work together effectively?
How much will this cost?
Governance
CAO? SRO?
who decides winning bids?