Please enable JavaScript.
Coggle requires JavaScript to display documents.
AI PM - Coggle Diagram
AI PM
AI GOVERANCE
VENDOR - AI CLAUSES
- Model Change Notification
- Acceptance against our Evaluation Set
- Sustained performance floor & remediation
- Training data restriction
- Ownership of derived artifacts - Labeled corpus, Eval Set, Fine Tuned Weights
- Model provider chain Disclosure
- Explainability & Audit Evidence
- Bias & Fairness Testing
- Exit & Data Extraction
- Volume & Price Protection
- No autonoumous action (only agreed ones)
GATES
- PROPOSE: Is AI actually the right answer? (unstructured, volume to recover run costs, Risk tolerance if Wrong output)
- FEASIBIITY: PoC, our Eval Set, Accuracy Threshold, Data Quality Assessment, DPIA, Initial Model Risk classification
- INITIATE: Acceptance Framework agreed, Eval Set specified, AI Contract Clauses agreed
- DESIGN: Model Risk Assessment, DPIA, Security assurance, Human-In-The-Loop design with Confidence Thresholds, Explainabilty & Audit Evidence approach
- BUILD & TEST: Eval Set running in Pipeline, Threshold met per output type, Bias testing evidenced, UAT threshold met
OPERATE: Drift monitoring, Model change process operating, Rollback tested, Post Implementation Review scheduled with measures agreed in adavance
ACCEPTANCE FRAMEWORK
- List of outputs/fields that matter
- What counts as correct value for each field: Eval Set (defined by people who do the work everyday/ not vendor).
Threshold: percentage number per output type / field (ensure goes in Vendor Contract)
- Eval Set: Named, versioned, owned by Us, Used as Acceptance Criteria for Regression Pack for Probablistic system
- Residual Risk Owner: Name, knows they are doing it, behind them sit a senior manager with personal accountability
CONTROL OWNERS / Sign-Offs
- Acceptance/QA - Test Lead (signs off the release meets the agreed Threshold)
- Operational & Customer Risk - Business/Process Owner for affected function (own residual risk)
- Model Risk: whoever owns the Model Validation - Test Lead??
- Data Protection: DPO (inc DPIA)
- Information Security: InfoSec
- Commercial: PM (with the Finance Business Partner)
MODEL CHANGE NOTIFICATION
- Notification
- Impact Assessment
- Rerun the Evaluation Set
- Look for Regression as carefully as improvement
- Accept or Reject (if below threshold)
- Record it (in Change log & Steering pack)
DECISION RIGHTS - Decided by - Recorded In
- Change < £50k (no threshold or outcome impact): PM | Change Log + Steering Pack
- Change > £50k or Outcome impact: Steering Group | Steering minutes & Change Log
- Contingency Release: Steering Group, against Registered Risk | Contingency Register
Whether Release meets Acceptance threshold: Acceptance Owner (/Test Lead) | Test Summary Rpt & Acceptance Record
- Whether to accept Residual Risk & Go Live: Residual Risk Owner, Risk consulted | Go/NoGo record
- Stage Gate approval: Steering Group, Portfolio Board if portfolio impact | Gate approval record
- Breach of Tolerance: Escalated, not decided at project level | Exception Report
- Stop or Pause the project: Portfolio Board on SRO recommendation | Portfolio Board minutes
GOVERNANCE
MEETINGS
- SRO catch up (Weekly 30m)
- Steering Group (Monthly)
- Delivery meeting (Weekly - Team, Vendor, Integrator, Dependencies & Blockers)
- Vendor Delivery review (weekly)
- Vendor SLA review (monthly)
- Vendor Contract review (3 monthly)
- RAID log review meeting (monthly)
- Risk review (monthly)
- Model Review
- Security Review
- Design Authority (At Stage Boundaries - Arch & Integration approval)
- Portfolio Board (Quarterly or on Exception) - At exception above tolerance or Cross-Portfolio dependencies
- Finance Business Partner Catch up
- Product / Sprint Review
- Go/No Go meetings
- Post Implementation Review
ARTIFACTS
- Plan
- Budget
- Benefit Tracking
- RAID log
- RACI matrix
- Vendor Contract
- Change Log (Decisions, CRs, Releases)
- Contingency Register
- Go/NoGo record
- Gate Approval record
RAID Log
- RISK - Date Raised, Category, Risk(cause,event,effect), Probability X Impact (= Score & Rating), Proximity, Response, Mitigating Action [for CONTINGENCY amounts], Owner, Target Date, Status, Reviewed Date, Contingency £ Held
- ASSUMPTION - Date Made, Assumption, Made By, Confidence, Impact if it proves false, Validation Action, Owner, Validated by, Status, Becomes a Risk (ref #)
- ISSUE
- DEPENDENCIES
- DECISIONS (needs to be logged somewhere)
REPORTING
- Steering Pack (send 3 days before)
- Weekly Project Status Report for SRO
- PMO Portfolio return
- Exception Report (within 5 days of a forecast breach)
STEERING PACK
- Decisions needed
- RAG status
- Milestones
- Current Spend vs Forecast
- Forecast on completion vs Planned
- Risk & Issues
- Dependencies
EXCEPTION REPORT
- Tolerance Breached
- Forecast or Actual
- Consequence if unaddressed
- Cause
- What I have already done
- Options
- Recommendation
- Decision Required
- Revised Forecast
- Who else needs to know
ROLES (& Accountable for)
- Snr Responsible Owner (SRO): Business Case & Benefit Realisation
- Digital Delivery Director: Delivery function, sets my tolerance, receives exceptions
- Portfolio Lead: Portfolio prioritisation, dependencies, PMO standards
- PM: Delivery within tolerance; single route for Supplier instuructions
- Acceptance Owner: Whether release meets agreed threshold
- Residual Risk Owner: The consequence of a wrong output reaching a customer or a decision
- Benefit Owner: The budget line the reduces. Must be the budget holder
BUDGET
-
DELEGATED TOLERANCE
- Cost Tolerance: +-5 %
- Schedule Tolerance: +-4 wks
- Change Authority: £50k
- Scope: within the approved outcome
- Contingency: 10% held by Steering Group
- Exception trigger: Forecast breach
- Exception route: within 5 working days
-
-