- Solution Evaluation
- To assess the performance of and value delivered by a solution in use by enterprise
- To rcm removal of barriers or constraints that prevent the full realization of the value
Stages of development:
- Prototypes or Proofs of Concept
- Pilot or Beta releases
- Operational releases
- Analyze the actual value being delivered
- Identify limitations which may be preventing value from being realized
- Make recommendations to increase the value of the solution
May combine:
- performance assessments, tests, and experiments
- objective and subjective assessments of value
=> generally focus on a component of an enterprise rather than the entire enterprise
Tasks:
- Measure solution performance: determine the most appropriate way to assess the solution's performance
- Analyze performance measures: understand the value solution delivers to the enterprise and SH & determine if it's meeting current business needs
- Assess solution limitations: investigates issues within the scope of a solution
- Assess enterprise limitations: investigates issues outside the scope of a solution
- Rcm actions to increase solution value: identify and define actions the enterprise can take to increase the value delivered by a solution
Inputs
Implemented solution (external)
6.1. Current state description
6.2. Business Objectives
6.2. Potential value
Outputs
Solution Performance measures
Solution Performance analysis
Soluiton limitation
Enterprise limitation
Recommend actions
8.1. Measure Solution Performance
.1. Purpose: to define performance measures and use the data collected to evaluate the effectiveness of a solution in relation to the value it brings
.2. Description
Determine the value of a newly deployed/existing solution
The measure used depend on the solution, context and how the org defines value
Performance may be assess through:
- KPIs aligned with enterprise measures
- G&O for a project
- process performance targets
- tests for a software application
.3. Inputs
- Business Objectives
- Implemented solution (external)
.4. Elements
.4.1. Define Solution Performance Measures
BA determine if current measures exist, or if methods for capturing them are in place
BA ensure that performance measures are accurate, relevant and elicit any additional measures identified by SHs
Common sources of measures: business goals, objectives and business processes
May be influenced/imposed by 3rd parties: solution vendors, government body or other regulatory organizations
Types and nature of the measurements are considered when choosing the elicitation method
May be:
- Quantitative
- Qualitative
4.2. Validate Performance Measures
Ensure that the assessment of solution performance is useful
Specific performance measures should align with any higher-level measures that exist within the context affecting the solution
Sponsor or any SH with decision making authority can decide measures used to evaluate soluiton performance
4.3. Collect Performance Measures
BA consider:
- Volume or Sample Size
- Frequency and Timing
- Currency: recently > older data
8.2. Analyze Performance Measure
.1. Purpose: to provide insights into the performance of a solution in relation to the value it brings
.2. Description
The measures collected in 8.1 often require interpretation (understanding) and synthesis to derive meaning and to be actionable
To meaningfully analyze performance measures:
- BA require a thorough understanding of the potential value that SH hope to achieve with the solution
- Consider variables: G&O, KPIs, the risk tolerance of both SHs and the enterprise, other stated targets
.3. Inputs
Potential Value
Solution Performance Measure
.4. Elements
.4.1. Solution Performance vs Desired Value
A solution might be high performing, but contributes lower value than expected (or than contribution in the past) OR low performing but potentially valuable
=> BA examines the collected measures to assess their ability to help SH understand the solution's value
If the measures are not sufficient to help SHs determine solution value, BA:
- collect more measurements or
- treat the lack of measures as a solution risk
.4.2. Risk
.4.3. Trends
BA consider the time period when the data was collected to guard against anomalies (~abnormal) and skewed trends when analyzing performance data
Any pronounced and repeated trends are noted
.4.4. Accuracy
BA test and analyze data collected by the performance measures to ensure their accuracy
To be considered accurate and reliable, the results of performance measures should be reproducible and repeatable
.4.5. Performance Variances
Performance Variances = the difference btw expected and actual performance
Root cause analysis may be necessary to determine the underlying causes of significant variances within a solution
.8.3. Assess Solution Limitations
.3.1. Purpose: to determine the factors internal to the solution that restrict the full realization of value
.3.2. Description
identify the root causes for under-performing and ineffective solutions and solution components
may be performed at any point during the solution life cycle: development/full implementation/existing solution => regardless of the timing, the assessment activities are similar and involve the same considerations
.3. Inputs
- Implemented solution (external): a solution that exists
- Solution performance analysis
.4. Elements
.4.1. Identify internal solution component dependencies
- Solutions often have internal dependencies that limit the performance of the entire solution to the performance of the least effective component
- BA identify solution components which have dependencies on other solution components, then determine if there is anything abt those dependencies or other components that limit solution performance and value realization
.4.2. Investigate Solution Problems
- BA identify problems in a solution/solution component by examining instances where the outputs from the solution are below an acceptable level of quality or where the potential vlaue is not being realized.
- Problems may be indicated by an inability to meet a stated goal, objective or RQM, or a failure to realize a benefit that was projected furing the tasks Define Change Strategy or Rcm Actions to Increase Solution Value
.4.3. Impact Assessment
- BA review identified problems -> assess the effect on the operation of the org or the ability of the solution to deliver its potential value => requires:
- determining the severity of the problem
- the probability of the re-occurence of the problem
- the impact on the business operations
- the capacity of the business to absorb the impact
- BA determine which problems must be resolved, which can be mitigated (make sth less serious) through other activities or approaches, which can be accepted
Other activities or approaches may include additional quality control measures, new or adjusted business processes or additional support for exceptions to the desired outcome
BA assess risks specifically to the solution and potential limitations of the solution.
.8. Outcome
Solution limitation
8.4. Assess Enterprise Limitations
.1. Purpose: to determine how factors external to the solution are restricting value realization
.2. Description
Solution may operate across various org within an enterprise => have many interactions and interdependencies
Solutions may also depend on environmental factors that are external to the enterprise
Enterprise limitations: culture, operations, technical components, SH interests or reporting structures
May be performed at any point during the solution life cycle
.3. Inputs
- Current state description
- Implemented (or constructed) solution (external)
- Solution performance analysis
.4. Elements
.4.1. Enterprise Culture Assessment
- Evaluates the extent to which the culture can accept a solution
- If cultural adjustments are needed to support the solution, the assessment is used to judge the enterprise's ability and willingness to adapt to these cultural changes
- BA perform cultural assessment + evaluate internal and external SH
.4.2. SH Impact Analysis
- Provide insight into how the solution affects a particular SH group
BA consider:
- Functions: the process in which the SH uses the solution: inputs SH provides into the process, how the SH uses the solution to execute the process, what outputs the SH receives from the process.
- Locations: the geographic locations of the SH interacting with the solution
- Concerns: the issues, risks, and overall concerns the SH have with the solution
4.3. Org structure changes
- The use of a solution and the ability to adopt a change can be enabled or blocked by formal and informal relationships among SHs
- The reporting structure may be too complex/simple to allow a solution to perform effectively
- Assessing if the org hierarchy supports the solution is a key activity
- BA consider informal relationships + formal structure
4.4. Opertional Assessment
- To determine if an enterprise is able to adapt to or effectively use a solution
- Identify which processes and tools within the enterprise are adequately equipped to benefit from the solution, and if sufficient and appropriate assest are in place to support it
BA consider: p180
.8. Outputs
Enterprise limitation
8.5. Recommend Actions to Increase Solution Value
.1. Purpose
To understand the factors that create differences btw potential value and actual value, and to rcm a course of action to align them
.2. Description
Focus on:
- Understanding the aggregate of the performed assessment
- Identifying alternatives and actions to improve solution performance and increase value realization
Identify how a solution should be replaced, retired or enhanced
Consider long-term effects and contributions of the solution to SHs
Include rcmdations to adjust the organization to allow for maximum solution performance and value realization
.3. Inputs
- Enterprise Limitation
- Solution Limitation
.4. Elements
.4.1. Adjust Solution Performance Measures
When the performance of solution is acceptable but not support to fulfill business O&G => require more appropriate measures
.4.2. Recommendations
Do nothing:
- When the value of a change is low relative to the effort required to make the change
- When the risks of change significantly outweigh the risks of remaining in current state
Org change:
is a process of managing attitudes about, perceptions fo and participation in the change related to the solution
refer to a process and set of tools for managing change at an org level
BA may help to develop rcmdations for changes to the org strucutre or personnel
Possible rcmdations:
- automating/simplifying the work people perform
- improving access to info
Reduce Complexity of Interfaces => imporve understanding
Eliminate Redundancy: different SH groups may have common needs that can be met with a single soluiton => reducing cost of implementation
Avoid Waste: to completely remove activities that not add value and minimize activities that not contribute to the final product directly
Identify Additional Capabilities:
- Solution options may offer capabilities to the org above and beyond those identified in the RQMs
- These capabilities are not of immediate value to the org but have the potential to provide future value
Retire the Solution
consider the replacement of a solution/solution component
Additional factors that may impact the decision of replacement/retirement of a solution:
- Ongoing cost vs initial investment
- Opportunity cost
- Necessity
- Sunk cost
.8. Outputs: Recommended Actions