Moment Flagging (How can we get more information? (Feedback for…
How can we get more information?
prediction heuristics need feedback as well
get more customer feedback on this :check:
let user label it
Smart notes feedback :red_cross:
(decision is to separate from moment flagging)
was this smart note helpful?
relevance and accuracy
moments removed from smart notes
where do we store it?
real time stored seperately
whole moment rejection :red_cross:
Users more likely to provide bad moment feedback
false positives more valuable to DS team
Thumbs up a moment :check: :star:
when I click thumbs down I expect a prompt of some sort to happen that allows me to explain or detail why I'm flagging it. This should be optional.
later version consideration
A/B test :check:
list of options on why it was wrong/bad
possibly an internal tool
may be too much for moments
Feedback for transcripts
flag only as version 1?
Flag the conversation
Flag a line/section :check:
reps not able to edit feedback
text box for correction
just thumbs down
same user experience
feedback before roll out
get mocks to power users early :check: :star:
get Donald's input
Brian and Donald sit down together :star:
Closing the loop
How does a user know we have actually taken action on their feedback?
Positive reinforcement messages
Share information internally on what we are using the info for, which are being improved etc. :check: :star:
broken down by organization possibly users
Customer success team handle communication to customers :check:
When users give us feedback, we will concentrate our training and testing on those orgs
To go further we store data about improvement for a specific moment (from the DS report we already generate). E.g. "you told us positive feedback had errors, and we improved it X% for your org in the last N weeks. Thank you for your feedback!". It would probably be an email or notification
Users need feedback on their actions
Don't want to give the impression it will be instant
indicate to a user that a moment has been flagged so multiple users aren't flagging the same things :star:
removing bad moments from searches (not for V1)
remove from elastic search?
how does this influence other things downstream
explore is harder
If I click Thumbs Down it should stay highlighted so others known that it's an issue :check:
Removing bad moments from call summaries :star:
investigate more detail on removal :check:
how do we handle accidental clicks?
undo for a period after the click
can do in a call summary :check:
More detailed feedback at the beginning to encourage engagement
1st 5 clicks vs later clicks :check:
Simple thank you for providing feedback :check: :star:
When someone click on thumbs up "thank you for your feedback. It will be used to improve our models for your calls"
Flash a text label after thumbs down, saying: "thank you for your feedback. We have removed this incorrect moment from your call, and will use this as training data to improve our algorithms"
Users need more context and guidance
"Feedback mode" :red_cross:
Help pages/training on what flagging moments is/does and why they should do it
David and Ali will work together
includes info about how we will use the feedback
One time training/walkthrough (on initial login post-release) :check: :star:
Broadcast banner, notification of update :check: :star:
First week after release have it be "in your face"
It's not clear to a user that they can give feedback on moments. For example, a user needs to hover over the moment, and then assume that the thumbs-up means something. I'd prefer a label at the top that says "Provide Feedback" or "Moment Not Right?" and has omnipresent thumbs down (until we have both thumbs up and thumbs down implemented)
Labels on the thumbs up/down to understand what they do
Internal team members need to be able to give moment feedback
Is this treated any differently from customer feedback?
staged roll out
who can view/flag
proxy user flag
Store org of user
store name of user that is giving the feedback
DS team needs to be able to consume the data to make improvements
What information does the DS team need to make use of the feedback
Brian mocked up a data schema
entire moment object is included
how is the data stored
plan is to put it in big query
Brian to share info with David
staging and prod