Users need more context and guidance
Labels on the thumbs up/down to understand what they do
It's not clear to a user that they can give feedback on moments. For example, a user needs to hover over the moment, and then assume that the thumbs-up means something. I'd prefer a label at the top that says "Provide Feedback" or "Moment Not Right?" and has omnipresent thumbs down (until we have both thumbs up and thumbs down implemented)
Help pages/training on what flagging moments is/does and why they should do it
One time training/walkthrough (on initial login post-release) :check: :star:
First week after release have it be "in your face"
Broadcast banner, notification of update :check: :star:
includes info about how we will use the feedback
David and Ali will work together
"Feedback mode" :red_cross:
Users need feedback on their actions
Flash a text label after thumbs down, saying: "thank you for your feedback. We have removed this incorrect moment from your call, and will use this as training data to improve our algorithms"
When someone click on thumbs up "thank you for your feedback. It will be used to improve our models for your calls"
Simple thank you for providing feedback :check: :star:
More detailed feedback at the beginning to encourage engagement
1st 5 clicks vs later clicks :check:
indicate to a user that a moment has been flagged so multiple users aren't flagging the same things :star:
Removing bad moments from call summaries :star:
can do in a call summary :check:
how do we handle accidental clicks?
undo for a period after the click
investigate more detail on removal :check:
If I click Thumbs Down it should stay highlighted so others known that it's an issue :check:
removing bad moments from searches (not for V1)
explore is harder
how does this influence other things downstream
remove from elastic search?
Don't want to give the impression it will be instant
feedback before roll out
get Donald's input
Brian and Donald sit down together :star:
get mocks to power users early :check: :star:
Closing the loop
How does a user know we have actually taken action on their feedback?
To go further we store data about improvement for a specific moment (from the DS report we already generate). E.g. "you told us positive feedback had errors, and we improved it X% for your org in the last N weeks. Thank you for your feedback!". It would probably be an email or notification
When users give us feedback, we will concentrate our training and testing on those orgs
Customer success team handle communication to customers :check:
Share information internally on what we are using the info for, which are being improved etc. :check: :star:
broken down by organization possibly users
Positive reinforcement messages
How can we get more information?
Feedback for transcripts
same user experience
just thumbs down
text box for correction
reps not able to edit feedback
flag only as version 1?
Flag a line/section :check:
Flag the conversation
when I click thumbs down I expect a prompt of some sort to happen that allows me to explain or detail why I'm flagging it. This should be optional.
may be too much for moments
possibly an internal tool
list of options on why it was wrong/bad
A/B test :check:
later version consideration
Thumbs up a moment :check: :star:
Users more likely to provide bad moment feedback
false positives more valuable to DS team
whole moment rejection :red_cross:
Smart notes feedback :red_cross:
(decision is to separate from moment flagging)
moments removed from smart notes
where do we store it?
real time stored seperately
relevance and accuracy
was this smart note helpful?
prediction heuristics need feedback as well
let user label it
get more customer feedback on this :check:
DS team needs to be able to consume the data to make improvements
how is the data stored
plan is to put it in big query
staging and prod
Brian to share info with David
What information does the DS team need to make use of the feedback
Brian mocked up a data schema
entire moment object is included
Internal team members need to be able to give moment feedback
Is this treated any differently from customer feedback?
store name of user that is giving the feedback
Store org of user
staged roll out
proxy user flag
who can view/flag