7 [IR] Relevance Feedback

Relevance

Relevance Feedback

Goals

know the basic idea of relevance feedback and query expansion

know the formulas of relevance feedback in vector space model and probabilistic model

1

Relevance is the basis for evaluating IR

Both recall and precision depends on "relevance"

Relevance is difficult to define precisely

The full set of relevant documents is never known

2

A relevant document is one that a person judges as useful in the context of a specific information need

Two related concepts

Topical relevance

Utility

assume that relevance is

solely a property of the internal mechanism of the retrieval system

is related the content of the document, so it is objective

is the result of the match between a query and the document representation

virtually ignoring the role of user

limitations

assume relevance to a query = relevance to a need

Assume relevance to be objective

assume relevance is static, not change

But is useful for evaluating IR system and algorithms

No user around, so cheap to run experiments

Relevance of docs is static, so can run experiements many time

Relevance of documents only related to the content and the mechanisms inside IR system, so provide basis for comparing effectiveness of different retrieval systems

Topical relevance is an important factor in users' relevance judgment

concentrate on "aboutness"

not concentrate on "aboutness", but on "usefulness"

Basic settings

Search system heavily reply on queries for finding relevant docs

But a query only approximates user's information need

User initial query is often short and poor approximation

People can improve query when seeing relevant and non-relevant docs

Procedure of relevance feedback

Basic procedure

  1. a user issue a (short, simple) query
  1. The system returns an initial set of retrieval results
  1. The user marks some returned documents as relevant or not relevant
  1. The system computes a better representation of the information need based on the user feedback
  1. The system displays a revised set of retrieval results

Types of relevance feedback

interactive relevance feedback: feedback information obtained from the user

Explicit relevance feedback

Implicit relevance feedback

let user mark relevant and irrelevant documents

system attempt to infer user intentions based on observable behavior

Blind relevance feedback or pseudo relevance feedback (doesn't always help)

feedback in absence of any evidence, explicit or otherwise

system assumes that the top ranked documents as relevant docs

How to use relevance feedback

assume that there is an optimal query

relevance feedback helps to bring user's query closer to the optimal one

how

Term reweighting

boost weights of terms from relevant documents

query expansion

add terms from relevant documents to the query

Rocchio