RE: NDCG

[Nick Craswell <nickcr@microsoft.com>]

> I think I will leave the log base as 2

Minor point: The base of the log changes DCG by a constant multiplier, which cancels out when you calculate NDCG, so NDCG is completely unaffected by the log base.  I think.

DCG = sum_r ( gain / log2(r) )
    = sum_r ( gain / ( log(r) / log(2) ) )
    = sum_r ( log(2) * gain / log(r) )
    = log(2) * sum_r ( gain / log(r) )

Nick.

> -----Original Message-----
> From: ireval@nist.gov [mailto:ireval@nist.gov] On Behalf Of Ian
> Soboroff
> Sent: 21 November 2007 14:45
> To: Multiple recipients of list
> Subject: Re: NDCG
>
>
> Stephen Robertson <ser@microsoft.com> writes:
>
> > You might like to note that the original version has the rather
> > curious property that rank 2 gets no discount.  A version commonly
> > used in Microsoft is adjusted to give a discount even on rank 2.  How
> > flexible are you making your implementation?  Any gain
> > function/discount function/truncation point?
>
> A few people are using such a variant that discounts by log(rank + 1).
> I like this approach better myself, but at this point I want to make
> sure I've caught all the bugs and to do that I want to make sure I
> match
> the specification of the measure in the ToIS article.
>
> My implementation is based on trec_eval, and will allow specifying gain
> values per relevance level.  I think I will leave the log base as 2, as
> I have not seen much research that shows much sensitivity in that
> parameter.  Truncation will be as usual in trec_eval (using the -M
> option).
>
> I don't know if this will make it into the official trec_eval, but if
> not I'll probably post my patch someplace.
>
> Ian
>