Jan Urbański <wulczer@wulczer.org> writes:
>> I think the only relevance of stopwords to the current problem is that
>> *if* stopwords have been removed, we would see a Zipfian distribution
>> with the first few entries removed, and I'm not sure if it's still
>> really Zipfian afterwards.
> That's why I was proposing to take s = 0.07 / (MCE-count + 10). But that probably doesn't matter much.
Oh, now I get the point of that. Yeah, it is probably a good idea.
If the input doesn't have stopwords removed, the worst that will happen
is we'll collect stats for an extra 10 or so lexemes, which will then
get thrown away when they don't fit into the MCE list. +1.
regards, tom lane