"Martin Brown" wrote in message
ups.com...
On May 10, 4:23 pm, "David Kane" wrote:
"Martin Brown" wrote in message

oups.com...

Agreed. But equally when the experiment has a systematic error due to
using a relatively shallow fixed depth (but reproducible) searching to
score the moves played it doesn't take much intuition to conclude that
an engine that cannot annotate club level games accurately at that
level is completely out of its depth on superGMs.

I'd wager that this method would give generally meaningful results
for club players, *despite* the fact that it will inaccurately analyze
certain positions. It's a pity that the authors did not apply
the method to the games of players with different ELO. That's

I agree entirely. Crafty would have enough headroom over most club
players that the relatively small errors it made would not matter
compared to the fairly gross blunders that determine the outcome of
games at our level.

I do not believe this is true when it tries to rate world class
players who are intrinsically much stronger and have significantly
better stategic intuition and positional understanding than the
engine.

an easy and obvious extension that would have gone a long way
to validating the worth of the method.

The argument that the method is refuted by finding one position
that the computer analyzes incorrectly is false. There are analogous

I am not saying that. Although it is more fun to study interesting key
positions in top level GM games with deep engine analysis than to
focus on the mundane obvious wood pushing moves that no GM will ever
get wrong.

issues in ELO rating: which games should be rated, and what is the
significance of each game.

If you look back to the start of this thread I originally said I
thought the engine probably had managed something like an accurate
assessment. That was before I read the bit in the original paper that
said it was hobbled to 12 ply fixed search. I still think that is true
as far as blunder rate is concerned, but not so for accuarcy in non-
blunder play.

The proof of the pudding is in the eating. A claim that
an analytical method is meaningful must be supported with evidence,
and that is true whether you are talking about "average
error analyzed by 12 ply Crafty" or some sophisticated calculation
based on 20-ply Hydra analyses.

The paper lacks any supporting evidence and therefore
its conclusions are dubious. However, I consider the method
highly interesting and worthy of discussion.


I revised my view after experimenting with Crafty at 12ply annotating
a few of my already Shredder10 (30s/move)annotated games. The
experiment is not difficult to do. Perhaps you will see a different
result?

Regards,
Martin Brown

PS Aplogies if this is posted twice, but from here it looks like
Google dropped it on the floor again.