"Martin Brown" wrote in message
oups.com...

Agreed. But equally when the experiment has a systematic error due to
using a relatively shallow fixed depth (but reproducible) searching to
score the moves played it doesn't take much intuition to conclude that
an engine that cannot annotate club level games accurately at that
level is completely out of its depth on superGMs.

I'd wager that this method would give generally meaningful results
for club players, *despite* the fact that it will inaccurately analyze
certain positions. It's a pity that the authors did not apply
the method to the games of players with different ELO. That's
an easy and obvious extension that would have gone a long way
to validating the worth of the method.

The argument that the method is refuted by finding one position
that the computer analyzes incorrectly is false. There are analogous
issues in ELO rating: which games should be rated, and what is the
significance of each game.