On May 10, 1:22 am, Martin Brown
wrote:

Well if it has to be open source then Fruit 2.1 (~2780) might be
another alternative to try against Crafty (~2670). An extra 100 points
and a bit less materiallistic evaluation would be closer to human GM
level play. Fruit 2.2.1 just about stumbled onto that tricky line that
Phil Innes sets so much stall by engines not finding before I pulled
the plug.

Lots of commercial chess programs will have a database of "tricky"
positions with "model" answers, just to fool people into rating them
higher. Tricks of the trade.



So in other words you would be happy to see different results
if we ran the experiment again next year, with twice as much RAM, a
bigger cache, more disc, and the next version of the OS, even if it's
the same version of Rybka? Or do you just not see the value of an
experiment that *could* be re-run on *any* computer, from a 20-yo
PDP-11 [though it would take a long time ...] to whatever super-
duper system we might have 20 years hence [when it might run in
a few seconds rather than a CPU-year] and give identical results
as the basis for further experiments?

Exact reproducibility probably isn't so important here. Getting the
maximum accuracy of the move evaluation function for the limited
amount of time available is the key. Fixed depth does not do that.


I disagree. Normalization, see one of my posts in this 200 post
thread.


Very possibly it is. Perhaps it isn't. We won't know
unless someone runs the experiment. My own gut feeling is that
G&B's basic results are probably pretty stable, and would be
reproduced by any decent engine at any decent ply level, though
I would not be at all surprised if a few of the 70-80 centipawn
"blunders" turned out well at greater depth and a few non-blunders
turned out to be dubious. Swings, roundabouts. BICBW.


I don't think they are swings and roundabouts though. GM level games
are littered with precisely the sort of positions that chess engines
find really difficult to score accurately. And they usually occur at
pivotal moments.

A pivotal moment is immaterial if you use normalization. As I
explained in a post in this thread, the fact that a player enters a 60
move mating net set by his opponent, unseen by Crafty with a 14 ply
move horizon, is immaterial since at some point Crafty will see the
mating net (namely, 7 moves before checkmate) and rate the losing
player lower than the winning player.


OK, perhaps the implication is that they should have stopped
there and then. But if historical Elo ratings are of interest, then
I see no reason why another objective measure of *something* need not
be. They *do* have an objective measure. It *does* seem that their
results correlate well with *some* quality that we can recognise in
the play of Capablanca, Petrosian, Tal, etc. Their methodology is
at least interesting, even if flawed.

Agreed. The experiment is worth repeating with a much stronger
engine.

Yes, agreed. As I posted 47.5 posts ago, for very close, nearly tied
rankings, the stronger chess program might make a difference. But for
clear demarcation breakpoints, such as between Capa and Kramnik versus
Karpov and Kasparov, a stronger chess engine doesn't matter.


The problem comes if we all
start to take it too seriously. It's a semi-amateur investigation
that resulted in a conference paper and a somewhat light-hearted
summary at a commercial chess site. I have no problem with that.

Nor do I. I think it mostly has found the players with the lowest
blunder rate fairly convincingly.

Regards,
Martin Brown

And, chess being 99% tactics (say many GMs, including Tarrach or
Teichman), the player with the lowest blunder rate is often the best
champion. Blunders = Function(overall strength). In fact, a study
from a few years ago found that the difference in most moves between a
patzer and a GM was not so much in the unexpected move made, but
rather in the fact GMs blundered far less than a Class C player.

RL