Martin Brown wrote:
There is quite often a systematic difference between the absolute
value of the evaluation function of different engines on a given
position, but the relative difference between alternate continuation
moves is what really matters as far as the decision making goes. Yes a
stronger engine will find new resources, but provided the weaker
engine is given sufficient time it can still make useful insights into
a game. That is my main worry with Crafty here - it doesn't always see
far enough into the future because its tree pruning is a lot more
conservative than Shredder or Rybka.

Usually moves where engines evaluations radically disagree are well
worth investigating to see why.

This is a worry, but you really need to play with Rybka some, to
understand what I am saying. You need to follow several games with
Rybka and your engine of choice. You will find numerous positions
where the programs will disagree violently (over 100cp) over the
favorite moves. (Realize that many times it is not that different).

It is precisely that difference where "strength" lies. Different
engines simply do not come up with the same moves given enough time.
Rybka seems to dramatically show that, and it is dramatically stronger.

That alone should provide enough of a question as to the results here.
The fact is that we don't know when the engines will be strong enough to
represent the "truth".

There may not be a "the truth" to be found...only successive
approximations to it given our computational limitations. Like
peeling an onion each time you make the engines an order of magnitude
more powerful or add enhanced heuristics you allow deeper searching of
the game tree that may alter the outcome. However, we have now crossed
the point where the best computer programs are demonstrably better at
match play than humans. Computer aided by a human in freestyle mode
and where the blunder rate from human error is essentially nil is
stronger still.

Thank you. That is my point. But you are trying to determine some sort
of "truth" by comparing to Crafty's hobbled play.

Working back from the tablebases where absolute knowledge and theorem
proof is possible may allow some further progress, but the storage
requirements and intense computational effort needed even for the
important 7 men tablebases is so great that it is only likely to be
done in a research lab. Having said that in 2 decades the size of
removable storage has gone from 360kb to 2GB (5000x) and consumer
grade hard disks from 10MB to 1TB (100000x). If this trend continues
then affordable PetaByte storage might well be available by 2030.

Don't let physics get into the way. We will probably be storing stuff
into the strings by then!

I will say that I do not use Crafty for day-to-day analysis so I don't
have an opinion other than that you need to remember in ELO that the
difference between 2500 and 2800 is vast, and the difference between
2800 and ~ 3100 is as vast. It is not 10% better, it is closer to think
of it as TWICE as good. Or more likely to win MOST of the time. It is
a HUGE difference.

You should also note that in engine vs engine games there is a
tendency for the strongest commercial engines to include a few tricks
in their prepared opening book that exploit known weaknesses in other
engines. This makes it a bit unfair to older engines that are not
heavily maintained and prepared for engine-engine matches.

Yes, and you can take the exact same opening book and have the same
issue between these engines. The interesting thing about Rybka, it
really is *that* strong.

Ultimately the point is, that they could have made their measurements in
the client rather than the engine. They could have modified, any sort
of client with source code available to do this. Then they could have
used multiple engine choices to ask their questions.