In article WHC3i.1755$xP.1292@trndny04,
"Chess One" wrote:

does the result make sense to strong chess players?

The problem here is what's known as confirmation bias.

The test agrees with what our intuition tells us, therefore it must be a
good and valid test. You use the test to check your intuition and your
intuition to check the test - it's perfectly circular reasoning.

But our intuition can be wrong.

You avoid confirmation bias by setting up rigorous standards for a test
before you run it, which was clealry not done in this case.

eg, if I was going to run this test, this is how I'd do it.

1. Select the players to be tested.
2. Since we're trying to measure player at their peak, pick five major
tournaments which they won (or slightly more, or slightly less, to even
out the sample size, aiming for the same number of games). A public
consensus of each player's best tournaments would be a reasonable
starting point. Tournaments - as opposed to matches - should give us
more variability in the types of positions reached, and thus help wash
out the bias if the computer struggled with some positions more than
others.
3. Analyze the game with the strongest available engine, with enough
analysis time that it would be expected to be competitive with top
Grandmasters today. Probably you would not include analysis of the first
5-10 moves of each game (although we'd have to find a logical, objective
methodology to mark the starting point of each game, which is not easy.
You want to be very careful about not scoring a player down because his
taste in openings is different from the computer's, and you want to
avoid giving later players credit for working in an era of higher
quality opening theory).

If you wanted to test this test, before you ran it, you could pick
specific tournaments and matches, and see how you did. Running all the
games from, say, the Zurich, 1953 candidates tournament and seeing if it
picked out Smyslov as the best player would be very interesting. If it
didn't, we'd be stuck with an interesting decision, to decide if the
program was inaccurate, or if the tournament did a poor job of selecting
the best player.