Martin Brown wrote:

help bot wrote:

But the program advertised at the Chessbase Web site
called "Shredder" claims to come with the endgame TBs
on DVD, highly compressed so they won't gobble up
your hard drive or require a massive (free) download.
That is what I was referring to.

Specifically they are so highly compressed that the entire ALL345 TB
in Shredders format is only 140MB ... It will fit entirely in system
ram with corresponding performance advantages over disk based full
size Nalimov tablebases which are roughly 7GB

Fifty times smaller than compressed Nalimov tablebases?

That's a big reduction in size. Any chance that this is some
sort of marketing hype and some information is missing?
Is the compression algorithm known? It has to be fast to be
suitable for decompressing on the fly from a compressed image
in memory.





--
G o o g l e F o o d : http://www.guymacon.com http://www.guymacon.com/
Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon
Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon
Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon

On Jun 22, 12:47 pm, Guy Macon http://www.guymacon.com/ wrote:
Martin Brown wrote:

help bot wrote:

But the program advertised at the Chessbase Web site
called "Shredder" claims to come with the endgame TBs
on DVD, highly compressed so they won't gobble up
your hard drive or require a massive (free) download.
That is what I was referring to.

Specifically they are so highly compressed that the entire ALL345 TB
in Shredders format is only 140MB ... It will fit entirely in system
ram with corresponding performance advantages over disk based full
size Nalimov tablebases which are roughly 7GB

Fifty times smaller than compressed Nalimov tablebases?

Yes. The sizes are 3-4-5 160MB and 3-4 1.4MB after rechecking.

I would really like them to do a 3-4-5-6 version (not memory resident)
but I reckon it would be worth splitting the difference with them on
the amount of disk needed to store full 6men Nalimovs.

That's a big reduction in size. Any chance that this is some
sort of marketing hype and some information is missing?

I haven't seen it fail yet. I would love to know how they have done
it. The best I can think of is a simple fast move generator and a list
of exceptions. The Nalimov TBs are as compressed as lossless
compression can manage given their original starting representation,
so I am guessing that Shredderbases use a novel representation of the
TBs. Their files entropy is slightly less compressed in the formal
sense than the Nalimovs.
(ie in princimple you could compress them with eg PKZIP and shave a
few more percent off the file size).

Is the compression algorithm known? It has to be fast to be
suitable for decompressing on the fly from a compressed image
in memory.

Indeed. AFAIK almost nothing has been written about it. Trade secret.
Seems to work OK though.

Does anyone know how it has been acheived?

Regards,
Martin Brown

On Fri, 22 Jun 2007 08:40:14 -0700, Martin Brown
wrote:

On Jun 22, 12:47 pm, Guy Macon http://www.guymacon.com/ wrote:
Martin Brown wrote:

help bot wrote:

But the program advertised at the Chessbase Web site
called "Shredder" claims to come with the endgame TBs
on DVD, highly compressed so they won't gobble up
your hard drive or require a massive (free) download.
That is what I was referring to.

Specifically they are so highly compressed that the entire ALL345 TB
in Shredders format is only 140MB ... It will fit entirely in system
ram with corresponding performance advantages over disk based full
size Nalimov tablebases which are roughly 7GB

Fifty times smaller than compressed Nalimov tablebases?

Yes. The sizes are 3-4-5 160MB and 3-4 1.4MB after rechecking.

I would really like them to do a 3-4-5-6 version (not memory resident)
but I reckon it would be worth splitting the difference with them on
the amount of disk needed to store full 6men Nalimovs.

That's a big reduction in size. Any chance that this is some
sort of marketing hype and some information is missing?

I haven't seen it fail yet. I would love to know how they have done
it. The best I can think of is a simple fast move generator and a list
of exceptions. The Nalimov TBs are as compressed as lossless
compression can manage given their original starting representation,
so I am guessing that Shredderbases use a novel representation of the
TBs. Their files entropy is slightly less compressed in the formal
sense than the Nalimovs.
(ie in princimple you could compress them with eg PKZIP and shave a
few more percent off the file size).

Is the compression algorithm known? It has to be fast to be
suitable for decompressing on the fly from a compressed image
in memory.

Indeed. AFAIK almost nothing has been written about it. Trade secret.
Seems to work OK though.

Does anyone know how it has been acheived?

Regards,
Martin Brown

Shredderbases don't store the distance to mate as Nalimov databases
do, they only tell you whether a position is won, lost or drawn.
Therefore, there is less information to be stored. They are more akin
to something like Daniel Shawul's bitbases
(http://wbec-ridderkerk.nl/html/details/Scorpio.html) than to Nalimov
or Chessmaster tablebases.

Tony

On Jun 22, 8:09 pm, Tony M wrote:

Is the compression algorithm known? It has to be fast to be
suitable for decompressing on the fly from a compressed image
in memory.

Indeed. AFAIK almost nothing has been written about it. Trade secret.
Seems to work OK though.

Does anyone know how it has been acheived?

Regards,
Martin Brown

Shredderbases don't store the distance to mate as Nalimov databases
do, they only tell you whether a position is won, lost or drawn.
Therefore, there is less information to be stored. They are more akin
to something like Daniel Shawul's bitbases
(http://wbec-ridderkerk.nl/html/details/Scorpio.html) than to Nalimov
or Chessmaster tablebases.


Hey, just knowing the distance to mate is no good
if a chess engine cannot solve the mate. They need
to also include (at least) one "best move", don't they?
That way the fetching engine can play the move, and
then be one move closer to the goal. Repeat as
necessary.

-- help bot

On Fri, 22 Jun 2007 17:36:06 -0700, help bot
wrote:

On Jun 22, 8:09 pm, Tony M wrote:

Is the compression algorithm known? It has to be fast to be
suitable for decompressing on the fly from a compressed image
in memory.

Indeed. AFAIK almost nothing has been written about it. Trade secret.
Seems to work OK though.

Does anyone know how it has been acheived?

Regards,
Martin Brown

Shredderbases don't store the distance to mate as Nalimov databases
do, they only tell you whether a position is won, lost or drawn.
Therefore, there is less information to be stored. They are more akin
to something like Daniel Shawul's bitbases
(http://wbec-ridderkerk.nl/html/details/Scorpio.html) than to Nalimov
or Chessmaster tablebases.


Hey, just knowing the distance to mate is no good
if a chess engine cannot solve the mate. They need
to also include (at least) one "best move", don't they?
That way the fetching engine can play the move, and
then be one move closer to the goal. Repeat as
necessary.

-- help bot



Knowing the distance to mate, or avoid mate, or whether a position is
a draw, is plenty good.

A chess program that uses a distance to mate tablebase generates all
possible legal moves from a given position, uses the tablebase to
figure out the distance to mate for the resulting positions from each
of those moves, and then chooses the move with the resultant position
that has the shortest distance to mate. A tablebase doesn't store
moves, only positions and a value for the position. It is up to a
program that uses tablebases to generate the moves and figure out what
to do with them. Clear as mud?

Tony

On Sat, 23 Jun 2007 01:59:30 GMT, Tony M wrote:

A tablebase doesn't store
moves, only positions and a value for the position. It is up to a
program that uses tablebases to generate the moves and figure out what
to do with them.

It seems to me that just having whether the position is a win, loss,
or draw would help, in the leaf evaluations. Then it could worry
about playing it when it gets there.
--
Replace you know what by j to email

On Fri, 22 Jun 2007 22:39:49 -0400, Jud McCranie
wrote:

On Sat, 23 Jun 2007 01:59:30 GMT, Tony M wrote:

A tablebase doesn't store
moves, only positions and a value for the position. It is up to a
program that uses tablebases to generate the moves and figure out what
to do with them.

It seems to me that just having whether the position is a win, loss,
or draw would help, in the leaf evaluations. Then it could worry
about playing it when it gets there.

That is how bitbases operate, and win/loss/draw information can
certainly prove useful in leaf evaluations. A tablebase that stores
distance to mate can provide more useful information in certain
situations.

Take, for example, a position in a KBB vs. KN endgame. Generally,
these endgames are a win for the stronger side, but a mate can take as
long as 60 or 70 moves (I'm aware of the FIDE 50 move rule, bear with
me here). That would be too deep for a search to ferret out. There
can be a couple of dozen moves that lead to mate, some to shorter
mates, some to longer mates.

If a program only knows about wins, losses and draws, how is it to
know which of the moves is best, since they all lead to winning
positions? If it chooses a move at random, it can get caught in a
cycle of moving from one winning position to another, without actually
getting closer to mate. This is the type of situation where distance
to mate comes in handy.

Tony

On Sat, 23 Jun 2007 03:14:42 GMT, Tony M wrote:

getting closer to mate. This is the type of situation where distance
to mate comes in handy.

I see your point. In some cases just an evaluation would be helpful.
In others (hard endgames) it isn't sufficient.
--
Replace you know what by j to email

Jud McCranie wrote:

It seems to me that just having whether the position is a win, loss,
or draw would help, in the leaf evaluations. Then it could worry
about playing it when it gets there.

There is a 'but' the the database and the engine has to be in synch.
If there are positions rated as win or draw in the database that the engine
doesn't agree about, and can't maintain, the database may end up acting like
a black hole, dragging the engine into a position it can't handle.
(A similar problem exists in Nalimov endgame tables, of course, particularly
in drawn situations.)

In general, it would be interesting to try to research alternative storage
methods. I remember, a long time ago, there was some discussion if it would be
practicable to use some kind of classification engine (such as an ANN), but
I think it went off on some subtangent.

--
Anders Thulin anders*thulin.name http://www.anders.thulin.name/

On Jun 22, 9:59 pm, Tony M wrote:


Shredderbases don't store the distance to mate as Nalimov databases
do, they only tell you whether a position is won, lost or drawn.
Therefore, there is less information to be stored. They are more akin
to something like Daniel Shawul's bitbases
(http://wbec-ridderkerk.nl/html/details/Scorpio.html) than to Nalimov
or Chessmaster tablebases.

Hey, just knowing the distance to mate is no good
if a chess engine cannot solve the mate. They need
to also include (at least) one "best move", don't they?
That way the fetching engine can play the move, and
then be one move closer to the goal. Repeat as
necessary.

-- help bot

Knowing the distance to mate, or avoid mate, or whether a position is
a draw, is plenty good.

A chess program that uses a distance to mate tablebase generates all
possible legal moves from a given position, uses the tablebase to
figure out the distance to mate for the resulting positions from each
of those moves, and then chooses the move with the resultant position
that has the shortest distance to mate. A tablebase doesn't store
moves, only positions and a value for the position. It is up to a
program that uses tablebases to generate the moves and figure out what
to do with them. Clear as mud?


As mud, yes! Okay, this knowledge will speed up the
search and allow for accurate evaluations, but you still
have to be able to figure out the (or one of the) winning
move.

For instance, I "know" that KQ vs. KR is generally a
win, and I know that the position score is therefore "a
"win, but is Fritz going to just resign, or do I need to
play a move before notching up the point? Of course
I need to play a move -- duh!

No, I think this in-memory trick is just for speeding
up the search. Somewhere on disk the full tablebases
need to be stored so the program can fetch the move.
Otherwise there will be many cases where the correct
evaluation is plugged in, but the engine cannot win the
game in practice.

-- help bot