source: liacs/dbdm/dbdm_1/0433373_rickvanderzwet.txt@ 317

Papers read:
  - J. Gray, The Next Database Revolution. SIGMOD 2004, pp. 13-18, June 2004. 
  - Hans-Peter Kriegel, et al.  Future trends in data mining. Data Mining and
    Knowledge Discovery [1384-5810] 2007 vol:15 iss:1 pg:87 

Current database limits [Gray, 2004] and the current limits in data-mining
[Kriegel, 2007] technology does not solely focus on the technological barriers of
current datasets, with regards to memory, latency and storage. But also focus
on a way on how to process this data efficiently. [Gray, 2004] talks about the
use of interfaces to connect databases to the clients e.g. providing direct
interfaces to the clients using SOAP calls for example. But the use of
distributed databases how not gone mentioned. But first things first:

= Data-mining & Usability = 

Data-mining nowadays focus on subset solutions, with less attempt to generalize
the effort for mass-use. The tools and methods provided and used are merely the
building blocks for algorithms with focus on subset solutions with a well known
datasets or a lot of sanitized and known meta-data. 

With the ever increasing amount of data gathered and stored, generated human
understandable results (if any result at all) becomes harder and harder. The
underlying technique for generating results is often not to be explained by
logic human reasoning. Making the results hard to justify or even explain,
leaving potential good algorithms and strategies unused. 

(Near) future should show us whether we are capable of extracting results which
are of added value to understanding the process instead of showing heuristics,
allowing us to reason further about what is going on inside an process.


= Memory based databases with a file based backend =

Reducing and elimination latency to the database objects on specific media has
been always been a major focus within the design of algorithms of database query
automation. Recent technology inventions and improvements has lead to
developments allowing us to run any average small size database fully into the
memory system. Hence reducing access to every object within the database to a
equal level, making the latency decisions in algorithms obsolete, clearing the
path for a new type of algorithm design focusing of spanning the whole data-set
as fast possible. 

Together with a full-memory database, comes the process of designing the
database in such way that it can be mirrored on persistent media for obvious
reasons (power failure, transport, backup, revisions). Instead of taking the
traditional block level disk access approach new disks comes with ability to do
clever queuing and latency reducing actions of file based objects. Future will
show whether block based access (memory database) with a file based storage
will be one of the possibles and how to cope best with large databases sets.

= Distributed databases =

One area not covered by [Kriegel,2007] and [Gray,2004] it the development of
several Peta-bytes datasets (like the genome databases) that needed to be
accessed by many concurrent clients trough out the world, so link-layer latencies
comes in the picture. 

Finding ways of enabling this datasets for all clients at an acceptable/uniform
access time it something getting a major importance in the future as datasets
are rapidly growing due to the development of new sensors and image/video based
storage and more of those datasets have a heavily shared nature as more
research and business will be gathering and sharing from multiple (geographical)
locations, but are in need of centralized query interfaces.