source: liacs/dbdm/dbdm_1/0433373_rickvanderzwet.txt@ 422

Papers read:

  - J. Gray, The Next Database Revolution. SIGMOD 2004, pp. 13-18, June 2004. 

  - Hans-Peter Kriegel, et al.  Future trends in data mining. Data Mining and

    Knowledge Discovery [1384-5810] 2007 vol:15 iss:1 pg:87 



Current database limits [Gray, 2004] and the current limits in data-mining

[Kriegel, 2007] technology does not solely focus on the technological barriers of

current datasets, with regards to memory, latency and storage. But also focus

on a way on how to process this data efficiently. [Gray, 2004] talks about the

use of interfaces to connect databases to the clients e.g. providing direct

interfaces to the clients using SOAP calls for example. But the use of

distributed databases how not gone mentioned. But first things first:



= Data-mining & Usability = 



Data-mining nowadays focus on subset solutions, with less attempt to generalize

the effort for mass-use. The tools and methods provided and used are merely the

building blocks for algorithms with focus on subset solutions with a well known

datasets or a lot of sanitized and known meta-data. 



With the ever increasing amount of data gathered and stored, generated human

understandable results (if any result at all) becomes harder and harder. The

underlying technique for generating results is often not to be explained by

logic human reasoning. Making the results hard to justify or even explain,

leaving potential good algorithms and strategies unused. 



(Near) future should show us whether we are capable of extracting results which

are of added value to understanding the process instead of showing heuristics,

allowing us to reason further about what is going on inside an process.





= Memory based databases with a file based backend =



Reducing and elimination latency to the database objects on specific media has

been always been a major focus within the design of algorithms of database query

automation. Recent technology inventions and improvements has lead to

developments allowing us to run any average small size database fully into the

memory system. Hence reducing access to every object within the database to a

equal level, making the latency decisions in algorithms obsolete, clearing the

path for a new type of algorithm design focusing of spanning the whole data-set

as fast possible. 



Together with a full-memory database, comes the process of designing the

database in such way that it can be mirrored on persistent media for obvious

reasons (power failure, transport, backup, revisions). Instead of taking the

traditional block level disk access approach new disks comes with ability to do

clever queuing and latency reducing actions of file based objects. Future will

show whether block based access (memory database) with a file based storage

will be one of the possibles and how to cope best with large databases sets.



= Distributed databases =



One area not covered by [Kriegel,2007] and [Gray,2004] it the development of

several Peta-bytes datasets (like the genome databases) that needed to be

accessed by many concurrent clients trough out the world, so link-layer latencies

comes in the picture. 



Finding ways of enabling this datasets for all clients at an acceptable/uniform

access time it something getting a major importance in the future as datasets

are rapidly growing due to the development of new sensors and image/video based

storage and more of those datasets have a heavily shared nature as more

research and business will be gathering and sharing from multiple (geographical)

locations, but are in need of centralized query interfaces.