Data Management: In-Memory Could Be the Mother of All Wisdom

As I review my series of #100linesOnBIDW blogs over the last couple of weeks, I found myself looking at the Data Management posting. I covered when to apply schemas, Big Data, and data governance. What I left out was technical implementation concepts for data management systems like row vs. column orientation; in-memory vs. spinning disk primary storage; and symmetric multiprocessing (SMP) vs. massively parallel processing (MPP). Processing and storage were the “developments” of 2012. I left 2013 for the “how to use” Data Management platforms.

One of the technical concepts that intrigues me though is spinning disk or flash drives vs. in-memory as primary means of storage for data management platforms. Spinning disk is approaching a price point that should be considered “cheap”…err… financially inexpensive. In-memory is smoking fast…errr… achieves a superior level of performance. Using flash drives, “spinning disk” is getting much faster. With the falling price of memory and the increase in maximum server memory approaching 1TB per blade, in-memory is not only fast, but becoming more affordable and applicable to multiple use cases.

The ancient Greek playwright Aeschylus is credited as saying:

“Memory is the mother of all wisdom.”

It is entirely possible that in-memory databases will be that gateway to all “wisdom” as memory capacity continues to rise and memory prices continue to decline. However, with that being said, let us look at a couple of “talking points” relating to in-memory databases for technical architects and database administrators:

• Can you be in-memory and ACID compliant? One of the keys to database management systems is the ability to be ACID compliant. This is an area where strictly in-memory data stores fall down. If all information is in memory and you have a failure of electricity, processor or hardware; the information is gone and the data store fails the “durability” aspect of ACID. This is where having a level of storage on disk becomes important. While you can have a pure in-memory data store, most in-memory database management systems have made accommodation for durability by using some form of spinning disk. Speaking of disk…
• Can you be in-memory and have rotating disks? As I mentioned above, ACID compliant in-memory databases need disks for durability. However, there are other uses for disk in an in-memory environment. I prefer that architects have the facilities to manage which data “lives” in memory and which can be placed on disk. As memory capacity on individual server hardware continues to grow, architects will have less need to manage hot-warm-cold data within in-memory databases. However, until then you need some form of disk.
• Does it matter how you mix your in-memory with your spinning, and non-spinning (aka flash), disks? Solid-state drive (SSD) drives provide an interesting option for in-memory database administrators and architects. For memory “challenged” environments, SSD has the ability to serve as the “warm” bridge between in-memory and spinning disk. However, in enterprise environments SSD presents constraints (aka a nice way to say limitations) that spinning disks don’t have. Architects should be aware of those issues before putting all their eggs in the SSD basket.

What say the readers?

Are in-memory databases on your radar? If so, are you concerned about the durability of in-memory processing? Do you consider an in-memory data store to be “pure” if there are disks in the architecture?

Provide your comments below and/or ping me via twitter at @JohnLMyers44 with the hashtag #100LinesOnBIDW.

In addition, I will be participating in a webinar on March 26 on the topic of comparing and contrasting in-memory database platforms. For more information, you can register here.

Next week, I will cover Business Analytics.