KeyValue explicit timestamps vs memstoreTS

综合技术 2012-09-17 阅读原文

post I tried to summarize HBase's ACID implementation. This topic was also the main subject of my

I talked to a few folks afterwards, and it seems there is some general confusion about the memstoreTS (and MVCC) and how it relates to the explicit timestamp stored with each KeyValue.

The key difference is that the timestamp is part of the data model. Each KeyValue has a rowkey, a column identifier, a value... and a timestamp.

An application can retrieve and set the timestamp, decide as of which timestamp(s) it wants to see a certain value, etc.
It should be considered as just another part of a column (a KeyValue pair to be specific) that happens to have special meaning.

The memstoreTS on the other hand is a construct purely internal to HBase. Even though it is called a timestamp (TS), it has nothing to do with time, but is just a monotonically increasing "transaction number". It is used to control visibility (MVCC) of a "transaction" in HBase (i.e. a Put or Delete operation), so that changes are not partially visible.

Remember that the explicit timestamp is part of the application data that gets changed by a transaction.

Say you have two concurrent threads that update the same columns on the same row (a row is the unit at which HBase provides transactions):

Thread1, Put: c1 = a, c2 =b

Thread2, Put: c1 = x, c2 =y

That's cool. One of the threads will be first, HBase makes sure (via the memstoreTS) that Puts are not partially visible, and also makes sure that all explicit timestamp are set to the same value for the same Put operation.

So any client (looking at the latest timestamp) will either see (a,b) or (x,y).

Things get more confusing when a client sets the explicit timestamp of (say) a Put. Again consider this example where two concurrent threads both issue a Put to the same columns of the same row, but set different timestamps:

Thread1, Put: c1(t1) = a, c2(t2) =b

Thread2, Put: c1(t2) = x, c2(t1) =y


So what will a client see?

The first observation is still that each Put is seen in its entirety, or not at all.

If a Get just retrieves the latest versions and does so after both Thread1 and Thread2 finished it will see:

c1(t2) = x, c2(t2) = b

This is where many people get confused.

It looks like we're seeing mixed data from multiple Puts. Well, in fact we do, but we asked for only the latest versions, which include values from different transactions; so that is OK and correct.

To understand this better let's perform a Get of all versions. (i.e. call setMaxVersions() on your Get). Now we'll potentially see this:

  • nothing (if the Get happens to be processed before either the Puts).
  • c1(t1) = a, c2(t2) = b (Thread1 was processed first, and Thread2 is not yet finished)
  • c1(t2) = x, c2(t1) = y (Thread2 was processed first, and Thread1 is not yet finished)
  • c1(t1) = a, c1(t2) = x, c2(t1) = y, c2(t2) = b (both Thread1 and Thread2 finished)

We see that each Put indeed is either completely visible or not at all, and that HBase correctly avoid visibility of a partial Put operation.

In addition we can confirm that the timestamp is part of the data model that HBase exposes to clients, and the transaction visibility has nothing to do with the application controlled/visible timestamps.

Lastly in this scenario we also see that it is impossible for a client to piece together the state of the row at precise transaction boundaries.

Assuming both Thread1 and Thread2 have finished, a client can request the state as of t1 (in which case it gets c1(t1) = a, c2(t1) = y) or as of t2 (in which case it'll see c1(t2) = x, c2(t2) = b).

This behavior is the same as you'd expect from a relational database - the current state is the result of many past transactions; with the extra mental leap that the timestamp is actually part of the data model in HBase.

责编内容by:HBase 【阅读原文】。感谢您的支持!


Key value store in Perl Integrated cross-platform database / key-value store for C # ...
An Overview of SQL and NoSQL with it’s Pros and Co... Structured query language or SQL is used as a medium of communication with the r...
HubbleDotNet 开源全文搜索数据库项目–内存索引... 经过2周多的努力,HubbleDotNet 的内存索引功能终于搞好了。有了内存索引,搜索不再去读硬盘,实时性大大提高了。hubble的内存索引不同于lucene...
缓存服务器 一、缓存服务器介绍 NoSQL(NoSQL = Not Only SQL ),意即“不仅仅是SQL”,泛指非关系型的数据库,随着互联网web2.0网站的兴起...
Top 10 Databases You Should Learn in 2018 A database is must need for any software development and which database to choos...