////
/**
 *
 * Licensed to the Apache Software Foundation (ASF) under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership.  The ASF licenses this file
 * to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
////

[[hbase_mob]]
== Storing Medium-sized Objects (MOB)
:doctype: book
:numbered:
:toc: left
:icons: font
:experimental:
:toc: left
:source-language: java

Data comes in many sizes, and saving all of your data in HBase, including binary
data such as images and documents, is ideal. While HBase can technically handle
binary objects with cells that are larger than 100 KB in size, HBase's normal
read and write paths are optimized for values smaller than 100KB in size. When
HBase deals with large numbers of objects over this threshold, referred to here
as medium objects, or MOBs, performance is degraded due to write amplification
caused by splits and compactions. When using MOBs, ideally your objects will be between
100KB and 10MB (see the <<faq>>). HBase ***FIX_VERSION_NUMBER*** adds support
for better managing large numbers of MOBs while maintaining performance,
consistency, and low operational overhead. MOB support is provided by the work
done in link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339]. To
take advantage of MOB, you need to use <<hfilev3,HFile version 3>>. Optionally,
configure the MOB file reader's cache settings for each RegionServer (see
<<mob.cache.configure>>), then configure specific columns to hold MOB data.
Client code does not need to change to take advantage of HBase MOB support. The
feature is transparent to the client.

MOB compaction

MOB data is flushed into MOB files after MemStore flush. There will be lots of MOB files
after some time. To reduce MOB file count, there is a periodic task which compacts
small MOB files into a large one (MOB compaction).

=== Configuring Columns for MOB

You can configure columns to support MOB during table creation or alteration,
either in HBase Shell or via the Java API. The two relevant properties are the
boolean `IS_MOB` and the `MOB_THRESHOLD`, which is the number of bytes at which
an object is considered to be a MOB. Only `IS_MOB` is required. If you do not
specify the `MOB_THRESHOLD`, the default threshold value of 100 KB is used.

.Configure a Column for MOB Using HBase Shell
----
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400}
hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400}
----

.Configure a Column for MOB Using the Java API
====
[source,java]
----
...
HColumnDescriptor hcd = new HColumnDescriptor(“f”);
hcd.setMobEnabled(true);
...
hcd.setMobThreshold(102400L);
...
----
====

=== Configure MOB Compaction Policy

By default, MOB files for one specific day are compacted into one large MOB file.
To reduce MOB file count more, there are other MOB Compaction policies supported.

daily policy  - compact MOB Files for one day into one large MOB file (default policy)
weekly policy - compact MOB Files for one week into one large MOB file
montly policy - compact MOB Files for one  month into one large MOB File

.Configure MOB compaction policy Using HBase Shell
----
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
hbase> create 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}

hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'daily'}
hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'weekly'}
hbase> alter 't1', {NAME => 'f1', IS_MOB => true, MOB_THRESHOLD => 102400, MOB_COMPACT_PARTITION_POLICY => 'monthly'}
----

=== Configure MOB Compaction mergeable threshold

If the size of a mob file is less than this value, it's regarded as a small file and needs to
be merged in mob compaction. The default value is 1280MB.

====
[source,xml]
----
<property>
    <name>hbase.mob.compaction.mergeable.threshold</name>
    <value>10000000000</value>
</property>
----
====

=== Testing MOB

The utility `org.apache.hadoop.hbase.IntegrationTestIngestWithMOB` is provided to assist with testing
the MOB feature. The utility is run as follows:
[source,bash]
----
$ sudo -u hbase hbase org.apache.hadoop.hbase.IntegrationTestIngestWithMOB \
            -threshold 1024 \
            -minMobDataSize 512 \
            -maxMobDataSize 5120
----

* `*threshold*` is the threshold at which cells are considered to be MOBs.
   The default is 1 kB, expressed in bytes.
* `*minMobDataSize*` is the minimum value for the size of MOB data.
   The default is 512 B, expressed in bytes.
* `*maxMobDataSize*` is the maximum value for the size of MOB data.
   The default is 5 kB, expressed in bytes.


[[mob.cache.configure]]
=== Configuring the MOB Cache


Because there can be a large number of MOB files at any time, as compared to the number of HFiles,
MOB files are not always kept open. The MOB file reader cache is a LRU cache which keeps the most
recently used MOB files open. To configure the MOB file reader's cache on each RegionServer, add
the following properties to the RegionServer's `hbase-site.xml`, customize the configuration to
suit your environment, and restart or rolling restart the RegionServer.

.Example MOB Cache Configuration
====
[source,xml]
----
<property>
    <name>hbase.mob.file.cache.size</name>
    <value>1000</value>
    <description>
      Number of opened file handlers to cache.
      A larger value will benefit reads by providing more file handlers per mob
      file cache and would reduce frequent file opening and closing.
      However, if this is set too high, this could lead to a "too many opened file handers"
      The default value is 1000.
    </description>
</property>
<property>
    <name>hbase.mob.cache.evict.period</name>
    <value>3600</value>
    <description>
      The amount of time in seconds after which an unused file is evicted from the
      MOB cache. The default value is 3600 seconds.
    </description>
</property>
<property>
    <name>hbase.mob.cache.evict.remain.ratio</name>
    <value>0.5f</value>
    <description>
      A multiplier (between 0.0 and 1.0), which determines how many files remain cached
      after the threshold of files that remains cached after a cache eviction occurs
      which is triggered by reaching the `hbase.mob.file.cache.size` threshold.
      The default value is 0.5f, which means that half the files (the least-recently-used
      ones) are evicted.
    </description>
</property>
----
====

=== MOB Optimization Tasks

==== Manually Compacting MOB Files

To manually compact MOB files, rather than waiting for the
<<mob.cache.configure,configuration>> to trigger compaction, use the
`compact` or `major_compact` HBase shell commands. These commands
require the first argument to be the table name, and take a column
family as the second argument. and take a compaction type as the third argument.

----
hbase> compact 't1', 'c1’, ‘MOB’
hbase> major_compact 't1', 'c1’, ‘MOB’
----

These commands are also available via `Admin.compact` and
`Admin.majorCompact` methods.

=== MOB architecture

This section is derived from information found in
link:https://issues.apache.org/jira/browse/HBASE-11339[HBASE-11339]. For more information see
the attachment on that issue
"link:https://issues.apache.org/jira/secure/attachment/12724468/HBase%20MOB%20Design-v5.pdf[Base MOB Design-v5.pdf]".

==== Overview
The MOB feature reduces the overall IO load for configured column families by storing values that
are larger than the configured threshold outside of the normal regions to avoid splits, merges, and
most importantly normal compactions.

When a cell is first written to a region it is stored in the WAL and memstore regardless of value
size. When memstores from a column family configured to use MOB are eventually flushed two hfiles
are written simultaneously. Cells with a value smaller than the threshold size are written to a
normal region hfile. Cells with a value larger than the threshold are written into a special MOB
hfile and also have a MOB reference cell written into the normal region HFile.

MOB reference cells have the same key as the cell they are based on. The value of the reference cell
is made up of two pieces of metadata: the size of the actual value and the MOB hfile that contains
the original cell. In addition to any tags originally written to HBase, the reference cell prepends
two additional tags. The first is a marker tag that says the cell is a MOB reference. This can be
used later to scan specifically just for reference cells. The second stores the namespace and table
at the time the MOB hfile is written out. This tag is used to optimize how the MOB system finds
the underlying value in MOB hfiles after a series of HBase snapshot operations (ref HBASE-12332).
Note that tags are only available within HBase servers and by default are not sent over RPCs.

All MOB hfiles for a given table are managed within a logical region that does not directly serve
requests. When these MOB hfiles are created from a flush or MOB compaction they are placed in a
dedicated mob data area under the hbase root directory specific to the namespace, table, mob
logical region, and column family. In general that means a path structured like:

----
%HBase Root Dir%/mobdir/data/%namespace%/%table%/%logical region%/%column family%/
----

With default configs, an example table named 'some_table' in the
default namespace with a MOB enabled column family named 'foo' this HDFS directory would be

----
/hbase/mobdir/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
----

These MOB hfiles are maintained by special chores in the HBase Master rather than by any individual
Region Server. Specifically those chores take care of enforcing TTLs and compacting them. Note that
this compaction is primarily a matter of controlling the total number of files in HDFS because our
operational assumptions for MOB data is that it will seldom update or delete.

When a given MOB hfile is no longer needed as a result of our compaction process it is archived just
like any normal hfile. Because the table's mob region is independent of all the normal regions it
can coexist with them in the regular archive storage area:

----
/hbase/archive/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/
----

The same hfile cleaning chores that take care of eventually deleting unneeded archived files from
normal regions thus also will take care of these MOB hfiles.

=== MOB Troubleshooting

==== Retrieving MOB metadata through the HBase Shell

While working on troubleshooting failures in the MOB system you can retrieve some of the internal
information through the HBase shell by specifying special attributes on a scan.

----
hbase(main):112:0> scan 'some_table', {STARTROW => '00012-example-row-key', LIMIT => 1,
hbase(main):113:1*     CACHE_BLOCKS => false, ATTRIBUTES => { 'hbase.mob.scan.raw' => '1',
hbase(main):114:2*     'hbase.mob.scan.ref.only' => '1' } }
----

The MOB internal information is stored as four bytes for the size of the underlying cell value and
then a UTF8 string with the name of the MOB HFile that contains the underlying cell value. Note that
by default the entirety of this serialized structure will be passed through the HBase shell's binary
string converter. That means the bytes that make up the value size will most likely be written as
escaped non-printable byte values, e.g. '\x03', unless they happen to correspond to ASCII
characters.

Let's look at a specific example:

----
hbase(main):112:0> scan 'some_table', {STARTROW => '00012-example-row-key', LIMIT => 1,
hbase(main):113:1*     CACHE_BLOCKS => false, ATTRIBUTES => { 'hbase.mob.scan.raw' => '1',
hbase(main):114:2*     'hbase.mob.scan.ref.only' => '1' } }
ROW                        COLUMN+CELL
 00012-example-row-key     column=foo:bar, timestamp=1511179764, value=\x00\x02|\x94d41d8cd98f00b204
                           e9800998ecf8427e19700118ffd9c244fe69488bbc9f2c77d24a3e6a
1 row(s) in 0.0130 seconds
----

In this case the first four bytes are `\x00\x02|\x94` which corresponds to the bytes
`[0x00, 0x02, 0x7C, 0x94]`. (Note that the third byte was printed as the ASCII character '|'.)
Decoded as an integer this gives us an underlying value size of 162,964 bytes.

The remaining bytes give us an HFile name,
'd41d8cd98f00b204e9800998ecf8427e19700118ffd9c244fe69488bbc9f2c77d24a3e6a'. This HFile will most
likely be stored in the designated MOB storage area for this specific table. However, the file could
also be in the archive area if this table is from a restored snapshot. Furthermore, if the table is
from a cloned snapshot of a different table then the file could be in either the active or archive
area of that source table. As mentioned in the explanation of MOB reference cells above, the Region
Server will use a server side tag to optimize looking at the mob and archive area of the correct
original table when finding the MOB HFile. Since your scan is client side it can't retrieve that tag
and you'll either need to already know the lineage of your table or you'll need to search across all
tables.

Assuming you are authenticated as a user with HBase superuser rights, you can search for it:
----
$> hdfs dfs -find /hbase -name \
    d41d8cd98f00b204e9800998ecf8427e19700118ffd9c244fe69488bbc9f2c77d24a3e6a
/hbase/mobdir/data/default/some_table/372c1b27e3dc0b56c3a031926e5efbe9/foo/d41d8cd98f00b204e9800998ecf8427e19700118ffd9c244fe69488bbc9f2c77d24a3e6a
----

==== Moving a column family out of MOB

If you want to disable MOB on a column family you must ensure you instruct HBase to migrate the data
out of the MOB system prior to turning the feature off. If you fail to do this HBase will return the
internal MOB metadata to applications because it will not know that it needs to resolve the actual
values.

The following procedure will safely migrate the underlying data without requiring a cluster outage.
Clients will see a number of retries when configuration settings are applied and regions are
reloaded.

.Procedure: Stop MOB maintenance, change MOB threshold, rewrite data via compaction
. Ensure the MOB compaction chore in the Master is off by setting
`hbase.mob.file.compaction.chore.period` to `0`. Applying this configuration change will require a
rolling restart of HBase Masters. That will require at least one fail-over of the active master,
which may cause retries for clients doing HBase administrative operations.
. Ensure no MOB compactions are issued for the table via the HBase shell for the duration of this
migration.
. Use the HBase shell to change the MOB size threshold for the column family you are migrating to a
value that is larger than the largest cell present in the column family. E.g. given a table named
'some_table' and a column family named 'foo' we can pick one gigabyte as an arbitrary "bigger than
what we store" value:
+
----
hbase(main):011:0> alter 'some_table', {NAME => 'foo', MOB_THRESHOLD => '1000000000'}
Updating all regions with the new schema...
9/25 regions updated.
25/25 regions updated.
Done.
0 row(s) in 3.4940 seconds
----
+
Note that if you are still ingesting data you must ensure this threshold is larger than any cell
value you might write; MAX_INT would be a safe choice.

. Perform a major compaction on the table. Specifically you are performing a "normal" compaction and
not a MOB compaction.
+
----
hbase(main):012:0> major_compact 'some_table'
0 row(s) in 0.2600 seconds
----

. Monitor for the end of the major compaction. Since compaction is handled asynchronously you'll
need to use the shell to first see the compaction start and then see it end.
+
HBase should first say that a "MAJOR" compaction is happening.
+
----
hbase(main):015:0> @hbase.admin(@formatter).instance_eval do
hbase(main):016:1*   p @admin.get_compaction_state('some_table').to_string
hbase(main):017:2* end
“MAJOR”
----
+
When the compaction has finished the result should print out "NONE".
+
----
hbase(main):015:0> @hbase.admin(@formatter).instance_eval do
hbase(main):016:1*   p @admin.get_compaction_state('some_table').to_string
hbase(main):017:2* end
“NONE”
----
. Run the _mobrefs_ utility to ensure there are no MOB cells. Specifically, the tool will launch a
Hadoop MapReduce job that will show a job counter of 0 input records when we've successfully
rewritten all of the data.
+
----
$> HADOOP_CLASSPATH=/etc/hbase/conf:$(hbase mapredcp) yarn jar \
    /some/path/to/hbase-shaded-mapreduce.jar mobrefs mobrefs-report-output some_table foo
...
19/12/10 11:38:47 INFO impl.YarnClientImpl: Submitted application application_1575695902338_0004
19/12/10 11:38:47 INFO mapreduce.Job: The url to track the job: https://rm-2.example.com:8090/proxy/application_1575695902338_0004/
19/12/10 11:38:47 INFO mapreduce.Job: Running job: job_1575695902338_0004
19/12/10 11:38:57 INFO mapreduce.Job: Job job_1575695902338_0004 running in uber mode : false
19/12/10 11:38:57 INFO mapreduce.Job:  map 0% reduce 0%
19/12/10 11:39:07 INFO mapreduce.Job:  map 7% reduce 0%
19/12/10 11:39:17 INFO mapreduce.Job:  map 13% reduce 0%
19/12/10 11:39:19 INFO mapreduce.Job:  map 33% reduce 0%
19/12/10 11:39:21 INFO mapreduce.Job:  map 40% reduce 0%
19/12/10 11:39:22 INFO mapreduce.Job:  map 47% reduce 0%
19/12/10 11:39:23 INFO mapreduce.Job:  map 60% reduce 0%
19/12/10 11:39:24 INFO mapreduce.Job:  map 73% reduce 0%
19/12/10 11:39:27 INFO mapreduce.Job:  map 100% reduce 0%
19/12/10 11:39:35 INFO mapreduce.Job:  map 100% reduce 100%
19/12/10 11:39:35 INFO mapreduce.Job: Job job_1575695902338_0004 completed successfully
19/12/10 11:39:35 INFO mapreduce.Job: Counters: 54
...
        Map-Reduce Framework
                Map input records=0
...
19/12/09 22:41:28 INFO mapreduce.MobRefReporter: Finished creating report for 'some_table', family='foo'
----
+
If the data has not successfully been migrated out, this report will show both a non-zero number
of input records and a count of mob cells.
+
----
$> HADOOP_CLASSPATH=/etc/hbase/conf:$(hbase mapredcp) yarn jar \
    /some/path/to/hbase-shaded-mapreduce.jar mobrefs mobrefs-report-output some_table foo
...
19/12/10 11:44:18 INFO impl.YarnClientImpl: Submitted application application_1575695902338_0005
19/12/10 11:44:18 INFO mapreduce.Job: The url to track the job: https://busbey-2.gce.cloudera.com:8090/proxy/application_1575695902338_0005/
19/12/10 11:44:18 INFO mapreduce.Job: Running job: job_1575695902338_0005
19/12/10 11:44:26 INFO mapreduce.Job: Job job_1575695902338_0005 running in uber mode : false
19/12/10 11:44:26 INFO mapreduce.Job:  map 0% reduce 0%
19/12/10 11:44:36 INFO mapreduce.Job:  map 7% reduce 0%
19/12/10 11:44:45 INFO mapreduce.Job:  map 13% reduce 0%
19/12/10 11:44:47 INFO mapreduce.Job:  map 27% reduce 0%
19/12/10 11:44:48 INFO mapreduce.Job:  map 33% reduce 0%
19/12/10 11:44:50 INFO mapreduce.Job:  map 40% reduce 0%
19/12/10 11:44:51 INFO mapreduce.Job:  map 53% reduce 0%
19/12/10 11:44:52 INFO mapreduce.Job:  map 73% reduce 0%
19/12/10 11:44:54 INFO mapreduce.Job:  map 100% reduce 0%
19/12/10 11:44:59 INFO mapreduce.Job:  map 100% reduce 100%
19/12/10 11:45:00 INFO mapreduce.Job: Job job_1575695902338_0005 completed successfully
19/12/10 11:45:00 INFO mapreduce.Job: Counters: 54
...
        Map-Reduce Framework
                Map input records=1
...
        MOB
                NUM_CELLS=1
...
19/12/10 11:45:00 INFO mapreduce.MobRefReporter: Finished creating report for 'some_table', family='foo'
----
+
If this happens you should verify that MOB compactions are disabled, verify that you have picked
a sufficiently large MOB threshold, and redo the major compaction step.
. When the _mobrefs_ report shows that no more data is stored in the MOB system then you can safely
alter the column family configuration so that the MOB feature is disabled.
+
----
hbase(main):017:0> alter 'some_table', {NAME => 'foo', IS_MOB => 'false'}
Updating all regions with the new schema...
8/25 regions updated.
25/25 regions updated.
Done.
0 row(s) in 2.9370 seconds
----
. After the column family no longer shows the MOB feature enabled, it is safe to start MOB
maintenance chores again. You can allow the default to be used for
`hbase.mob.file.compaction.chore.period` by removing it from your configuration files or restore
it to whatever custom value you had prior to starting this process.
. Once the MOB feature is disabled for the column family there will be no internal HBase process
looking for data in the MOB storage area specific to this column family. There will still be data
present there from prior to the compaction process that rewrote the values into HBase's data area.
You can check for this residual data directly in HDFS as an HBase superuser.
+
----
$ hdfs dfs -count /hbase/mobdir/data/default/some_table
           4           54         9063269081 /hbase/mobdir/data/default/some_table
----
+
This data is spurious and may be reclaimed. You should sideline it, verify your application’s view
of the table, and then delete it.
