---
---
= Clustered Cache
ifndef::sourcedir39[]
include::{includedir}/common.adoc[]
endif::sourcedir39[]

ifdef::notBuildingForSite[]
include::{includedir}/menu.adoc[]
endif::notBuildingForSite[]

== Introduction

Distributed caching allows you to harness additional benefits of horizontal scale-out, without losing on low latency offered by local on-heap tiers.

[ditaa]
....
+------------------------------+            +------------------------------+
|          Application         |            |          Application         |
|                          cBE7|            |                          cBE7|
|+----------------------------+|            |+----------------------------+|
||        Cache Manager   c7DE||            ||        Cache Manager   c7DE||
||+--------------------------+||            ||+--------------------------+||
|||           Cache      c7AE|||            |||           Cache      c7AE|||
|||+------------------------+|||            |||+------------------------+|||
||||    Heap Tier       cFA4||||            ||||        Heap Tier   cFA4||||
|||+------------------------+|||            |||+------------------------+|||
||||                        ||||            ||||                        ||||
||||      Off Heap Tier     ||||            ||||      Off Heap Tier     ||||
||||                    cFA4||||            ||||                    cFA4||||
|||+------------------------+|||            |||+------------------------+|||
||||     Clustered Tier cA8D||||<--+    +-->||||     Clustered Tier cA8D||||
|||+------------------------+|||   |    |   |||+------------------------+|||
||+--------------------------+||   |    |   ||+--------------------------+||
|+----------------------------+|   |    |   |+----------------------------+|
+------------------------------+   |    |   +------------------------------+
                                   |    |
                                   V    V
                      +------------------------------+
                      |       Terracotta Server      |
                      |                          c7AE|
                      |+----------------------------+|
                      ||       Cache Clustered      ||
                      ||        Tier Manager    cF55||
                      |+----------------------------+|
                      ||                            ||
                      ||          Off Heap          ||
                      ||        Data Storage        ||
                      ||                        cFA4||
                      |+----------------------------+|
                      +------------------------------+
....
 * Hot data is cached locally, hotter data in faster tiers
 * Data cached by one application instance is available to all cluster members.
 * Full data is available in the cluster.
 * One or more mirror servers may be deployed to provide HA

To enable clustering with Terracotta, you will have to deploy a Terracotta server configured with clustered cache storage.
For convenience Ehcache 3.1 introduced a downloadable kit that contains the Terracotta Server and also the required client libraries.

You will then need to configure a cache manager to have clustering capabilities such that the caches it manages can utilize the clustered storage.
Finally, any caches which should be distributed should be configured with a clustered storage tier.

[[concepts]]
== Clustering Concepts

In this section we discuss some Terracotta clustering terms and concepts that you need to understand before creating cache managers and caches with clustering support.

Server off-heap resource::

Server off-heap resources are storage resources defined at the server.
Caches can reserve a storage area for their cluster tiers within these server off-heap resources.

Cluster Tier Manager::

The Ehcache _Cluster Tier Manager_ is the server-side component that gives clustering capabilities to a cache manager.
Cache managers connect to it to get access to the server's storage resources so that the clustered tiers of caches defined in them can consume those resources.
An Ehcache cluster tier manager at the server side is identified by a unique identifier.
Using the unique identifier of any given cluster tier manager, multiple cache managers can connect to the same cluster tier manager in order to share cache data.
The cluster tier manager is also responsible for managing the storage of the cluster tier of caches, with the following different options.

Dedicated pool::

Dedicated pools are a fixed-amount of storage pools allocated to the cluster tiers of caches.
A dedicated amount of storage is allocated directly from server off-heap resources to these pools.
And this storage space is used exclusively by a given cluster tier.

Shared pool::

Shared pools are also fixed-amount storage pools, but can be shared by the cluster tiers of multiple caches.
As in the case of dedicated pools, shared pools are also carved out from server off-heap resources.
The storage available in these shared pools is strictly shared.
In other words, no cluster tier can ask for a fixed-amount of storage from a shared pool.

Sharing of storage via shared pools does not mean that the data is shared.
That is, if two caches are using a shared pool as their clustered tier, the data of each cache is still isolated but the underlying storage is shared.
Consequently, when resource capacity is reached and triggers eviction, the evicted mapping can come from any of the cluster tiers sharing the pool.

Here is a pictorial representation of the concepts explained above:

[ditaa]
....
+-----------------------------------------+
|         Primary Server Resource         |
|                (196 GB)                 |
|                                         |
|  /---------------\   /---------------\  |
|  | Shared Pool A |   | Shared Pool B |  |
|  |    (32 GB)cBEC|   |    (24 GB)cBEC|  |
|  \---------------/   \---------------/  |
|                                         |
|  /---------------\   /---------------\  |
|  | Fixed Pool C  |   |  Fixed Pool D |  |
|  |    (32 GB)cA8D|   |    (16 GB)cA8D|  |
|  \---------------/   \---------------/  |
|                                     cFAA|
+-----------------------------------------+
|        Secondary Server Resource        |
|                 (96 GB)                 |
|                                         |
|  /---------------\   /---------------\  |
|  | Shared Pool E |   |  Fixed Pool F |  |
|  |    (28 GB)cBEC|   |    (12 GB)cA8D|  |
|  \---------------/   \---------------/  |
|                                     cF55|
+-----------------------------------------+
....

[[starting-server]]
== Starting the Terracotta Server

The snippet below defines two offheap resources named `primary-server-resource` and `secondary-server-resource` having
sizes `128MB` and `96MB` respectively:

[listing]
----
offheap-resources=primary-server-resource:128MB,secondary-server-resource:96MB
----

This can either be defined in config properties file or during server startup.

Assuming that you have the clustered Ehcache kit available locally, start with extracting the *ehcache-clustered* kit.
Change to your extracted directory and then execute the *start-tc-server* script located under `$KIT_DIR/server/bin` to start the Terracotta server.
You will then need to activate the cluster using `activate` command of config tool which is located under `$KIT_DIR/tools/bin`.

NOTE: You will need to have `JAVA_HOME` point to a Java 8 installation while starting the Terracotta server.

Check for the below `INFO` log to confirm if the server started successfully,
`Terracotta Server instance has started up as ACTIVE node on 0:0:0:0:0:0:0:0:9410 successfully, and is now ready for work.`

[[clustered-cache-manager]]
== Creating a Cache Manager with Clustering Capabilities

After starting the Terracotta server, as described in the previous section, you can now proceed to create the cache manager.
For creating the cache manager with clustering support you will need to provide the clustering service configuration.
Here is a code sample that shows how to configure a cache manager with clustering service.

[source%nowrap,java,indent=0]
----
include::{sourcedir39}/clustered/client/src/test/java/org/ehcache/clustered/client/docs/GettingStarted.java[tag=clusteredCacheManagerExample]
----

<1> Returns the `org.ehcache.config.builders.CacheManagerBuilder` instance.
<2> Use the `ClusteringServiceConfigurationBuilder` static method `.cluster(URI)` for connecting the cache manager to the clustered storage at the `URI` specified that returns the clustering service configuration builder instance.
Sample `URI` provided in the example is pointing to the clustered storage instance named "my-application" on the Terracotta server (assuming the server is running on localhost and port 9410).
<3> Auto-create the clustered storage if it doesn't already exist. We also allow auto-create on reconnection since the cluster is not persistent.
<4> Returns a fully initialized cache manager that can be used to create clustered caches.
<5> Close the cache manager.

[[cache-manager-config]]
== Cache Manager Configuration and Usage of Server Side Resources

This code sample demonstrates the usage of the concepts explained in the previous section in configuring a cache manager and clustered caches by using a broader clustering service configuration:

[source%nowrap,java,indent=0]
----
include::{sourcedir39}/clustered/client/src/test/java/org/ehcache/clustered/client/docs/GettingStarted.java[tag=clusteredCacheManagerWithServerSideConfigExample]
----

<1> `defaultServerResource(String)` on `ClusteringServiceConfigurationBuilder` instance sets the default server off-heap resource for the cache manager.
From the example, cache manager sets its default server off-heap resource to `primary-server-resource` in the server.
<2> Adds a resource pool for the cache manager with the specified name (`resource-pool-a`) and size (28MB) consumed out of the named server off-heap resource `secondary-server-resource`.
A resource pool at the cache manager level maps directly to a shared pool at the server side.
<3> Adds another resource pool for the cache manager with the specified name (`resource-pool-b`) and size (32MB).
Since the server resource identifier is not explicitly passed, this resource pool will be consumed out of default server resource provided in Step 3.
This demonstrates that a cache manager with clustering support can have multiple resource pools created out of several server off-heap resources.
<4> Provide the cache configuration to be created.
<5> `ClusteredResourcePoolBuilder.clusteredDedicated(String, long, MemoryUnit)` allocates a dedicated pool of storage to the cache from the specified server off-heap resource.
In this example, a dedicated pool of 32MB is allocated for clustered-cache from `primary-server-resource`.
<6> `ClusteredResourcePoolBuilder.clusteredShared(String)`, passing the name of the resource pool specifies that `shared-cache-1` shares the storage resources with other caches using the same resource pool (`resource-pool-a`).
<7> Configures another cache (`shared-cache-2`) that shares the resource pool (`resource-pool-a`) with `shared-cache-1`.
<8> Creates fully initialized cache manager with the clustered caches.

NOTE: When a cache is allocated a block of memory from a shared pool, it is retained forever and would never get
reallocated to another cache sharing the pool.

[[cluster-tier-manager-lifecycle]]
== Ehcache Cluster Tier Manager Lifecycle

When configuring a cache manager to connect to a cluster tier manager there are three possible connection modes:

[source%nowrap,java,indent=0]
----
include::{sourcedir39}/clustered/client/src/test/java/org/ehcache/clustered/client/docs/GettingStarted.java[tag=clusteredCacheManagerLifecycle]
----
<1> In auto create mode if no cluster tier manager exists then one is created with the supplied configuration.
If it exists and its configuration matches the supplied configuration then a connection is established.
If the supplied configuration does not match then the cache manager will fail to initialize.
<2> In auto create on reconnect mode we additionally support auto creation of any necessary entities when reconnecting to a cluster.
This behavior is useful in an non-persistent cluster in case the cluster loses its state due to a restart (planned or accidental).
<3> In expected mode if a cluster tier manager exists and its configuration matches the supplied configuration then a connection is established.
If the supplied configuration does not match or the cluster tier manager does not exist then the cache manager will fail to initialize.
<4> In config-less mode if a cluster tier manager exists then a connection is established without regard to its configuration.
If it does not exist then the cache manager will fail to initialize.

[[clustered-cache]]
== Configuring a Clustered Cache

=== Clustered Storage Tier

[source%nowrap,java,indent=0]
----
include::{sourcedir39}/clustered/client/src/test/java/org/ehcache/clustered/client/docs/GettingStarted.java[tag=clusteredCacheTieredExample]
----

<1> Configuring the heap tier for cache.
<2> Configuring the cluster tier of dedicated size from server off-heap resource using `ClusteredResourcePoolBuilder`.

The equivalent XML configuration is as follows:

[source%nowrap,xml,indent=0]
----
include::{sourcedir39}/clustered/client/src/test/resources/configs/docs/ehcache-clustered.xml[tag=tieringSample]
----

<1> Specify the heap tier for cache.
<2> Specify the cluster tier for cache through a custom service configuration from the clustered namespace.

=== Specifying consistency level

Ehcache offers two levels of consistency:

Eventual::

This consistency level indicates that the visibility of a write operation is not guaranteed when the operation returns.
Other clients may still see a stale value for the given key.
However this consistency level guarantees that for a mapping `(K, V1)` updated to `(K, V2)`, once a client sees `(K, V2)` it will never see `(K, V1)` again.

Strong::

This consistency level provides strong visibility guarantees ensuring that when a write operation returns other clients will be able to observe it immediately.
This comes with a latency penalty on the write operation required to give this guarantee.

[source%nowrap,java,indent=0]
----
include::{sourcedir39}/clustered/client/src/test/java/org/ehcache/clustered/client/docs/GettingStarted.java[tag=clusteredCacheConsistency]
----

<1> Specify the consistency level through the use of additional service configuration, using _strong_ consistency here.
<2> With the consistency used above, this `put` operation will return only when all other clients have had the corresponding mapping invalidated.

The equivalent XML configuration is as follows:

[source%nowrap,xml,indent=0]
----
include::{sourcedir39}/clustered/client/src/test/resources/configs/docs/ehcache-clustered.xml[tag=consistencySample]
----

<1> Specify the consistency level through a custom service configuration from the `clustered` namespace.

=== Clustered Cache Expiry

Expiry in clustered caches work with an exception that `Expiry#getExpiryForAccess` is handled on a best effort basis for cluster tiers.
It may not be as accurate as in the case of local tiers.

=== Clustered Unspecified Inheritance

We have included an option which allows a cache to be created inside the cache manager without having to explicitly define its cluster tier resource pool allocation.
In order to use this feature the cluster tier must already have been created with either a _shared_ or _dedicated_ resource pool.

In this case the definition of the cluster resource is done simply with a `clustered()` resource pool.
This effectively means _unspecified_ and indicates you expect it to exist already.
It will then inherit the clustered resource pool as it was configured when creating the cluster tier.

This option provides many benefits.
The main benefit is it simplifies clustered configuration by allowing clustered resource pool configuration to be handled by one client,
then all subsequent clients can inherit this configuration.
In addition, it also reduces clustered pool allocation configuration errors.
More importantly, sizing calculations only need to be done by one person and updated in one location.
Thus any programmer can use the cache without having to worry about using matching resource pool allocations.

The example code below shows how this can be implemented.

[source%nowrap,java,indent=0]
----
include::{sourcedir39}/clustered/client/src/test/java/org/ehcache/clustered/client/docs/GettingStarted.java[tag=unspecifiedClusteredCacheExample]
----

<1> Configure the first cache manager with auto create on reconnect
<2> Build a cache configuration for a clustered `dedicated` resource pool
<3> Create cache `my-dedicated-cache` using the cache configuration
<4> Configure the second cache manager as _expecting_
<5> Build a cache configuration for a clustered _unspecified_ resource pool, which will use the previously configured clustered _dedicated_ resource pool.
<6> Create cache with the same name `my-dedicated-cache` and use the clustered _unspecified_ cache configuration
