Terracotta Management Console

The Terracotta Management Console (TMC) is a web-based administration and monitoring application for Terracotta products.

To confirm the version of the TMC you are running, and for other information about the TMC, click About on the toolbar.

Setting Up User Accounts

When you first connect to the TMC, the authentication setup page appears, where you can choose to run the TMC with authentication or without. Authentication can also be enabled/disabled in the TMC Settings panel once the TMC is running.

If you do not enable authentication, you can connect to the TMC without being prompted for a login or password. Note that with authentication disabled, the TMC cannot connect to secured nodes.

If you enable authentication, the following choices appear:

• .ini file – Simple built-in role-based authentication.
• LDAP – Use with LDAP server.
• Microsoft Active Directory – Use with an Active Directory server.

Instructions for setting up connections to LDAP and Active Directory are available with the form that appears when you select the LDAP or Active Directory. Note that with any authentication enabled, the TMC can connect only to secured nodes.

Simple Account-Based Authentication

Authentication based on using built-in role-based accounts backed by a .ini file is the simplest scheme offered by the TMC. When you choose .ini-file authentication, a setup page appears for initializing the two accounts that control access to the TMC:

• Administrator – This account has read-write control over all aspects of the TMC, including accounts and connections.
• Operator – This read-only account can view statistics and other information about configured connections. This account cannot add or edit accounts or connections.

After you enter a username and password for each account, click Setup. You will be logged out, and can then log in using the new credentials.

Inactivity Timeout

Once a user logs in, there is no default timeout for inactivity. To set a default timeout for inactivity, uncomment the following block in web.xml and set the timeout value (in minutes) using the <param-value> element:

  <context-param> 
    <description>
      After this amount of time has passed without user activity, the user will be
      automatically logged out.
    </description>
    <param-name>idleTimeoutMinutes</param-name> 
    <param-value>30</param-value> 
  </context-param>

The TMC User Interface

A view of the TMC is shown below. Note that display panels and the connection-groups drop-down menu appear if an active (connected) connection group is available and selected.

Managing Connections

When you initially log on to the TMC, only default connection groups with default connections exist. If a node that can be monitored is running on localhost at the port specified by one of the default connections, then that default connection will appear as an active connection. Other default connections appear as an unavailable (inactive) connections.

You can create and edit connections and connection groups using the Connections panel. To open the Connections panel, click Preferences on the tool bar. You can also create new connections directly by clicking New Connection on the tool bar. Connections are assigned to connection groups to simplify management tasks.

Working With Connections

Connections allow you to monitor and administer nodes (both clustered and standalone) using the TMC.

Adding a Connection

To add a new connection, follow these steps:

1. Click New Connection on the tool bar.
2. Enter the location of the node you want to monitor, then click Next.

A screen appears confirming the agent found at the given location. If no agent is found, a warning appears, and no connection can be set up. Note that the location is relative to the machine running the Terracotta Management Server (TMS). The default location, "localhost", is the machine the TMS is running on, and may not be the machine your browser is running on.

The location is a URI in the following form:

<scheme>:<host-address>:<port>

A typical URI will appear similar to:

http://my-ehcache-host:9888

If the URI is for a server in a Terracotta Server Array, all other nodes participating in the cluster are automatically found. It is not required to create separate connections for those other nodes.

3. Enter a name to identify the connection.
4. Click Create Connection to save the new connection or Cancel to discard the new connection.

Editing and Deleting Connections

Managed connections that appear in the connections list can be edited or deleted.

To delete an existing connection, click Preferences on the toolbar to view the Connections panel. Locate the connection under its connection group in the Configured Connections list and click the red X next to that connection's name. A dialog allows you to confirm or cancel the delete operation.

To edit a connection, follow these steps:

1. Click Preferences on the tool bar.
2. In the Connections panel, click the pencil icon for the connection you want to edit.

3. Edit the connection's location, group, and name.

You can choose a group for the connection from the menu of existing groups, or create new connection group for the new connection. If you create a new group, you must enter a name for the group in the provided field.

4. Enter a connection timeout or accept the default value.

The connection timeout limits the time for successfully establishing a connection to the node. This ensures that the TMC does not hang waiting for a connection in the case where the node is unreachable.

5. Enter a read timeout or accept the default value.

The read timeout limits the time the TMC waits for data from a connected node. This ensures that the TMC does not hang waiting for a connection in the case where the node is unresponsive.

6. Click Save Changes to save the new values or Cancel to revert to the original values.

Monitoring Connection Groups

For every configured connection group, you can display a mini dashboard to view group status.

Each TSA connection-group dashboard displays the number of connected active (green) and mirror (blue) servers. It also displays the number of clients connected to that TSA. Certain other server states may also be indicated on the dashboard, including server starting (yellow) and server unreachable (red).

Each standalone connection group dashboard displays its number of configured connections and the number currently connected.

Each dashboard has a control drop-down menu with commands applicable to that dashboard and its associated connection group. For example, to hide a connection group's dashboard, choose Hide This Connection from the group's dashboard control menu. The connection group's connections are unaffected by hiding the dashboard. To restore the dashboard to the connections, click Preferences from the tool bar, then enable Show in Dashboard checkbox for that group.

Managing Application Data

To manage the application data of nodes in a connection group, select the group, then click the Application Data tab. Each Application Data panel has a CacheManager and Scope menu to select which CacheManagers and nodes supply the data for that panel.

Overview Panel

The Overview panel displays health metrics for CacheManagers and their caches, including certain cache statistics to help you track performance and resource usage across all CacheManagers.

Real-time statistics are displayed in a table with the following columns:

• Hit Ratio – The ratio of cache hits to get attempts. A ratio of 1.00 means that all requested data was obtained from the cache (every put was a hit). A low ratio (closer to 0.00) implies a higher number of misses that result in more faulting of data from outside the cache.
• Hit Rate – The number of cache hits per second. An effective cache shows a high number of hits relative to misses.
• Local Disk Hit Rate – The fault rate (data faulted in from the local disk).
• Local Heap Hit Rate – The rate of local (in-heap) hits (no faulting).
• OffHeap Hit Rate – The rate of local (off-heap) memory hits. Available only with BigMemory.
• Miss Rate – The number of cache misses per second. An effective cache shows a high number of hits relative to misses.
• Local Disk Miss Rate – The fault rate (data faulted in from remote source).
• Local Heap Miss Rate – The rate of local (in-heap) misses (causing faulting).
• OffHeap Miss Rate – The rate of local (off-heap) memory misses (causing faulting). Available only with BigMemory.
• Size – Overall data size (in entries).
• Local Heap Size – Overall data size (in entries) in the local heap.
• Local Disk Size – Overall data size (in entries) on the local disk.
• Local OffHeap Size – Overall data size (in entries) in local memory (off-heap). Available only with BigMemory.
• Local Heap Bytes – Overall data size (in bytes) in the local heap.
• Local Disk Bytes – Overall data size (in bytes) on the local disk.
• Local OffHeap Bytes – Overall data size (in bytes) in local memory (off-heap). Available only with BigMemory.
• Average Get Time – The average time for executing a get operation.
• Average Search Time – The Average Search Time graph displays how long each search operation takes (as well as the current values for that search time). This historical graph provides a view into how quickly cache searches are being performed; a correlation between how long searches are taking (Average Search Time) and how many are executed (Search Rate) may be seen over time.
• Search Rate – The search-rate graph displays how many searches per second are being executed (as well as the current values for that rate). This historical graph provides a view into how many cache searches are being performed; a correlation between how long searches are taking (Average Search Time) and how many are executed (Search Rate) may be seen over time.
• Put Rate – The number of cache puts executed per second. The number of puts always equals or exceeds the number of misses, since every miss leads to a put. In addition, updates are also counted as puts. Efficient caches have a low overall put rate.
• Remove Rate – The rate of element eviction.
• Update Rate – The number of updates to elements in the cache, per second. A high number of updates implies a high eviction rate or rapidly changing data.
• Expiration Rate – The number of elements reaching expiration in the cache, per second. Expired elements are not automatically evicted.
• Eviction Rate – The number of elements being evicted from the cache, per second. Evicted elements are expired, or evicted based on a usage algorithm when size limits are exceeded.
• Transaction Rollback Rate – A JTA historical graph used for displaying the rollback rate (as well as the current values for that rate) for transactional caches.
• Transaction Commit Rate – A JTA historical graph used for displaying the transaction commit rate (as well as the current values for that rate) for transactional caches.
• Writer Queue Length – The Write-Behind historical graph displays the total number of writes in the write-behind queue or queues (blue line), as well as the current value.

To choose the types of statistics displayed in the table, click Configure Columns to open a list of available statistics. Choose statistics (or set the option to display all statistics), then click OK to accept the change. The table immediately begins to display the chosen statistics.

To sort the table by a specific statistic, click the column head for that statistic.

Charts Panel

The Charts panel graphs the same statistics available in the Overview panel. This is useful for tracking performance trends and discovering potential issues.

In addition to being able to select a CacheManager and scope for the displayed data, you can also select a specific cache (or all caches) for the selected CacheManager.

Each historical real-time graph plots the appropriate metrics along the Y axis against system time (X axis). To view the value along a single point on a graph, float the mouse pointer over that point. This also displays the units used for the statistic being graphed.

To choose the type of statistic graphed by a particular chart, click the chart's corresponding Configure link to open a list of available statistics. Choose a statistic, then click OK to accept the change. The chart immediately begins to graph the chosen statistic.

Sizing Panel

The Diagnostics panel provides information on the usage of the heap, off-heap, and disk tiers by the caches of the selected CacheManager. To view tier usage by any active CacheManager, select that CacheManager from the CacheManager drop-down menu.

Usage by Tier

The Relative Cache Sizes by Tier table displays usage of the tier selected from the Tier drop-down menu. The table has the following columns:

• Cache – The name of the cache. An icon indicates whether the cache is distributed ( ) or not ( ).
• Size (MB) – The size of the cache's data in megabytes.
• % of Used – Percent of the total storage allotted to the cache that is currently used for cache data.
• Entries – The total number of cache entries.
• Mean Entry Size (bytes) – An estimate of the average size of each cache entry.

Click a row in the table to set the cache-related tier graphs to display values for the named cache.

Usage Graphs

The panel shows the following bar graphs:

• CacheManager Usage by Tier – Overall usage of each tier.
• Cache Usage by Tier – Usage of each tier by the selected cache. Choose the cache from the Selected Cache drop-down menu.
• Cache Miss Rate by Tier – The rate of cache misses at each tier. Usage of each tier by the selected cache. Choose the cache from the Selected Cache drop-down menu.

Float the mouse pointer over a bar to display an exact usage value. Click a tier's bar to display values for that tier in the Relative Cache Sizes by Tier table.

The Selected Cache Menu

The Selected Cache drop-down menu determines which cache is shown in the cache-related tier graphs and highlighted in the Relative Cache Sizes by Tier. The menu also indicates if the cache uses size-based (ARC) or entry-based sizing.

Management Panel

The Management panel displays a table listing the selected CacheManager's caches for the selected connection.

The following controls are available for the selected CacheManager:

• View Config – Displays the contents of the CacheManager's configuration file. The text in this window can be copied.
• Edit Config – Opens a dialog where you can edit the values of MaxBytesLocalDisk (if the local disk is being used) and MaxBytesLocalHeap.
• Disable All – Disables the operations of all of the listed caches at once. If caches are disabled, becomes Enable All, which can enable the operations of all of the listed caches at once.
• Clear Caches – Wipes the data of all of the listed caches at once.

The table's columns display the following attributes and metrics:

• Cache – The name of the cache. To view this cache's configuration, click View Config. Click Edit Config to open a dialog where you can edit the values of the following parameters:
  • MaxBytesLocalDisk
  • MaxBytesLocalHeap
  • MaxEntriesLocalHeap
  • TimeToIdleSeconds
  • TimeToLiveSeconds
• Enabled – Indicates whether the cache is enabled ("true") or disabled ("false"). Click Disable to stop the operations of the cache. If the cache is disabled, click Enable to start the operations of the cache.
• Element Count – The total number of elements in the cache. Click Clear Cache to wipe the data of the cache.

Monitoring Clusters

The Monitoring tab is available only for cluster connection groups. You can use the features available under this tab to monitor the functioning of the cluster, as well as the functioning of individual cluster components.

Runtime Statistics

Runtime statistics provide a continuous feed of sampled real-time data on a number of server and client metrics. The data is plotted on graphs. Sampling begins automatically when a runtime statistic panel is first viewed, but historical data is not saved.

Use the Select View menu to set the runtime statistics view to one of the following:

• All Servers – Display aggregated statistics for the TSA.
• Specified mirror group – Display aggregated statistics for the selected mirror group.
• Specified server – Display runtime statistics for the selected server.
• All Clients – Display aggregated statistics for all of the cluster's clients.
• Specified client – Display runtime statistics for the selected client.

Specific runtime statistics are defined in the following sections. The cluster components for which the statistic is available are indicated in the text.

Live Object Count

Shows the total number of live objects in the cluster, mirror group, or server.

If the trend for the total number of live objects goes up continuously, clients in the cluster will eventually run out of memory and applications may fail. Upward trends indicate a problem with application logic, garbage collection, or a tuning issue on one or more clients.

Eviction Rate

Shows the number of entries being evicted from the cluster, mirror group, or server.

Expiration Rate

Shows the number of expired entries found (and being evicted) on the TSA, mirror group, or server.

Write Operation Rate

Shows the number of completed writes (or mutations) in the TSA or selected server. Operations can include evictions and expirations, so that large-scale eviction or expiration operations can cause spikes in the operations rate (see the corresponding evictions and expirations statistical graphs). This rate is low in read-mostly setups, indicating that there are few writes and little data to evict. If it drops or deviates regularly from an established baseline, it may indicate issues with network connections or overloaded servers.

Note that when a client is (or all clients are) selected, then this statistic is reported as the Write Transaction Rate, tracking client-to-server write transactions.

Off-heap Usage

Shows the amount, in megabytes or gigabytes, of maximum available off-heap memory and off-heap memory being used. This statistic appears only if BigMemory is in effect.

Fault/Flush Rate

Faults from off-heap or disk occur when an object is not available in a server's on-heap cache. Flushes occur when the heap or off-heap cache must clear data due to memory constraints. The fault/flush Rate statistic is a measure of how many objects (per second) are being faulted and flushed from and to lower tiers in response to application requests. Objects being requested for the first time, or objects that have been flushed from off-heap memory before a request arrives, must be faulted in from disk. High rates could indicate inadequate memory allocation at the server.

The client flush-rate statistic is a measure of how many objects are being flushed out of client memory to a Terracotta server. These objects are available in the Terracotta server if needed at a later point in time. A high flush rate could indicate inadequate memory allocation at the client.

The client fault-rate statistic is a measure of how many objects are being faulted into client memory from the server. A high fault rate could indicate poor locality of reference or inadequate memory allocation at the client.

Events

The Events panel displays cluster events received by the Terracotta server array. You can use this panel to quickly view these events in one location in an easy-to-read format, without having to search the Terracotta logs.

Note that, in addition to displaying events with the chosen severity level, all events with a higher severity level are also displayed. That means that if the INFO level is chosen, then all events with WARN and above are also displayed.

Administering Clusters

Configuration

Using subpanels, the Configuration panel displays the status, environment, and configuration information for the server or servers selected in the Cluster Node menu.

The Main subpanel displays the server status and a list of properties, including the server's IP address, version, license (capabilities), and restartability and failover modes. A specific server must be selected to view this subpanel.

The following additional subpanels are available:

• Environment – The server's JVM system properties.
• TCProperties – The Terracotta properties that the server is using.
• Process Args – The JVM arguments that the server was started with.
• TCConfig – The Terracotta configuration file that the server was started with.

Troubleshooting Clusters

You can get a snapshot of the state of each server and client in the Terracotta cluster using thread dumps. To display the console's thread-dumps feature, click Troubleshooting.

The thread-dump navigation pane lists completed thread dumps by date-time stamp. The contents of selected thread dumps are displayed in the right-side pane. To delete all shown thread dumps, click Clear All.

To generate a thread dump, follow these steps:

1. Choose the target of the thread dump from the Scope menu.
2. Click Take Thread Dump.

When complete, the thread dump appears in the thread-dumps navigation pane.

3. Expand the entry created in the thread-dumps navigation pane to see the thread-dump entry or entries.

The entries correspond to servers and clients included in the thread dump.

4. Click a server or client entry to display the thread dump for that server or client.

Servers that appear in the Scope menu but are not connected produce empty thread dumps.

Logging

The Logs panel controls logging.

Preferences

Click Preferences on the toolbar to open a dialog where global TMC settings can be configured.

Click the Polling tab to set the Polling Interval Seconds, which controls the granularity of polled statistical data. Note that shorter polling intervals can have a greater effect on the overall performance of the nodes being polled. o reset to default values, click Reset to Defaults.

Click the Security tab to set security. Use the Authentication Enabled checkbox to enable or disable authentication. If you enable authentication and confirm your choice, the TMC displays the account setup page. Connections to unsecured nodes may be lost. If you disable authentication, connections to secured nodes may be lost. To reset to default values, click Reset to Defaults.