Differences

This shows you the differences between two versions of the page.

--- alarm_analysis:data_retention [2019/11/14 15:54] – su
+++ alarm_analysis:data_retention [2023/12/29 13:42] (current) – su
@@ Line 1: / Line 1: @@
-====== Data Retention ======
+====== Alarm Analysis Data Retention ======
-By default, Alarm Analysis retains all data. The Alarm Analysis application is designed so that it can comfortably execute monthly reports irrespective of how much Alarm & Event history is stored. However, at some point you may wish to expire data to free storage capacity.
+By default, Alarm Analysis retains all data. The Alarm Analysis application is designed to comfortably execute monthly reports irrespective of how much Alarm & Event history is stored. However, at some point you may wish to expire data to free storage capacity.
 The article below provides guidance on configuring a data retention policy.
@@ Line 10: / Line 10: @@
 Alarm Analysis Import flow can be considered in two stages:
-**1. Collection**
+==== 1. Collection ====
-Source Alarm & Event (A&E) data arrives on the import flow and is stored for later processing.
+Source Alarm & Event (A&E) data arrives on the import flow and is stored for later processing. We call this the **A&E Raw Data Cache**.
-Data is stored in monthly indices determined by date of arrival. This may be different to the date of the actual event record cotained in the raw A&E message (remember, the data is unprocessed at this stage so we don't know what it contains).
+Data is stored in monthly indices determined by date of arrival. This may be different to the date of the actual event record contained in the raw A&E message (remember, the data is unprocessed at this stage so we don't know what it contains).
 For an ongoing (near real-time) data flow, the monthly indices of raw data will be closely aligned with A&E event time. However, exceptions may occur. For example, importing a large historical backlog.
@@ Line 20: / Line 20: @@
 The data is stored via the Intelligent Plant Big Data Service to indices with the following format:
- * [datasource].evt_YYYYMM \\ e.g. scadaopc.evt_201908
+  * [datasource].evt_YYYYMM \\ e.g. scadaopc.evt_201908
-**2. Processing**
+==== 2. Processing ====
-The source data stored in stage 1 undergoes cleaning and transformation to generate an Alarm Analysis record optimised for analytics and aggregation.
+The source data stored in stage 1 undergoes cleaning and transformation to generate an Alarm Analysis record optimised for analytics and aggregation. This is the **Alarm Analysis database**.
 Data is stored in monthly indices determined by the A&E event date. The index name format is:
- * [Company].[Asset].msg_YYYYMM \\ e.g. oilco.osprey.msg_201908
+  * [Company].[Asset].msg_YYYYMM \\ e.g. oilco.osprey.msg_201908
+----
 ===== The case for keeping unprocessed data =====
@@ Line 36: / Line 37: @@
 Once data has been processed, it //could// be removed from the collection store. However, this would prevent  re-processing of historic data. This can be pertinent in the early days of configuring an Alarm Analysis Import Flow, where rules are often revised.
-Only remove unprocessed data if satisfied that historic re-processing of historic is not a requirement.
+Only remove unprocessed data if satisfied historic re-processing of historic is not a requirement.
+----
 ===== Configuring a Data Retention Policy =====
-  - Open App Store Connect and navigate to Real-Time Data
+. On App Store Connect server, Open App Store Connect (http://localhost:2006/Service) \\
+Then navigate to "Real-Time Data > Data Sources".
+{{:app_store_connect:asc_01.png?300|}}
+. Select the Alarm Analysis data source for which a Data Retention Policy is required. \\
+Navigate to "Actions > Configure Data Source Settings".
+{{:alarm_analysis:dataretention01.png?300|}}
+. Scroll to "Data Retention" settings.
+{{:alarm_analysis:dataretention02.png?300|}}
+The following settings are available:
+^ Setting ^ Description ^ Example ^
+^ Delete Expired Indices | Enable a data retention policy that deletes expired indices. \\ \\ Only enable when absolutely sure all other config is correct as this will attempt to permanently delete data. \\ \\ | True or False |
+^ Index Names | Indices to be included in deletion checks. \\ \\  Wildcards should be employed with care and only used for date suffixes. \\ \\  Comma separate multiple names. \\ \\  Do not modify when Data Retention is enabled. \\ \\ | scadaopc.evt_* |
+^ Retention Period | Specify data retention period in months. \\ \\ Do not modify when Data Retention is enabled. \\ \\  | 6 |
+----
+===== Schedule =====
+The Data Retention Policy (once enabled) is executed daily at 1am.
-  - Select an A