Differences

This shows you the differences between two versions of the page.

--- alarm_analysis:data_retention [2019/11/14 14:15] – su
+++ alarm_analysis:data_retention [2023/12/29 13:42] (current) – su
@@ Line 1: / Line 1: @@
-====== Data Retention ======
+====== Alarm Analysis Data Retention ======
-By default, Alarm Analysis retains all data. The Alarm Analysis application is designed so that it can comfortably execute monthly reports irrespective of how much Alarm & Event history is stored. However, at some point you may wish to expire data to free storage capacity.
+By default, Alarm Analysis retains all data. The Alarm Analysis application is designed to comfortably execute monthly reports irrespective of how much Alarm & Event history is stored. However, at some point you may wish to expire data to free storage capacity.
 The article below provides guidance on configuring a data retention policy.
-===== How Alarm Analysis Stores Data =====
+===== How Alarm Analysis stores data =====
+Alarm Analysis Import flow can be considered in two stages:
+==== 1. Collection ====
+Source Alarm & Event (A&E) data arrives on the import flow and is stored for later processing. We call this the **A&E Raw Data Cache**.
+Data is stored in monthly indices determined by date of arrival. This may be different to the date of the actual event record contained in the raw A&E message (remember, the data is unprocessed at this stage so we don't know what it contains).
+For an ongoing (near real-time) data flow, the monthly indices of raw data will be closely aligned with A&E event time. However, exceptions may occur. For example, importing a large historical backlog.
+The data is stored via the Intelligent Plant Big Data Service to indices with the following format:
+  * [datasource].evt_YYYYMM \\ e.g. scadaopc.evt_201908
+==== 2. Processing ====
+The source data stored in stage 1 undergoes cleaning and transformation to generate an Alarm Analysis record optimised for analytics and aggregation. This is the **Alarm Analysis database**.
+Data is stored in monthly indices determined by the A&E event date. The index name format is:
+  * [Company].[Asset].msg_YYYYMM \\ e.g. oilco.osprey.msg_201908
+----
+===== The case for keeping unprocessed data =====
+Once data has been processed, it //could// be removed from the collection store. However, this would prevent  re-processing of historic data. This can be pertinent in the early days of configuring an Alarm Analysis Import Flow, where rules are often revised.
+Only remove unprocessed data if satisfied historic re-processing of historic is not a requirement.
+----
+===== Configuring a Data Retention Policy =====
+. On App Store Connect server, Open App Store Connect (http://localhost:2006/Service) \\
+Then navigate to "Real-Time Data > Data Sources".
+{{:app_store_connect:asc_01.png?300|}}
+. Select the Alarm Analysis data source for which a Data Retention Policy is required. \\
+Navigate to "Actions > Configure Data Source Settings".
+{{:alarm_analysis:dataretention01.png?300|}}
+. Scroll to "Data Retention" settings.
+{{:alarm_analysis:dataretention02.png?300|}}
+The following settings are available:
+^ Setting ^ Description ^ Example ^
+^ Delete Expired Indices | Enable a data retention policy that deletes expired indices. \\ \\ Only enable when absolutely sure all other config is correct as this will attempt to permanently delete data. \\ \\ | True or False |
+^ Index Names | Indices to be included in deletion checks. \\ \\  Wildcards should be employed with care and only used for date suffixes. \\ \\  Comma separate multiple names. \\ \\  Do not modify when Data Retention is enabled. \\ \\ | scadaopc.evt_* |
+^ Retention Period | Specify data retention period in months. \\ \\ Do not modify when Data Retention is enabled. \\ \\  | 6 |
+----
+===== Schedule =====
+The Data Retention Policy (once enabled) is executed daily at 1am.
-Alarm & Event data is stored in montly indices. Depending on how Alarm & Import has been configured, there may be two types of monthly index.
-  * **Alarm Analysis Message** \\ The cleaned and transformed Alarm & Event record. Optimized for big data operations.
-    * **Source Event Cache** \\ A raw Alarm & Event arrives on the import flow. Minimal parsing is undertaken - only enough to allow the record to be stored along with meta information such as "where it came from" and "time of arrival".
-The **Source Event Cache** //could// be removed once the raw record has been transformed to an Alarm Analysis record. However, this may prevent future retaining the source object facilitates re-import, which can be useful - particularly when first definining and configuring transformation rules. If the //Source Event Cache// it may be difficult (or impossible((Alarm Analysis supports many different types of import flow. A source data provider may remain available for future retrieval (e.g. a datafile or database repostitory). However, in other cases the event data source may be ephemeral (e.g. OPC), and it is the responsibility of Alarm Analysis' to //collect// as well as process.)) to apply transformation rules to historic data if the Source Event Cache has been removed.