User Tools

Site Tools


alarm_analysis:data_retention

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision
Previous revision
alarm_analysis:data_retention [2019/11/15 14:46] sualarm_analysis:data_retention [2023/12/29 13:42] (current) su
Line 12: Line 12:
 ==== 1. Collection ====  ==== 1. Collection ==== 
  
-Source Alarm & Event (A&E) data arrives on the import flow and is stored for later processing.+Source Alarm & Event (A&E) data arrives on the import flow and is stored for later processing. We call this the **A&E Raw Data Cache**.
  
-Data is stored in monthly indices determined by date of arrival. This may be different to the date of the actual event record cotained in the raw A&E message (remember, the data is unprocessed at this stage so we don't know what it contains). +Data is stored in monthly indices determined by date of arrival. This may be different to the date of the actual event record contained in the raw A&E message (remember, the data is unprocessed at this stage so we don't know what it contains). 
  
 For an ongoing (near real-time) data flow, the monthly indices of raw data will be closely aligned with A&E event time. However, exceptions may occur. For example, importing a large historical backlog. For an ongoing (near real-time) data flow, the monthly indices of raw data will be closely aligned with A&E event time. However, exceptions may occur. For example, importing a large historical backlog.
Line 25: Line 25:
 ==== 2. Processing ====  ==== 2. Processing ==== 
  
-The source data stored in stage 1 undergoes cleaning and transformation to generate an Alarm Analysis record optimised for analytics and aggregation.+The source data stored in stage 1 undergoes cleaning and transformation to generate an Alarm Analysis record optimised for analytics and aggregation. This is the **Alarm Analysis database**.
  
 Data is stored in monthly indices determined by the A&E event date. The index name format is: Data is stored in monthly indices determined by the A&E event date. The index name format is:
alarm_analysis/data_retention.txt · Last modified: 2023/12/29 13:42 by su