alarm_analysis:data_retention
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
alarm_analysis:data_retention [2019/11/15 14:46] – su | alarm_analysis:data_retention [2023/12/29 13:42] (current) – su | ||
---|---|---|---|
Line 12: | Line 12: | ||
==== 1. Collection ==== | ==== 1. Collection ==== | ||
- | Source Alarm & Event (A&E) data arrives on the import flow and is stored for later processing. | + | Source Alarm & Event (A&E) data arrives on the import flow and is stored for later processing. We call this the **A&E Raw Data Cache**. |
- | Data is stored in monthly indices determined by date of arrival. This may be different to the date of the actual event record | + | Data is stored in monthly indices determined by date of arrival. This may be different to the date of the actual event record |
For an ongoing (near real-time) data flow, the monthly indices of raw data will be closely aligned with A&E event time. However, exceptions may occur. For example, importing a large historical backlog. | For an ongoing (near real-time) data flow, the monthly indices of raw data will be closely aligned with A&E event time. However, exceptions may occur. For example, importing a large historical backlog. | ||
Line 25: | Line 25: | ||
==== 2. Processing ==== | ==== 2. Processing ==== | ||
- | The source data stored in stage 1 undergoes cleaning and transformation to generate an Alarm Analysis record optimised for analytics and aggregation. | + | The source data stored in stage 1 undergoes cleaning and transformation to generate an Alarm Analysis record optimised for analytics and aggregation. This is the **Alarm Analysis database**. |
Data is stored in monthly indices determined by the A&E event date. The index name format is: | Data is stored in monthly indices determined by the A&E event date. The index name format is: |
alarm_analysis/data_retention.txt · Last modified: 2023/12/29 13:42 by su