Differences

This shows you the differences between two versions of the page.

--- big_data_service:best_practice [2022/11/17 15:45] – [Server Constraints] su
+++ big_data_service:best_practice [2022/11/17 16:22] (current) – [What to do if you are approaching advised limits?] su
@@ Line 27: / Line 27: @@
 ==== Sharding ====
+Data in elasticsearch are stored in shards and the size and number of shards will impact performance and stability.
 To obtain a list of all the shards on an elasticsearch node, open the following URL on your App Store Connect server.
@@ Line 44: / Line 46: @@
 A default installation of App Store Connect allocates only 1GB of RAM to Elasticsearch. If the App Store Connect is importing Alarm Analysis data, it will create 2 shards per calendar month - thus reaching the recommended limit within 10 months. Even a node configured with the max 32GB RAM will reach its limit of 640 indices by 5 years if recording Alarm Analysis data for 5 assets.
+Exceeding these limits may lead to issues index corruption. For Alarm Analysis, this could manifest in the loss of monthly blocks of data.
+===== What to do if you are approaching advised limits? =====
+Most of the options below require specialist knowledge of elasticsearch. Consult with the Intelligent Plant team if unsure how to proceed.
+**Data Source Configuration** \\
+Change data index settings that determine how data is stored. IP Historian exposes index type and sharding properties. \\
+NB. Modifying these settings on a live system is not advised.
+**Scale Up** \\
+Increase server RAM and allocating more to the Big Data process.
-This may result in spurious index corruption issues.
+**Scale Out** \\
+Introduce more elasticseach nodes on dedicated infrastructure.
+**Data Archiving** \\
+Either remove old data, or migrate old data to another App Store Connect instance dediated to archiving.
+**Data Consolidation** \\
+Re-index smaller data nodes together. For instance, consolidate Alarm Analysis monthly indices into single (one shard) indices. \\
+NB. The new index would need alias names of all the indices replaced in order to support application function.