alarm_analysis:resilient_data_transfer
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
alarm_analysis:resilient_data_transfer [2018/02/05 14:30] – [1. PIN: Data Core Node] su | alarm_analysis:resilient_data_transfer [2023/12/29 13:41] (current) – su | ||
---|---|---|---|
Line 3: | Line 3: | ||
===== Overview ===== | ===== Overview ===== | ||
- | Intelligent Plant produces highly configurable tools for capturing and analyzing data. We provide solutions that can be deployed in high-security/ | + | Before reading this guide, you should |
- | This document describes best practice for a particular use-case: | + | In these scenarios, it is necessary to configure Data Core installations to act as a data relay. Moreover, if the connectivity between relays is unreliable, further configuration is required to guarantee resilient data transfer. |
- | //Collect **Process Control Alarm & Event** data for **Alarm Analysis** | + | This guide focuses |
+ | Data is gathered from serial feed and relayed via a Data Core Node from the Process Information Network to an App Store Connect on the Business Network. The relay is configured for Resilient Data Transfer. | ||
- | ===== Terminology ===== | + | {{ : |
- | | Alarm Analysis | Intelligent Plant application that processes Alarm & Event data to produce interactive EEMUA (Engineering Equipment and Materials Users Association) grade reports and analysis. | | + | A two box installation is required, with the following |
- | | Data Core Node | Intelligent Plant' | + | |
- | | Industrial App Store | Cloud portal for industrial apps (including online editions of Intelligent Plant applications). | | + | |
- | | App Store Connect | An instance of Data Core Node pre-configured for secure connections with the Industrial App Store. | | + | |
- | | PCN | Process Control Network. | | + | |
- | | PIN | Process Information Network. | | + | |
- | | BN | Business Network. Also referred to as the Enterprise Network. | | + | |
- | | TCP | Transmission Control Protocol. A network communication protocol. | | + | |
+ | {{ : | ||
- | ===== Network Topology ===== | + | * Data Core listens for events (TCP Printer Stream). Arriving events are parsed and sent... |
+ | - On an immediate data relay to App Store Connect (Fast TCP Out) | ||
+ | - To a local data repository (Big Data Sink) | ||
- | {{ : | + | * A second process on the Data Core Node polls the local data repository for stored events (Big Data Source) and sends to... |
+ | - a further data relay that awaits an Ack from the App Store Connect (TCP Out). If Ack is not received, data is resubmitted. | ||
- | | + | |
- | - This physical architecture guarantees one-way data-flow. \\ \\ | + | |
- | - The Data Core node installation on the PIN includes: | + | |
- | - TCP Printer Port listener (listening for input from the Serial to Ethernet Converter) | + | |
- | - Alarm & Event Collector (collecting A&E data from the PCN) | + | |
- | - a TCP Out Channel for data transfer \\ \\ | + | |
- | - The Data Core TCP Out Channel (on the PIN) initiates an authenticated TCP connection with App Store Connect (on the BN). Data transmitted on the TCP channel is signed | + | |
- | - App Store Connect acts a local data processor and supports connections to the Industrial App Store. \\ \\ App Store Connect incudes: | + | |
- | - TCP Listener | + | |
- | - Alarm & Event Collector (collecting A&E data from the Alarm Analysis server) | + | |
- | - Alarm Analysis Processing (converting A&E data to Alarm Analysis records) \\ \\ Alarm Analysis data remains on the Business Network. \\ \\ | + | |
- | - App Store Connect initiates a connection to the Industrial App Store using Microsoft' | + | |
- | - The Industrial App Store: a cloud portal of industrial applications that can connect to business data sources through App Store Connect. \\ \\ | + | |
- | - Users inside and outside the Business Network (e.g. office and home workers) access Alarm Analysis via the Internet. They log-in to the Industrial App Store and select the Alarm Analysis app. \\ NB. App Store users must be granted access to the Alarm Analysis data source (configured on the Business Network App Store Connection). \\ For instructions on sharing data, refer to the App Store Wiki: Share Data with other App Store Users. \\ \\ | + | |
- | ===== Deployment | + | While connectivity between the PIN and BN is good, all events will reach the destination twice. The fast stream means data arrives quickly, the second stream guarantees resilience. App Store Connect consolidates the two streams |
- | The network topology above assumes data crosses into the PIN via a serial printer feed. | ||
- | In order to collect data on the printer stream and transfer to the BN we need to undertake the following: | ||
- | - Install | + | ====== Step by Step Install |
- | - Install App Store Connect on the PIN | + | |
- | The configuration of these //Nodes// is as follows: | + | **1. Install Data Core Node on PIN** |
- | {{ :data_core:resilienttransfer_02.png |}} | + | For detailed instructions on how to deploy a Data Core node, see: [[data_core:stand-alone_installation]]. |
+ | **2. Data Core Node Configuration** | ||
+ | Configure the following Data Core components (assume default settings unless otherwise stated). | ||
- | + | For detailed instructions on how to create an Event Source to Sink subscription, see: [[data_core: | |
- | + | ||
- | + | ||
- | + | ||
- | ==== 1. PIN: Data Core Node ==== | + | |
- | + | ||
- | For detailed instructions on how to deploy Data Core node, see: [[data_core: | + | |
- | + | ||
- | Configure the following Data Core components (assume default settings unless otherwise stated): | + | |
^ //TCP Printer Stream// ^^ | ^ //TCP Printer Stream// ^^ | ||
+ | ^ Type | TCP Printer (Event Source) | | ||
^ Description | Listen and parse data arriving on TCP channel | | ^ Description | Listen and parse data arriving on TCP channel | | ||
- | ^ Type | TCP Printer (Event Source) | + | ^ Disabled |
^ TCP Port | 9000 | | ^ TCP Port | 9000 | | ||
^ Message Delimiter | New Line {\n} | | ^ Message Delimiter | New Line {\n} | | ||
Line 74: | Line 51: | ||
^ // Fast TCP Out// ^^ | ^ // Fast TCP Out// ^^ | ||
+ | ^ Type | TCP Event Sink (Event Sink) | | ||
^ Description | Immediate data transfer to the Business Network | | ^ Description | Immediate data transfer to the Business Network | | ||
- | ^ Type | TCP Event Sink (Event Sink) | | + | ^ Disabled |
^ TCP Server Host | [ IP Address of server hosting "Fast TCP In" ] | | ^ TCP Server Host | [ IP Address of server hosting "Fast TCP In" ] | | ||
^ TCP Server Port| 11000 | | ^ TCP Server Port| 11000 | | ||
Line 84: | Line 62: | ||
^ //Big Data Sink// ^^ | ^ //Big Data Sink// ^^ | ||
- | ^ Description | Save data to the Big Data Store | | ||
^ Type | Big Data Event Sink (Event Sink) | | ^ Type | Big Data Event Sink (Event Sink) | | ||
+ | ^ Description | Save data to the Big Data Store | | ||
+ | ^ Disabled | False | | ||
^ Big Data URL | http:// | ^ Big Data URL | http:// | ||
^ // | ^ // | ||
^ //Big Data Source// ^^ | ^ //Big Data Source// ^^ | ||
+ | ^ Type | Big Data Event Source (Event Source) | | ||
^ Description | Retrieve collected data from Big Data Store | | ^ Description | Retrieve collected data from Big Data Store | | ||
- | ^ Type | Big Data Event Source (Event Source) | + | ^ Disabled |
+ | ^ Paused | False | | ||
^ Big Data URL | http:// | ^ Big Data URL | http:// | ||
^ Index Filter | tcpprinterstream.evt_* | | ^ Index Filter | tcpprinterstream.evt_* | | ||
Line 98: | Line 79: | ||
^ // TCP Out// ^^ | ^ // TCP Out// ^^ | ||
+ | ^ Type | TCP Event Sink (Event Sink) | | ||
^ Description | Resilient data transfer to the Business Network | | ^ Description | Resilient data transfer to the Business Network | | ||
- | ^ Type | TCP Event Sink (Event Sink) | | + | ^ Disabled |
- | ^ TCP Server Host | [ IP Address of server hosting "Fast TCP In" ] | | + | ^ TCP Server Host | [ IP Address of server hosting "TCP In" ] | |
^ TCP Server Port| 11000 | | ^ TCP Server Port| 11000 | | ||
- | ^ Username | [Service account with access to server hosting "Fast TCP In" ] | | + | ^ Username | [Service account with access to server hosting "TCP In" ] | |
- | ^ Password | [Service account password with access to server hosting "Fast TCP In" ] | | + | ^ Password | [Service account password with access to server hosting "TCP In" ] | |
^ Check Response | Yes | | ^ Check Response | Yes | | ||
^ // | ^ // | ||
+ | **3. Install App Store Connect on BN** | ||
- | ===== Data Relay across 3-Zone Network Architecture ===== | + | For detailed instructions on how to deploy App Store Connect, see: [[data_core: |
- | The following example illustrates relaying data across a 3 zone network architecture. In each zone a Data Core Node is installed. | ||
- | {{: | + | **4. App Store Connect Configuration** |
- | Alarm & Event data enters Data Core Node 1 via a serial printer-port cable connected to a Serial to Ethernet Converter. This physical architecture guarantees one-way data-flow from Process Controllers to Data Core. | + | Configure the following |
- | A **Fast Flow** relays data across the zones via a fire-and-forget process providing near-real-time data on the destination server. | + | For detailed instructions |
- | On a separate **resilient flow** the database on the source server is poled and new data relayed via a guaranteed delivery process. This provides a complete data record on the destination server. | + | ^ //TCP In// ^^ |
+ | ^ Type | TCP Event Source (Event Source) | | ||
+ | ^ Description | Receive | ||
+ | ^ Disabled | False | | ||
+ | ^ TCP Server Port| 11000 | | ||
- | The database on the destination server consolidates | + | ^ //Big Data Sink// ^^ |
+ | ^ Type | Big Data Event Sink (Event Sink) | | ||
+ | ^ Description | Save data to the Big Data Store | | ||
+ | ^ Disabled | False | | ||
+ | ^ Big Data URL | http:// | ||
+ | ^ Big Data Refresh Interval | 5s | | ||
+ | ^ // | ||
+ | |||
+ | |||
+ | **5. Firewall Settings** | ||
+ | |||
+ | Firewalls will need to allow passage for the following protocols on ports: | ||
+ | |||
+ | ^ Firewall ^ Requirements ^ | ||
+ | | BN:Internet Network Firewall | TCP Port 443 open to outbound traffic | ||
+ | | Computer hosting App Store Connect | Windows Firewall TCP Port 443 open to outbound traffic \\ TCP Port 11000 open to inbound traffic | | ||
+ | |PIN:BN Network Firewall | TCP Port 11000 open to outbound traffic | | ||
+ | | PCN:PIN Network Firewall | No inbound access required. | | ||
- | ===== Data Core Installation ===== | + | ==== 5. Testing |
- | Stand-Alone installations | + | Assuming that the TCP Printer Stream configured above is listening to an active Alarm & Event stream, we should see evidence |
- | If the intention | + | A quick test is to execute a URL search query. |
+ | |||
+ | 1. Log on to servers hosting | ||
+ | |||
+ | Open web browser and enter | ||
+ | < | ||
+ | |||
+ | A " | ||
< | < | ||
- | App Store Connect is an instance of a Data Core Node pre-configured for secure connection with the Industrial App | + | health status index pri rep docs.count docs.deleted store.size pri.store.size |
- | Store. | + | green open |
</ | </ | ||
- | ===== Data Core Configuration ===== | + | A " |
- | Configure the following Event Source and Event Sink components and subscriptions. | + | 2. Log on to servers hosting App Store Connect |
- | {{:alarm_analysis:resilientdatatransfer7.png|}} | + | Open web browser and enter |
+ | < | ||
- | ^ //TCP Printer Stream// ^^ | + | You should expect to see something like: |
- | ^ Type | TCP Printer (Event Source) | | + | |
- | ^ TCP Port | 9000 | | + | |
- | ^ Message Delimiter | New Line {\n} | | + | |
- | ^ Maximum Characters per Scan | 4000 | | + | |
- | ^ //Fast TCP Out// ^^ | + | < |
- | ^ Type | TCP Event Sink (Event Sink) | | + | health status index pri rep docs.count docs.deleted store.size pri.store.size |
- | ^ TCP Server Host | [ IP Address of server hosting "Fast TCP In" ] | | + | green open |
- | ^ TCP Server Port| 11000 | | + | </ |
- | ^ Username | [Service account with access to server hosting "Fast TCP In" ] | | + | |
- | ^ Password | [Service account password with access to server hosting "Fast TCP In" ] | + | |
- | ^ Check Response | False | | + | |
- | ^ //Retrieve Data// ^^ | + | A " |
- | ^ Type | Big Data Event Source (Event Source) | | + | |
- | ^ Big Data URL | http:// | + | |
- | ^ Index Filter | tcpprinterstream.evt_* | | + | |
- | ^ Sleep Period | 30 | | + | |
- | ^ Lag | 60| | + | |
- | ^ //TCP In// ^^ | + | A " |
- | ^ Type | TCP Event Source (Event Source) | | + | |
- | ^ TCP Server Port| 11000 | | + | |
- | ^ //TCP Out// ^^ | ||
- | ^ Type | TCP Event Sink (Event Sink) | | ||
- | ^ TCP Server Host | [ IP Address of server hosting " | ||
- | ^ TCP Server Port| 11000 | | ||
- | ^ Username | [Service account with access to server hosting " | ||
- | ^ Password | [Service account password with access to server hosting " | ||
- | ^ Check Response | True | | ||
- | ^ //Store Data// ^^ | ||
- | ^ Type | Big Data Event Sink (Event Sink) | | ||
- | ^ Big Data URL | http:// | ||
- | ^ Big Data Refresh Interval | 5s | | ||
- | ===== Firewall Requirements ===== | ||
- | Network and server firewalls must be open for TCP traffic on the ports configured above. | + | |
+ | |||
+ | |||
+ | |||
+ | |||
+ | ==== 6. Next Steps ==== | ||
+ | |||
+ | So far, we've moved Alarm & Event data across a network. We are now ready to configure Alarm Analysis processing. | ||
+ | |||
+ | For more info, see [[alarm_analysis: | ||
+ | |||
alarm_analysis/resilient_data_transfer.txt · Last modified: 2023/12/29 13:41 by su