02 - Design and Develop Data Processing

Use complete as output mode outputMode("complete") when you want to aggregate the data and output the entire results to sink every time.

It is similar to the complete with one exception; update output mode outputMode("update") just outputs the updated aggregated results every time to data sink when new data arrives.

Use append as output mode outputMode("append") when you want to output only new rows to the output sink.

Creating a RANGE RIGHT

Creating a RANGE LEFT

In some cases, the changed data within a period in your source data store can be easily to sliced up (for example, LastModifyTime, CreationTime). In some cases, there is no explicit way to identify the delta data from last time you processed the data. The Change Tracking technology supported by data stores such as Azure SQL Database and SQL Server can be used to identify the delta data.

SYS_CHANGE_VERSION

Provide a value in time span format and both negative and positive offsets are allowed. This property is mandatory if the trigger is depending on itself and in all other cases it is optional. Self-dependency should always be a negative offset. If no value specified, the window is the same as the trigger itself.

Use the surrogate key transformation to add an incrementing key value to each row of data. This is useful when designing dimension tables in a star schema analytical data model. In a star schema, each member in your dimension tables requires a unique key that is a non-business key.

A cache sink is when a data flow writes data into the Spark cache instead of a data store. In mapping data flows, you can reference this data within the same flow many times using a cache lookup. This is useful when you want to reference data as part of an expression but don't want to explicitly join the columns to it. Common examples where a cache sink can help are looking up a max value on a data store and matching error codes to an error message database.

The assert transformation enables you to build custom rules inside your mapping data flows for data quality and data validation. You can build rules that will determine whether values meet an expected value domain. Additionally, you can build rules that check for row uniqueness. The assert transformation will help to determine if each row in your data meets a set of criteria. The assert transformation also allows you to set custom error messages when data validation rules are not met.

Use the Alter Row transformation to set insert, delete, update, and upsert policies on rows. You can add one-to-many conditions as expressions. These conditions should be specified in order of priority, as each row will be marked with the policy corresponding to the first-matching expression. Each of those conditions can result in a row (or rows) being inserted, updated, deleted, or upserted. Alter Row can produce both DDL & DML actions against your database.

Create a consumer group on your IoT hub. Create and configure an Azure Stream Analytics job to read temperature telemetry from your consumer group and send it to Power BI. Create a report of the temperature data in Power BI and share it to the web.

You perform the following steps in this tutorial: Prepare the source data store Create a data factory. Create linked services. Create source, sink, and change tracking datasets. Create, run, and monitor the full copy pipeline Add or update data in the source table Create, run, and monitor the incremental copy pipeline

A cloud part that is responsible for the job definition: users define inputs, output, query, and other settings, such as out of order events, in the cloud.

A module running on your IoT devices. The module contains the Stream Analytics engine and receives the job definition from the cloud.

Supported stream input types are: Edge Hub Event Hub IoT Hub Supported stream output types are: Edge Hub SQL Database Event Hub Blob Storage/ADLS Gen2

For both inputs and outputs, CSV and JSON formats are supported.

Manufacturing safety systems must respond to operational data with ultra-low latency. With Stream Analytics on IoT Edge, you can analyze sensor data in near real-time, and issue commands when you detect anomalies to stop a machine or trigger alerts.

Mission critical systems, such as remote mining equipment, connected vessels, or offshore drilling, need to analyze and react to data even when cloud connectivity is intermittent.

Azure IoT Edge moves cloud analytics and custom business logic to devices so that your organization can focus on business insights instead of data management.

Azure IoT Edge allows you to deploy complex event processing, machine learning, image recognition, and other high value AI without writing it in-house. A

Installs and update workloads on the device. Maintains Azure IoT Edge security standards on the device. Ensures that IoT Edge modules are always running. Reports module health to the cloud for remote monitoring. Manages communication between downstream leaf devices and an IoT Edge device, between modules on an IoT Edge device, and between an IoT Edge device and the cloud.

Create an Azure Stream Analytics job to process data on the edge. Connect the new Azure Stream Analytics job with other IoT Edge modules. Deploy the Azure Stream Analytics job to an IoT Edge device from the Azure portal.

When you create an Azure Stream Analytics job to run on an IoT Edge device, it needs to be stored in a way that can be called from the device. You can use an existing Azure Storage account, or create a new one now.

Use the Append Variable activity to add a value to an existing array variable defined in a Data Factory or Synapse Analytics pipeline.

Tumbling window triggers are a type of trigger that fires at a periodic time interval from a specified start time, while retaining state. Tumbling windows are a series of fixed-sized, non-overlapping, and contiguous time intervals. A tumbling window trigger has a one-to-one relationship with a pipeline and can only reference a singular pipeline.

Data integration scenarios often require customers to trigger pipelines based on events happening in storage account, such as the arrival or deletion of a file in Azure Blob Storage account. Data Factory and Synapse pipelines natively integrate with Azure Event Grid, which lets you trigger pipelines on such events.

Azure Blob storage, Azure Data Lake Storage Gen2, and Azure SQL Database are currently supported as input sources for reference data.

Event Hubs, IoT Hub, Azure Data Lake Storage Gen2 and Blob storage are supported as data stream input sources.

A data stream is an unbounded sequence of events over time. Stream Analytics jobs must include at least one data stream input.

Reference data is either completely static or changes slowly. It is typically used to perform correlation and lookups.