DP-203 References

DP-203 References

113 bookmarks
Newest
Delegate access with a shared access signature - Azure Storage
Delegate access with a shared access signature - Azure Storage
A shared access signature (SAS) is a URI that grants restricted access rights to Azure Storage resources. You can provide a shared access signature to clients who should not be trusted with your storage account key when you want to delegate access to certain storage account resources.
A user delegation SAS, introduced with version 2018-11-09. A user delegation SAS is secured with Azure AD credentials. This type of SAS is supported for the Blob service only and can be used to grant access to containers and blobs.
·docs.microsoft.com·
Delegate access with a shared access signature - Azure Storage
Managed identities for Azure resources
Managed identities for Azure resources
An overview of the managed identities for Azure resources.
Managed identities provide an identity for applications to use when connecting to resources that support Azure Active Directory (Azure AD) authentication. Applications may use the managed identity to obtain Azure AD tokens. With Azure Key Vault, developers can use managed identities to access resources. Key Vault stores credentials in a secure manner and gives access to storage accounts.
a managed identity is a service principal of a special type that can only be used with Azure resources. When the managed identity is deleted, the corresponding service principal is automatically removed.
·docs.microsoft.com·
Managed identities for Azure resources
Column-level security for dedicated SQL pool - Azure Synapse Analytics
Column-level security for dedicated SQL pool - Azure Synapse Analytics
Column-Level Security allows customers to control access to database table columns based on the user's execution context or group membership, simplifying the design and coding of security in your application, and allowing you to implement restrictions on column access.
Column-Level security allows customers to control access to table columns based on the user's execution context or group membership.
·docs.microsoft.com·
Column-level security for dedicated SQL pool - Azure Synapse Analytics
What is Azure role-based access control (Azure RBAC)?
What is Azure role-based access control (Azure RBAC)?
Get an overview of Azure role-based access control (Azure RBAC). Use role assignments to control access to Azure resources.
Azure role-based access control (Azure RBAC) helps you manage who has access to Azure resources, what they can do with those resources, and what areas they have access to.
·docs.microsoft.com·
What is Azure role-based access control (Azure RBAC)?
Grant limited access to data with shared access signatures (SAS) - Azure Storage
Grant limited access to data with shared access signatures (SAS) - Azure Storage
Learn about using shared access signatures (SAS) to delegate access to Azure Storage resources, including blobs, queues, tables, and files.
A shared access signature (SAS) provides secure delegated access to resources in your storage account. With a SAS, you have granular control over how a client can access your data.
three types of shared access signatures
A user delegation SAS is secured with Azure Active Directory (Azure AD) credentials and also by the permissions specified for the SAS. A user delegation SAS applies to Blob storage only.
A service SAS is secured with the storage account key. A service SAS delegates access to a resource in only one of the Azure Storage services: Blob storage, Queue storage, Table storage, or Azure Files.
An account SAS is secured with the storage account key. An account SAS delegates access to resources in one or more of the storage services. All of the operations available via a service or user delegation SAS are also available via an account SAS.
·docs.microsoft.com·
Grant limited access to data with shared access signatures (SAS) - Azure Storage
Incrementally copy data using Change Tracking using Azure portal - Azure Data Factory
Incrementally copy data using Change Tracking using Azure portal - Azure Data Factory
In this tutorial, you create an Azure Data Factory with a pipeline that loads delta data based on change tracking information in the source database in Azure SQL Database to an Azure blob storage.
In some cases, the changed data within a period in your source data store can be easily to sliced up (for example, LastModifyTime, CreationTime). In some cases, there is no explicit way to identify the delta data from last time you processed the data. The Change Tracking technology supported by data stores such as Azure SQL Database and SQL Server can be used to identify the delta data.
SYS_CHANGE_VERSION
·docs.microsoft.com·
Incrementally copy data using Change Tracking using Azure portal - Azure Data Factory
Create tumbling window trigger dependencies - Azure Data Factory & Azure Synapse
Create tumbling window trigger dependencies - Azure Data Factory & Azure Synapse
Learn how to create dependency on a tumbling window trigger in Azure Data Factory and Synapse Analytics.
Provide a value in time span format and both negative and positive offsets are allowed. This property is mandatory if the trigger is depending on itself and in all other cases it is optional. Self-dependency should always be a negative offset. If no value specified, the window is the same as the trigger itself.
·docs.microsoft.com·
Create tumbling window trigger dependencies - Azure Data Factory & Azure Synapse
Surrogate key transformation in mapping data flow - Azure Data Factory & Azure Synapse
Surrogate key transformation in mapping data flow - Azure Data Factory & Azure Synapse
Learn how to use the mapping data flow Surrogate Key Transformation to generate sequential key values in Azure Data Factory and Synapse Analytics.
Use the surrogate key transformation to add an incrementing key value to each row of data. This is useful when designing dimension tables in a star schema analytical data model. In a star schema, each member in your dimension tables requires a unique key that is a non-business key.
·docs.microsoft.com·
Surrogate key transformation in mapping data flow - Azure Data Factory & Azure Synapse
Sink transformation in mapping data flow - Azure Data Factory & Azure Synapse
Sink transformation in mapping data flow - Azure Data Factory & Azure Synapse
Learn how to configure a sink transformation in mapping data flow.
A cache sink is when a data flow writes data into the Spark cache instead of a data store. In mapping data flows, you can reference this data within the same flow many times using a cache lookup. This is useful when you want to reference data as part of an expression but don't want to explicitly join the columns to it. Common examples where a cache sink can help are looking up a max value on a data store and matching error codes to an error message database.
·docs.microsoft.com·
Sink transformation in mapping data flow - Azure Data Factory & Azure Synapse
Assert data transformation in mapping data flow - Azure Data Factory
Assert data transformation in mapping data flow - Azure Data Factory
Set assertions for mapping data flows
The assert transformation enables you to build custom rules inside your mapping data flows for data quality and data validation. You can build rules that will determine whether values meet an expected value domain. Additionally, you can build rules that check for row uniqueness. The assert transformation will help to determine if each row in your data meets a set of criteria. The assert transformation also allows you to set custom error messages when data validation rules are not met.
·docs.microsoft.com·
Assert data transformation in mapping data flow - Azure Data Factory
Alter row transformation in mapping data flow - Azure Data Factory & Azure Synapse
Alter row transformation in mapping data flow - Azure Data Factory & Azure Synapse
How to update database target using the alter row transformation in the mapping data flow in Azure Data Factory and Azure Synapse Analytics pipelines.
Use the Alter Row transformation to set insert, delete, update, and upsert policies on rows. You can add one-to-many conditions as expressions. These conditions should be specified in order of priority, as each row will be marked with the policy corresponding to the first-matching expression. Each of those conditions can result in a row (or rows) being inserted, updated, deleted, or upserted. Alter Row can produce both DDL & DML actions against your database.
·docs.microsoft.com·
Alter row transformation in mapping data flow - Azure Data Factory & Azure Synapse
Designing tables - Azure Synapse Analytics
Designing tables - Azure Synapse Analytics
Introduction to designing tables using dedicated SQL pool.
A hash distributed table distributes rows based on the value in the distribution column. A hash distributed table is designed to achieve high performance for queries on large tables.
A replicated table has a full copy of the table available on every Compute node. Queries run fast on replicated tables since joins on replicated tables don't require data movement. Replication requires extra storage, though, and isn't practical for large tables.
A round-robin table distributes table rows evenly across all distributions. The rows are distributed randomly. Loading data into a round-robin table is fast. Keep in mind that queries can require more data movement than the other distribution methods.
Staging Use round-robin for the staging table.
Dimension Use replicated for smaller tables. If tables are too large to store on each Compute node, use hash-distributed.
Fact Use hash-distribution with clustered columnstore index. Performance improves when two hash tables are joined on the same distribution column.
By default, dedicated SQL pool stores a table as a clustered columnstore index. This form of data storage achieves high data compression and query performance on large tables.
A heap table can be especially useful for loading transient data, such as a staging table which is transformed into a final table.
An external table points to data located in Azure Storage blob or Azure Data Lake Store. When used with the CREATE TABLE AS SELECT statement, selecting from an external table imports data into dedicated SQL pool.
A temporary table only exists for the duration of the session. You can use a temporary table to prevent other users from seeing temporary results and also to reduce the need for cleanup.
·docs.microsoft.com·
Designing tables - Azure Synapse Analytics
Real-time data visualization of data from Azure IoT Hub – Power BI
Real-time data visualization of data from Azure IoT Hub – Power BI
Use Power BI to visualize temperature and humidity data that is collected from the sensor and sent to your Azure IoT hub.
Create a consumer group on your IoT hub. Create and configure an Azure Stream Analytics job to read temperature telemetry from your consumer group and send it to Power BI. Create a report of the temperature data in Power BI and share it to the web.
·docs.microsoft.com·
Real-time data visualization of data from Azure IoT Hub – Power BI
Incrementally copy data using Change Tracking using PowerShell - Azure Data Factory
Incrementally copy data using Change Tracking using PowerShell - Azure Data Factory
In this tutorial, you create an Azure Data Factory pipeline that copies delta data incrementally from multiple tables in a SQL Server database to Azure SQL Database.
You perform the following steps in this tutorial: Prepare the source data store Create a data factory. Create linked services. Create source, sink, and change tracking datasets. Create, run, and monitor the full copy pipeline Add or update data in the source table Create, run, and monitor the incremental copy pipeline
·docs.microsoft.com·
Incrementally copy data using Change Tracking using PowerShell - Azure Data Factory
Azure Stream Analytics on IoT Edge
Azure Stream Analytics on IoT Edge
Create edge jobs in Azure Stream Analytics and deploy them to devices running Azure IoT Edge.
A cloud part that is responsible for the job definition: users define inputs, output, query, and other settings, such as out of order events, in the cloud.
A module running on your IoT devices. The module contains the Stream Analytics engine and receives the job definition from the cloud.
Supported stream input types are: Edge Hub Event Hub IoT Hub Supported stream output types are: Edge Hub SQL Database Event Hub Blob Storage/ADLS Gen2
For both inputs and outputs, CSV and JSON formats are supported.
Manufacturing safety systems must respond to operational data with ultra-low latency. With Stream Analytics on IoT Edge, you can analyze sensor data in near real-time, and issue commands when you detect anomalies to stop a machine or trigger alerts.
Mission critical systems, such as remote mining equipment, connected vessels, or offshore drilling, need to analyze and react to data even when cloud connectivity is intermittent.
·docs.microsoft.com·
Azure Stream Analytics on IoT Edge
What is Azure IoT Edge
What is Azure IoT Edge
Overview of the Azure IoT Edge service
Azure IoT Edge moves cloud analytics and custom business logic to devices so that your organization can focus on business insights instead of data management.
Azure IoT Edge allows you to deploy complex event processing, machine learning, image recognition, and other high value AI without writing it in-house. A
Installs and update workloads on the device. Maintains Azure IoT Edge security standards on the device. Ensures that IoT Edge modules are always running. Reports module health to the cloud for remote monitoring. Manages communication between downstream leaf devices and an IoT Edge device, between modules on an IoT Edge device, and between an IoT Edge device and the cloud.
·docs.microsoft.com·
What is Azure IoT Edge
Tutorial - Stream Analytics at the edge using Azure IoT Edge
Tutorial - Stream Analytics at the edge using Azure IoT Edge
In this tutorial, you deploy Azure Stream Analytics as a module to an IoT Edge device
Create an Azure Stream Analytics job to process data on the edge. Connect the new Azure Stream Analytics job with other IoT Edge modules. Deploy the Azure Stream Analytics job to an IoT Edge device from the Azure portal.
When you create an Azure Stream Analytics job to run on an IoT Edge device, it needs to be stored in a way that can be called from the device. You can use an existing Azure Storage account, or create a new one now.
·docs.microsoft.com·
Tutorial - Stream Analytics at the edge using Azure IoT Edge