This post demonstrates how to extend the metadata contained in the Data Catalog with profiling information calculated with an Apache Spark application based on the Amazon Deequ library running on an EMR cluster. You can query the Data Catalog using the AWS CLI. You can also build a reporting system with Athena and Amazon QuickSight to query and visualize the data stored in Amazon S3.

Thoughts on packaging python libraries — Note This is about packaging libraries, not applications. ⸻ All the advice here is implemented in a project template (with full support for C...

Devart Excel Add-in for PostgreSQL allows connecting Microsoft Excel to PostgreSQL data, import data from Excel to PostgreSQL, edit these data and save them back to PostgreSQL.

This is the unofficial documentation for Obsidian plugin development.

Obsidian: A knowledge base that works on local Markdown files.

If the original data is spread out in a number of files, we can read them in parallel (on multi-core) machines to save time. Ideally. However, it seems a bit tricky with python.

Python Flask tutorial showing IntelliSense, debugging, and code navigation support in Visual Studio Code, the best Python IDE.

Obsidian is an amazing tool for developer and developer advocates. You can use it as a second brain, for content creation, and more.

This thoroughly revised guide demonstrates how the flexibility of the command line can help you become a more efficient and productive data scientist. You’ll learn how to combine small yet powerful command-line tools to quickly obtain, scrub, explore, and model your data. To get you started, author Jeroen Janssens provides a Docker image packed with over 100 Unix power tools—useful whether you work with Windows, macOS, or Linux.

This tip is focused on the concepts of using Git Workflow with SQL Database Projects.

I was tinkering with the idea of finding the right way to help others to identify the resources that give you bang for the buck when it comes to upskilling yourself in data engineering… and this is what I came up with. The best data engineering courses, learning materials, data engineering training.

Introduction

Many organizations are benefiting from the performance gains and ease of administration of the Amazon Aurora database platform. The idea of breaking free from commercial licenses, backup administration, and data center maintenance is always a welcome thought. But where to begin, and how complex is the journey to migrate from SQL Server to Amazon Aurora […]

This article reviews the differences between stored procedures and functions in Postgres and the types of functionality they provide.

In this article, we explain the concept of Stored Procedures, and how to write them effectively in PostgreSQL 9.6. PostgreSQL is an open source database

The book that teaches SQL to developers: Learn to replace thousands of lines of code with simple queries!

The book that teaches SQL to developers: Learn to replace thousands of lines of code with simple queries!

Business logic is supposed to be the part of the application where you deal with customer or user facing decisions and computations. It is often argued that this part should be well separated from the rest of the technical infrastructure of your code. Of course, SQL and relational database design is meant to support your business cases (or user stories), so then we can ask ourselves if SQL should be part of your business logic implementation. Or actually, how much of your business logic should be SQL?

Part I. Tutorial Welcome to the PostgreSQL Tutorial. The following few chapters are intended to give a simple introduction to PostgreSQL, relational …

Learn PostgreSQL quickly through a practical PostgreSQL tutorial designed for database administrators and application developers.

We here are very big fans of Postgres as a database and believe it is often the best database for the job. For many though, working with and maintaining Postgres involves a steep learning curve. This guide is designed as an aid for beginners and experienced users to find specific tips and explore tools available within Postgres. If you want to take the quick & painless approach instead, there are great services available for Postgres hosting.

One common misconception about DataOps is that it is just DevOps applied to data analytics. While a little semantically misleading, the…

Nobody wants to look at a dashboard or make decisions with yesterday’s data. We live in a world where real-time information is a…

Learn how to harness the power of Obsidian to create your connected-thoughts library. Make Obsidian your personal idea engine

Obsidian Publish!

Data engineering makes data more useful and accessible by sourcing & preparing the data to be handled by data scientists. Learn more about the responsibilities.