Open Source Infrastructure

GitHub - spotify/luigi: Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.

GitHub - Netflix/metaflow: Build and manage real-life data science projects with ease!

GitHub - flyteorg/flyte: Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.

GitHub - dagster-io/dagster: An orchestration platform for the development, production, and observation of data assets.

GitHub - PrefectHQ/prefect: The easiest way to coordinate your dataflow

GitHub - apache/airflow: Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

Hydra / The Open Source Data Warehouse

GitHub - duckdb/duckdb: DuckDB is an in-process SQL OLAP Database Management System

GitHub - apache/doris: Apache Doris is an easy-to-use, high performance and unified analytics database.

GitHub - apache/kylin: Apache Kylin

GitHub - apache/pinot: Apache Pinot - A realtime distributed OLAP datastore

GitHub - apache/druid: Apache Druid: a high performance real-time analytics database.

GitHub - ClickHouse/ClickHouse: ClickHouse® is a free analytics DBMS for big data

GitHub - apache/hbase: Apache HBase

GitHub - edgedb/edgedb: A graph-relational database with declarative schema, built-in migration system, and a next-generation query language

GitHub - confluentinc/ksql: The database purpose-built for stream processing applications.

GitHub - milvus-io/milvus: Vector database for scalable similarity search and AI applications.

GitHub - apache/accumulo: Apache Accumulo

GitHub - OpenTSDB/opentsdb: A scalable, distributed Time Series Database.

GitHub - basho/riak: Riak is a decentralized datastore from Basho Technologies.

GitHub - apache/couchdb: Seamless multi-master syncing database with an intuitive HTTP/JSON API, designed for reliability

GitHub - apache/cassandra: Mirror of Apache Cassandra

GitHub - apache/flume: Mirror of Apache Flume

GitHub - Paradigm4/SciDB-Py: Python wrapper for SciDB queries

GitHub - influxdata/influxdb: Scalable datastore for metrics, events, and real-time analytics

GitHub - MariaDB/server: MariaDB server is a community developed fork of MySQL server. Started by core members of the original MySQL team, MariaDB actively works with outside developers to deliver the most featureful, stable, and sanely licensed open SQL server in the industry.

GitHub - cockroachdb/cockroach: CockroachDB - the open source, cloud-native distributed SQL database.

GitHub - redis/redis: Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs, Bitmaps.

GitHub - mongodb/mongo: The MongoDB Database

GitHub - mysql/mysql-server: MySQL Server, the world's most popular open source database, and MySQL Cluster, a real-time, open source transactional database.