Today in any organization, they are looking for ways to leverage their data and considering developing a data platform. Hence they are looking for tools and systems to help build such platform.

I did a thought process of researching about the components of a data platform and it pointed me to the following:

• Data Governance
• Extract Transform Tools (ETL) or Task Management
• Data Warehousing
• Business Analytics and Dashboarding
• Data Annotation and Quality Assurance

Here are the following tools which I found during my search of doing the exercise.

These cover cloud and open source systems which maybe useful to anyone doing their research.

Data governance:

• Cloud
  • Azure Purview: https://azure.microsoft.com/en-us/products/purview
  • Google Dataplex: https://cloud.google.com/dataplex
  • aws lake formation: https://docs.aws.amazon.com/lake-formation/latest/dg/how-it-works.html
• Open Source:
  • Apache Atlas: https://atlas.apache.org/#/
  • Amunsden: https://www.amundsen.io/amundsen/
  • Datahub: https://datahubproject.io/

ETL

• Cloud
  • Azure Data Factory: https://azure.microsoft.com/en-us/products/data-factory/
  • Amazon Data pipeline: https://aws.amazon.com/datapipeline/
  • Google Cloud fusion: https://cloud.google.com/data-fusion/
• Open source:
  • Airbyte: https://airbyte.com/
  • Apache Nifi: https://nifi.apache.org/

Data Warehousing

• Cloud
  • Amazon Redshift: https://aws.amazon.com/redshift/features/?nc=sn&loc=2&dn=1
  • Azure Synapse Analytics: https://learn.microsoft.com/en-us/azure/synapse-analytics/overview-what-is
  • Google Big Query: https://cloud.google.com/bigquery/docs/introduction
• Open source:
  • Dremio: https://www.dremio.com/
  • Apache Hive: https://hive.apache.org/
  • Apache Druid: https://druid.apache.org/

Business Analytics and Dashboarding

• Cloud
  • PowerBI: https://powerbi.microsoft.com/en-us/guidedtour/power-platform/power-bi/1/1
  • GCP Looker: https://cloud.google.com/looker/
  • Amazon Quicksight: https://aws.amazon.com/quicksight/
• Open Source
  • Superset: https://superset.apache.org/
  • Metabase: https://www.metabase.com/

Data Annotation and Quality Assurance

• Open Source
  • Label Studio: https://labelstud.io/
  • Cleanlab: https://github.com/cleanlab/cleanlab

Geospatial

• Open Source
  • Sedona: https://github.com/apache/sedona

Some of the tools above overlap with other tools. It is up to the individuals to do their research in setting up these tools.

---