O’Reilly Report

Automating Data Transformations

In the past decade, the Modern Data Stack emerged, evolved, and enabled many organizations to ingest, store, and analyze more data than ever.

Automation enabled efficiency and scale across the data cycle, with one exception: data transformations.

To transform data, many organizations still use inflexible GUI tools or rely on manual code-first solutions that require the work of highly skilled engineers. Neither of those options scales.

In this report, we present a way to combine the flexibility of code-first with the ease-of-use of GUI to enable scale through automation.

You will learn:

  • How to optimize the transformation layer with Data Architecture as a Service (DAaaS) and Data as a Product (DaaP).
  • How using metadata at the column level enables automation and revolutionizes data transformations.
  • How to create trust in your data and data teams across your entire organization.


Table of Contents

  • Today’s Modern Data Stack
    • What Is the MDS?
    • Managing Data as a Product (DaaP)
    • Basic Terms and Concepts in the MDS
    • Automation in the MDS
  • A Renaissance in Data Transformation
    • Why Data Transformations Matter
    • Data Transformation: Existing Solutions
      • SQL Plus Orchestration Tooling
      • Code-First
      • GUI-First
    • Data Transformations: Finding the Golden Middle
      • Hybrid Approaches
  • Delivering Value with Data Transformations Through Automation
    • Principles of Data Value
      • Product-First
      • Column-First
    • Optimizing the Transformation Layer
      • Enabling Analytics at Scale
      • DAaaS
    • Culture Shift
      • Democratizing Data Transformation
      • Implementation

The truth about automation, repeated throughout history, is that it does not eliminate work. It merely shifts the type of work we do, often to something more challenging and enjoyable. If it now takes three analysts to build infrastructure instead of ten, the other seven will still have plenty to do—focusing on higher-level tasks like implementing new patterns, optimizing design, and tackling new creative problems that were previously consumed by manual process and day-to-day operations.

About the Authors

Satish Jayanthi
CTO and Co-Founder, Coalesce

As Co-Founder and CTO at Coalesce, Satish has designed and built the company’s data automation software. Prior, Satish was the Sr. Solutions Architect at WhereScape, a leading provider of data automation software, where he met his co-founder Armon.

Armon Petrossian
CEO and Co-Founder, Coalesce

Armon Petrossian is the CEO and Co-Founder of Coalesce. Prior to that, Armon was part of the founding team at WhereScape in North America, where he served as national sales manager for almost a decade.

Explore next-gen data transformations for yourself

Experience the power of Coalesce with a 30-day trial