Posts – data-in-flight

maestro 0.3.0

New methods for getting schedule status and return values from pipelines

R

data pipelines

orchestration

maestro

packages

release

maestro 0.3.0 introduces principled methods for obtaining a schedule table, pipeline status, and artifacts (i.e., return values). This is now possible because of a major…

maestro 0.2.0

New tags for specifying hours, days, and months for pipelines

R

data pipelines

orchestration

maestro

packages

release

maestro 0.2.0 brings with it new features for scheduling pipelines on specific hours, days, or months. With a few new tags, we have so much more versatility for scheduling.

The Data Decontamination Mindset

Why you should assume your organization’s data is radioactive

data cleaning

data engineering

ingestion

opinion

Data engineers deal with data from a variety of sources. Typically, you have very little control over these sources. Best case scenario is the data comes from an API that…

Deploying a Maestro Project to Google Cloud Platform

How to deploy a maestro project to GCP

R

data engineering

data pipelines

orchestration

maestro

deployment

cloud

tutorial

In the previous post we saw how maestro helps orchestrate data pipelines in a single R project. For maestro to be useful you need to deploy it somewhere and these days that…

Hello maestro

A new package for orchestrating data pipelines in R

R

data engineering

data pipelines

orchestration

maestro

packages

deployment

release

I’m thrilled to announce the release of maestro, a new package for developing, scheduling, and monitoring data pipelines. Here, I’ll walk through what maestro does and why…

Prepare for Takeoff

news

My new blog Data In Flight is about sharing my experiences working with real data on a small scale. In my experience I’ve found that exciting things happen when you have…