Cart before horse

The analyst I described yesterday is a real-life scenario that I lived. It’s filled with invigorating highs (”They liked my analysis”), frustrating lows (” They want me to rebuild the analysis with changes”), and lots of murky waters in between (…trying to cobble together analysis from scratch each day).

There is another way.

Build a scalable, repeatable, maintainable, and automated data platform.

Scalable: The solution can serve all users who need it. All the relevant data is available (not just what fits in your pandas data frame).

Repeatable: The process exists as code and is persisted in a repository and can be repeated identically without fail. The computer repeats the process, not an analyst trying to remember the steps they wrote down.

Maintainable: When bugs happen there is a central place where the error can be fixed. The data, logic, and rules live in a central environment (central for your team, or maybe the whole company). It doesn't live on someone's laptop, private SharePoint, or in an email somewhere.

Automated: It's scheduled to run without manual intervention. No one needs to sign on at 9 pm on Friday night to deliver a report the CEO wants.

These are the building blocks for your data platform. Any successful analytics program takes each of them seriously.

Those with a software engineering background will laugh at the simplicity of the above list because software engineering is a far more mature discipline than the scrappy world of data, data analytics, and data engineering.

Analytics without a data platform is running on a hamster wheel.

Analytics without a data platform is putting the cart before the horse.

It was good to see you today,

Sawyer