Getting it on the table.
This week we are talking about how your data team can move faster. So far we’ve covered source control and environment strategy.
Today we are looking at how you deliver data products to your data consumers.
All the prep work, detailed configuration changes, optimizations, enhanced visualizations, and new metrics somehow have to make it into the hands of the users.
It’s about delivery.
Moving food from the kitchen to a guest’s table is the process of delivery. It requires integrating food from different parts of the kitchen (protein, vegetables, starch, etc.) onto the same plate. Then that plate goes through a series of checks before it’s delivered to the table. The nerds among us will call this Continuous Integration (CI) and Continuous Deployment (CD).
When you only have one element of your solution - perhaps an Excel file - then the integration part is easy. There’s nothing else to integrate.
But nearly all of your data solutions will include multiple parts, which means changes to one could impact several other parts. A process to test and integrate changes to these solutions in a designated location is crucial. Did that pipeline update break our BI report? What about the view definition change? I wonder if changing this database table name will disrupt anything?
In addition to the constant integration of new changes (every couple hours or at least once a day), your team needs to regularly ship that code. Land it in the hands of users.
That’s deployment/delivery. I’ve worked with some data teams where deploying new changes is an hours or even days-long process and they only deploy once a month or so. The best data teams deploy multiple times a day.
The simplest form of CI/CD is refreshing the data in an Excel file and emailing it to a business user. But most solutions are more complex than that.
Your team needs to invest time and energy into this area if you ever want to scale, deliver quickly, or ensure data quality.
If you need help talking through this process for your team…
I’m here,
Sawyer
