Making some changes

This week we are talking about moving faster as a data team.

It’s your job to make changes, add new things and remove old things.

Reports, dashboards, database tables, python scripts, and ETL pipelines. You and your team will touch some or all of these objects every day.

But who touched what? When did that report get changed? Two people tried to change that at once, which change counts?

When you are a team of one (or a part-time data person), it’s easy to assume you can just keep track of everything on your own. Until someone asks you about a code change you made 6 months ago (or 6 weeks ago) and you can’t remember why that occurred. If you are like me, you’ve slept a few times since then and obviously forgotten everything.

But the pain points become clear as the complexity grows.

• Two team members and three reports.

• Five team members, two databases, and thirty-five dashboards.

• Three teams, a data lakehouse, ML models, and way too many dashboards.

This is why source control is fundamental for data teams.

At its most basic form, source control keeps version history of your files (like Sharepoint, Excel or Google Docs might do). But for software and data teams, source control allows multiple team members to work with multiple files at once in isolated environments, enabling them to integrate their changes together, while keeping a record of what changes occurred and when (with notes about the change). And roll back changes when needed.

One of the highest ROI activities for your team is taking a small step toward source control. This week,

• Review existing source control solutions. Do you have anything? Are they being used or ignored? What % of your code solutions do they cover?

• If you are starting from zero, pick one area of your work and find a source control solution. The quick and dirty solution is Sharepoint. But plan to pursue more robust solutions in the near future.

• Work with your team to revise or establish processes for integrating working with source control into their development work.

Then hit reply and tell me about your progress or barriers.

I’m here,

Sawyer

from The Data Shop