The Data Daily
Less than 2 minutes to read each morning.
Not sure if you want in? Read the archives below.
5 days a week since May 1st, 2023.
Smooth and fast
Do your agile ceremonies help you ship data products faster?
Great. Keep going.
Do your agile ceremonies slow you down?
Then stop it.
Don't have agile processes but feel like you are moving slowly?
It might be time to start.
Processes should make you smooth and fast.
If it’s creating friction then it’s causing exhaustion on your team and disappointment among your stakeholders.
I’m here,
Sawyer
from The Data Shop
p.s. Don’t know if it’s making you faster or slower? It’s time to measure before and after to find out.
Everything is a nail
As an early career data professional, I thought data was a magic potion to solve all the problems of the business, politics, education, and climate.
Thats how the courses and hype-fluencers sold data to me.
“Data is everywhere”. “Data is the new oil”. “Data can answer all of our questions.” “Data & AI can 10x your business!”
I viewed data as a hammer. And every problem in the world is a nail. And I need to hit them all with data.
The more experienced you are in the data space the more aware of the limitations of data. Data is messy. Ambiguous. Inconclusive.
It’s only part of the equation. Or maybe not even helpful until much later on in the process. Humans get in the way so often.
The more comfortable you are with what data can and cannot do or what problems it can and cannot solve, the better equipped you are to help your organization.
When you are honest about the limitations of data, you are more likely to gain the trust of your stakeholders.
Moreso, and counterintuitively, limitations and constraints create far more freedom and creativity.
Sure, you can think of data as a hammer. But opening your eyes to see uses for screwdrivers, sandpaper, bandsaws, planes, and chisels makes the hammer even better.
I’m here,
Sawyer
from The Data Shop
A superpower for you
Asking good questions is a superpower.
Projects fail because of poor requirements.
Partnerships fail because of a lack of understanding and interest.
User testing fails because use cases were ignored or overlooked.
The dashboard collects cobwebs because the business problem isn’t solved.
The sale fell through because core needs were missed.
The point where we fail is often the point we…
stopped asking questions.
asked lazy questions.
skipped follow-up questions.
stopped listening to responses.
assumed we knew their answer already.
The simplest superpower you can cultivate. That doesn’t require X-ray vision, a flying suit of iron, a shield of Vibranium, or powers from another universe.
Is caring enough to keep asking questions.
Until you understand. And the other person knows you understand.
Sawyer
from The Data Shop
What's so hard about data architecture and business intelligence?
Did you miss the Data Roundtable Live Stream from yesterday? You can catch the replay here.
(Because it's on replay you can even watch it on 1.5x or 2x. I'm not offended)
With my friends Ahmad and James we talked about:
Data Architecture basics
What's changed in Data Architecture over the last 20 years
What are some tactics for understanding Power BI performance bottlenecks?
How has Microsoft Fabric changed the game for data professionals in BI or Architecture?
What are the core challenges that data leaders face?
and more...
It was good to see you today,
Sawyer
from The Data Shop
Nothing good comes from winter.
Pre-s: The Data Roundtable - Free Webinar Livestream is today on LinkedIn at 12/3 pm ET. We are chatting about Data Architecture, Data Leadership, and Business Intelligence in addition to taking questions from attendees. Join us!
----------
It’s winter in my homeland of Michigan. For 4-5 months a year, it’s cold and dead. When there’s no snow on the ground (like most of this winter), the world is a stale brown as all the plants go dormant. The sun disappears for longer at night and hides behind clouds for most of the days.
Nothing good could come from winter.
But if you wander into the trees behind my house - you will need warm boots this time of year - and navigate down the south side of the lot you will run into a thick maple tree tucked next to the path. Journey a bit further and you will find more maples. There are dozens of small maple samplings, but it’s the mature ones you should pay attention to.
The maples look just as brown and lifeless as the rest of the forest and ground foliage, yet something hides beneath the surface. “Tapping” a maple tree involves drilling a small hole into the side of a mature trunk and inserting a metal spile. Pay attention, you might see maple sap begin to drip out before you’ve even finished drilling the hole.
We collect the sap into buckets and boil the sap down into a very highly concentrated liquid that we celebrate as maple syrup. We only tap 12 trees although there are plenty more around. Despite how dead and dry everything looks in the winter, a dozen maple trees produce too much sap for us to boil and store. There’s more than enough to go around. An abundance. With dozens more maple samplings growing up each year.
Here’s the thing.
The maple sap only “runs” and drips out of the taps because of the winter. It’s the hard freezes and traps the water and gasses into the center of the tree. When the temperature drifts above freezing during the day time (usually right about this time of year) those gasses in the center of the tree expand and push the sweet sap out of the tree.
Better have your tap ready. It’s worth collecting. It’s sweetest and best to collect it before the trees begin budding for spring.
Nothing good could come from winter?
Sometimes winter is the only way something good appears. During the winter there might be far more happening beneath the surface than you realize. And if you wait until you see evidence of spring, you might miss it.
You might be in the winter season. Either personally, or professionally. Your data team might be in the winter season.
Beauty can come from winter. You just need to know where to look.
I’m here,
Sawyer
from The Data Shop
How quickly can you recover from mistakes?
Pre-s: The Data Roundtable - Free Webinar Livestream tomorrow on LinkedIn at 12/3 pm ET. We are chatting about Data Architecture, Data Leadership, and Business Intelligence in addition to taking questions from attendees. Join us!
--------
Recently I made a mistake in a daily email.
Not a typo. Those happen all the time. But a material mistake that was important. I was announcing the launch of The Technical and Strategic Data Leader course and linking to the registration page. It’s the biggest launch I’ve done with The Data Shop. But I messed up the link to the main landing page. Not once, but twice in the email. Consequentially, the first people who got the email and clicked the link were sent to a “preview” link.
Within 20 minutes someone alerted me to the issue (thanks, Cristobal!). I was confused about why the mistake was there, but then I reviewed my steps in preparing the email announcement and realized my mistake. Thankfully, I was able to make some updates and have the links working correctly after a few more minutes.
Mistakes will happen on your data team. It’s not if, but when.
Part of your work as a data leader is preparing for mistakes. How to diagnose them. How to resolve them. And how to minimize their occurrence in the future.
A dashboard is showing incorrect data.
A pipeline failed due to an incorrect parameter value.
A report fails to update due to a schema change in the source.
Users are unable to access the database because of credential changes
Do you have a history of changes that you can review to see when the error was introduced?
What was the process for reviewing the pull request?
Is there a data quality check in place for that use case?
What were the table counts before and after that pipeline run? Do we have monitoring of those jobs?
And, most importantly, how quickly can you deploy a fix to production?
These are the core questions a great data team can answer.
They have CI/CD, source control, data quality processes, etc.
They have processes to handle mistakes.
Great data teams aren't surprised by mistakes.
They expect mistakes to happen.
Are you ready for the next mistake?
I’m here,
Sawyer
from The Data Shop
Human connection
Online communication is noisier than ever.
Personal connection is rarer than ever.
This is true in your professional life and personal life.
Your teammates are overwhelmed by impersonal content distribution and an inhuman culture of profits and layoffs.
Your career seems to hang on the arbitrary AI reviewing your resume.
Your friends and family are longing for a technology-free connection and someone to care about their ideas, dreams, and emotions.
I enjoy limited pieces of social media (mostly LinkedIn). But my main goal is to move past the noise of public broadcasts to see the humans behind that platform.
I write this daily email for you. Not to add to the noise in your life, but hopefully to spark creativity and connection.
And in 2024 I’m focused on creating spaces for humans to connect. For me at The Data Shop, that means data people.
Three things are coming up that I need to tell you about. Spaces that allow us to move beyond broadcasts and more human connection.
1. The Data Roundtable: Free Livestream Tuesday, February 27th at 12 pm PT / 3 pm ET
Along with James Serra and Ahmad Chamy we are answering your questions about Data Architecture, Data Leadership, and Business Intelligence. Streaming live on Linkedin. Link to follow next week.
2. Free Office Hours: March 4th, 11 am PT / 2 pm ET
*** Open to email subscriber list only ***
My virtual office doors are open today for an hour if you have any questions on anything data. Or if you just want to talk about your spring break plans, tapping maple trees, remote work, or the 2024 Presidential Election (JK, not going there), I’m here for that too.
Drop in whenever and stay for however long you’d like.
There’s no catch. No sales pitch. No hard feelings if no one comes.
Just trying to help where help might be needed.
Link to follow.
3. The Technical and Strategic Data Leader - Beginning March 12th
A six-week cohort-based learning experience for Data Leaders, Data Architects, or those responsible for data at their company. This experience is highly interactive between the course mentors and participants. Expect to actively engage with your peers throughout the course.
I hope to connect with some of you at one of these three events.
Sawyer
from The Data Shop
Don’t try to explain it to me
Tax season is starting in the USA.
You might be hiring an accountant to do taxes for you.
If you do here are two things we expect from an accountant.
Taxes are filed with as little hassle as possible from us.
Taxes filed with us paying as little tax as is legally possible.
That’s it.
I don’t care…
What forms the accountant fills out.
What deductions they choose to take.
What tax laws have changed since last year.
How city, state, and federal returns are different.
I expect the accountant to know all that, handle all that and please don’t try to explain it to me.
Or expect me to make decisions about it.
-----------------------------------------------
When a business team comes to you with a data request here are two things they expect from you.
That the data will be delivered with as little hassle from them as possible.
That it will be accurate and meet their needs in the most effective way.
That’s it.
They don’t care…
What Python library you use
What data quality checks are in place
How you had to redesign the pipeline to accommodate a new data source.
How bad the previous data engineer messed everything up and you had to fix it.
They expect you to know all that. And to make the best decisions related to it so they can get the results they need.
I’m glad you are here,
Sawyer
from The Data Shop
Music and data
You know your music genre.
The genre that just checks all the boxes for you. It might be nostalgic, inspiring, relaxing, or energizing. Or all of those things at once.
You know award shows, the history, the terminology, and the genre-defining artists.
How jazz found its roots in New Orleans over a hundred years ago in the African American communities.
Or nuances of baroque, classical, and romantic genres that developed in Europe over the last few hundred years.
What’s your music genre?
You know your data genre as well.
That genre of data that you built your career on. The patterns, styles, and technologies feel familiar and energizing for you.
You know the open source projects, the thought leaders, the terminology, and the industry-defining moments over the last few decades.
Genres like streaming analytics. Batch data warehousing. Web analytics. Business intelligence. Data governance. MLOps.
What’s your data genre?
The innovators in music are those who borrow and integrate from different genres. A hip-hop artist is inspired by jazz. A rock musician is inspired by classical music. The folk artists embracing elements of techno.
People like Ray Charles, Bob Dylan, and Johnny Cash.
Same with data.
The innovators in data are those who expand beyond their safe and comfortable data genre. They ideate on streaming patterns with data warehouses. They explore the flexibility of web analytics when they build data governance patterns.
When you are comfortable with your data genre, it takes some courage to step outside the box and start innovating.
It’s what the data world needs.
It was great to see you today,
Sawyer
from The Data Shop
The layoffs
I spoke with a friend a couple of weeks ago who’s a manager at his company. Leadership told him to compile a list of names for layoffs—around 25% of his team.
“I have to find the people most crucial to our team first. The people we can’t live without”. He told me.
It’s a brutal situation for everyone involved. And while companies vary a lot in how they determine layoff criteria it always boils down to some form of “we can survive without these people” and “we can’t survive without these people”.
As you hear constant echoes around your industry, or as you weather layoffs at your own company, take a minute to return to first principles.
Paying people for inputs is expensive. Logging time. Lots of effort. Doesn’t move the objectives.
Paying people for outputs is ambiguous. Lots of story points delivered. Plenty of lines of code. But does it matter?
Paying people for outcomes is rational. It’s the only thing that justifies that salary.
It’s possible your inputs contribute to outcomes.
It’s possible your outputs further the desired outcomes.
But if you’d rather not leave leadership wondering about what outcome you contributed, then don’t leave it in the realm of “it’s possible”.
I’m glad you are here,
Sawyer
from The Data Shop
The salesperson
You might think you are a data engineer, data analyst, or Director of Data.
But you might also be a salesperson.
It’s a common three-step sales pattern. See a need. Design a solution that meets the need. Then sell the solution. Invite the buyer into it. Call to action.
“Frustrated by salt and snow muck covering your car in the winter? Come through our car wash”
“Looking for ways to improve your physical flexibility? Join me for this fitness class each Wednesday morning.”
“Hoping to retire sometime before you move into the retirement home? Call me for a financial retirement analysis”
Your organization has dozens - nah hundreds - of problems, needs, and opportunities.
Many won’t be solved by data.
But the astute data team is watching closely for the ones that data can speak to. Can improve. Can reduce. Can maximize. Can influence.
And then focuses efforts on crafting data solutions to that problem, need, or opportunity.
Once that beautiful data product, mapped precisely to that aching business problem, is ready then you get to sell. It doesn’t have to be sleezy or cheesy.
If you are meeting a real problem in the business, your sales pitch about this data solution will be ever-so-welcome.
If you build what they want. They will…ahem…want it.
When you build the right thing? Selling is much easier. When you don’t know the problem, opportunity or need? Selling the data solution is much like hiking through mud.
Embrace the seller mindset.
And enjoy all the benefits of offering a solution that meets many people’s needs.
It was good to see you today,
Sawyer
from The Data Shop
In that order
Pre-s: 2024 is moving fast. Our first cohort of The Technical and Strategic Data Leader kicks off next month. The early bird discount ends today. Register today with code EARLYBIRD250 to save $250.
------------------------------------
Problems and opportunities are tackled in this order:
People -> processes -> tools.
Like a building.
People are always the foundation.
Processes are the structure and frame.
Tools are the electrical, plumbing, and finishing touches.
As a leader, you will often see problems or opportunities. Something that requires you to execute something new or change course. Always start in this order.
In what ways are our people creating, limiting or empowering this opportunity or problem?
How are our processes creating, limiting or empowering this opportunity or problem?
What role do our tools play in creating, limiting or empowering this opportunity or problem?
Fixing your tool stack first is like changing the light fixture in a house with a crumbling foundation.
I’m here,
Sawyer
from The Data Shop
The YouTube comment
Last week our garage door keypad stopped working. It’s one of those boxes that mount outside your garage and, with a four-digit code, opens your garage door. Very handy for foot traffic like kids coming home from school. The keypad was flashing all sorts of lights when you punched in the code, but the garage door didn’t budge.
I started with the owner's manual and attempted to resync the outside keypad with the garage door lift mounted on the ceiling of my garage. It’s a 6 step process to reset the code. One step is identifying an opener ID from a list of 12 different models, styles, and manufacturing dates listed. After staring at my model for a while, I guessed that my model was ID 4.
Then I walked through the steps to reset the keypad pin. Nothing. I tried again. And again. Each time I looked closer at the instructions unsure if I was misreading a key step.
By the 6th time, I pulled up YouTube. The most popular video was 2 minutes long and walked through the steps I had been doing (correctly). I was at a loss. And frustrated. I don’t have the time, energy, or patience for this (mostly patience).
Then I glanced at the top-voted comment below the video.
“If you have a yellow button then your model ID is 12”.
I could have hugged that commentor (Thanks, yezwhat87! You are a true hero).
I went through the steps again using ID 8 and it worked perfectly.
Here’s the thing.
Your leaders are swimming in information. Inundated with data, content, opinions, perspectives, and instructions. They have crucial responsibilities. And they don’t have the time, patience, or energy to mine through potential permutations or potential scenarios to predict results. They can’t scour the spreadsheets and dashboards looking for a crucial piece of information.
They need the key piece of data. Pinned as the top comment.
That’s the job of a data team. To deliver the right data, at the right moment, in the right way.
Just like YouTube commenter yezwhat87!
Sawyer
from The Data Shop
Leading is lonely.
Leading is lonely.
When you are on a team, you have friends, peers, and colleagues with whom you stand shoulder to shoulder.
When you lead, there is no one at your shoulder.
When you are on a team, it’s easier to share struggles, and frustrations, and ask for help. Your shoulder-mates welcome it.
When you lead, you are prone to swallow your struggles and frustrations. Should you feel safe enough to ask for help, the path for finding it is opaque.
When you are on a team, the dreams, vision, and strategy are laid out for you to embrace and execute.
When you lead, you carry the burden of staring into the future to envision something new. And crafting a compelling narrative that others will follow.
Whether you lead a data team of 4 or are a leader of data leaders, the challenge of loneliness is the same.
Today might feel great.
Tomorrow might not.
I’ll be here in your inbox on both of those days.
Hit reply anytime,
Sawyer
from The Data Shop
Sawyer
p.s. In leadership and looking for a space to collaborate and commiserate with industry peers? Join our intensive 6-week course on data leadership. Details here.
Sunk cost bias - Data Edition
The sunk cost bias describes humans' poor ability to make rational decisions when we’ve invested meaningful time, energy, or money into something.
We make irrational decisions because we don’t want to “waste” the time, energy, and money we’ve invested, and so we continue on.
Even when abandoning the activity would better serve them.
Sunk cost bias is fundamentally a backward-looking problem. It betrays how heavily humans rely on the past to make decisions about the future.
I need to stick with this career path because I’ve spent x years and $y on education.
As bored as I am with this book, I should finish it because I bought it for $15 and spent a few hours already reading it.
The question we miss is this: Is the future of this path still the one I want?
This shows up in data teams all the time.
We invested thousands of dollars in on-prem servers a couple of years ago. Despite how good moving to the cloud would be for our business, we don’t want to waste our investment.
I’ve spent three days trying to get this code to work. I found an easier framework to use, but it would require rewriting the whole thing and I don’t want to waste the time I spent over the last three days.
We sent out the whole team to get training on XYZ Shiny new data tool. It would be a shame to waste the thousands of dollars we spent on training, even though now it seems like the tool may not be the best fit for us.
Reducing the effects of sunk cost bias (I doubt you can ever eliminate it) requires a resilient forward-looking commitment. It’s never too late to change course.
What is the best next step now?
I’m here,
Sawyer
from The Data Shop
p.s. It can be really hard to see your own sunk cost bias. Inviting outside voices into the equation is often the most effective option.
Chickens and Eggs
My friend Rob is a backyard chicken farmer. He lives in the heart of the city neighborhood with a backyard you can mow in 10 minutes. But he found room for chickens.
Last month we nursed a beverage together and talked about why he raises his chickens.
“Sawyer, brown eggs taste so much better”, he told me.
“Last month I accidentally bought brown eggs at the grocery store instead of the usual white eggs” I replied. “I noticed no difference whatsoever. Except that I paid more”.
“Dude, you just don’t get it”.
And he’s right I don’t get it.
And there it became clear. Rob values the taste, aesthetics, and experience of raising, collecting, and eating brown eggs.
I don’t.
What is valuable can vary dramatically between people.
And between business units, organizations, and companies.
Real-time streaming data ingestion? Incredibly valuable for one team. Complete waste of money for another.
All historical data in hot storage and available for on-demand queries? In demand daily at one company. Ignored and costly at another.
Not only does value vary. It also has a steep cliff. When a few factors change, the same idea, product, or design goes from invaluable to worthless.
You will only know what’s valuable by deeply understanding your customers, stakeholders, or leadership.
Your team needs to be designed to quickly adapt to deliver what’s valuable to business.
What you read on LinkedIn, in Harvard Business Review, or in a Medium blog post about “business value” has to be adopted for your specific context.
If you need help on the journey to value,
I’m here,
Sawyer
Confidence
Your confidence and certainty in a decision will always be a sliding scale.
Somewhere between terrified and cocky.
One side represents immense uncertainty and fear about the likely outcome.
The other side comes in with an overabundance of confidence and certainty.
Either extreme can doom you as a data leader.
You are constantly making decisions.
Who to hire, fire, and promote.
Technology tools to buy, trial, or deprecate.
Architecture design to implement, adjust, or retire.
Stakeholders request to prioritize, reduce scope, or reject.
How to hire consultants and contractors and for what scope of work.
Too much confidence and you ignore the inherent uncertainty and “unknowables” in every decision
Too much fear and you under-index on what you can know and the cost required to reverse the choice.
Three questions to ask moderate or increase your confidence in a decision.
What is the impact of this decision? Deciding what to order for lunch and deciding whether to have kids have dramatically different impacts on your life.
What would it cost (in time, money, stress) to reverse this decision? Changing a project management tool takes less effort than changing data warehouse technologies.
What do I not know about this decision? What parts of it are out of my control? Think about all the third parties who will influence what happens after the decision is made. Software vendors, contractors, stockholders, and senior team members.
Lead well my friends.
The data team needs you.
Sawyer
Sawyer
from The Data Shop
You wouldn't...
You wouldn’t give a big presentation without running a few dry runs through your slide deck and material.
You wouldn’t try out a recipe for the first time when you have an important guest coming over for dinner.
You wouldn’t buy a car site unseen and without a test drive.
You wouldn’t even buy a widget on Amazon without reading a few reviews.
You wouldn’t hire a VP, manager, or junior developer without several rounds of interviews.
You wouldn’t pick a paint color for your living room without analyzing paint chips and probably painting a few test spots.
-----------
And yet, many, many data teams make bug fixes and code changes directly in production.
They think it’s easier. Quicker. Perhaps they think the fallout from a mistake isn’t that bad.
Or they don’t know a better way. It’s too cumbersome to have separate environments.
“CI/CD” gives you shivers.
There is a better way. It’s not easier to skip this stuff. And the side effects of prod failures are worse than you think.
Great data teams don’t skip the foundation.
Or everything else crumbles.
I’m here,
Sawyer
from The Data Shop
Hidden Measures
Psss.... did you hear? Yesterday The Technical and Strategic Data Leader launched. It's a six-week (10+ hours of live content) cohort-based learning experience for Data Leaders, Data Architects, or those responsible for data at their company. Starting March, 12th 2024. Registration details.
Now to your regularly scheduled daily email:
------------------------
Everybody is happiest when they know how to win.
Nobody has ever argued over who won the Super Bowl.
But college football has more than a dozen (depending on who is counting) years where who won the national championship is debatable.
While it can lead to long conversations between fans and commentators, nobody thinks not knowing who the champion is is a good thing. It’s frustrating for fans, players, coaches, and administrators (just ask the 2023 Florida State University Football team)
A data project of any size should have clear rules for knowing when you’ve won.
Nobody should be arguing over if the project was a success.
Without clear expectations of what “winning” looks like, everybody will intuitively come up with their own ideas about what a championship means to them. Getting rid of that old system feels like a win to one of the data engineers. Retaining the exact look, feel, and useability of a report is a win for operations. Under budget is a win for finance. On-time (or ahead of schedule) is a win for the project manager.
At the end of the project, everyone walks away with different ideas about who won, or if the project was a success.
The measures for success were hidden. Everybody held them tightly in their heads - and assumed everyone else felt the same way.
Without the whole team agreeing on what “winning” looks like.
You are plowing rocky soil and planting crops riddled with weeds.
I’m here,
Sawyer
from The Data Shop
I’m not a plumber
Pre-s: I built something for you. Been working on it for a few months. Been thinking about it for years. I'm very excited to share it with you.
It's launching Monday, Feb 5th right here in your inbox at 8 am ET. See you then.
----------------
“The business doesn’t understand data”
“My stakeholders don’t understand how complicated data work is”
This is the sentiment that shows up in my inbox.
And I always ask: “Why does that matter?”
I don’t understand plumbing.
I don’t understand how complicated an electrician's work is.
But I do know that I want hot water, working toilets, and lights that work when I flip the switch.
No electrician or plumber has shown up at my house trying to explain to me the details of how they do their job.
Neither have they asked my opinion about what types of wires to use, where to put the wires, or what size pipes to use. Partly because I wouldn’t know what to say. But mainly because I don’t care.
The business knows what it wants. Running water. An outlet to plug their phone into. And a kitchen sink to wash dishes in.
If you start asking them where to run the pipes, how much voltage they should use, or the type of tools you should use, you are guaranteed to have a problem.
It was good to see you today,
Sawyer
from The Data Shop