Why are you so slow?

Speed and performance of a data platform are often core business needs.

• Waiting for a business intelligence report to load.

• Paying for long-running data pipelines or queries.

• Enduring daily data refreshes instead of hourly.

If you are facing slow performance and delayed data delivery to business teams, here are some places to start.

Reasons why data platform performance suffers.

• Wrong technology.

  • NoSQL instead of SQL or Spark. OLTP instead of OLAP. Direct Query instead of Import. Or vice versa. Limited caching.

• Poor modeling.

  • Dimensional vs. relational. Aggregate vs. atomic grains. Mismanagement of business keys, surrogate keys, and source system keys.

• Poor workload isolation.

  • Mismanagement of Individual users, group reporting, or ETL workload permissions. One user’s query crowds out other key workloads slowing down teams of people or other high priority tasks.

These can be highly technical challenges and design decisions, but they are rooted in removing friction from business users' data experience.

The design that could have worked last month or last year, may no longer serve the business well.

Always be architecting.

Always be designing.

Always be modeling.

I’m here,

Sawyer

“Why even bother?”

Last month my wife and I spent an afternoon raking fall leaves in our back yard and on the side of our house.

However, we have a ton of large trees around our house and many of them still cling to their leaves. And so the next weekend, with some additional help from my dad, we raked leaves again. This time in the front yard.

The next morning, between the strong winds and rain that blew through, a heavy blanket of leaves once again coated both our front and back yard.

It looked exactly like it did before.

But we were more demoralized than before. Spending a few hours over a couple of weekends raking leaves, only to have the same amount of work present in our yard a few days later. I hopelessly look across my yard thinking “Why even bother?”

What surprised me is how much different this feels from other tasks. Throughout the summer I have to mow the grass every week. Yet I don’t feel the same disappointment or frustration when the grass gets longer.

Or other mundane tasks. Like washing dishes. Doing laundry. Or taking a shower. Every time you complete the task, it’s only a matter of days or hours before the task is waiting for you again. But, in contrast to raking leaves, I don’t bemoaningly wonder “Why even bother?”

There are two kinds of tasks. Terminal and Chronic.

We subconsciously assign every task to one of those two buckets.

Chronic tasks are serial events that we expect to perform over and over again. Mowing the lawn, doing the dishes, taking a shower. These tasks are a part of our routines and habits. The satisfaction is lower but the pain of accomplishing them is lower too.

Terminal tasks are infrequent or one-time events that we only expect to perform once. Stain the deck once every few years. Raking fall leaves once a year every November. You rest easy and enjoy a sense of satisfaction after completing what you think is a terminal task. Knowing you are done for a while. Terminal tasks have high completion satisfaction and high completion pain.

It’s frustrating when a task you thought was terminal is actually chronic. This is how it feels when leaves continue to fall after you’ve already raked the yard twice.

How you approach, budget, and plan for the work on your Data Team is highly dependent on whether you view the tasks as terminal or chronic.

...

Is data platform architecture a chronic or terminal task?

What about data modeling?

Dashboard development?

Gathering requirements from stakeholders?

...

If you view these tasks as terminal tasks then you will experience an increased level of frustration when you have to perform them again.

When you realize you need to model your data differently are you frustrated - “We already did that last year!”

Or gather more requirements from stakeholders - “That was completed last quarter”

Treating chronic tasks as terminal is short-sighted and morale-draining.

But if you realize they are chronic,

you are better equipped to last

for the long-term

Sawyer

Value creation

A small wording change can help.

Sometimes you find value.

Sometimes you deliver value.

But a more accurate way to describe what you and your data team do is

Create

See if this mindset helps you.

Someone who finds value is a guy on the beach with a metal detector.

Someone who delivers value is a delivery truck driver.

You find something that's already there. Or you move something from Point A to Point B.

If neither of those images appeals to you try this:

You can create value.

Create is something an entrepreneur, innovator, or visionary does.

No longer are you looking for something that already exists. You create something that's never existed.

No longer are you moving stuff from here to there. You create something to shape the future.

Look back at the work your team completed last quarter.

Rewrite the narrative in your head.

You didn’t just find or deliver.

You created.

It was good to see you today,

Sawyer

Data Platform Decision - Data Ideas Podcast

Last week I had the opportunity to join my friend Dustin on his Data Ideas podcast.

We discussed data platform architecture, the popularity of the cloud, deciding between data warehouses, databases, and data lakehouses, how to measure success as a data team, and some of my personal journey.

Listen in here:

​Apple Podcast​
​Spotify

I'm here,

Sawyer

Inclusive Data Architecture Choices

Many choices for your data platform aren’t exclusive.

This applies to Data Modeling

And Data Architecture.

Data Architecture Training material: “First, choose whether you want an Extract-Transform-Load (ETL) or an Extract-Load-Transform (ELT) data architecture”

Data Architect in the wild: “We currently have an ETLTLTTTL pattern but that might change next week”

ETL or ELT is not a single choice. It will likely be ETLLTL or ELTLTTL or some combination that no one has implemented yet.

Understand core frameworks for data platforms. Learn data modeling theory. Lean ETL/ELT patterns.

Then mix and match. Combine. Invert.

Implementing a pattern isn’t the aim.

Being invaluable to the business is.

I’m here,

Sawyer

A different type of choice

How you model your data is a core part of your data platform.

Many data modeling options sit before you.

You can view it as an exclusive choice.

I pick A or B.

I choose pattern X or Z.

...

Or you can view it as an inclusive choice.

I pick A and B

I chose patterns X and Z.

...

The industry vendors or gurus will beat the drum on exclusive choice.

“It’s star-schema or bust!”

“The Data Vault 2.0 is the most robust practice available”.

But Data Modeling is and not or. It’s an inclusive choice.

Your data architecture could very likely include:

• Data Vault and Kimball Star Schema

• Kimball Star Schema and OneBigTable

• Inmon Warehouse and Kimball Star Schema

• Data Vault and OneBigTable

Data modeling is not a one-time choice. Nor is it a single choice.

Your data model revolves around business objectives which means your model looks like and more than or.

I’m here,

Sawyer

from ​The Data Shop

What are my options?

Respect your stakeholders enough to give them options.

No one wants to be boxed in.

When a request comes in an immediate solution might come to mind. Resist the urge to respond with just that option. It might take some work, but spend some time thinking about three options to solve the problem.

There might be timeline options:

• I do this option in the next hour. This other option in the next week. Or this third option which will take a month.

They might be preference options

• I can do this in red, blue, or green.

They might be priority options:

• I can do this first, second, or third in relation to our other priorities.

If you want to understand your stakeholders better. And improve stakeholder satisfaction. And build trust.

Respect them by giving them options.

I can solve this problem with A, B, or C.

I’m here,

Sawyer

from ​The Data Shop

Four Seasons of A Data Team

In Michigan, we are turning the corner from fall to winter and our beautiful fall leaves have left the branches and covered everything in sight. I've been busy the last few weeks with a variety of outdoor tasks in preparation for winter.

Between raking an excessive amount of leaves, cleaning up door toys and projects, and pulling the ice melt and snow shovel out from hiding, I’ve been thinking about changing seasons. I love living in a part of the country that experiences all four seasons of the year in bold intensity. Each season offers a different rhythm, wardrobe, activities, pace of work, family experiences, and foods. Each has distinct memories and stirs a rainbow of emotions.

A data team goes through similar seasons.

Although a team likely won’t experience each of these seasons in one calendar year. The seasons could last for several months or even years. But identifying the phases you walk through, where you are right now, and where you are headed in the next few months focuses your attention.

Spring: This season is marked by new. New builds, new team members, new tools implemented, new data system, new leadership. Teams in this season will find themselves experiencing surprisingly warm days, followed by days of bitter cold, while slowly marching toward the new growth of spring flowers, green grass, and sunshine.

Summer: A season of adventure, exploration, and rewarding experiences. The newness that began to take root in the spring is reaching maturity and the benefits are being realized. The new technical tool is paying dividends. The new leadership has found stability, built trust with the team, and is seeing the results of the new vision and process in place.

Fall: A season of pruning, cutting back, and harvesting. The team in a “fall season” might be retiring legacy tools, reducing team size, or deprecating an on-prem system. Just as a tree shedding its leaves in the fall is part of a natural and healthy pattern, it's the same for a data team. Old is being set aside and, for the moment, nothing new is taking its place.

Winter: A season of maintenance and survival. Due to being short-staffed, limited budget, or organizational challenges, new initiatives are on pause. The team moves focuses on “keeping the lights on”. The team feels an overall lack of clarity about what’s next, what the new priorities will be or how long winter will last. Initially, it can feel like a respite, but over time, everyone begins to long for spring.

These seasons might last for weeks, months, or even years. They naturally follow each other in this order.

Where is your team at right now?

Where are you headed?

Sawyer

from The Data Shop

Vanity Metrics

Data leaders are tempted to chase all kinds of vanity metrics.

• Speed of pipelines.

• Amount of data.

• Number of cool technology logos.

• Team headcount.

They are shiny.

They go on resumes.

You can compare them against peers and competitors.

And boast about on Linkedin/X/Reddit.

These are vanity metrics.

They are fun to count. And there might be real reasons why you are tracking them. Perhaps a faster pipeline is worth the developer hours it will take to optimize it. Perhaps ingesting more data will improve a business team’s process. Maybe a few more headcount will allow you to invest in a key priority. Maybe that fancy Modern Data Stack tool will improve your processes around data transformation.

Tracking those metrics will only matter if they are helping you track something else. Something more important.

How valuable you are to the business.

That's a metric worth your time.

I’m here,

Sawyer

from ​The Data Shop

What if what they want is wrong?

What if what the business stakeholder wants is wrong?

Often strategy isn't clearly communicated across the org and so teams are working from different directives. Pulling in different directions. Operating on different assumptions.

Two possible approaches:

• Point to a shared goal: Sometimes you can talk it through with them, leverage the clearly identified goals from leadership, and perhaps end up in better alignment with your business stakeholders.

• Give them what they want, with caveats: Other times you might need to invest some time giving them what they want. But with a clear evaluation plan. Something like "We are willing to work toward delivering what you asked for, but before we do, can we define how we will know if it worked/was the right thing? That way we can revisit in a couple weeks/months and see if we hit the goals we wanted"

Who's right or wrong isn't the primary question.

Sometimes assumptions have to be tested.

Building trust through collaboration is often inefficient.

I'm here,

Sawyer

from ​The Data Shop

You never get to quit

What if you never get to quit?

Or retire.

Or sell.

We endure so much frustration, stress, and boredom in our work grasping at the idea of a two-day weekend, a week-long vacation, or an eventual retirement.

But what would you do with that frustration, stress, and boredom if you don’t get to quit? If there wasn’t an escape.

What kind of data team would that look like?

Build that.

My overt mission at The Data Shop is to help data teams become invaluable to the business.

The covert mission behind that front is to help data teams become places people don’t want to quit. Or escape. Because if the business matters to you, and you find a way to be invaluable to the business, well…that tends to generate energy, excitement, and fulfillment.

I’m here for the long haul.

And I’m here to make your work in data as great as possible.

For as long as possible.

I'm glad you're here,

Sawyer

Build The Right Thing > Build The Thing Right

Engineers and technical professionals are full of opinions on how to build technology.

• Microservices.

• Idempotent pipelines.

• Metadata-driven data ingestion.

• Bar charts instead of pie charts.

• Surrogate keys over business keys.

• Data Vault warehouse modeling.

• Trunk based development.

There is no end to this list.

The conversations on LinkedIn and at the local tech meetup swirl around debating how to build it right.

But most of you don't care about that.

You are aware of these conversations, but they aren't what keeps you up at night. You are stuck thinking about a harder and more important question.

How do we build the right thing?

Building the thing right is a vanity project if you haven't first established that you are building the right thing.

• How do we build something that will best help our stakeholders?

• How do we decide between the competing needs of the business?

• Which of these data products ideas will grow stale and unused and which will empower the marketing team for the next fiscal year?

You can figure out microservices, data modeling techniques, or visualization choices later.

It's the leader's task to first pick the right thing to build.

You aren't alone.

I’m here,

Sawyer

What’s valuable about data?

Data is valuable when it makes your business work well.

That means data is valuable when

…it describes what happened before.

…it offers visibility into what is happening now.

…it improves workflows.

…it shows you mistakes.

…it answers a question quickly

…it builds trust between you and your customers

…it surfaces risk.

…it quantifies risk.

…it removes risk.

…it enables leaders to plan next quarter

…it enhances understanding of a customer base.

Nothing on that list is technology. It’s business.

Your turn. What about your data makes your business work well?

I'm here,

Sawyer

from The Data Shop

Why is it valuable?

How much is this house worth?

3 bed 2 bath 1,800 square feet.

Not sure?

How about some more details?

Two-car garage. 1/5 acre lot. Walking distance to elementary schools

Still not sure?

Of course. It’s impossible to even guess without one more key piece of data.

Location.

The price could vary from $120k in ​Youngstown OH​

to $2.7 Million in ​Newport Beach, CA​.

One key feature makes one house worth 22x more than another house of the same size.

Without understanding that variable you might try adjusting other things to create value. Add another bathroom. Paint the exterior. Remodel the kitchen. Add 3 bedrooms in a major expansion project. But you will never get a house in Youngstown to be worth what is in Newport Beach.

People are happy to pay more to live near the beach in Southern California. Way more. This is so obvious to most of us. “Duh, Sawyer. Of course, California is a more expensive location than Ohio”

But it’s the difference between knowing THAT something is valuable and knowing WHY something is valuable is.

Only when you understand that location is the key variable, it’s far easier to define, design and create value.

If real estate and location seem obvious, try this with something else.

Google search is more valuable than Bing search. Why?

A hot dog at Yankee Stadium costs more than a hot dog at Costco. Why?

A Mercedes costs more than a Volvo. Why?

A band t-shirt at a concert costs 3x what a t-shirt of the same size and material does at Walmart. Why?

Being invaluable to the business means not just knowing what’s valuable.

But why.

Thanks for being here,

Sawyer

from ​The Data Shop

Tell them about the value

Knowing what’s valuable is step one.

Step two is communicating it.

It’s time for you to write quarterly updates to your boss.

It’s time to present the executive sponsor of the project with an update.

It’s time to present the client with a final read-out at the close of the project.

It’s time to pitch the board on an initiative that requires a large financial investment.

…

You can write about your input.

“I billed this many hours”

“I hired 5 new team members”

…

You can write about your outputs.

“I delivered a new web application”

“We built a new database for reporting”

…

Or you can write about outcomes.

“We decreased customer onboarding time by 35%”

“We improved the accuracy end of month financial reporting by 3%”

…

The first two require the reader to connect lots of dots to get to value.

The value of your work is only visible in the third one.

Connect the dots for your reader.

Don’t make them try to figure out the value you created by giving them a list of inputs and outputs.

I’m here,

Sawyer

from ​The Data Shop

“This will change your life”

“Let me tell you why this product is going to change your life” - Sleazy Salesperson

Or at least a suspicious salesperson.

Our skepticism creeps in:

What do they know about my life?

Why do they think I want to change something?

What’s wrong with what I already have?

The end result? A very unlikely sale.

We do this with our data products all the time.

“Let me tell you how this dashboard is going to make your job easier”

“Let me tell you why you will love this new business intelligence tool”

“Let me tell you how this data model will make analytics amazing for you”

The end result? A skeptical business stakeholder.

But

Wouldn’t it be great if you were so attuned to the needs of your stakeholders that when you delivered a data product they said TO YOU:

“Let me tell you why this dashboard is going to change my life”

“Let me tell you why I love this new business intelligence tool”

“Let me tell you how this data model is making analytics amazing for me”

Well, then you might really have something.

I’m here

Sawyer

from ​The Data Shop​

A bad desired outcome

You could have a very clear desired outcome.

And deliver that outcome.

Exceeding your stakeholder's expectations.

And still fail.

...

Because it was a bad desired outcome.

You grew profits and burned out your employees.

You pushed products and damaged your communities.

You accommodated your customers every need and went out of business.

...

Purdue Pharma expressed a desire to increase sales and product revenue. They hired McKinsey & Co, the world's most prestigious and influential consulting firm, to help achieve that outcome.

McKinsey crushed that goal and “Turbocharge” sales of one key product.

OxyContin.

McKinsey recently paid $600 million in settlement fees to 49 states because of their role in helping Purdue Pharma further opioid sales. The effects of achieving that outcome and the ensuing opioid crisis have been (and will continue to be) devastating.

Having a clear desired outcome matters.

Having the right desired outcome matters more.

I’m glad you are here,

Sawyer

from ​The Data Shop

Scary

It’s time for something scary (it’s October 31st after all).

I shuttered as I read this part of the Agile manifesto:

“Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.”

Terrifying, right?

That’s like saying “Rewrite the speech the night before the conference” when you learn your topic doesn’t fit the theme.

Or you need to enlarge all the doorways in a house just when the drywall was hung because you found out it needs to be handicap accessible.

Often it feels too scary. Too big of a delay. Too much stress.

We'd rather deliver the wrong thing, on schedule, as planned than welcome changing requirements.

Only when you are committed to outcomes and not activities or deliverables can you do this.

If you are scared or resistant to changing requirements, pause and ask yourself why.

If you ever need help welcoming

changing requirements

for your data team

I’m here,

Sawyer

Guessing at a desired outcome

When people aren’t given clarity about the desired outcome they make up their own.

They often make good guesses.

“We should make more money”

“We should make this run faster”

“We should automate this”

Like I said, good guesses.

Your team is smart. They will get it right some of the time.

But if the team had a shared and clear vision of the outcome it might look like this:

“We are discontinuing that product line”

“That process is legacy and won’t be used any more”

“This revenue stream holds the highest potential. We can’t invest over there anymore.”

“That service isn’t profitable. We are losing money on every sale.”

What looked like progress now looks like confusion.

Leaders don’t let their team guess at the goal.

I’m here,

Sawyer

from ​The Data Shop

We need you, leaders

Value is impossible without leadership.

We need you, leaders.

Managers, Directors, VPs and CxOs.

We are looking for you to tell us:

• Where are we going?

• What’s the desired future state?

• What’s most important for us to pursue?

• What are the measurements for when we will get there?

It’s on the leader's shoulders to answer these questions and communicate them to the team.

A failure to deliver business value is a failure of leadership.

The team is hungry to execute the vision.

To distribute benefits to the people and communities we engage with.

To produce evidence-able positive effects on the business.

Absent leadership, everyone swims in different directions.

A bug fix here.

A report here.

A database query there.

No one knows if they’ve hit value.

Or just staying busy.

I’m here,

Sawyer

from ​The Data Shop