Small data

People love talking about “Big Data”.

But most data engineers or analysts will never need to think about datasets larger than a terabyte.

It’s easy to get infatuated with reading blogs about how Uber, Meta, or Linkedin builds scalable data systems and machine learning models on petabytes of data.

Design best practices for big tech companies rarely translate well to the middle and small markets. How does Stripe’s streaming pipeline with millions of events a second relate to your nightly batch data jobs?

It doesn’t.

If you are in the middle or small market, try this instead:

• Talk with peers at similar-sized companies doing it well.

• Go to conferences with similar-sized or companies are sharing their use cases and success.

• Engage consultants and specialists who have designed numerous solutions at your size and scale.

Your company can be remarkably successful without ever needing to scale a data system like TikTok.

I’m here,

Sawyer