The art of Data Analysis

Transforming volume to value

The communication speed between humans is limited to 10 bits per second. This rate shows how fast we can process and understand information.

Visual comprehension is not much quicker. Our brain cannot transform images into cognitive meaning way faster than 10 bit/s.

(source: Make it Clear by Patrick Henry Winston)

What is Data Analysis

In short, the goal of data analysis is to tweak raw data into useful information.

We clean, transform, visualize, and model data to discover something hidden in the raw mess. If we did a great job, we may form conclusions and make decisions with the support of our findings.

At the end of the day, it is some sort of communication. We want the data to communicate to us. And we may need to communicate findings to others.

Let's do some math

If we calculate with 10 bits/second:

• In a one-minute conversation, approximately 75 bytes of information is communicated.

• In an hour-long lecture, about 4,500 bytes (or roughly 4.4 kilobytes) of information is communicated.

• Assuming an average reading time of 6 hours for a book, this would amount to 27,000 bytes (or about 26.4 kilobytes) of information.

(This is what our brain can understand/interpret)

Imagine that you have a dataset and you need to do a presentation about it.

Your manager is too busy, so you only have 10 minutes to share your findings.

In a 10-minute presentation, at the human communication speed limit, we can communicate approximately 750 bytes of information.

It is not rare to work with 100Gb+ of data as an analyst. 100 gigabytes = 858,993,459,200 bits = 107,374,182,400 bytes.

100 GB is approximately 143,165,577 times larger than 750 bytes.

That's a huge gap between the amount of data we can analyze and the amount we can communicate. That gap needs to be filled by analysts.

Effective analysis is the art of compressing data to get through an incredibly small 10 bit/s bottleneck.

What can we take away?

Even if you work with Terabytes of data the goal should be to make the communication simple. Bring it down to the 10 bits/s level. Volume is not always = more value.

You cannot show all the nuances of a huge dataset. Always remember the speed at which we can process information. We are dumb, nowhere near close to the computer. Focus on what’s most relevant, and build a story around that.