All the IDEs have built-in tab completions and chat interfaces to discuss the codebase with AI. Some others were built around AI coding entirely. You don’t even need a text editor anymore, just a chat interface, and AI will edit all the code files for you.

Vibecoding is a hot topic today, and people have started to use it even in their work or production code.

But where does it lead?

If you tried AI coding, you probably noticed that it is really hard to go back. You actually need to think again. Humans are lazy. We don’t want to think anymore. It’s so comfortable to “prompt code” and note “problem solve code”.

And whenever you must go back to general coding, it feels so painful. We got addicted to AI, and we have withdrawal symptoms without it. It’s like going to the toilet without your phone. You cannot poop if you don’t scroll TikTok.

I need to use the documentation even for the basic syntax. I could maybe code a basic for loop, but I don’t trust my brain anymore. AI degraded my skills, so it’s time to do something.

I wrote a post about slowing down, but it is really hard to slow down today. Here is what I plan to do

How to fight against AI degrading our skills

No AI coding sessions

In the morning, work on a project without AI. Switch off tab completion, no AI chat, or for hard mode, switch on airplane mode. That’s elite-level OG coding. Force yourself not to use AI.

You must understand your code, and it forces you to problem-solve again. This makes coding fun again.

Start small, build hobby projects, read the documentation, or even Stack Overflow. (Yes, that site still exists.) You almost need to relearn coding.

Coding challenges without AI

If you don’t want to start an entire hobby project, just go to a challenge site like HackerRank or LeetCode. These are great sites for quick practice problems where you can get back into the habit of no AI coding.

You don’t even need to change your regular coding environment to switch off AI. Their built-in coding editor works perfectly for Non-AI practice.

I just found Advent of Code, and I will try to solve the challenges myself to learn.

Type out the AI response

This is the middle ground. Using AI, but no copy-paste allowed.

Pasting AI output without reading it is the real problem. If you type it out, you force yourself to read it, and while typing, you can understand the code. It’s not problem solving, but at least the code understanding part is there.

If you only listen to English podcasts and read in English, your speaking will not develop. Same with coding. You actually need to “problem-solve code” to develop real skills. But understanding is still better than nothing.

I am sure you know at least one person who is never busy. They have regular breaks away from the keyboard, they do only a few meetings a week, they never work overtime, and they are happy to go with you for a quick chat over a coffee.

Probably the output of the 2 person above is the same, or even the second person is more effective.

Busyness is a personality thing, and 9 out of 10 busy people are just unorganised. Of course, it can be affected by company culture, but even within the same environment, I have met both personality types. Also, everyone has busy periods, and sometimes you cannot avoid them. But generally, being busy is lame.

I think people should aim to be part of the never-busy group.

Being busy

The busy people usually have their own great mantras for why they are so busy.

“The management will notice how hard I try.”

“I am so motivated to be promoted, so I work a lot.”

“Let’s organize a call so we can go into the deep details and understand the process.”

“I am a maximalist, and everything needs to be perfect.”

“I have no time to rest, the deadline is so close.”

Let me tell you this:

• Management only cares about the outputs and KPIs.

• This person will be the first to burn out.

• ‘Deep understanding’ of the process usually leads to overcomplicating things. More below.

• Perfect means balance in many aspects. Busy people usually hunt perfect in a few aspects and sacrifice others.

• A 20-minute break will not necessarily result in a 20-minute delay. Great ideas can come in great breaks.

Busy people are also usually loud. They want to be visible, and they want to show how much they do. But their loudness is just noise and not actual important buzz. They want to be part of every conversation because they like to hear about things, but most of the time, they cannot add to the conversation.

They tend to micromanage. They want to know everything, but they have no time for the real details, so they know nothing. They see only the surface and usually have no value to add.

They are busy because they chose to be so. Being busy can be a decision for most people, or at least everyone should have some sort of influence on how busy they are. Being busy is not equal to being better in your job. It will have many consequences that are usually not positive. Busy people usually sacrifice work-life balance, rest, and are more stressed and harder to work with.

Busy people are present in the most calm environment as well, but the hustle culture can multiply their bad habits.

Being not so busy

Less busy people are not less busy because they are not motivated or lazy. In fact, they are usually less busy because they already took the time to learn stuff (like automation, the tools, time management, etc., see below) and chose to be less busy.

People are usually less busy because:

They optimize for output and not bullshit. Planning and discussing projects is a must, but when you spend more time in meetings than actually implementing, the process is flawed.

They automate their processes. When you have automations working for you, you have more time for other things, simple.

They know what tools to use for a given task. For a simple data analysis, Excel will do the job. For multi-line text editing or list comparison, use VS Code. You don’t need the fancy, shiny tool to be effective at work.

They spend time learning the tools they work with. Learn the keyboard shortcuts, learn the basic functions in Excel, spend some time on YouTube, and watch how experienced people use the tools you also use.

They organize their days effectively. Time-management is also a skill. Calendar boxing and organized to-do lists can help to be organized. Find something that fits your needs and personality. The internet is full of productivity tips.

They are quiet, but they are good listeners. They focus on the results, and they are successful because of them and not because of the noise they cause.

They know how to allocate resources and assign tasks. The broad picture is enough for them to add clear instructions that lead to a calm, but productive environment.

Subscribe now

Hotkeys

I use my numpad as a macropad. It allows me to assign specific numbers to specific apps. Windows has this function built in, but it is not as reliable as AHK. Crazy how a scripting tool can interact with Windows better than Windows itself.

OpenPBI() 
{
    if WinExist("ahk_exe PBIDesktop.exe") {
        winactivate
    }
    else
        Run("C:\Program Files\Microsoft Power BI Desktop\bin\PBIDesktop.exe")
}

OpenCode() 
{
    if WinExist("ahk_exe Code.exe") {
        winactivate
    }
    else
        Run("Code.exe")
}

<^Numpad2:: OpenPBI()
<^Numpad6:: OpenCode()

I like to organize AHK scripts according to the above structure, having functions and hotkeys separated. You could have a hotkey → script, but that is not as easy to search through.

I have 2 functions here, OpenPBI is the first. This function checks if I have a Power BI instance open, and if I do, it will activate the window. If I don’t have one open, then it will run the exe for me.

This function alone is not a working code; I need to assign it to a hotkey. In this case, it is Left Ctrl + Numpad2. The other function does the same thing with VS Code. Previously, I was a big Alt + Tab guy, but that changed with AHK. It makes it super easy to navigate between apps, and it is smooth as well. In AHK scripts ^ = Ctrl

OpenEdgeNewWindow() {
    if WinExist("ahk_exe msedge.exe")
    {
        WinActivate
        Send("^t")
    }
    else
    {
        Run("C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe")
        Send("^t")
    }
}

<^!Numpad1:: OpenEdgeNewWindow()

The above one is an extended version of the previous code. It will open Edge, but also send the keyboard keys Ctrl + t. Ctrl + t opens a new window in most browsers. So if I press Left Ctrl + Alt + Numpad1, a new browser window will open. In AHK scripts ! = Alt.

SearchSQLBI()
{
    Send("+!{Right}")
    Sleep(10)
    Send("^c")
    Sleep(100)
    Run("https://dax.guide/" . A_Clipboard)
}

^!Numpad5:: SearchSQLBI()

If you code DAX a lot, the above can be super useful, since in the built-in PBI editor, there is no great documentation for the functions. The code will select the word currently under the cursor, and it will open the DAX guide with the given word in the link. I click into the function name, hit the shortcut, and boom, I have the docs right in front of me:

Another great one is the DAX Code Formatter shortcut I have:

^+f::  ; Ctrl + Shift + F
{
    try {
        ; Copy selected text
        Send("^c")
        Sleep(200)  ; Wait for clipboard to update

        daxCode := A_Clipboard

        ; URL encode the DAX code
        encodedDax := UriEncode(daxCode)

        ; Build the full URL
        url := "https://www.daxformatter.com/?embed=1&l=long&s=true&fx=" . encodedDax

        ; Open in Browser
        Run url
    }
}

; Function to URL-encode text
UriEncode(str) {
    static chars := "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz-_.~"
    out := ""
    Loop Parse, str
    {
        ch := A_LoopField
        if InStr(chars, ch)
            out .= ch
        else
            out .= "%" . Format("{:02X}", Ord(ch))
    }
    return out
}

I select the DAX code, hit the shortcut, and it will open the DAX Formatter page with the code. I just need to copy-paste the result.

Extend functionality with a GUI

I usually need to transform text into uppercase, lowercase or title case. Here is a script I use to make it faster:

Edittext()
{
    MyMenu := Menu()
    MyMenu.add("Uppercase", Menuhandler)
    MyMenu.add("Lowercase", Menuhandler)
    MyMenu.add("Title", Menuhandler)
    MyMenu.Show()
}

Menuhandler(Item, *) {
    A_Clipboard :=""
    Send "^c"
    ClipWait

    switch Item{
        case "Uppercase":
            SendText(StrUpper(A_Clipboard))
        case "Lowercase":
            SendText(StrLower(A_Clipboard))
        case "Title":
            SendText(StrTitle(A_Clipboard))
    }
}

I select the text, use the shortcut, and a GUI will pop up so I can choose what type of transformation I want:

Built-in GUI for AHK is dynamite. You can extend these functionalities a lot. I hope this was a great inspiration, and you might find something useful in these. Let me know if you did so and have some other ideas for automation.

Subscribe now

Is there a better feeling than automating repetitive tasks? I freaking love it. This got me into programming, and to this date, that gives me the most joy. Knowing that I never need to do the same boring thing is dynamite.

I work on Windows, and I use it on my personal computer as well. I know... I am not a real computer enthusiast until I move to a proper Linux distro, but that will probably come sooner or later.

Until then, I am really grateful for AutoHotkey. The day I installed it on my work profile, my life changed. It is so powerful that every user who cares about time-saving should use it. Super easy to create beginner-friendly scripts. It is a great way to learn programming as well.

In the coming posts, I will share some scripts I have, starting with hotstrings.

Let me know if you see room for improvement or how you use automation for everyday tasks.

Hotstrings

Let’s start with an example:

::btw::by the way

The abbreviation btw will be automatically replaced with "by the way" when I trigger it with an ending character such as Space.

Using the logic above, you can create shortcuts to your name, email, or even longer strings that you type many times, like some general email templates.

::_mail::Johndoe@gmail.com
::_name::John Doe

::text1::
(
Any text between the top and bottom parentheses is treated literally.
By default, the hard carriage return (Enter) between the previous line and this one is also preserved.
    By default, the indentation (tab) to the left of this line is preserved.
)

Basically, Hotstrings are simple text replacement tools. They are easy to create and super fast with AHK.

I also have a few SQL-specific hotstrings:

::sel::SELECT
::seldis::SELECT DISTINCT
::sdis::SELECT DISTINCT
::cw::CASE WHEN
::wh::WHERE 
::gr::GROUP BY
::gb::GROUP BY

You can add your most-used tables and filters. Most SQL editors provide this functionality, but AHK is super fast, and the hotstrings are easy to define, so I would always default to it instead of creating these in the SQL editor.

Modifiers and Hotif

You can define hotstrings with several modifiers and their combination:

C - Case-sensitive trigger
* - Trigger even without an ending character
? - Trigger inside other words
O - Omit the ending character

There is also another awesome functionality, the Hotif modifier.

You can wrap your hotstrings (and hotkeys) between Hotifs, and your code becomes context sensitive.

For example, the below will limit my hotstrings to Power BI. This feature allows me to define the same hotstrings for different apps with different actions.

Here is the example that combines the hotifs and modifiers:

#Hotif winactive("ahk_exe PBIDesktop.exe")

:O:cal::CALCULATE(){Left}
:O:cr::COUNTROWS(){Left}
:O:ret::RETURN{Shift down}{Enter}{Shift up}
:CO:var::VAR _
:CO:VAR::VAR _
::sel::SELECTEDVALUE(){Left}
::sv::SELECTEDVALUE(){Left}
::kf::KEEPFILTERS(){Left}
::rf::REMOVEFILTERS(){Left}
::treatval::TREATAS(VALUES( )){Left}{Left}
::swtrue::SWITCH(TRUE(), ){Left}{Shift down}{Enter}{Shift up}

#Hotif

Another feature present in this code is the key input action. For example, when I type cal in Power BI and trigger the action with a space, AHK will “type” CALCULATE() then it will simulate the Left keypress to go into the brackets.

Autocorrect

Hopefully, you now see how hotstrings can change the way you code or type in general, but the biggest improvement AHK hotstrings can provide is a unique autocorrect script that can be adjusted to address your frequent typos.

Here is a great repository that already has thousands of corrections, and you can easily extend the list. It has a built-in GUI as well, so you can quickly add some words or substrings.

https://github.com/kunkel321/AutoCorrect2

This script is genius, it works in the background seamlessly, and is quick and efficient.

Hotstrings are just the entry-level stuff that AHK provides. Next time, I will share some Hotkeys that I use with more advanced scripts.

Now I feel like it is a time for a comeback.

What happened?

I just had a super busy year on the work and personal front as well, but I also just wanted to be away from content.

Probably most of you are here because you followed me on Twitter. That was my main platform before it became a shitshow. That is where I felt at home, and that was really my style: Short content packed with visuals.

Moving to Substack was hard. Even though it is a nice platform with a lot of potential and great creators all around, it’s needless to say that it is different. In my “blogging“ journey, I always created threads, short tweets, and now I am staring at a blank wall that waits for me to fill it with endless words. No character limits, just the empty editor.

When I write here, I feel that the content should be long and well-edited, like in a book. Twitter was loose and comfy, while Substack feels like the elite club, and I need to be well-dressed to belong here.

I didn’t like it originally, but now I see why this place might still be a good fit for me.

I learned to slow down. Twitter is also like TikTok, a constant fast-paced dopamine hit. That’s how I consumed and created content. I ran 365 sprints a year with my daily content. I thought creating on Substack is the ultimate marathon. The truth is probably in the middle. While it’s a “longer distance” to publish something here, it’s not a full marathon. And I feel way more comfortable running those distances than I did a year before.

We are living in an extremely fast-paced world. Tech is changing with every blink, AI is faster than a Ferrari, and slowing down here feels strange. But I must do it, I feel like. Running super fast all the time will not train you to run the longer distances.

How did I arrive here?

I work a lot in Power BI these days, and I needed to deepen my DAX knowledge. The best resources for that are the Definitive Guide to DAX Book and the SQLBI articles in general. And if you work with Power BI, it might not surprise you, but advanced DAX is haaaard. Every problem will hit you with a new challenge, and when you feel you learned something, it will just keep you grounded and humble. I need to reread articles and chapters all the time, and with each reread, I notice something new.

The authors make it very clear in the intro that this should be the way to do it:

The journey of learning DAX taught me that vibe coding is not the way. Programming in general requires you to master the basics and cannot jump right into the advanced stuff. No quick free lunch in this field, and writing good content might be the same.

What will come?

Previously, my content defined my days; now my days will define the content.

I will not push content if I have nothing to tell, just to send something out. I will not force myself to do things just because that’s great content.

We will still learn together, but I will write about things or projects that I am working on. Last year, I was too lazy, so it will also be some sort of motivation for me.

I want to learn about D3.js, Django, RAG, Dash, so many things on my list, but it will not be a rush. I will do some projects while learning and document the journey. If you are here for it and want to follow, then great, thank you. But if you don’t want to see it, please unsubscribe.

Let's explore the options for making the output numbers more readable and easier to work with.

The user experience

Look at this:

15315633662

That is fifteen billion, three hundred fifteen million, six hundred thirty-three thousand, six hundred sixty-two. (Thank you ChatGPT)

To make it easier to read, here are some separators added: 15,315,633,662

But that is still a "hard to read" number. At this scale, the last few digits are probably not that relevant to many users. That's why we usually see numbers like:

15.32 billion

15.3B

15,316 million

Depending on how we want to communicate and round our number, all the above can present the same long value we have.

To achieve an output like the above, you can use Babel.

It is a Python library that assists in internationalizing and localizing applications.

To format numbers, you can use the format_compact_decimal method:

That looks way better now.

Your experience

To make formatting even more confusing, here are two numbers that can mean the same thing:

1,099.98

1.099,98

Note the commas and dots. The first is a US standard. It uses a comma for thousands and a dot for decimals. But in the EU, it is usually the other way around. For example, in Germany, they use a dot for thousands and a comma for decimals.

You can use Babel to resolve this confusion:

• parse_decimal('1,099.98', locale='en_US') → converts it to 1099.98 as a float.

• parse_decimal('1.099,98', locale='de') → also converts to 1099.98, respecting the German format.

Now, imagine a situation when you need to work with files that are coming in from two different standards. Date formatting, currency formatting, and time zones... Pain in the neck for sure.

Babel has some sort of solution to all the above scenarios.

I was working on a project that required pretty number outputs, and my research led to Babel. Easy solution, would recommend.

Feature engineering is when we modify or create columns or variables in our data.

You can use it in simple analysis as well, and you should. Here is why:

(At the end, I will show how I used it in my latest analysis)

Data is usually limited

We always say that more (quality) data is better. But data availability is usually limited. Sometimes it is intentional so we don’t store “unnecessary” info, or sometimes the source has gaps.

In either case, feature engineering can help you a lot. If data has gaps, you can fill them, and feature engineering is a must. In other cases, feature engineering can add some spice to the analysis.

The power of feature engineering

If you see a dataset that has only a few columns, don’t think immediately that it is not good for analysis. From a few useful columns, you can make (engineer) a lot of helpful features.

Here are some ideas for this tiny table:

From timestamp:

• hour, day, weekday, month, is_weekend

• time_of_day (e.g. morning/afternoon/evening/night)

• session_duration (time diff between user activities)

• days_since_last_activity

• is_peak_hour (based on business-defined hours)

From amount:

• amount_bucket (e.g. low/medium/high spender)

• relative_amount (compared to the user’s average)

• rolling_avg_amount (per user over a period)

• spending_growth (compared to the previous spend)

Then, from these new columns, you can go even further, combine 2-3 columns, and build up features like a tree.

You have a lot of options to play with your data:

• Create new features:

  • From date_of_birth, calculate age

• Encode:

  • Convert education_level into ordinal values (e.g. HS=1, BSc=2, MSc=3)

• Scale, normalize

• Bin:

  • Categorize transaction_amount into “low”, “medium”, “high”

  • Group age into buckets: “18–25”, “26–35”, etc.

• Extract:

  • From full_name, extract first_name

  • From timestamp, extract weekday

• Aggregate:

  • Count number_of_purchases per user

  • Calculate average_order_value per customer

• Text manipulation

• Handle missing data:

  • Use a binary flag column income_missing to indicate NAs

The more features you have in the raw data, the easier feature engineering is. You can exponentially increase the number of features, and with that, the value of your analysis.

To be good at feature engineering, you must have great domain knowledge. Everyone can subtract two dates from each other, but the real heroes create features a rookie would never think about.

Never underestimate a basic table, since with feature engineering, you can bring it to life!

A real example

I am working with the subscribers’ data from Substack.

One feature I added is the engagement score:

I used only existing columns from the original df, but with this new feature, I can do a lot more interesting analysis.

Substack uses the ISO 2 codes to describe a country, but Plotly likes ISO 3 codes.

Here is a function to create the data I need:

Now I can use it with plotly to create charts like this:

And here are a few more:

1. Get the day of week from the subscription_date

2. Get the mail provider

3. Get the number of days since the user opened the last mail

The data Substack provides is meh, but with a bit of feature engineering, we can level it up.

Identifiability spectrum

When we anonymize data, we want it to be different from the original format, so it is not easy to recognize or identify. But it has levels. Some anonymization techniques will only make it harder to recognize the original data, while others make it impossible.

For example, if you know the stock ticker of a company is ‘GOOGL’, then you know that we talk about Google (or in this case, the company behind it: Alphabet Inc.). This ticker is a direct ID.

Indirect IDs are like the CEOs of a company. If you know only the fact that Elon Musk is the CEO of the company, then you need another identifier to limit the companies to one.

As you move further down on this spectrum, you will find it harder and harder to get back to the original data, and at the end, we have totally anonymized data that we cannot identify.

Other aspects to consider

Identifiability is probably the most important aspect in anonymization (btw, typing this word kills me…), but other things we should consider:

• Data should be analyzable

• Reversibility

• Live or static

• Deterministic or nondeterministic

When we anonymize data, we don’t want to lose the ability to analyze it. For example, if a dataset has correlation in it, then we cannot randomly change the data and lose its statistical features.

For internal use, you probably want the anonymization to be reversible. But for external parties, you usually must make the data unidentifiable, so end users cannot reverse it back to the original format.

Live anonymization is when you are not changing the entire data, only the subset that you want to access at a time. For static anonymization, data is already stored in the anonym format.

And finally, a deterministic method is where A is always changed to B. In a nondeterministic method, A can be B in the first run, but C in the second run.

Now that we know about the aspects to consider, let’s see the techniques we can use.

Generalization

When we generalize, we replace a value with a less specific but consistent value.

For numbers, you can use ranges, and for categorical data, you can use a broader category:

You can also level them:

Suppression

In this technique, we make the data invisible. We either remove it or keep it, but hide it.

If we extend the example above and add gender to the table:

And then we suppress the gender column, we get this:

You can apply all these techniques conditionally or combined with other techniques.

Distortion

We use this technique mainly with numerical data. The original data is distorted by noise. For example, we can add a random number to each value. The formula to use can be super simple, but complicated as well.

We would need to remove the Original Salary column, but I left it here for easier understanding.

If we want to keep the option to analyse the data, the distorted values must be plausible.

The above table applies a bad distortion and makes the salary implausible.

The salary can’t be negative, the Nurse and Cardiologist salaries are unrealistic, so an analysis of these data would be misleading.

Swapping

In swapping, we simply change the values between records:

The goal is to make the sensitive value unlinkable to the individual. Swapping must be controlled, because the data can be misleading again if we make it randomly.

It is more useful for homogeneous groups. Like within a category, you can more easily swap the subcategory.

Now we cannot link the person to the original job, but the context is still logical.

Masking

Masking replaces part of a data value with a placeholder but keeps the format:

It is great if we want to keep the pattern recognizable, so the readability stays in the data.

It is a super broad and really interesting topic. We just scratched the surface, but I might go deeper into it later. See you there as well.

Reading documentation is a skill. If you are early in your career, this habit will help you build your documentation reading muscles.

If you are more experienced and think you know a lot, I would still suggest doing this regularly. Every time I reread the documentation for a Python topic or DAX formula, I find something new. You will not learn the nuances in the first few reads.

Consider the read_csv method in pandas:

It has many parameters, and people usually use it only to load in the data, but you can start your data prep with it. To understand it well, you need to come back a few times.

Plus point: Think about how you could use the new knowledge in your current project.

Plus 2 points: Ask ChatGPT to give you some problems and practice the new knowledge, or implement it in your current project.

2. Review a chart

The media is full of visualizations nowadays. When you see one, don’t just look at it as a regular person. Look at it from the data and visualization perspective.

Ask: What works? What doesn’t? How would I improve it? Why did they choose this chart type? Why this color?

You can use the r/dataisbeautiful subreddit for this exercise. You know what? Let’s do one together.

Here is the first chart that comes up for me at the moment on the subreddit:

https://www.reddit.com/r/dataisbeautiful/comments/1jv43ei/oc_us_balance_of_trade/

And here are my quick, unfiltered, thinking-loud thoughts about it:

" The title and subtitle are short, but enough for the story. The labels are clear, but the shades of colors might be too close to each other. Bigger contrast may be better - I am not sure about their color blindness score either, but they fit the background well. The Axes are clear, the distance is consistent - They don’t want to lie or form the narrative with axes manipulation. The stacked bar chart helps to see the net effect. Long enough timeframe to see the trend. The data source is also added. Straight to the point, no fluff, can build a great story around it. 10/10"

Plus 2 points: Try to recreate the chart.

3. Read articles/newsletters or watch videos

This doesn’t require any description. Watch tutorials or read posts about the tools you use or general data topics, but don’t stop there.

Sometimes, take a rest from data and learn about other relevant things like communication, presentation, business knowledge, or other soft skills that are still necessary in the field.

Plus point: Share them with friends and colleagues - Learn together

Plus 2 points: Create summaries, write notes, and share them online (see 5. point).

If we talk about business knowledge 👇

4. Translate a Business Question into Data Terms

The difference between good and excellent data professionals is the impact they make. To be impactful, you need to know the connection between business and data. This practice will help to improve here.

Take a hypothetical business question like Why are sales down in Q2? or Which customers are likely to churn? and ask: What data would I need? What metrics would I use?

Plus point: Document these business-to-data translations so you can use them as “templates”. If a similar question pops up, you just need to pull them from your pocket, and no need to start from zero.

5. Write a Data Journal

I started writing to learn. Writing on X and on Substack helped me a lot throughout my career.

Writing will help you build technical skills. You need to do research, create code snippets, go deep into topics, etc., to have a quality output.

It will also help with soft skills like presentation and clear communication. Framing your ideas effectively is something that’s really hard to master, but writing will improve it for you unnoticeably.

Another advantage is the community aspect. Writing made it possible for me to talk to awesome data experts and call them friends. From a random person, you can elevate yourself to the next shelf, and other creators will look at you differently.

Plus point: If you already have a blog or something, share with us!

Final thoughts

Spend 5 minutes on each of the above exercises every day (or a few times a week) and your career will skyrocket! Short but focused practices can add up in the long run. Of course, if you go for plus points, it can take a lot more time, but it’s an investment.

Do you have a data habit that you do daily? Let us know in the comments.

After the short internship, I got a full-time position in the team. I did the same boring shit, but for more money. Then Covid hit, and I became a zombie staring at Excel sheets for a few hours a day from home office. I hated it, and I wanted something more exciting.

“There must be a better way to do all this.” I thought, and luckily, the company organized a VBA training internally. “Ohh, this will be a better way.” I realized after I googled what the hell VBA is.

I enrolled in the course, and that was the best thing I ever did!

We had 4 sessions about the very basics. Variables, Loops, Functions. It was a programming 101 with VBA. It got me hooked. At the end of the course, we got a script that was able to send automated emails. It felt magical.

For the final project, I automated one of my tasks, saving me 1 hour every week. It was super simple, but I knew that this was something I needed to dig deeper into.

Programming was different back then. We didn’t have ChatGPT, so I learned VBA the OG way. Googling every single line and spending hours on Stack Overflow. It was painful, but I had a mission to automate all my tasks at least partially. And I did. The more I automated, the more I could spend on learning, and the more I learned, the more I automated.

In 6 months, I maxed out the options I had, so I started to look for something more challenging. A Data Analyst role opened internally. It required some BI skills and SQL proficiency. I had none of those, but I still applied.

In the interview, I presented my VBA projects. It showed how passionate I was and my ability to learn. They told me that if in the next few weeks I get decent with SQL, I will get the job.

Projects, 100/10, would recommend and must-haves. You may have a lot of badges on your CV from 1–2 hours Udemy courses, but can you speak about a project you worked on and solved real-world problems? I did not have the SQL badges from Udemy yet, but I solved problems before. No matter which career path you choose, work on some projects!

I bought a SQL course from Udemy. and spent 3–4 hours a day to learn it. I felt the spark with SQL again. I got the job. Maybe the little cute cat helped to stay on the course.

It was a beginner-friendly role, but I learned a lot of SQL and some basics of QLIK. That is a BI Tool that is not so popular for a reason. 2/10 would not recommend. But at least I could tell later that I had some experience with BI.

I knew that the next major step would be to learn Python. I watched some courses on Youtube and took notes. During my journey, I started to follow people on social media, and I heard that if you want to deeply understand something, teach it. I found some other people who did “learning in public” and I gave it a shot myself.

I studied Statistics, Pandas, Matplotlib, basics of ML, and a bunch of other topics. Sharing my journey into data also taught me communication, marketing, design, and business stuff. Blogging is another great thing I did for my career. 10/10 would recommend.

Data Analysis is not just about the data. We need to sell our work, and we need to sell ourselves on interviews, for example. In data, hard skills are overrated, and soft skills are underrated. We should spend more time on communication, sales, and storytelling skills early in our careers. Even if we are not directly working within those areas, the skills are general. You can be the most talented coder if you cannot present your results effectively.

In the meantime, I got accepted to a Swedish university, so I did my masters in Finance there. I wanted to be a Financial Data Analyst. I studied Finance at the Uni, and I studied Data online in front of more and more people.

You may ask why no data at a university? I felt that a Financial degree would be more beneficial later in my career with self-taught data knowledge. I did my niche knowledge at an institution and studied data on my own. Niche knowledge is also really important. If your niche is Biochemistry in data, then the Biochemistry part will be super hard to learn alone.

But if you want to be in Machine Learning, the path without an ML degree will be way harder.

That’s how my career and content game started, and now here we are. I am working in Risk Reporting, so I can use all the knowledge I collected throughout the last few years. I am still learning a lot of data, and I share it here.

To sum it up, here is my roadmap in order of learning the skill:

Excel
VBA
SQL
BI Basics
Python
Python packages for data
Visualization
Statistics

Next on my list:

DAX (since I mainly use Power BI now)
Data Modeling
Cloud

Do I recommend others to follow this roadmap? You probably know the answer. It is always: “It depends.”

I think it has a great progression towards Analytics Engineering, where I am heading.

If you want to be a Data Analyst, it is a great path. You would learn from the ground up. VBA is not a must, but Excel knowledge can be super beneficial. Learning how to work with tabular data, how pivots work, how functions work will be useful. Then, picking up SQL, BI, and Python will set you up for a junior role.

For Data Science and Machine Learning, Math, Statistics, Linear Algebra, and similar topics should be a big part of the roadmap. These are harder skills to master, so self-taught ML Engineers are rare animals. If you want to go down this route, you should consider a degree from a great college.

They say there are no Junior Data Engineers. If this is true, being a Data Analyst is a great starting point, and then you can double down on SQL, data modeling, pipelines, and Cloud.

Coming from data, I also know SQL. But I don't know anything about web frameworks or backend development, and I could go on with this list for an hour.

I realised that if I want to build data apps, I need to learn some of those things. One of the first topics that came up is ORM.

What is an ORM?

ORM (Object Relational Mapping) is a programming technique that lets you interact with a relational database using the objects and classes in your programming language instead of writing raw SQL.

In the ORM context, classes are the tables, and objects are the rows from the table.

Here is an example table transformed into Python objects:

There are many options for ORM in Python:

1. SQLAlchemy

2. Peewee

3. The Django ORM
   and a lot more...

Let's see some code examples:

If we want to query the data in SQL, we would do something like this:

SELECT * FROM users WHERE age > 30;

The same in ORM would look like this:

results = session.query(User).filter(User.age > 30).all()

Filter is our Where statement, All is the *, and Query is the Select from.

You can also replicate group by and use aggregate functions.

SELECT AVG(age) FROM users;

The above SQL is equal to the below ORM:

avg_age = session.query(func.avg(User.age)).scalar()

And similarly for Group by:

SELECT age, COUNT(*) FROM users
WHERE age > 25
GROUP BY age;

results = session.query(User.age, func.count()) \
    .filter(User.age > 25) \
    .group_by(User.age) \
    .all()

As you can see, we can easily achieve the same thing with pure Python code, keeping the Object Oriented look and feel.

This is the biggest advantage of ORM. You don't need to jump languages, and it makes fast and clean development possible.

Also, if the underlying database were to change, for example, from SQLite to MySQL, ORM would still work the same way.

If you are super comfortable in Python and don't know a lot about SQL, it is a great alternative, but there are disadvantages as well.

With more complex queries, the syntax can be harder to write, and the execution may slow down. That's why it's a good idea to be ready to go back to raw SQL.

When we use ORM, we shift the data manipulation into the application itself, we do not handle it separately. It's not an issue if the app design is ok, but it can be chaotic.

Conclusion

I realised during reading and learning about ORMs that I should stay with raw SQL. I already know that well, and I am comfortable working with it.

I see the debate on the internet about which is better. Just like with everything else, there are advantages and disadvantages to both concepts. But the biggest advantage of SQL over ORM for me is that I know it already. Why would I learn the deep details of a new concept if I can easily replace it with SQL?

Coming from data, in this case, is good. On the other hand, if you are leaning towards ORMs, I would suggest to still learn SQL. ORM is great, but for complex problems, SQL will be superior. Also, the logic (and even the main syntax) is similar, so being familiar with SQL will not hurt.

The biggest conclusion for me was this:

Every shiny new framework or tool looks good in a simplified, small example.

Don’t fall for it and try to learn everything.

Focus on battle-tested tools that have stood the test of time.

In this case, it is SQL for me. It was great to learn the concept and basics of ORM, but it's not worth going deeper.

I will talk differently to my grandma and my manager.

Defining the audience should be the first step in every visualization project. The target audience will guide every following decision.

A visualization for the Sales team will be something about trends and revenue, while a presentation to HR should be about salary and diversity.

You may not need to break everything down to ELI5 level if the audience has a technical background.

Part of this question should be the goal of the audience. You may find the perfect language and tools for your communication, but your result will be 0 if you talk about apples when they want to hear about oranges.

So when you have a clear picture of the WHO, you need the WHAT.

What’s the one thing I want them to remember tomorrow?

You may feel tempted to show that you can use the latest tool to create the most colorful infographic with 50 metrics on one chart.

The result will be fancy, but not impactful. You want results that will stick with the audience, and they will leave with action items based on your presentation.

You can make strong visualizations in Excel easily. Don’t need to reinvent the wheel, keep it simple and effective.

Casey Neistat said:

“The best steak houses serve their filet on a plate with nothing else. Shitty franchises cover theirs in sauce and other shit to distract you from the fact you’re eating dog food.”

https://ckarchive.com/b/lmuehmhn4337vtd7kkm78c8r5e000cg

I read the above newsletter about marketing and realized how close it is to visualization. They are about the same thing: One story that we want to sell to the audience and make it stick.

Finding that one thing may be harder than putting together the fancy visual. All those tools give you comfort, and you believe that if you put one more element on the chart, it will be better. Usually not the case.

After a point, it is not about technical skills, but domain knowledge and understanding what can drive the most impact.

The best copywriters write short but memorable lines, and the best billboards are simple:

https://marketingexamples.com/ads/shot-on-iphone

One sentence with a great picture can tell more than a bunch of facts. Translate this idea into visualization:

A few more examples:

https://x.com/The_AdProfessor/status/1876270402156310688

Marketing is about persuasion. If your visuals are good, it will persuade the audience and drive good decisions.

The core principles are the same. Stay within the borders of simplicity, but still be creative!

I found a video with James Serra, an expert in Data Architecture, and this rabbit hole is deeeeep. I scratched the surface for you, so here are my notes from the video and his blog:

Data Warehouse

We don’t want to access the data directly from its sources. These sources can be customer information, product information, or purchase data from the same system/app, but they may be in different locations. We need a copy of the data in one place, and we store this copy in the DW.

The DW will be the single version of truth for historical data. We don’t want to run reports or analyses on the sources directly.

A DW is optimized for reading, meaning that it is easier to pull data from it than from the source systems. The source systems are usually optimized for writing, meaning that it is easier to input the data.

Source systems are usually not self-explanatory, but we can restructure and rename tables and fields, so data becomes more organized in a DW.

A DW is schema-on-write, meaning the data’s structure is defined and enforced when the data is written into the system. Before you load data into the DW, you must decide what tables you need, their columns, data types, and relationships. When data comes in it must adapt to this schema.

Data quality and consistency are high because you’re controlling what goes in. Changes in the source systems will not cause issues in reports or analyses.

DWs are for all areas like Finance, HR, and Sales, and the data in them is very detailed, while Data Marts are built on DWs and contain only single subject area subsets.

Data Lake

A Data Lake is schema-on-read. You can load any kind of data raw and unstructured, in its original format. You define the schema when you read or query the data, not before.

DLs are good for storing huge data at low cost. You can put in it everything “just in case” you need it later. You may never touch a part of it.

You can explore if the data is useful or not before making the first modifications on it. It may be the case that it turns out that the data is not useful at all, so no waste of time and cost in processing it.

It can handle large files quickly and can be useful also for archiving the data.

Modern Data Warehouse

In the MDW we combine the two structures above and it has 5 major steps:

1. Ingest

2. Store - Data Lake

3. Transform

4. Model

5. Visualize/ML

In the first step, we import the data from many sources into the lake in raw format.
In the MDW concept, the data is cleaned in the lake, so steps 2 and 3 are happening simultaneously. We will have raw data and a clean data layer in the lake.

Some end users, like Data Scientists, can already access data from the lake. For other users, the cleaning happening in step 3 may not be “clean enough”.

Step 4 models the data and puts it into a relational data warehouse. In some cases even star schema is created, so the analytics/reporting team could work with the data more easily.

In step 5 end users can drag and drop data fields into reports or do analytics since all the joins and preparation are done at this stage.

Data Fabric

Data Fabric adds functionality to the MDW.

For example, we can add an API to our cleaned data, or use advanced security features for the Data Lake and Warehouse.

It is an additional layer on top of our existing MDW.

Data Mesh

Decentralization is a buzzword nowadays. Data cannot be a niche where we don’t use it.

Date Mesh is a decentralized data architecture. It is not a technology, but a concept. All the domains could use the same data lake, hence the lake would become decentralized.

In a Data Mesh, each domain has ownership over its data, and they make it available to other domains as a product. This can lead to a lot of chaos, so ideally, there is a central IT team that helps to create standardized processes for each domain.

A great Data Mesh is hard to build and it could take years. You need the whole organization to be on the same page for their data architecture decisions.

Which to use?

There are no clear ‘if this then that’ answers to this question. It depends on the industry, budget, and people, but mostly the size and type of the data. As you can see from the notes above, the architecture gets more and more complex and their capacity increases as well.

You can also have a combination of all the above.

Sources:

https://www.jamesserra.com/

It depends…

Wait, how can the answer be that it depends?

The answer depends on who you ask, Python or Excel.

45728 vs 45729. But why is that?

It is because of a bug/feature in Excel (and some other apps).

The Leap Year Bug

Lotus 1-2-3 was a spreadsheet app from Lotus Software (later IBM). It was released on 26 January 1983 (15388 days ago, according to my spreadsheet), and IBM shut it down in 2013. It looked like this:

It was popular in the ‘80s, just like Excel is today. But there was a big issue with it.

Lotus 1-2-3 assumed that 1900 was a leap year. But it was not.

To define a leap year, we can use this logic:

• Divisible by 4 → It’s a leap year

• Except if it’s also divisible by 100 → it’s not a leap year

• Unless it’s divisible by 400 → it is a leap year

Or logically:

(Y % 4 == 0) and ((Y % 100 != 0) or (Y % 400 == 0))

1900 is divisible by 4 and 100 but not by 400 → Not a leap year.

2000 is divisible by all of them, so it is a leap year.

The engineers forgot about this, so Lotus had this bug.

Excel deliberately replicated the bug to maintain compatibility with Lotus, and to maintain compatibility with previous versions, Excel still has the bug.

What about Python?

Python doesn’t have this bug, so we have the differences in the results above.

Let’s check what happens if we want to import a dataset into pandas with the fake leap year in it.

Book1 doesn’t contain the Excel bug, while Book2 does.

No issue with Book1 as it is a regular CSV, but for Book2, it cannot convert the column to a datetime datatype; it remains an object.

The difference is also visible on the outputs of the dfs.

Here is another example from Python:

from datetime import date

today = date.today()
later = today.replace(year = today.year + 1)

If today is 2025-03-14, then:

• today becomes date(2025, 3, 14)

• later becomes date(2026, 3, 14)

since we added one year. This code works until you run it on February 29.

Generally, the leap year bug has two categories.

• Those that lead to error conditions, like the python code above.

• Those that lead to incorrect data, bad aggregations, or “off by one” problems.

Code and apps may crash due to the errors raised by Category 1. They are hard to miss.

Category 2, however, is the silent killer; it will not crash apps but can be dangerous if not handled correctly:

• Calculating “yearly averages” without accounting for that extra day in a leap year (366 vs. 365 days).

• Querying a database for events between February 1 and February month end, but forgetting to involve February 29 for leap years.

• Add “1 year” to February 29, and you land on March 1 or February 28. This can impact billing, invoices, etc.

The bug can cause huge issues.

In 2024, for example

• Payment terminals at ICA grocery and pharmacy stores across Sweden failed to process transactions on February 29th.

• Street lighting in Paris, France, was inadvertently turned off at midnight at the start of February 29th.

Some more examples here.

Why no correction?

According to Microsoft, the disadvantages of changing this behaviour outweigh the advantages.

With a change, all the dates would decrease by 1 day as they would take out a previous day from the system.

Since the compatible applications are “used to” this behaviour, changing it would require changes in all other programs as well.

In his new book, Effective Visualization, Matt Harrison discusses the CLEAR framework to make effective visuals that serve the purposes above. CLEAR stands for:

C - Color
L - Limits
E - Explanation
A - Audience
R - References

Color

Color is the biggest asset for visualization. You can define/change a lot of things with it.

The best use of color is to guide the eye of the viewer. You can emphasize your story by graying out the “everything else” and keeping the most important points colorful.

We can also use colors based on already existing associations. For example, we tend to attach blue to cold and red to warm. Red is bad, Green is good, and so on.

Matt suggests using 3 colors maximum. More can be overwhelming. Also, you need to think about colorblind people and how the chart will look in black and white or grayscale.

Limits

As a beginner, I wanted to create the fanciest visuals ever. I thought infographics that have a bunch of data on them were cool. But now I understand that the simpler, the better. Limits give you simplicity. Again, we are aiming for story, not art.

You should use the most basic charts like line plots, bar plots, scatter plots, and histograms. The audience will be familiar with these chart types, and familiarity makes understanding easier. For a new, fancy chart, they need to understand how the chart works first, then the story.

A simple visual like the above is easy to understand; it takes 3 seconds. But what happened here is a crime:

Your eyes are probably jumping around, trying to figure out where to look, and you get frustrated after 15 seconds of looking at the “visualization”.

There are no hard rules here. Remove unnecessary things where possible, but there is no problem with adding labels if you need them to tell the story.

Two plots are better than one crowded plot! Creating more visuals is ok until you keep them simple and they add to the story. Sometimes, it’s impossible to pack everything into one visual.

Use complex or novel charts only when absolutely necessary. There is no issue with coming up with new ideas, but keep it simple and consider understandability.

Explanation

Sometimes, the visual itself is enough to get the point, but titles, subtitles, legends, and labels can help.

Colors guide the eye, text guides the understanding. The shorter, the better; they can even be words, not whole sentences.

Consider the placement of your annotations. They should be close to the data they explain. Colors for text can be applied here as well, but stay within the limits mentioned above.

Arrows, circles, and callout values are other visual elements that can help with the exxpanation.

Audience

Knowing the audience is a must. You will not use jargon or niche words if you know that the audience cannot understand them. The sport chart above is kind of niche since someone not familiar with Tennis may have a hard time understanding it.

If you create visuals for a wide audience, keep your wording on grandma’s level. Simple always win like Serena Williams did.

You can use the same dataset to create different stories for different audiences. The sales team may be interested in revenue trends, while HR requires a story on salaries.

References

Cite the source! Tell the data source and what types of modifications you made to it.

By linking the source, you allow others to work on the same data. This is key for replicating the analysis or even improving it.

Where to find good examples?

FiveThrirtyEight collects its best charts every year. Here is the 2019 edition (Good examples in this post are from these collections)

Other big news sites like the NY Times and The Economist have awesome charts that follow the most important rules. Use these as inspiration and learn from their great visuals!

Sources:

Effective Visualization: Exploiting Matplotlib & Pandas by Matt Harrison

https://fivethirtyeight.com/features/our-51-best-and-weirdest-charts-of-2021/
https://fivethirtyeight.com/features/the-56-best-and-weirdest-charts-we-made-in-2019/
https://visme.co/blog/bad-infographics/

There are different types of number generators, the two main categories are:

• Pseudo-Random Number Generators (PRNGs)

• True Random Number Generators (TRNGs)

Let’s discuss each. But first, the basics:

Random numbers

One single random number is a value that we chose from the available options. All options have the same probability to be chosen, meaning, the distribution is uniform.

A sequence of random numbers means that we repeat the process above and the choices are not impacting future choices, so the drawing is independent.

Randomness is important in many fields and also used for various purposes in Data Science:

• Data sampling

• Data splitting

• Simulations

Randomness has great advantages, but it is hard to make a computer act randomly. They follow the instructions no matter what. And that’s good. The same input should result in the expected output and not a random mess. Imagine that you open up the browser and type google.com, and it would open up a random website for you.
We need some methods to make them act more randomly.

True Random Number Generators

The real world is fortunately highly unpredictable, so we can use it as an input for computers. TRNGs usually use physical events to create the randomness. The events can vary from simple, like how you move your mouse to more advanced like nuclear radioavtive events involving quantum mechanics.

The most fun way to create randomness is to create chaos.

Cloudflare uses the coolest TRNGs. They have many lavalamps installed in the HQ office. One lavalamp alone is super random itself, as the liquid in it can take up infinite forms, but they have many of them. Moreover, since they are installed on one of the companies main corridors, people passing the lamps can add to the chaos.

A camera watches the lobby 24/7 and take photos. Since computers store photos as series of numbers, every photo is a truely random number.

In the London office Cloudflare uses pendulums. The unpredictable swings coming from these things mixed with the incoming lights and shadows make the installation super random.

Here is a great illustration of how random a single pendulum can be.

Another office from Cludflare includes the hanging rainbow mobiles that react to changes in its environment like door openings, and ambient light. As you can see, the bigger the chaos is the better for randomness.

And finally, here is a great illustration showing how these sources turn into a random number that is impossible to predict.

Pseudo-Random Number Generators

The number generators we work with in our applications like Excel, Python are PRNGs. They cannot use chaos like the above. PRNGs use mathematical algorithms to create sequences that looks random. But the numbers are not truly random, they follow a predetermined sequence.

When I wrote “looks random” I meant if you see the numbers written out in a sequence. But when we visualize them, the story is different.

Bo Allen created the above visual to show that how a bad combination of language (PHP), operating system (Windows), and function (rand()) can cause some obvious patterns. (With the same code, different operating systems get different results)

We can see that random generators can be very bad, but the best are really good. The quality varies a lot.

Generally, PRNGs are an efficient way to create randomness since these algorithms are fast. Also, the deterministic nature can be advantageous.

We see the seed parameter in many codes, because the seed provides reproducibility. If you set it to be the same, the sequence created will be the same. Meaning, we can recreate our randomness for every run and test other parts of the code without changes in data.

Without reproducibility, testing is more challenging. If each execution produced entirely different random values, it would be difficult to determine whether changes in the output were caused by modifications in the code or simply by a new set of random inputs.

These algorithms are good for modeling and simulations, but not the best solution in cases where numbers must be truly unpredictable, like in gambling or data encryption.

PHPs documentation also highlights this. And it explains why rand() is more restiricted in Windows.

Conclusion

Going down this rabbit hole I realised how important randomness can be and how fun or creative solutions run behind super secure systems. As said, computers are not random at all without some random inputs. But what about humans?

Check this out:

Sources:
How do lava lamps help with Internet encryption? | Cloudflare

https://blog.cloudflare.com/harnessing-office-chaos/

https://www.random.org/randomness/

https://www.random.org/analysis/#visual

https://en.wikipedia.org/wiki/Lavarand

Popularity of the first name Stevie correlates with Netflix’s stock price (NFLX) (r=0.996)

Statistically, they are correlating, but we know it’s only pure chance and no connection. But how do we know? We use logical reasoning.

What is the difference between logic and statistics? They represent two different models for understanding the world. Each has strengths and weaknesses

Statistics

It uses math and works with probabilities, generalizing from patterns in data instead of relying on rules. It is good at making decisions based on previous examples and making predictions.

The data can be huge, even terabytes, and actually, Statistics thrives in data-rich environments where patterns can be learned. Machine learning, for example, depends heavily on statistical inference.

But…

• It can’t prove absolute correctness - only probabilities of correctness.

• You will have a hard time with unreliable data.

• It can lead to misleading conclusions. See the spurious correlation above.

Logic

When you have a logical reasoning properly set up you can rely on its certainty, the conclusion follows the premises. For this reason, it’s usually easier to understand a logical system.

When there is no data, you must rely on logical reasoning. It is good in no-data or bad-data environments.

But…

• When solving logical problems, the more variables you have, the harder the solution becomes. Logic does not scale well.

• Logical rules are strict and cannot be easily adapted to new data.

• Cannot handle probability

Human Reasoning

Neither approach is perfect, so we use both every day.

Here are a few examples:

We use logic to check if the door is locked. This is a binary (true/false) problem. If we remember locking the door and know it cannot unlock itself, we can logically conclude it’s locked.
“I turned the key and heard the lock click, so the door must be locked.”

The above example is called deductive reasoning. The structure follows
General principle → Specific conclusion.

or its reverse:

But that is too strict and we usually have more complex problems. Here are another examples:

When should we go to sleep? We rely on experiences to estimate how much sleep we need and how tired we will feel the next day. So we use statistics.
“When I sleep less than 6 hours, I feel exhausted the next day, so I’ll go to bed now.”

Deciding what to eat for lunch can be a mix. If we have dietary restrictions, we use logic (“I can’t eat peanuts because I’m allergic”). But if we’re choosing between two restaurants, we might rely on experiences (statistics).

Most real-life decisions involve a mix of both logical and statistical reasoning, but also emotions and other factors like domain knowledge can impact the decision.

Consider this situation:

You are sitting in a café, enjoying your breakfast. You reach for your cup and realize that your wallet is missing from the table. A moment later you notice a man in a hoodie walking toward the exit, looking suspicious. Maybe he stole your wallet?

If you quickly jump to conclusions, your reasoning may follow the above-mentioned straightforward deductive reasoning:

1. Premise 1: If someone takes my wallet, it will no longer be on the table.

2. Premise 2: My wallet is no longer on the table.

3. Conclusion: Someone must have taken it.

This reasoning has one major gap: the conclusion assumes only one possible explanation.

The wallet could have fallen under the table, you forgot that you put it into your bag and so on. We need a practical explanation that involves possibilities:

In this case:

• A = “The wallet was stolen.”

• B = “The wallet is missing.”

• The fact that B is true might increase the plausibility of A, but it does not prove it.

We can make similar reasonings more subtle:

Notice that we went from true or false statements to plausible.

Let’s take a look at the correlation example again. How do we know that the relationship is not causal?

We apply logical reasoning and real-world knowledge (that comes from previous experiences - statistics):

• Is there a believable chain of events where names can influence stock price? Not really

• Is there a known mechanism that could prove this? Not really

• Do we have reverse causality? Could Netflix’s stock price changes influence the number of babies named Stevie? Or vice versa. Not really

• Is there a hidden factor that might affect both? Unlikely

Reasoning of Machines

AI, similarly to us, uses both statistical and logical reasoning.

Chatbots predict the most probable next word in a sentence based on data. Fraud Detecting models look at past transactions and assign a probability score to determine if a transaction is fraudulent.

Medical models diagnose diseases based on predefined logical conditions. Game-playing robots use if-then rules to make moves.

Of course, some models can combine them. Self-driving cars use logic for traffic rules (stop at a red light) and statistics for decision-making (predict if the pedestrian will cross).

What is the difference between brain and brAIn then?

János Neumann said in 1948,

“You insist that there is something a machine cannot do. If you tell me precisely what it is that a machine cannot do, then I can always make a machine which will do just that!”

When we say machines cannot think, we say that we cannot describe a problem in detail. Machines can think if we can define what thinking is.

And they can be better than us! As mentioned above, statistics scales well, because we can use GBs of data to analyze. But humans cannot handle that much info in their brains. Also, logic cannot scale well. People can understand simple logical trees, but a logical tree with 100 outcomes will set your brain on fire, machines can handle those as well.

Again, it’s only on us to define our problems well, so machines can solve them.

If you go to a University website and check their statistics you will probably find a measure called “average class size”. It is a simple average but can be misleading.

Different measures have different meanings for other perspectives. Consider this example:

In a tiny University, there are only two classes. One is a large course with 90 freshmen, while the other is a small one with 10 seniors.

What is the average class size?

It depends on who you ask. If you ask the University then the answer will be 50. (90+10)/2. And that is correct, but can be misleading to students. For the University a course is a course and they are equally weighted.

If you ask a student, the answer will be totally different. A senior will say that 50 is way too many because they only have 10 people in a class. A freshman will say that 50 is not true when they are sitting with 89 people around them in every class.

90% of the University students feel that they have bigger classes than the average.

To understand the student perspective we need to do a weighted average calculation.

90 people said that the class size is 90 and 10 said that the class size is 10. That’s why we need to “duplicate” the numbers. Since we have 100 students the “real average from the student perspective” will be 82. (8200/100)

You always hear from friends: “Ohh the beach is always so crowded. There are thousands of people there.” Then the statistics say that the average of visitors is only a few hundred. Now you know the explanation. When the beach is not crowded your friends are not there to notice it.

Friendship paradox

A similar pattern can occur with friends.

Red has 1 friend, Yellow has 3, and Green and Blue have 2 friends each. This means 8 friends divided by 4 = 2 friends on average.

But what if we interview each color and ask how many friends their friends have? Here are the answers:

R: Y has 3
Y: R has 1, G has 2, and B has 2.
B: Y has 3, and G has 2
G: Y has 3, and B has 2.

If we add up the numbers the result is 18 and the reported names are 8 (note: we are not dividing by 4. We need to sum the number of names in the answers).

18/8 equals 2.25 (>2). On average friends have more friends than the colors themselves.

Let’s take a closer look at the answers. Y occurs 3 times, R occurs once, G and B occurs 2 times each. The number of occurrences equals their number of friends. So we go back to the weighted average calculation. More popular friends are listed more and contribute more to the score.

Explained through code

Let’s create some Python code to observe this in action. You can find the code at the bottom of this post!

This code represents a Probability Mass Function (PMF), which is a way to describe the probability of different numbers of friends in the population.

The _normalize method calculates the total number of people and divides each count by the total to get a probability distribution. We have {2: 5, 5: 8, 10: 14}, where the keys are the number of friends and the values are the number of people who have that many friends. The total number of people is 5 + 8 + 14 = 27. So, the probability for each friend count:

• P(2 friends) = 5 / 27

• P(5 friends) = 8 / 27

• P(10 friends) = 14 / 27

The Mean method calculates the mean number of friends in the actual distribution.

The Bias method creates a biased version of the PMF, where people with more friends are sampled more often. This is done by multiplying the probability by the number of friends, just like we did in the examples above.

Then we finally plot the results:

Mean number of friends (actual): 12.60. This tells us, “If you randomly pick a person, how many friends do they have?”

Mean number of friends (observed by friends): 16.69. This tells us, “If you randomly pick a friend from someone’s friend list, how many friends do they have?” People with more friends show up more often in this count.

The actual average number of friends is lower than the perceived number of friends.

PMF helps to compare the two distributions easily. The biased PMF is always shifted to the right (towards bigger numbers), meaning the observed friend count is higher.

Main sources:

• Friends You Can Count On - The New York Times

• The ‘class size paradox’: How individual- and group-level perspectives differ, and why it matters in research | Pew Research Center

• Probability mass functions

Code:

import numpy as np
import matplotlib.pyplot as plt

class Pmf:
    """A simple class for Probability Mass Function (PMF) operations."""
    def __init__(self, data, label=''):
        self.label = label
        self.pmf = self._normalize(data)

    def _normalize(self, data):
        total = sum(data.values())
        return {k: v / total for k, v in data.items()}

    def Mean(self):
        return sum(k * p for k, p in self.pmf.items())

    def Bias(self):
        """Bias the PMF so that people with more friends are sampled more often."""
        biased_pmf = {k: v * k for k, v in self.pmf.items()}  # Multiply by k (friend count)
        total = sum(biased_pmf.values())
        return {k: v / total for k, v in biased_pmf.items()}

    def Plot(self, biased_pmf):
        x = list(self.pmf.keys())
        y_actual = list(self.pmf.values())
        y_biased = list(biased_pmf.values())

        plt.bar(x, y_actual, alpha=0.6, label="Actual Distribution")
        plt.bar(x, y_biased, alpha=0.6, label="Observed Distribution (biased)", color='red')
        plt.xlabel("Number of Friends")
        plt.ylabel("Probability")
        plt.legend()
        plt.show()

# Define a hypothetical friend distribution
friend_distribution = {
    2: 5,   # 5 people have 2 friends
    5: 8,   # 8 people have 5 friends
    10: 14, # 14 people have 10 friends
    15: 6,  # 6 people have 15 friends
    20: 10, # 10 people have 20 friends
    25: 5   # 5 people have 25 friends
}

pmf = Pmf(friend_distribution, label="actual")
biased_pmf = pmf.Bias()

print(f"Mean number of friends (actual): {pmf.Mean():.2f}")
print(f"Mean number of friends (observed by friends): {sum(k * v for k, v in biased_pmf.items()):.2f}")

# Plot actual vs biased distribution
pmf.Plot(biased_pmf)

Let’s collect some best practices for handling NULL in SQL:

IS NULL Condition

One mistake beginners think is that NULL is a value that we can compare to numbers or other data types. But regular comparison operators like “=”, “<>”, “<“, “>” are not working with NULL in SQL. A NULL cannot be compared to another NULL either. NULL = NULL is FALSE because NULL means “unknown,” so two unknown values cannot be compared.

Here is a great exercise to understand the NULL logic:

employees table contains the following values:

SELECT * FROM employees where department_id = NULL;

As mentioned above, = will not work with NULL, so the result will be an empty table (no rows returned).

If you are thinking with the ‘general’ binary programming logic then you would expect that running the two lines below will return all the rows:

SELECT * FROM employees WHERE department_id = 2;

SELECT * FROM employees WHERE department_id <> 2;

But that’s wrong. The results are these:

Poor Charlie is missing since NULL <> 2 is UNKNOWN.

To get all the values we need to add the following:

SELECT * FROM employees WHERE department_id IS NULL

The IS NULL condition is used to return rows that contain the NULL values in a column. Its variation IS NOT NULL will return all non NULL values.

ISNULL() is different!

ISNULL()function is different from IS NULL. Adding to the confusion different Database systems have different functionality assigned to them.

ISNULL(column, default_value) replaces NULL with a given value in SQL Server.

SELECT name, ISNULL(department_id, 'Not Assigned') AS department
FROM employees;

In MySQL ISNULL() is used to test whether an expression is NULL or not. It returns 1 if the expression is NULL, and 0 otherwise.

SELECT name, ISNULL(department_id) AS is_null
FROM employees;

In MySQL if we want to replace NULL we need to use the IFNULL() function. Confusing right?

IFNULL(column, default_value) replaces NULL values with a specified default in MySQL.

NULLIF()

NULLIF(expr1, expr2) returns NULL if two expressions are equal, otherwise it returns the first expression. It is basically the same as:

CASE WHEN (expr1 = expr2)
THEN NULL
ELSE V1 END

COALESCE()

And finally, the one I use the most. You can pass many parameters into COALESCE() and it will return the first non NULL value. It checks all the values from left to right. This can replace most of the functions mentioned above.

Let’s look at some examples:

SELECT name, COALESCE(phone_number, 'N/A') AS phone_number
FROM customers;

If phone_number is NULL, it will be replaced with 'N/A'

We can add more parameters to it:

SELECT employee_id, name, COALESCE(home_phone, work_phone, mobile_phone, 'No phone available') AS contact_number
FROM employees;

For John the first non NULL value going from left to right is work_phone, but for Mark since all are NULL we replace them with a default value.

You can also use it in calculations. NULL will impact all calculations. If you include a NULL, the result will be NULL as well.

If a column might have NULL values in a calculation, you can replace them with a default:

SELECT order_id, price, discount, COALESCE(price - discount, price) AS final_price
FROM orders;

If discount is NULL, price - discount would result in NULL.

COALESCE(price - discount, price) ensures that if discount is NULL, the original price is used.

Without COALESCE() we would be screwed.

SELECT order_id, price, discount, 
       price - discount AS final_price
FROM orders;

If you are running a webshop you probably don’t want to show NULL as the final price of a product.

Be careful out there with NULL and use the techniques mentioned above!

Which is your favorite ice cream flavor?

Simple right? You expect a single flavor people will add as an answer.
After the survey period you sit down to record the answers into a database. Reaching this question you realize that the question was not simple at all, or at least the answers are not helpful.

Missing values

A customer answers your questions but leaves the favorite flavor answer box empty. You were expecting a choice, but they never gave one. You record because you don’t know their preference.

Unknown Value

The customer wrote something down, but you cannot read it. The answer was provided but got lost. You add NULL because the information exists, but it is unknown.

Miscellaneous Value

A customer walks in and lists chocolate, vanilla, or strawberry. Now you finally have somewhat good answers, but you expected a singular flavor. You record NULL because the preference doesn’t fit neatly into the expected answers.

Inapplicable Value

You see this answer: Pepperoni Pizza. That’s a great choice for sure, but not for an ice cream. You didn’t ask about favorite food. What do you do? You store NULL because no valid response applies.

Let’s talk SQL

In all the above cases, the system just sees NULL - but NULL doesn’t tell you why the data is missing.

If you’re trying to calculate the most popular flavor, do you count NULLs as “vanilla”? Do you ignore them? Should you assume they were “chocolate”?

NULL ads complication to databases. It is a special value and has no data type, meaning it can appear in text, numerical, or any other columns as well. The biggest confusion comes from the fact that NULL can mean 4 things in SQL. It can be missing, unknown, miscellaneous, or inapplicable value in the data. NULL makes filtering, aggregating, and comparing data tricky because it isn’t truly a value.

Just like you at the ice cream shop with the answers, an SQL database struggles to interpret NULL correctly in queries. NULL behaves unpredictably when used in calculations and other operations. Staying at the ice cream shop example:

NULL in Math

Imagine trying to count how many scoops of each flavor were sold today. But some orders are NULL in the “scoops” column. What happens when you add them up? The NULL will destroy the calculation and your result will be NULL .

NULL in Comparisons

You want to check if two customers ordered the same flavor. If one order is NULL , SQL doesn’t say “yes” or “no” - it says UNKNOWN. It’s like asking, “Did two customers order the same thing?” and getting the response: “I have no idea.” That’s SQL’s three-valued logic in action—TRUE, FALSE, and UNKNOWN.

NULL in Sorting

When sorting orders by flavor, NULL sometimes appear at the top, sometimes at the bottom, depending on the database settings. It’s like rearranging your menu board but having mystery items that randomly jump to the top or bottom whenever you sort the list.

NULL in Grouping

They group for some operations but not for others.
It’s like reviewing your sales and seeing:

• Vanilla: 50 orders

• Chocolate: 40 orders

• NULL: ??? (How many? Do they even belong together?)

What to do then?

If you are doing an online survey instead, you can set more limitations. People can only choose one value from a list of your flavors, you can mark the answer as a must, and so on.

The best problem-solving is prevention. In SQL you can add a NOT NULL constraint to avoid NULL and all the confusion that comes with it.

But if you end up with NULL in the DB, then you need to handle them. That’s a different story, so see you in the next post!