According to some estimates between 50% to 80% of the work of a data scientist is spent collecting and preparing data, what the New York Times calls janitor work[1]. When we consider the iterative nature of the data science process (refer to The Data Science Process ), we see each cycle typically repeats the data preparation step. As our understanding of the data evolves as well as the refinement of the model, we find ourselves often going back to further develop the data. While data preparation has never been an easy process, in a big data world the greater variety of data and data sources makes it all the more difficult. These sources rarely store or present data in a structure that facilitates analysis. To address this issue, we need to tidy the data. Let me explain…
Data Science
The Data Science Process
We live in a world where larger and larger volumes of varied data types are coming at us in ever increasing speeds, i.e. we live in a world of big data. In order to make sense of big data, we have turned to data science. Data Science is a tool employed by the transliterate to transform data into information.
From Data Literacy to Transliteracy
From Data Literacy to Transliteracy Understanding necessary skills for data democratization.
Overfitting / Underfitting – How Well Does Your Model Fit?
Supervised machine learning is inferring a function which will map input variables to an output variable. Let’s unpack this definition a bit with an example. Say that we are a bank that wants to determine to whom we should give a loan. The objective, therefore, is to infer a function that examines a set … Continue reading Overfitting / Underfitting – How Well Does Your Model Fit?
The School of Athens
Recently, I was asked by a friend why I chose The School of Athens as the image for this blog. Frankly, I could not think of anything that could be more appropriate. Let me explain… Between 1509 and 1511, Raphael had been given a commission to paint four reception rooms in the public portion of … Continue reading The School of Athens
They Want to Get Rid of Me!! – Rise of the Citizen Data Scientist
They want to get rid of me. Wait, let me rephrase that, they want to get rid of us!! I can’t blame them, if I were them I would want to get rid of us too. Our friends at Gartner started all this talk about citizen data scientists.
Open Data Science
Everyone is talking about data science. One study found that 96% of companies believe that data science is integral to the success of their business. Yet, most of these organizations (70%) are not realizing its full potential. They cite such factors as poor data quality, lack of talent, and access to proper tools and technology[1]. … Continue reading Open Data Science
What is the Philosophy of Data Science (and Should Data Scientists Care)?
Recently I had read a LinkedIn article by Kalyan Sambhangi in which he asked where are all the data philosophers.[1] He makes a good point, if data science is truly a science shouldn’t the philosophy of science be applicable to it as well; i.e. the philosophy of data science. I agree; it should. So, let … Continue reading What is the Philosophy of Data Science (and Should Data Scientists Care)?
Meditations on BI and Data Science 