# Entropy

## What is entropy?

Entropy is a core concept in Information Theory. It is defined as the average level of ‘information’, ‘surprise’, or ‘uncertainty’ inherent in the possible outcome of a random variable. For example, a coin flip has two possibilities so we can expect it to have lower entropy than a dice, which has six possibilities. Below, we define the equation for Entropy, sometimes called Shannon’s Entropy:

$H(X) = -\Sigma_{i=1}^{n} P(X_{i}) logP(X_{i})$

Here is a simple function for calculating entropy. Let’s use it to compare between a coin and a dice.

entropy <- function(prob){
s = -sum(prob * log2(prob))
return(s).

cat('\n The entropy of a fair coin toss is: ',entropy(rep(1/2,2)))

##
##  The entropy of a fair coin toss is:  1

cat('\n The entropy of a fair dice roll is: ',entropy(rep(1/6,6)))

##
##  The entropy of a fair dice roll is:  2.584963


As expected, the coin toss has lower entropy because there are fewer possibilities compared to the dice roll.

Let’s expand our definition of entropy a bit more by conceptualizing its relationship to information. In Information Theory, we can think of information as being stored in or transmitted as variables. We can obtain information from a variable just by observing its value. The entropy of any variable is the “amount of information” contained within the variable. However, the amount of information is not determined by the number of different values. Instead, it is determined in proportion to the average level of ‘uncertainty’.

For instance, we can take a book and see that a book with more pages could have more information than a book with less pages. However, if you have already ready the larger book, then there is no level of ‘uncertainty’ and therefore, no amount of information. Likewise, the information in a variable is tied to the amount of uncertainy or surprise. Another way of thinking of it is that lower probability events have more information, amd higher probability events have less information.

## Exploring entropy in mtcars

Let’s use the mtcars data set to test to explore Entropy.

data("mtcars")

##                    mpg cyl disp  hp drat    wt  qsec vs am gear carb
## Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
## Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
## Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
## Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
## Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
## Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1


We can calulate the entropy of the engine type. Another way of stating the question is asking how pure is this data set with respect to engine type? If the data set only have 4 cylinder cars, then it is 100% pure.

freq <- table(mtcars$cyl)/length(mtcars$cyl) #Find the frequency for each type of engine
cyl_prob <- as.data.frame(freq)[,2]
e0 <- entropy(cyl_prob)
cat('\n The entropy is:', e0)

##
##  The entropy is: 1.530994


Since the entropy is non-zero, it is impure and not 100% for any type of engine. Let’s subset the data and test it with only 4 cylinder cars.

mtcars_subset <- mtcars[mtcars$cyl == '4',] freq <- table(mtcars_subset$cyl)/length(mtcars_subset$cyl) #Find the frequency for each type of engine cyl_prob <- as.data.frame(freq)[,2] cat('\n The entropy is:', entropy(cyl_prob))  ## ## The entropy is: 0  The entropy is zero because we know within this subset that there are only 4 cylinder cars. Suppose we want to know if we can reduce entropy by segmenting the data. We can use Information Gain to measure how much “information” a specified feature gives us about the data set. ## Information gain Information gain is defined as the reduction of entropy due to a change in the dataset. It is defined as follows: $IS(X, Y)= H(X) - H(X|Y)$ $H(X|Y) =\Sigma_{i=1}^{n} P(Y_{i}) H(X|Y_{i})$ Let’s take a look at an example. Here we will split the data into four bins and calculate the min() and max() for MPG. library(dplyr)  ## ## Attaching package: 'dplyr' ## The following objects are masked from 'package:stats': ## ## filter, lag ## The following objects are masked from 'package:base': ## ## intersect, setdiff, setequal, union  data <- mtcars data$bin<-cut(data[,'mpg'], breaks=4, labels=c(1:4))
summary_data <- data %>% group_by(bin) %>% summarise(
n=length(cyl),
min=min(mpg),
max=max(mpg).
summary_data

## # A tibble: 4 × 4
##   bin       n   min   max
##   <fct> <int> <dbl> <dbl>
## 1 1        10  10.4  15.8
## 2 2        13  16.4  21.5
## 3 3         5  22.8  27.3
## 4 4         4  30.4  33.9


Next, we will calculate the entropy for engine cylinder after segmentation and we will also calculate the probability for choosing a bin.

cyl_prob <- data %>% group_by(bin) %>% count(cyl) %>% summarise(e = entropy(n/sum(n)))  #Find the entropy of cylinder for each bin
summary_data <- right_join(summary_data, cyl_prob, c('bin'))
summary_data$p <- summary_data$n/nrow(data) #calculate probability for choosing a bin
summary_data

## # A tibble: 4 × 6
##   bin       n   min   max     e     p
##   <fct> <int> <dbl> <dbl> <dbl> <dbl>
## 1 1        10  10.4  15.8  0    0.312
## 2 2        13  16.4  21.5  1.42 0.406
## 3 3         5  22.8  27.3  0    0.156
## 4 4         4  30.4  33.9  0    0.125


Finally, we find our information gain.

IG <- e0-sum(summary_data$p*summary_data$e) #find the information gain
cat('\n The entropy is:', IG)

##
##  The entropy is: 0.954299


What does this all mean? To start, we were able to reduce entropy after segmentation. We also learn that some of the bins have an entropy of zero. This suggests that after segmentation, there is a decrease in amount of uncertainty. This means that by selecting any one of the bins, we may have a 100% chance to have cars with single type of engine. Let’s have a look at the data again.

data %>% group_by(bin, cyl) %>% summarise(n=length(cyl),
min=min(mpg),
max=max(mpg))

## summarise() has grouped output by 'bin'. You can override using the .groups argument.

## # A tibble: 6 × 5
## # Groups:   bin 
##   bin     cyl     n   min   max
##   <fct> <dbl> <int> <dbl> <dbl>
## 1 1         8    10  10.4  15.8
## 2 2         4     2  21.4  21.5
## 3 2         6     7  17.8  21.4
## 4 2         8     4  16.4  19.2
## 5 3         4     5  22.8  27.3
## 6 4         4     4  30.4  33.9


As you can see, bin 1, 3, and 4 only contain cars with a single type of engine. So this means that we are improving entropy over the complete data set by segmenting on MPG. In other words, if we were to draw a random car and knew the MPG, we have a good idea of how many cylinders the car has. So if given a car that had a MPG of 30.5, than we have a 100% chance of it being a 4 cylinder car.