Please enable JavaScript.

Coggle requires JavaScript to display documents.

Statistics - Coggle Diagram

- - - - The Minimum desired lift (also called the Minimum detectable effect)
      - The Baseline conversion rate
      - The Statistical significance threshold
  - - - converted visitors(посетители, совершившие целевое действие) этот термин используется для описания посетителей веб-сайта или онлайн-платформы, которые выполнили желаемое действие или достигли цели, установленной для них.
      - Generally, our metric(baseline conversion rate) will be the percent(or proportion) of users who take a certain action after interacting with one of our options. For instance:
        The percent of customers who buy a t-shirt after visiting one of two versions of a website
        The percent of users who click on one of two versions of an ad
  - - - Lift is generally expressed as a percent of the baselin conversion rate. Suppose that 6% of our customers currently buy socks on our website Sock Hops (that’s our baseline conversion rate). We think that a new website layout would increase this. Changing a website layout is hard, so we only think that it’s worth doing if at least 8% of our customers would buy socks on Sock Hops with the new layout. That means that we want to increase our conversions by 2%. To calculate lift:
        
        (click on me!)
        100 (new - old) / old
        100 (8 - 6) / 6
        33%
        Sock Hops’ desired lift is 33%.
  - - - Margin of error
      - Confidence level
      - Population size
      - Expected proportion
  - - - However, if the small town of Vancucumber is holding an election for a new mayor, and we want to project the results of the election, then the 1700 citizens would be the only important people. In this case, 1700 is the population size we would use in a sample size calculator.
  - - - As the expected proportion increases, it is rarer that we hit that proportion accurately with the random sample we choose.
        If we do not have historical data, we normally use 50%, which gives the most conservative (i.e., largest required) sample size.
- - - - Variance
        
        Variance is a descriptive statistic that describes how spread out the points in a data set are.
        
        Mathematically, we can write this comparison as
        difference=(X−μ)2
        Where X is a single data point and the Greek letter mu is the mean
        
        Full aormula of variance is:
        Variance is usually represented by the symbol sigma squared.
        We start by taking every point in the dataset — from point number 1 to point number N — and finding the difference between that point and the mean.
        Next, we square each difference to make all differences positive.
    - - Standard deviation is computed by taking the square root of the variance.
  - - - In first method we exlude the Q2 in calculations. In second method we include the Q2 wehn traing to calculate Q1 and Q2
        
        First method:
        [−108,4,8,15,16,23,42]
        Using the first method, we found Q1 to be 4.
        Second Method:
        when calculating Q1 using this new method we would now find the median of this dataset:
        [−108,4,8,15]
        Using this method, Q1 is 6.
  - - - The interquartile range is displayed in a commonly-used graph — the box plot. In a box plot, the ends of the box are Q1 and Q3. So the length of the box is the IQR.
        
        межквартильный размах (IQR) охватывает 50% данных в распределении.
        
        Проверьте, находится ли число внутри интервала между Q1 и Q3, используя следующее условие:
        
        Если число >= Q1 - 1.5 IQR и число <= Q3 + 1.5 IQR, то оно находится в рамках IQR.
        
        Если число < Q1 - 1.5 IQR или число > Q3 + 1.5 IQR, то оно не находится в рамках IQR и может считаться выбросом.
        
        Если число находится внутри интервала между Q1 и Q3, то оно считается типичным значением, находящимся в рамках центральной части данных. Если число не попадает в этот интервал, оно может считаться выбросом или необычным значением.
  - - - The individual measurements on Monday(32 ft), Tuesday(35 ft), and Wednesday(31ft) are called samples. A sample is a subset of the entire population. The mean of each sample is the sample mean and it is an estimate of the population mean(32.8 ft).
        
        For a population, the mean is a constant value no matter how many times it’s recalculated. But with a set of samples, the mean will depend on exactly what samples we happened to choose. From a sample mean, we can then extrapolate the mean of the population as a whole.
    - - In real life, we probably won’t be able to collect lots of repeated samples, but luckily we can use something called the Central Limit Theorem, which tells us that a sample mean is more likely to be similar to the population mean if the sample size is large.
    - - A null hypothesis is a statement that the observed difference is the result of chance.
        
        Sample of null hypothesis:
        What is the probability that the two population means are the same and that the difference we observed in the sample means is just chance?
      - Hypothesis testing is a mathematical way of determining whether we can be confident that the null hypothesis is false.
    - - In statistical hypothesis testing, we concern ourselves primarily with two types of error.
        
        false positive(Type 1) это ошибка, когда вы говорите, что есть эффект или различие, когда его на самом деле нет.
        
        false negatives(Type 2) это ошибка, когда вы говорите, что эффекта или различия нет, когда он на самом деле есть.
    - - a p-value is the probability that we yield the observed statistics under the assumption that the null hypothesis is true.
        p-value should small enough to reject the null hypothesis. Generally, we want a p-value of less than 0.05, meaning that there is less than a 5% chance that our results are due to random chance.
      - It is important that we choose the significance level before we perform our statistical hypothesis tests to yield a p-value.
        Choosing our significance level in advance helps keep us honest.
        we wait until after we see the results, we might pick our threshold such that we get the result we want to see.
      - P-значение показывает, насколько необычны или странны данные, и чем меньше P-значение, тем менее вероятно получить такие же или более странные (существенные) результаты, при условии, что нулевая гипотеза верна. Если P-значение очень низкое, это указывает на то, что наши данные сильно отличаются от того, что можно было бы ожидать, если бы нулевая гипотеза была верна.
  - - - Как и в случае со стандартным отклонением(std) у среднего абсолютного отклонения(MAD) есть правило похожее на 68,95,99.
        
        Для MAD справедливы следующие правила:
        
        Приблизительно 50% значений данных лежат в интервале [mean - MAD, mean + MAD].
        
        Приблизительно 75% значений данных лежат в интервале [mean - 1.5 MAD, mean + 1.5 MAD].
        
        Приблизительно 89.5% значений данных лежат в интервале [mean - 2 MAD, mean + 2 MAD].
- - - - A symmetric dataset has equal amounts of data on both sides of the peak. Both sides should look about the same.
    - - A skew-right dataset has a long tail on the right of the peak, but most of the data is on the left.
    - - A skew-left dataset has a long tail on the left of the peak, but most of the data is on the right.
  - - - In fact, here are a few more helpful rules for normal distributions:
        68% of our samples will fall between +/- 1 standard deviation of the mean.
        95% of our samples will fall between +/- 2 standard deviations of the mean.
        99.7% of our samples will fall between +/- 3 standard deviations of the mean
      - Suppose that we have a normal distribution with a mean of 50 and a standard deviation of 10. When we say “within one standard deviation of the mean”, this is what we are saying mathematically:
        lower_bound = mean - std
        = 50 - 10
        = 40
        upper_bound = mean + std
        = 50 + 10
        = 60
        It turns out that we can expect about 68% of our dataset to be between 40 and 60, for this distribution.
- - - - Popular Types(For more open (SciPy mind map)
        
        For categorical data, we will cover:
        Binomial Tests
        Chi Square
        
        For numerical data, we will cover:
        One Sample T-Tests
        Two Sample T-Tests
        ANOVA
        Tukey Tests
        
        How to choose test?
    - - Step 3: Determine the Null Distribution
        
        null distribution is the distribution (across repeated samples) of the statistic we are interested in if the null hypothesis is true
        
        we using a sample destribution
        
        SAMPLING DISTRIBUTIONS
        
        Central Limit Theorem
        
        The Central Limit Theorem establishes that the sampling distribution of the mean will be normally distributed (even if the original population was not normally distributed).
        
        Given that sample size(n) is sufficiently large (n > 30), the sampling distribution of the means will be normally distributed with mean approximately equal to the population mean μ
        
        Standard Error
        
        2 more items...
        
        Sampling Distributions
        
        A sampling distribution is obtained by taking a random sample of a certain size multiple times, taking a sample statistic, and plotting the distribution of this sample statistic.
        Другими словами, это распределение, которое показывает, какие значения статистики могут быть получены из различных случайных выборок из одной и той же генеральной совокупности
        Note that we can look at a sampling distribution for any statistic, like maximum or minimum.
        
        1 more item...
        
        Estimators
        
        Remember, you can have a sampling distribution for any sample statistic, including:
        
        Mean
        
        Median
        
        Min/Max
        
        Variance
        
        2 more items...
        
        confidence interval
        (Доверительные интервалы)
        
        Доверительный интервал - это диапазон значений, в пределах которого с определенной вероятностью находится неизвестный параметр популяции (например, среднее значение или доля), оцененный на основе выборки данных.
        A confidence interval is a range of values within which an unknown population parameter (like a mean or proportion), estimated from a sample, is likely to fall with a certain level of confidence.
        
        1 more item...
        
        Random sampling in Numpy
        
        random sampling in Python from np.array:
        data = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
        sample_size = 5
        sample = np.random.choice(data, sample_size, replace=False)
        
        Since collecting data for an entire population is often impossible, researchers may use a smaller sample of data to try to answer their questions.
        
        Let’s simulate the null distribution using our population. We can do this by:
        
        We should Taking many random samples of 100 scores, each, from the population
        
        Calculating and storing the average score for each of those samples
        
        Plotting a histogram of the average scores
        
        This yields the following distribution, which is approximately normal and centered at the population average. Plot on the graph the population mean(dotted line) and sample average.
      - Step 2: Define the Null
        and Alternative Hypotheses
        
        Null Hypotheses(H0)
        
        Average monthly store sales are $29.92k And any difference is gotten by chance or simple fluctuation.
        
        Alternative Hypotheses(Ha)
        
        should choose the alternative hypothesis before collecting any data
        
        Average monthly store sales are not equal to $29.92k.
        
        Average monthly store sales are less than $29.92k
        
        Average monthly store sales are more than $29.92k
      - one-sample t-tests are used for quantitative data to compare a sample mean to an expected population mean.
      - Example
        
        Suppose we have a store who sales product about 29.92 thousand per month. We make some changes and in these month we get 31.16 thousand.
      - Step 1: Ask a Question
        
        is this month significantly outperforming or it just simple flactuation?
      - Step 4: Calculate a P-Value (or Confidence Interval)
        
        P-value (p-значение) представляет собой вероятность получить такие же или более экстремальные результаты, чем наблюдаемые, при условии, что нулевая гипотеза верна.
        About confidence intervals:
        
        Option 1
        
        Average monthly store sales are more than $29.92k
        
        In this case, we want to know the probability of observing a sample average greater than or equal to 31.16 given that the null hypothesis is true
        We make some calculation and get 3.1% of the total distribution. This proportion, which is usually written in decimal form (i.e., 0.031), is called a p-value.
        
        We will not consider other option(we only onterested in one)
        
        Condidence interval
        
        percentiles = np.percentile(sample_means, [2.5, 97.5])
        print(percentiles)
        This leaves us with a range covering 95% of the data.
        If our observed statistic falls outside this interval, then we can conclude it is unlikely that the null hypothesis is true.
      - Step 5: Interpret the Results
        
        Since we got p-value(0.031) under significance threshold(0.05), we could reject the null hypothesis in favor of the alternative.
        The interpretation of this number is as follows:
        Случайно получить продажи $31.16k или больше при условии, что нулевая гипотеза верна(any difference is gotten by chance or simple fluctuation) довольно мало вероятно и равна 3.1%, что ниже порога значимости.
        
        Remember that we don't prove the alternative hypothesis, we reject the null hypothesis.
        
        Remember about type I errors (false positives) and type II errors (false negatives):
        
        false positive это ошибка, когда вы говорите, что есть эффект или различие, когда его на самом деле нет.
        
        false negatives это ошибка, когда вы говорите, что эффекта или различия нет, когда он на самом деле есть.
      - Step2.1: Define the Significance thresholds
        
        A common choice for this threshold, which is also sometimes referred to as Alpha, is 0.05 — but this is an arbitrary choice! Using a lower threshold means you are less likely to find significant results, but also less likely to mistakenly report a significant result when there is none.
        
        уровень порога значимости, который вы устанавливаете для вашего теста (например, 0.05), фактически определяет, насколько вы готовы рисковать совершением ложноположительных ошибок. Если вы устанавливаете порог 0.05, то вы говорите, что готовы принять риск ошибки первого рода в 5% случаев. То есть в 5% случаев вы будете отвергать нулевую гипотезу, даже если она верна. Это связано с балансом между нахождением реальных эффектов и минимизацией ошибок.
    - - One Sided or One-Tailed test
        
        Односторонний тест обычно используется, когда вам важно только одно направление различия, и вы не интересуетесь другим направлением. (GRATER THAN | LESS THAN)
      - Two Sided or Two-Tailed test
        
        Двусторонний тест используется, когда вам важно выяснить, есть ли какое-либо различие, но вы не делаете предположений относительно того, в каком направлении это различие может быть.(NO EQUAL TO ...)(in python as default using this type of testing)
    - - The sample was randomly selected from the population
        
        The individual observations were independent
        
        The data is normally distributed without outliers OR the sample size is large (enough)
    - - For example, we need to run 10 hypothesis tests .05 significance level for each test, then:
        
        When we run a hypothesis test for a single question, we have a 95% chance of getting the right answer (a p-value > 0.05) — and a 5% chance of making a type I error.
        
        When we run hypothesis tests for two questions, we have only a 90% chance of getting the right answer for both hypothesis tests (.95×.95 = 0.90) — and a 10% chance of making at least one type I error.
        
        When we run hypothesis tests for all 10 questions, we have a 60% chance of getting the right answer for all ten hypothesis tests (0.95^10 = 0.60) — and a 40% chance of making at least one type I error.
      - When running multiple tests, use a lower significance threshold (eg., 0.01) for each test to reduce the probability of making a type I error.
  - - - Linear regression is a powerful modeling technique that can be used to understand the relationship between a quantitative variable and one or more other variables, sometimes with the goal of making predictions.
        
        For example, linear regression can help us answer questions like:
        
        Is a mother’s height a good predictor of their child’s adult height?
        
        What is the relationship between apartment size and rental price for NYC apartments?
      - Equatatin of simple lineral regression
        
        a common choice for linear regression is ordinary least squares (OLS). In simple OLS regression, we assume that the relationship between two variables x and y can be modeled as:
        y = mx + b + error
      - Equation of a Line
        
        Like the name implies, LINEar regression involves fitting a line to a set of data points.
        Equation of a Line:
        y=mx+b
        
        x and y represent variables, such as height and weight
        
        b represents the y-intercept of the line. This is where the line intersects with the y-axis (a vertical line located at x = 0).
        
        m represents the slope. This controls how steep the line is. If we choose any two points on a line, the slope is the ratio between the vertical and horizontal distance between those points; this is often written as rise/run.
        Remember that slope can be thought of as rise/run — the ratio between the vertical and horizontal distances between any two points on the line.
        
        The following plot shows a line with the equation y = 2x + 12:
        
        plt.plot(x, m*x+b, color='red', label='Regression Line')
        
        Finding the "Best" Line
        
        "Ordinary least squares" (что можно перевести как "метод наименьших квадратов") — это способ, который помогает нам найти такую прямую линию (или кривую), чтобы сумма квадратов расстоянийtotal от этой линии до каждой точки была минимальной. Таким образом, это лучший способ представить общий тренд или направление всех точек на графике.
        
        We define “best” as the line that minimizes the total squared error(or loss function in ML) for all data points.
        
        Допустим, у вас есть данные о том, сколько часов студенты учились для экзамена и какие они получили оценки:
        hours_studied = np.array([1, 2, 3, 4, 5])
        grades_received = np.array([50, 60, 65, 70, 90])
        slope, intercept = np.polyfit(hours_studied, grades_received, 1)
        .# строим линию по формуле: y=mx+b
        predicted_grades = slope * hours_studied + intercept
        
        OR you can calculate INTERCEPT and SLOPE using this
      - quantitative prdictors
        
        Fitting a Linear Regression Model in Python
        (finding best intercept and slope in python)
        
        Suppose we have a dataset named body_measurements with columns height and weight.
        If we want to fit a model that can predict weight based on height, we can create the model as follows:
        
        import statsmodels.api as sm
        model = sm.OLS.from_formula('weight ~ height', data = body_measurements)
        results = model.fit()
        print(results.params)
        Output:
        Intercept -21.67
        height 0.50
        
        This tells us that the best-fit intercept is -21.67, and the best-fit slope is 0.50.
        
        To make a prediction, we need to plug in the intercept and slope to our equation for a line.
        The equation is: y=mx+b
        weight = 0.50 * height +(-21.67)
        
        To make a prediction, we can plug in any height. For example, we can calculate that the expected weight for a 160cm tall person is 58.33kg:
        weight = 0.50 * 160 + (-21.67)=58.33
        
        Or we can calculate the same using from results.params using their indices (0 and 1, respectively):
        print(results.params[1] * 160 + results.params[0])
        .# Output: 58.33
        
        We can also do this calculation using the .predict() method on the fitted model. . To predict the weight of a 160 cm tall person, we need to first create a new dataset with height equal to 160:
        newdata = {"height":[160]}
        print(results.predict(newdata))
        .# Output: 58.33
        
        Interpreting a Regression Model
        
        In previous we calulated thi data:
        Intercept -21.67
        height(slope) 0.50
        
        interpritation of Intercept -21.67
        
        The interpretation of the intercept is that a person who is 0 cm tall is expected to weigh -21 kg
        
        This is pretty non-sensical because it’s impossible for someone to be 0 cm tall and have negative weight!However, in other cases, this value does make sense and is useful to interpret.
        For example, the intercept for the OLS regression model predicting score based on hours_studied is 43. Interpritation is: a student who studied for 0 hours is expected to score a 43 on the test.
        
        interpritation of slope 0.50
        
        Therefore, the slope (which we previously calculated to be 0.50 kg/cm) is the expected difference in the outcome variable (weight) for a one unit difference in the predictor variable (height). In other words, we expect that a one centimeter difference in height is associated with .5 additional kilograms of weight.
        
        Another example, the slope for the OLS regression model predicting score based on hours_studied is 9.8. Interpritation is: for every additional 1 hour of studying, students are expected to score 9.8 points higher on the test.
      - Assumptions of Linear Regression
        
        The first assumption is that the relationship between the outcome variable and predictor is linear (can be described by a line).
        
        The next two assumptions (normality and homoscedasticity) are easier to check after fitting the regression. First, we need to calculate two things: fitted values and residuals.
        
        The fitted values(red dots) are the predicted weights for each person in the dataset that was used to fit the model, while the residuals(dotted line) are the differences between the predicted weight and the true weight for each person.
        
        fitted values
        
        В контексте простой линейной регрессии, это точки на регрессионной линии.
        We can calculate the fitted values using .predict() by passing in the original data. The result is a pandas series containing predicted values for each person in the original dataset:
        
        fitted_values = results.predict(body_measurements(origanal data))
        print(fitted_values.head())
        Output:
        0 66.673077
        1 59.100962
        2 71.721154
        3 70.711538
        4 65.158654
        
        residuals
        
        Это разница между наблюдаемыми(значениями зависимой переменной) и подогнанными значениями (то есть значениями, предсказанными моделью).
        The residuals are the differences between each of these fitted values and the true values of the outcome variable.
        They can be calculated by subtracting the fitted values from the actual values.
        
        We can perform this element-wise subtraction in Python by simply subtracting one python series from the other, as shown below:
        residuals = body_measurements.weight - fitted_values
        print(residuals.head())
        Output:
        0 -2.673077
        1 -1.100962
        2 3.278846
        3 -3.711538
        4 2.841346
        
        Normality assumption
        
        The normality assumption states that the residuals should be normally distributed.
        
        To check this assumption, we can inspect a histogram of the residuals and make sure that the distribution looks approximately normal (no skew or multiple “humps”):
        plt.hist(residuals)
        plt.show()
        
        If residuals appear normally distributed, leading us to conclude that the normality assumption is satisfied.
        
        If the plot instead looked something like skewed, we would be concerned that the normality assumption is not met
        
        After plotting data, make some tests to prove that your data is met normality assumption.
        
        if data large enough(more that 2000 observation)
        
        Тест Шапиро-Уилка
        
        Нулевая гипотеза (H0): Данные имеют нормальное распределение.
        Альтернативная гипотеза (H1): Данные не имеют нормального распределения.
        
        График Q-Q
        
        otherwise
        
        Тест Колмогорова-Смирнова
        
        Нулевая гипотеза (H0): Данные соответствуют нормальному распределению.
        Альтернативная гипотеза (H1): Данные не соответствуют нормальному распределению.
        
        График Q-Q
        
        Homoscedasticity assumption
        
        Homoscedasticity is a fancy way of saying that the residuals have equal variation across all values of the predictor variable.
        
        A common way to check this is by plotting the residuals against the fitted values.
        plt.scatter(fitted_values, residuals)
        plt.show()
        
        If the homoscedasticity assumption is met, then this plot will look like a random splatter of points, centered around y=0 (as in the example above).
        
        If there are any patterns or asymmetry, that would indicate the assumption is NOT met and linear regression may not be appropriate. For example:
        
        Тест Бройша-Пагана
      - Categorical Predictors(Binary)
        
        The simplest case of a categorical predictor is a binary variable (only two categories).
        
        For example, suppose we surveyed 100 adults and asked them to report their height in cm and whether or not they play basketball.
        We’ve coded the variable bball_player so that it is equal to 1 if the person plays basketball and 0 if they do not. A plot of height vs. bball_player is below:
        
        fitting the line
        
        We see that people who play basketball tend to be taller than people who do not. Just like before, we can draw a line to fit these points.
        
        To re-create the scatter plot with the best fit line, we could use the following code:
        
        .#Calculate group means:
        print(data.groupby('play_bball').mean().height)
        Output:
        0 169.016
        1 183.644
        
        .# Create scatter plot:
        plt.scatter(data.play_bball, data.height)
        .# Add the line using calculated group means:
        plt.plot([0,1], [169.016, 183.644])
        OR like that:
        plt.plot([0, 1], data.groupby('play_bball').mean().height[0], data.groupby('play_bball').mean().height[1])
        .# Show the plot:
        plt.show()
        
        1 more item...
        
        Fit and Interpretation
        
        Fit
        
        we can actually fit the model using statsmodels.api.OLS.from_formula() and find the slope and intercept, the same way we did for a quantitative predictor:
        
        model = sm.OLS.from_formula('height ~ play_bball', data)
        results = model.fit()
        print(results.params)
        Output:
        Intercept 169.016
        play_bball 14.628
        
        Interpritation
        
        intercept
        
        To interpret this output, we first need to remember that the intercept is the expected value of the outcome variable when the predictor is equal to zero.
        
        In this case, the intercept is therefore the mean height of non-basketball players.(169.016)
        
        slope
        
        The slope is the expected difference in the outcome variable for a one unit difference in the predictor variable. In this case, a one unit difference in play_bball is the difference between not being a basketball player and being a basketball player.
        
        Therefore, the slope is the difference in mean heights for basketball players and non-basketball players.(14.628)
        
        categorical predictor that has more than two categories
        
        This article will explore the implementation and interpretation of a single categorical predictor with more than two categories.
        
        The Data
        
        For now, we’ll only focus on two columns of this dataset:
        rent: the rental price of each apartment
        borough: the borough that the apartment is located in, with three possible values ('Manhattan', 'Brooklyn', and 'Queens')
        
        import pandas as pd
        rentals = pd.read_csv('rentals.csv')
        print(rentals.head(5))
        Output:
        index | rent | borough
        0 | 5295 | Brooklyn
        1 | 4020 | Manhattan
        2 | 16000 | Manhattan
        3 | 3150 | Queens
        4 | 2955 | Queens
        
        The X Matrix
        
        The X matrix is new dataset when we we use statsmodels.api.OLS.from_formula() to create a model. which is implemented behind the scenes in statsmodels
        
        We can see the X matrix for this model using patsy.dmatrices(), which is implemented behind the scenes in statsmodels:
        import pandas as pd
        import patsy
        rentals = pd.read_csv('rentals.csv')
        y, X = patsy.dmatrices('rent ~ borough', rentals)
        .# Print out the first 5 rows of X
        print(X[0:5])
        
        Output:
        [[1. 0. 0.]
        [1. 1. 0.]
        [1. 1. 0.]
        [1. 0. 1.]
        [1. 0. 1.]]
        
        1 more item...
        
        Implementation and Interpretation
        
        The formula for linear regression with categorical variables can be summarized as follows:
        y=β0 + β1 × X1 + β2 × X2+⋯+βn × Xn + ϵ
        y = depend variable
        β0 = intercept
        β1,β2, ... = slopes
        X1, X2, ... = independ variables(in our case 0s or 1s)
        ϵ = error or remainder
        
        Let’s now fit a linear regression model using statsmodels and print out the model coefficients:
        import statsmodels.api as sm
        model = sm.OLS.from_formula('rent ~ borough', rentals).fit()
        print(model.params)
        Output:
        Intercept 3327.403751
        borough[T.Manhattan] 1811.536627
        borough[T.Queens] -811.256430
        
        We can use the intercept and two slopes to construct the following equation to predict rent:
        rent=3327.4+1811.5×borough[T.Manhattan]−811.3×borough[T.Queens]
        
        1 more item...
        
        Getting the same results
        
        print(rentals.groupby('borough').mean())
        Output:
        borough | rent
        Brooklyn | 3327.403751
        Manhattan | 5138.940379
        Queens | 2516.147321
        
        Changing the Reference Category
        
        In the example above, we saw that 'Brooklyn' was the default reference category (because it comes first alphabetically), but we canchange the reference category to 'Manhattan' if we want to.
        
        model = sm.OLS.from_formula('rent ~ C(borough, Treatment("Manhattan"))', rentals).fit()
        print(model.params)
        Output:
        Intercept | 5138.940379
        C(borough, Treatment("Manhattan"))[T.Brooklyn] | -1811.536627
        C(borough, Treatment("Manhattan"))[T.Queens] | -2622.793057
        
        Other Python Libraries for fitting Linear Models
        
        It's could be pandas or scikit-learn. Pandas example:
        import pandas as pd
        rentals = pd.get_dummies(rentals, columns = ['borough'], drop_first = True)
        print(rentals.head())
        Putput:
        rent | borough_Manhattan | borough_Queens
        0 | 5295 | 0 | 0
        1 | 4020 | 1 | 0
        2 | 16000 | 1 | 0
        3 | 3150 | 0 | 1
        4 | 2955 | 0 | 1
        
        scikit-learn example:
        from sklearn.linear_model import LinearRegression
        X = rentals[['borough_Manhattan', 'borough_Queens']]
        y = rentals[['rent']]
        .# Fit model
        regr = LinearRegression()
        regr.fit(X, y)
        print(regr.intercept)
        print(regr.coef)
        
        Output:
        LinearRegression()
        [3327.40375123]
        [[1811.5366274 -811.25642981]]
      - MInd set to compleat a lineral regressing model
        
        Find data
        
        Choose variables
        
        Create scatter plot
        
        Calculate slope and intercept
        a. make interpritations of slope and intercept
        
        Calculate line across dots
        a. plot line on graph
        
        make prediction
        
        Calculate fitted values and residuals
        
        Try to approve assumptions of linear regression
        a. relationship must be linear
        b. Normality
        c. Homoscedasticity
      - Поиск данных: Этот шаг всегда является первым.
        
        Выбор переменных: Определяете зависимые и независимые переменные.
        
        Предварительная проверка предпосылок:
        a. Линейность отношений: На этом этапе уже можно провести первичную проверку на линейность, используя диаграммы рассеяния или другие графические методы.
        
        Создание диаграммы рассеяния: Это помогает визуально оценить связь между переменными.
        
        Расчет наклона и пересечения и интерпретация: Вычисляете параметры модели и интерпретируете их.
        
        Расчет линии регрессии и её отображение на графике: Подтверждение адекватности модели.
        
        Делание предсказания: Использование модели для прогнозов.
        
        Расчет подогнанных значений и остатков: Позволяет оценить качество подгонки модели.
        
        Подробная проверка предпосылок линейной регрессии:
        а. Линейность: Более тщательная проверка с использованием остатков модели.
        b. Нормальность остатков: Обычно проверяется с помощью графика квантилей-квантилей (QQ-plot) или теста Шапиро-Уилка.
        c. Гомоскедастичность: Проверка равенства дисперсий остатков, например, с помощью теста Бройша-Пагана или графического анализа.
- - - - Set theory is a branch of mathematics based around the concept of sets. In simple terms, a set is a collection of things.
        Notationally, mathematicians often represent sets with curly braces.
        When defining a set, we often use a capital letter:
        A = {1, 2, 3, 4}
        
        Sets also follow two key rules:
        
        Each element in a set is distinct.
        
        The elements in a set are in no particular order.
        
        Therefore, we can say:
        {1, 2, 3, 4, 5,} = {5, 4, 1, 2, 3}
        
        Sets can also contain subsets. Set A is a subset of set B if all the elements in A exist within B. For example:
        A = {1, 2, 3}
        B = {1, 2, 3, 4, 5}
      - Union
        
        The union of two sets encompasses any element that exists in either one or both of them.
        
        Объединение двух множеств A и B – это множество, содержащее все элементы, которые принадлежат хотя бы одному из множеств. Обозначается как A ∪ B.
      - Intersection
        
        The intersection of two sets encompasses any element that exists in both of the sets.
        
        Пересечение двух множеств A и B – это множество, содержащее все элементы, которые принадлежат и множеству A, и множеству B. Обозначается как A ∩ B.
      - Complement
        
        The complement of a set consists of all possible outcomes outside of the set.
        
        Дополнение множества A – это множество всех элементов, которые не принадлежат множеству A. Обозначается как A'.
    - - In probability, an experiment is something that produces observation(s) with some level of uncertainty.
        A sample point(Элементарный исход) is a single possible outcome of an experiment.
        A sample space is the set of all possible sample points for an experiment.
        A specific outcome (or set of outcomes) is known as an event and is a subset of the sample space. Событие представляет собой набор элементарных исходов или подмножество пространства элементарных исходов, которые удовлетворяют определенным условиям или критериям. Событие может состоять из одного или нескольких элементарных исходов.
        
        For example, suppose that we run an experiment where we flip a coin twice and record whether each flip results in heads or tails.
        There are four sample points in this experiment: two heads (HH), tails and then heads (TH), heads and then tails (HT), or two tails (TT).
        We can write the full sample space for this experiment as follows:
        S = {HH, TT, HT, TH}
        Three events we might look at in this sample space are:
        A = {HH}
        B = {TT}
        C = {HT, TH}
      - calculate the estimated probability
        
        Suppose we flipped two coins 1000 times. Outcomes:
        {HH}: 252
        {TT}: 247
        {HT}: 256
        {TH}: 245
        
        To calculate the estimated probability of any one outcome, we use the following formula:
        P(event) = Number of Times Event Occurred / Total Number of Trials
        
        So, the probability of two heads on two coin flips is approximately:
        P(HH) = 252/1000
        P(HH) = .252 or 25.2%
        I we want to be more confident in results, we should incrise number of trails(Law of Large Numbers)
      - Events
        
        Independence and Dependence
        
        Independence
        
        Two events are independent if the occurrence of one event does not affect the probability of the other.
        
        Exmple:
        Imagine that we flip a fair coin 5 times and get 5 heads in a row. Does this affect the probability of getting heads on the next flip? The fact that previous coin flips do not affect future ones is called independence.
        
        Dependence
        
        Two events are dependent if the occurrence of one event does affect the probability of the other.
        
        Example:
        Suppose we have a bag of five marbles: two marbles are blue and three marbles are red. If we take one marble out of the bag, what is the probability that the second marble we take out is blue?This describes two events that are dependent. The probability of grabbing a blue marble in the second event depends on whether we take out a red or a blue marble in the first event.
        
        Mutually Exclusive Events
        (Взаимоисключающие события)
        
        not mutually exclusive
        
        For example we rolling a six-sided die. If event A is rolling an odd number and event B is rolling a number greater than two, these events are not mutually exclusive. They have an intersection of {3, 5}.
        Any events that have a non-empty intersection are not mutually exclusive.
        
        mutually exclusive
        
        Two events are considered mutually exclusive if they cannot occur at the same time. For example, consider a single coin flip: the events “tails” and “heads” are mutually exclusive because we cannot get both tails and heads on a single flip.
        We can visualize two mutually exclusive events as a pair of non-overlapping circles.
        
        Addition Rule(Правило сложения)
        
        The Addition Rule is used to determine the probability that at least one of several events will occur.
        In another words Addition Rule describes the probability one event OR another event (or both) occurs.
        
        NOT mutually exclusive events
        
        P(A or B) = P(A) + P(B) - P(A and B, то есть A* B )
        
        Example:
        Допустим, у нас есть монетка, и мы хотим найти вероятность выпадения орла (А) или решки (В) при одном броске монеты.
        
        Вероятность выпадения орла P(А) = 0.5
        
        Вероятность выпадения решки P(В) = 0.5
        
        Вероятность выпадения и орла, и решки P(А и В) = 0 (потому что не могут выпасть и орел, и решка одновременно)
        
        Тогда по Правилу сложения:
        P(А или В) = 0.5 + 0.5 - 0 = 1
        
        mutually exclusive events
        
        P(A or B) = P(A) + P(B)
        
        Conditional Probability(Условная вероятность)
        
        Conditional probability measures the probability of one event(A) occurring, given that another one has already occurred(B).
        Formula: P(A|B)
        Notationally, we denote the word “given” with a vertical line "|".
        
        For example we have 5 marbles, 2 blue and 3 red. if we want to represent the probability that we choose a red marble given the first marble is blue, we can write:
        P(Red Second | Blue First) = 3/4
        
        Another example with multiplication rule:
        Suppose that the following is true (this is shown in the first set of branches in the diagram):
        
        20 percent of the population has strep throat.
        
        80 percent of the population does not have strep throat.
        
        Now suppose that we test a bunch of people for strep throat. The possible results of these tests are shown in the next set of branches:
        
        If a person has strep throat, there is an 85% chance their test will be positive and a 15% chance it will be negative. This is labeled as:
        P(+| ST) = 0.85
        and
        P(- | ST) = 0.15
        
        If a person does not have strep throat, there is a 98% chance their test will be negative and a 2% chance it will be positive. This can be labeled as:
        P(- | NO ST) = 0.98
        and
        P(+ | NO ST) = 0.02
        
        Finally, let’s look at the four possible pairs of outcomes that form the terminal branches of our diagram:
        P(A and B) = P(A) × P(B | A)
        
        P(ST and +) = 0.17
        P(ST and -) = 0.03
        P(NO ST and +) = 0.016
        P(NO ST and -) = 0.784
        Это вероятности того, что будет если взять случайного человека и тут показаны все возможные исходы(sample space)
        
        independent events
        
        Therefore, for independent events, we can say the following:
        P(A|B) = P(A)
        and
        P(B|A) = P(B)
        Для независимых событий A и B, условная вероятность P(A|B) (вероятность события A при условии, что событие B произошло) равна просто вероятности события A, потому что событие B никак не влияет на вероятность A.
        
        Multiplication Rule(Правило умножения)
        
        Multiplication Rule is used to find the probability that two or more events will occur at the same time.
        
        Dependent Events
        
        P(A and B) = P(A) * P(B | A)
        
        We have five marbles: two are blue, and three are red. We pick two marbles without replacement. What if we want to know the probability of choosing a blue marble first AND a blue marble second?
        P(Blue first and Blue second) = P(Blue first) * P(Blue second | Blue first)
        P(Blue first and Blue second) = 2/5(it's a 0.4) * 1/4(t's a 0.25) = 0.1
        
        Independent Events
        
        P(A and B) = P(A) * P(B)
        
        Let’s look at the simplest example: flipping a fair coin twice. Event A is that we get tails on the first flip, and event B is that we get tails on the second flip.
        P(Tails and Tails) = P(Tails) * P(Tails)
        P(Tails and Tails) = 0.5 * 0.5 = 0.25
        
        Bayes' Theorem
        
        Bayes' theorem is a mathematical tool that helps us reevaluate and reassess the probabilities of events using new information or knowledge that we already have.
        P(A | B) = (P(B | A) * P(A)) / P(B)
        "P(A|B) is the conditional probability of event A given that event B has already occurred."
        
        Example:
        Imagine that you are a patient who has recently tested positive for strep throat. You may want to know the probability that you HAVE strep throat(ST), given that you tested positive:
        P(ST | +)
        To calculate this probability, we will use Bayes' Theorem:
        P(B | A) = (P(A | B) * P(B)) / P(A)
        Using Bayes’ theorem:
        P(ST | +) = (P(+ | ST) * P(ST)) / P(+)
        We know that:
        
        P(+ | ST) = 0.85
        
        P(ST) = 0.20
        
        P(ST and +) = 0.17
        
        P(NO ST and +) = 0.016
        
        We need to calculate P(+) or probability that test positive(Мы складываем P(ST and +) и P(NO ST and +), потому что нам интересно знать общую вероятность получить положительный результат вне зависимости от того, есть ли заболевание или нет.):
        P(+) = P(ST and +) + P(NO ST and +)
        P(+) = 0.17 + 0.016 = 0.186
        
        Finally, if we plug all of these into the Bayes’ theorem formula, we get:
        P(ST | +) = (0.85 × 0.20) / 0.186 = 0.914
        There is a 91.4% chance that you actually have strep throat given you test positive.
    - - The Law of Large Numbers states that as you repeat a random experiment many times, the average outcome of those experiments gets closer and closer to the expected value. In simple terms, it means that the more you do something random over and over again, the closer the average result will be to what you would normally expect. For example, if you flip a coin many times, the more flips you make, the closer the ratio of heads to tails will be to 50% each.
        
        Проще говоря, если мы повторяем один и тот же случайный эксперимент много раз, то со временем среднее значение результатов будет все ближе к теоретическому или ожидаемому значению.
    - - There are several properties of expectation and variance that are consistent through all distributions
        
        Properties of Expectation
        
        The expected value of two independent random variables is the sum of each expected value separately:
        E(X + Y) = E(X) + E(Y)
        
        For example, if we wanted to count the total number of heads between 10 fair quarter flips and 6 fair nickel flips, the expected value combined would be 5 heads (from the quarters) and 3 heads (from the nickels) so 8 heads overall.
        
        Multiplying a random variable by a constant a changes the expected value to be a times the expected value of the random variable:
        E(aX) = aE(X)
        
        For example, the expected number of heads from 10 fair coin flips is 5. If we wanted to calculate the number of heads from this event run 4 times (40 total coin flips), the expected value would now be 4 times the original expected value, or 20.
        
        Adding a constant a to the distribution changes the expected value by the value a:
        E(X + a) = E(X) + a
        
        Let’s say that a test was given and graded, and the average grade was 78 out of 100 points. If the teacher decided to curve the grade by adding 2 points to everyone’s grade, the average would now be 80 points.
        
        Properties of Variance
        
        Increasing the values in a distribution by a constant a does not change the variance:
        Var(X + a) = Var(X)
        
        This is because the variance of a constant is 0 (there is no range for a single number).
        Let’s take the previous example with the teacher curving grades: though the expected value (or average) of the test changes from 78 to 80, the spread and dispersion (or variance) of the test scores stays the same.
        
        Scaling the values of a random variable by a constant a scales the variance by the constant squared:
        Var(aX) = a^2Var(X)
        
        X = [1, 3, 5, 7, 9]
        mean(X) = 5.
        var(X) = 8.
        Теперь предположим, что мы умножаем каждый элемент датасета X на 2(a):
        aX = [2, 6, 10, 14, 18]
        mean(aX) = 10.
        Теперь давайте применим правило: Var(aX) = a^2 × Var(X)
        В данном случае "a" равно 2 (потому что мы умножили на 2):
        Var(aX) = 2^2 × 8 = 32.
        
        The variance of the sum of two random variables is the sum of the individual variances:
        Var(X + Y) = Var(X) + Var(Y)
        
        This principle ONLY holds if the X and Y are independent random variables. Let’s say that X is the event getting a heads on a single fair coin flip, and Y is the event rolling a 2 on a fair six-sided die:
        Var(X) = 0.5 × (1 - 0.5) = 0.25
        Var(Y) = 0.167 × (1 - 0.167) = 0.139
        Var(X + Y) = Var(X) + Var(Y)
        Var(X + Y) = 0.25 + 0.139 = 0.389
  - - - In probability, we often use random variables to represent random events. For example, we could use a random variable to represent the outcome of a die roll: any number between one and six.
        Random variables must be numeric, meaning they always take on a number rather than a characteristic or quality.
        If we want to use a random variable to represent an event with non-numeric outcomes, we can choose numbers to represent those outcomes. For example, we could represent a coin flip as a random variable by assigning “heads” a value of 1 and “tails” a value of 0.
      - Random Variables In Python
        
        In this lesson, we will use random.choice(a, size = size, replace = True/False) from the numpy library to simulate random variables in python.
        
        The following code simulates the outcome of rolling a fair die twice:
        import numpy as np
        die_6 = range(1, 7)
        rolls = np.random.choice(die_6, size = 2, replace = True)
        print(rolls)
        .#OutPut
        [2, 5]
      - Types of Random variables
        
        Discrete Random Variables
        
        Random variables with a countable number of possible values are called discrete random variables.
        For example, rolling a regular 6-sided die would be considered a discrete random variable because the outcome options are limited to the numbers on the die.
        
        Probability Mass Functions(PMF)
        
        A probability mass function (PMF) is a type of probability distribution that defines the probability of observing a particular value of a discrete random variable.
        There are certain kinds of random variables (and associated probability distributions) that are relevant for many different kinds of problems.
        
        binomial distribution
        
        This is a distribution that models the number of successes in a series of independent binary (two possible outcomes) experiments.
        In binomial distribution we can calculate the probability of observing a specific value
        
        The binom.pmf() method from the scipy.stats library can be used to calculate the PMF of the binomial distribution at any value. This method takes 3 values:
        
        x: the value of interest
        
        n: the number of trials
        
        p: the probability of success
        
        1 more item...
        
        Binomial Distribution: Calculating Probability of a Range
        
        Is about finding the probability of observing a range of values for a discrete random variable
        For example, let’s say we flip a fair coin 5 times, and want to know the probability of getting between 1 and 3 heads.
        We can do it by hand flowing this equation:
        P(1 to 3 heads) = P(X = 1) + P(X = 2) + P(X = 3)
        
        1 more item...
        
        Expected Value of the Binomial Distribution
        
        Consider the following scenario: we flip a fair coin(p = 0.5) 10(n) times and count the number of heads we observe. How many heads would you expect to see?
        equation for the expected value of the binomial distribution:
        Expected(№ of Heads) = E(X) = n × p
        Expected(№ of Heads) = 10 × 0.5 = 5
        
        Variance of Binomial Distribution
        
        Let’s use the 10(n) fair coin flips example to try to understand how variance is calculated. Each coin flip has a certain probability of landing as heads or tails: 0.5 and 0.5, respectively.
        
        1 more item...
        
        Cumulative Distribution Function(CDF)
        Функция накопленной вероятности
        
        Instead of the probability of observing a specific value, the cumulative distribution function gives the probability of observing a specific value OR LESS.
        The value of a cumulative distribution function at a given value is equal to the sum of the probabilities lower than it, with a value of 1 for the largest possible number.
        
        Example:
        We showed how the probability mass function can be used to calculate the probability of observing less than 3 heads out of 10 coin flips by adding up the probabilities of observing 0, 1, and 2 heads. The cumulative distribution function produces the same answer by evaluating the function at CDF(X=2).
        In this case, using the CDF is simpler than the PMF because it requires one calculation rather than three.
        
        Calculating range using CDF
        
        We can use a CDF to calculate the probability of a specific range by taking the difference between two values from the CDF.
        For example, to find the probability of observing between 3 and 6 heads, we can take the probability of observing 6 or fewer heads and subtracting the probability of observing 2 or fewer heads. This leaves a remnant of between 3 and 6 heads.
        
        1 more item...
        
        Calculating range using CDF in Python
        
        1 more item...
        
        CDF in Python
        
        binom.cdf() method takes 3 values:
        
        x: the value of interest, looking for the probability of this value or less
        
        n: the sample size
        
        p: the probability of success
        
        2 more items...
        
        Poisson Distribution
        
        The Poisson distribution is another common distribution, and it is used to describe the number of times a certain event occurs within a fixed time or space interval.
        For example, the Poisson distribution can be used to describe the number of cars that pass through a specific intersection between 4pm and 5pm on a given day.
        It can also be used to describe the number of calls received in an office between 1pm to 3pm on a certain day.
        
        The Poisson distribution is defined by the rate parameter, symbolized by the Greek letter lambda, λ. Lambda represents the expected value — or the average value — of the distribution.
        Например, если "λ" равно 3, это означает, что в среднем мы ожидаем видеть 3 события в данном интервале времени или месте.
        
        Calculating exact value
        
        For example, suppose that we expect it to rain 10 times in the next 30 days. The number of times it rains in the next 30 days is “Poisson distributed” with lambda = 10. We can calculate the probability of exactly 6 times of rain as follows:
        
        1 more item...
        
        Calculating range
        
        For example, if we expect it to rain 10 times in the next 30 days, the number of times it rains in the next 30 days is “Poisson distributed” with lambda = 10. We can calculate the probability of 12-14 times of rain as follows:
        
        1 more item...
        
        Cumulative Distribution Function(CDF) in Poisson Distribution
        
        Calculating probability
        
        2 more items...
        
        Calculating range
        
        1 more item...
        
        expectancy(central tendency) and spread
        
        central tendency
        
        1 more item...
        
        spread
        
        1 more item...
        
        calculating perecetniles in Poisson desrtibution
        
        Use this method to calculate the number of defects that would put us in the 90th percentile for a given day. In other words, on 90% of days, we will observe fewer defects than this number:
        .# stats.poisson.ppf(percentile, lambda)
        print(stats.poisson.ppf(0.9, 7))
        Output:
        10.0
        
        Continuous Random Variables
        
        When the possible values of a random variable are uncountable, it is called a continuous random variable.
        These are generally measurement variables and are uncountable because measurements can always be more precise – meters, centimeters, millimeters, etc.
        
        Probability Density Function(PDF)
        Функция плотности вероятности
        
        Probability Density Functions define the probability distributions of continuous random variables and span across all possible values that the given random variable can take on.
        Проще говоря, PDF дает нам информацию о том, как вероятность "плотно" распределена вдоль диапазона значений непрерывной переменной. На графике PDF можно увидеть, как вероятность изменяется в зависимости от значений случайной переменной.
        
        When graphed, a probability density function is a curve across all possible values the random variable can take on, and the total area under this curve adds up to 1.
        
        In a probability density function, we cannot calculate the probability at a single point.
        Вероятность попадания в одну конкретную точку на непрерывной числовой оси (например, как 35 градусов Цельсия) для непрерывного распределения (например, нормального) равна нулю. Это связано с тем, что вероятность в непрерывном распределении определяется площадью под графиком функции плотности вероятности в данном диапазоне.
        Другими словами, когда мы говорим о вероятности в непрерывном распределении, мы рассматриваем интервалы (например, "от 34.5 до 35.5 градусов"), а не конкретные точки (например, "ровно 35 градусов").
        
        Cumulative Distribution Function IN PDF
        
        Сalculate the area under the curve using CDF
        
        To calculate the area of the blue region of curve in Python, we can use the norm.cdf(). This method takes on 3 values:
        
        x: the value of interest
        
        loc: the mean of the probability distribution
        
        scale: the standard deviation of the probability distribution
        
        2 more items...
        
        Calculating the range for continuous distributions
        
        This is essentially the same process as calculating the probability of a range of values for discrete distributions.
        
        1 more item...
- - - - Знакомство с данными: Ознакомление с набором данных, понимание его структуры, размерности, типов переменных и общего формата.
        
        Очистка данных: Выявление и обработка отсутствующих значений, дубликатов, выбросов и ошибок в данных.
        
        Визуализация данных: Построение различных графиков (например, столбчатые диаграммы, гистограммы, круговые диаграммы, разброса и т. д.) для визуального представления распределения данных и выявления закономерностей.
        
        Статистический анализ: Применение статистических методов для вычисления описательных статистик, таких как среднее, медиана, стандартное отклонение и корреляция между переменными.
        
        Выявление связей и зависимостей: Поиск взаимосвязей между различными переменными и выявление паттернов, трендов или корреляций.
        
        Формулирование гипотез: По результатам анализа данных можно выдвигать гипотезы о свойствах данных, которые позволяют провести более точное исследование.
        
        Принятие решений: EDA может помочь принять решения по выбору моделей машинного обучения, адаптировать стратегии анализа и предложить возможные подходы к решению конкретных задач.
      - EDA techniques
        
        Data inspection
        
        Numerical summarization
        
        Data visualization
        
        Data inspection
        Data inspection is an important first step of any analysis. This can help illuminate potential issues or avenues for further investigation.
        
        Example:
        print(data.head())
        
        Numerical summarization:
        For numerical data, this allows us to get a sense of scale, spread, and central tendency. For categorical data, this gives us information about the number of categories and frequencies of each.
        
        Example:
        data.describe(include = 'all')
      - EDA as a cyclical process
        
        Though EDA is commonly performed at the start of a project — before any analysis or model building — you may find yourself revisiting EDA again and again.
      - Exploratory Data Analysis (EDA) – это процесс исследования данных с целью изучения их структуры, выявления особенностей и закономерностей, а также проверки гипотез о данных. EDA является важным этапом в анализе данных, предшествующим более сложным статистическим методам и построению моделей.
    - - Categorical Variables
        
        Nominal variables, which describe something.
        When we want to describe something about the world, we need a nominal variable. Nominal variables are usually words (i.e., red, yellow, blue or hot, cold), but they can also be numbers (i.e., zip codes or user id’s).
        
        Ordinal variables, which have an inherent ranking.
        When our categories have an inherent order, we need an ordinal variable. Ordinal variables are usually described by numbers like 1st, 2nd, 3rd. Places in a race, grades in school, and the scales in survey responses (Likert Scales) are ordinal variables.
        
        Binary variables, which have only two possible variations.
        When there are only two logically possible variations, we need a binary variable. Binary variables are things like on/off, yes/no, and TRUE/FALSE. If there is any possibility of a third option, it is not a binary variable.
      - Quantitative Variables
        
        Numerical variables
        
        Continuous(measurements)
        For a variable to be continuous, there must be infinitely smaller units of measurement between one unit and the next unit. Length, time, and temperature are all good examples of continuous variables because they all increase continuously.
        
        Discrete(counts)
        For a variable to be discrete, there must be gaps between the smallest possible units. People, cars, and dogs are all good examples of discrete variables.
        
        Some variables depend on context to determine if they are continuous or discrete. Money and time can both be measured continuously or discretely.
        It is therefore always essential to understand how your data was created in order to represent it appropriately.
      - Changing Numerical Variable Data Types
        
        If a variable appears with the wrong data type, we can change it with the .astype() function.
        
        Best practices for data storage say that we should match the data type of the column with its real-world variable type.
        Therefore:
        Continuous (numerical) variables should usually be stored as the float data type because they allow us to store decimal values.
        Discrete (numerical) variables should be stored as the int datatype to represent mathematically that they are discrete.
      - Changing Categorical Variable Data Types
        
        Best practices for categorical data storage say that we should match the data type of the column with its real-world variable type. However, the types are a little more nuanced:
        
        Nominal variables could be represented as objects and as strings
        
        For example:
        cereal['name'] = cereal['name'].astype("string")
        
        Ordinal variables should be represented as objects, but pandas often guesses int since they are often encoded as whole numbers.
        
        The pandas .Categorical() method can be used to store data as type category and indicate the order of the categories.
        cereal['shelf'] = pd.Categorical(cereal['shelf'], ['bottom', 'mid', 'top'], ordered=True)
        Python recognize that the shelf column is an ordinal variable, it understands that top > mid > bottom.
        Output will be:
        [bottom, mid, top]
        Categories (6, object): [bottom < mid < top]
        
        Binary variables can be represented as bool, but pandas often guesses int or object data types.
      - One-Hot Encoding
        
        One-Hot Encoding (OHE) - это метод преобразования категориальных данных в числовой формат, который понятен компьютерам и может быть использован в различных алгоритмах машинного обучения.
        Когда мы применяем One-Hot Encoding к категориальным данным, каждая уникальная категория превращается в отдельный столбец, а для каждой строки данных устанавливается значение 1 в столбце, соответствующему категории, которой эта строка принадлежит, и 0 во всех остальных столбцах.
        
        . It is important to note that OHE works best when we do not create too many additional variables, as increasing the dimensionality of our dataframe can create problems when working with certain machine learning models.
        
        titanic = pd.get_dummies(data=titanic, columns=['Embarked'])
        print(titanic.head())
      - numerical codes associated with each category
        
        Метод cat.codes используется для преобразования каждого уникального значения в этом столбце в соответствующее числовое представление.
        например из этого:
        strongly disagree', 'disagree', 'neutral', 'agree', 'strongly agree'
        мы получим это:
        [0, 1, 2, 3, 4]
        
        For example we coud use tha for Likert scale. Supose we have colun in our survey data named higher_tax. In thic column stored survey responses like: 'strongly disagree', 'disagree', 'neutral', 'agree', 'strongly agree'. And we'd like to find out the median value of survey response.
        
        census['higher_tax'] = pd.Categorical(census['higher_tax'], ['strongly disagree', 'disagree', 'neutral', 'agree', 'strongly agree'], ordered=True)
        census['higher_tax'] = census['higher_tax'].cat.codes
        print(census['higher_tax'].median())
        output:
        2
        В данном случае медианный ответ 2 указывает на то, что большинство респондентов выбрали ответ "neutral" (нейтрально) на данный вопрос, так как это значение является "разделяющим" значение, которое делит данные пополам.
    - - Introduction
        Summary statistics are an important component of Exploratory Data Analysis (EDA) because they allow a data analyst to condense a large amount of information into a small set of numbers that can be easily interpreted.
        In order to decide what kind of summary statistic to use, it is important to consider two things:
        
        The question (and how many variables that question involves)
        
        The data (is it quantitative or categorical?)
      - Univariate Statistics
        
        Summary statistics that focus on a single variable are called univariate statistics.
        Univariate statistics can help us answer questions like:
        
        How much does a typical car cost?
        
        What proportion of cars have a manual transmission?
        
        How old is the oldest listed car?
        
        Each of these questions focuses on a single variable (selling_price, transmission, and year, respectively, for the above examples).
        
        Depending on the type of variable, different summary statistics are appropriate.
      - Quantitative Variables
        
        Central Location(Central Tendency)
        
        Mean: Also called the “average”; calculated as the sum of all values divided by the number of values.
        
        Median: The middle value of the variable when sorted.
        
        Mode: The most frequent value in the variable.
        
        Trimmed Mean: The mean excluding x percent of the lowest and highest data points.
        
        Sometimes we should polt our data to investigate it(shape)
        
        Mean:
        df.column.mean()
        
        Median:
        df.column.median()
        
        Mode:
        df.column.mode()
        (or mode()[0])
        
        Trimmed mean:
        from scipy.stats import trim_mean
        trim_mean(df.column, proportiontocut=0.1) # trim extreme 10%
        
        Spread
        
        Range: The difference between the maximum and minimum values in a variable.
        
        Inter-Quartile Range (IQR): The difference between the 75th and 25th percentile values.
        
        Variance: The average of the squared distance from each data point to the mean.
        
        Standard Deviation (SD): The square root of the variance. Например, если SD = 806253.40 ,то это означает, что значения датасете, в среднем, отклоняются от среднего значения на примерно 806253.40.
        
        Mean Absolute Deviation (MAD): это статистическая мера разброса данных, которая позволяет оценить, насколько значения в наборе данных отклоняются от их среднего значения. В простых словах, MAD показывает, насколько "в среднем" данные отличаются от их среднего значения.
        
        For highly skewed data or data with extreme outliers, we therefore might prefer to use IQR or MAD. For data that is more normally distributed, the variance and standard deviation are frequently reported.
        
        Range:
        df.column.max() - rentals.rent.min()
        
        Inter-Quartile Range (IQR):
        df.column.quantile(0.75) - df.column.quantile(0.25)
        OR
        from scipy.stats import iqr
        iqr(rentals.rent)
        
        Variance:
        df.column.var()
        
        Standard Deviation (SD):
        df.column.std()
        
        Mean Absolute Deviation (MAD):
        df.column.mad()
      - Categorical Variables(discrete)
        
        Ordinal
        
        For ordinal categorical variables, we may still want to summarize central location and spread.
        we should NOT calculate the mean of an ordinal categorical variable (or anything that relies on the mean, like variance, standard deviation, and MAD).
        
        Pandas:
        df.['column'].value_counts()
        AND
        df.['column'].value_counts(normalize=True)
        OR
        df.column.value_counts() / len(df.column)
        
        Nominal
        
        For nominal categorical variables (and ordinal categorical variables), another common numerical summary statistic is the frequency or proportion of observations in each category.
        This is often reported using a frequency table and can be visualized using a bar chart.
      - Bivariate Statistics
        
        In contrast to univariate statistics, bivariate statistics are used to summarize the relationship between two variables.
        They are useful for answering questions like:
        
        Do manual transmission cars tend to cost more or less than automatic transmission?
        
        Do older cars tend to cost less money?
        
        Are automatic transmission cars more likely to be sold by individuals or dealers?
        
        Depending on the type of variable, different summary statistics are appropriate.
        
        One Quantitative Variable and One Categorical Variable
        
        If we want to know whether manual transmission cars tend to cost more or less than automatic transmission cars, we are interested in the relationship between transmission (categorical) and selling_price (quantitative). To answer this question, we can use a mean or median difference.
        
        For example, we could calculate that the median price of automatic transmission cars is 100000 Rupees higher than for manual transmission cars.
        
        mean and median difference
        
        median difference
        
        median_GP = np.median(scores_GP)
        median_MS = np.median(scores_MS)
        print(median_GP) #Output: 11.0
        print(median_MS) #Output: 10.0
        print(median_GP-median_MS) #Output: 1.0
        
        mean difference
        
        mean_GP = np.mean(scores_GP)
        mean_MS = np.mean(scores_MS)
        print(mean_GP) #output: 10.49
        print(mean_MS) #output: 9.85
        print(mean_GP - mean_MS) #Output: 0.64
        
        Side-by-Side Box Plots
        
        Set data equal to the name of the dataset (students), x equal to the column name of your categorical variable (address), and y equal to your quantitative variable (G3).
        
        sns.boxplot(data = df, x = 'school', y = 'G3')
        plt.show()
        
        While overlapping histograms and side by side boxplots can convey similar information, histograms give us more detail and can be useful in spotting patterns that were not visible in a box plot (eg., a bimodal distribution).
        
        To answer the quastion: "How do we know whether this difference is considered small or large?", we should look on overlap between the boxes.
        If we see that there is a lot of overlap between the boxes, its seems to be not much difference between variables.
        if we see that the boxes barely overlap, its seems to bee much difference between variables.
        
        Inspecting Overlapping Histograms
        
        plt.hist(scores_GP , color="blue", label="GP", density=True, alpha=0.5)
        plt.hist(scores_MS , color="red", label="MS", density=True, alpha=0.5)
        plt.legend()
        plt.show()
        
        Exploring Non-Binary Categorical Variables
        
        Pair-wise comparisons for a quantitative and a non-binary categorical variable
        For example, suppose we want to know whether or not an association exists between math scores (G3) and (Mjob), a categorical variable representing the mother’s job. This variable has five possible categories: at_home, health, services, teacher, or other.
        For example, we can compare scores for students whose mothers work at_home or in health; at_home or other; at home or `services; etc.
        
        Remember, x equal to the column name of your categorical variable (Mjob), and y equal to your quantitative variable (G3).
        sns.boxplot(data = df, x = 'Mjob', y = 'G3')
        plt.show()
        
        Two Quantitative Variables
        
        If we want to know whether older cars tend to cost less money, we are interested in the relationship between year and selling_price, both of which are quantitative. To answer this question, we can use the Pearson correlation.
        
        For example, if we calculate that the correlation between year and selling_price is 0.4, we can conclude that there is a moderate positive association between these variables (older cars do tend to cost less money).
        
        Scatter Plots
        
        different kinds of associations can lead to different patterns in a scatter plot.
        
        plt.scatter(x = housing.price, y = housing.sqfeet)
        plt.xlabel('Rental Price (USD)')
        plt.ylabel('Area (Square Feet)')
        plt.show()
        
        Covariance
        
        Ковариация является мерой степени, с которой две переменные изменяются вместе.
        
        A positive covariance indicates that a larger value of one variable is associated with a larger value of the other.
        
        A negative covariance indicates a larger value of one variable is associated with a smaller value of the other.
        
        A covariance of 0 indicates no linear relationship.
        
        Ключевое ограничение ковариации заключается в том, что ее значение не нормировано и зависит от единиц измерения переменных, что делает сложным сравнение ковариаций разных пар переменных.
        Если вам нужно оценить связь между переменными рекомендуется использовать корреляцию
        
        To calculate covariance, we can use the cov() function from NumPy, which produces a covariance matrix for two or more variables:
        variable 1 | variable 2
        variable 1 | variance(variable 1) | covariance
        variable 2 | covariance | variance(variable 2)
        
        1 more item...
        
        Correlation
        
        Pearson Correlation (often referred to simply as “correlation”) is a scaled form of covariance. It also measures the strength of a linear relationship, but ranges from -1 to +1, making it more interpretable.
        
        Highly associated variables with a positive linear relationship will have a correlation close to 1.
        
        Highly associated variables with a negative linear relationship will have a correlation close to -1.
        
        Variables that do not have a linear association (or a linear association with a slope of zero) will have correlations close to 0.
        
        Generally, a correlation larger than about .3 indicates a linear association. A correlation greater than about .6 suggestions a strong linear association.
        If shape our scatter plot looks like curve or perfectly harizontal line, correlation = 0
        
        from scipy.stats import pearsonr
        corr_price_sqfeet, p = pearsonr(housing.price, housing.sqfeet)
        print(corr_price_sqfeet) #output: 0.507
        
        Two Categorical Variables
        
        If we want to know whether automatic transmission cars are more likely to be sold by individuals or dealers, we are interested in the relationship between transmission and seller_type, both of which are categorical. We can explore this relationship using a contingency table and the Chi-Square statistic.
        It is important to know that all the methods used in this article can be used with non-binary variables.
        
        Contingency Tables: Frequencies
        
        Contingency tables, also known as two-way tables or cross-tabulations, are useful for summarizing two variables at the same time.
        To assess whether there is an association between these two variables, we need to ask whether information about one variable gives us information about the other.
        
        influence_leader_freq = pd.crosstab(npi.influence, npi.leader)
        print(influence_leader_freq)
        
        Contingency Tables: Proportions
        
        influence_leader_freq = pd.crosstab(npi.influence, npi.leader)
        influence_leader_prop = influence_leader_freq/len(npi)
        print(influence_leader_prop)
        
        Marginal Proportions
        
        The proportion of respondents in each category of a single question is called a marginal proportion.
        Маргинальные пропорции - это показатели, которые показывают долю или процент какой-то группы внутри всего общего числа.
        Допустим, у нас есть таблица с данными о студентах в школе, и мы хотим узнать, сколько мальчиков и девочек есть в каждом классе и в общей школе.
        
        leader_marginals = influence_leader_prop.sum(axis=0)
        print(leader_marginals)
        influence_marginals = influence_leader_prop.sum(axis=1)
        print(influence_marginals)
        
        Expected Contingency Tables
        
        Вместо фактических наблюдаемых значений в ячейках таблицы ожидаемые таблицы содержат значения, которые можно было бы ожидать или предсказать, если бы никакой связи между категориями не существовало (т.е., если бы они были независимыми). Эти ожидаемые значения вычисляются на основе сумм и пропорций в таблице сопряженности, а также предположения о независимости между категориями.
        Ожидаемые таблицы сопряженности играют важную роль в статистическом анализе, особенно в тестах независимости, таких как Хи-квадрат тест. Сравнение наблюдаемых значений с ожидаемыми позволяет определить, есть ли значимая связь между категориями или переменными, или они независимы. Если наблюдаемые значения сильно отличаются от ожидаемых, это может указывать на наличие взаимосвязи или влияние между категориями.
        
        To calculate these expected proportions, we need to multiply the marginal proportions for each combination of categories:
        leader | influence
        no 0.484 | no 0.388
        yes 0.516 | yes 0.612
        [!!Open me!!]
        leader = no | leader = yes
        influence = no | 0.4840.388 = 0.188 | 0.5160.388 = .200
        influence = yes | 0.4840.612 = 0.296 | 0.5160.612 = 0.315
        
        These proportions can then be converted to frequencies by multiplying each one by the sample size (11097 for this data):
        [!!Open me!!]
        leader = no | leader = yes
        influence = no | 0.18811097 = 2087 | 0.20011097 = 2221
        influence = yes | 0.29611097 = 3288 | 0.31511097 = 3501
        
        1 more item...
        
        The Chi-Square Statistic
        
        Chi-Square statistic (статистика Хи-квадрат) является числовым выражением связи или ассоциации между переменными в таблице сопряженности
        Для 2x2 table значение 4 для статистики Хи-квадрат может рассматриваться как некая граница, после которой можно говорить о возможной связи между переменными. Однако, это не жесткое правило, и значение 4 просто указывает на более вероятное присутствие связи.
        (Это чем-то похоже на p-value и его границу 0,05)
        
        from scipy.stats import chi2_contingency
        chi2, pval, dof, expected = chi2_contingency(influence_leader_freq)
        print(chi2)
        output: 1307.88
        
        Often, when prints chi2 prints pval
    - - Introduction
        
        To choose visualization for EDA you must answer this quastions:
        
        What the question you want to answer (and how many variables that question involves).
        
        The data is quantitative or categorical?
      - Univariate analysis
        
        Quantitative variables
        
        Box plots (or violin plots)
        
        sns.boxplot(x='column', data=dataFrame)
        plt.show()
        
        histograms
        
        if you want to ge more detail histogram add some additional binы(bins=50), 50 bins instead of 10
        sns.displot(dataFrame.column, bins=10, kde=False)
        plt.show()
        
        Categorical variables
        
        Bar chart(this is prefer)
        
        sns.countplot(x='column', data=dataFrame, palette='winter')
        plt.show()
        
        Pie chart(look in matplotlib mind map)
        
        borough_labels = ["Manhattan", "Brooklyn", "Queens"]
        plt.pie(dataFrame.coulmn.value_counts(), labels = borough_labels)
        plt.show()
      - Bivariate analysis
        
        One quantitative variable and one categorical variable
        
        side-by-side box plots
        
        sns.boxplot(x='column', y='column1', data=dataFrame, palette='Accent')
        plt.show()
        
        overlapping histograms
        
        plt.hist(rentals.rent[rentals.borough=='Manhattan'], label='Manhattan', density=True, alpha=.5)
        plt.hist(rentals.rent[rentals.borough=='Queens'], label='Queens', density=True, alpha=.5)
        plt.hist(rentals.rent[rentals.borough=='Brooklyn'], label='Brooklyn', density=True, alpha=.5)
        plt.legend()
        plt.show()
        
        overlapping histograms
        
        sns.histplot(dataFrame, x = "column", hue = dataFrame.cloumn1l)
        plt.show()
        
        Two quantitative variables
        
        scatter plot
        
        sns.scatterplot(dataFrame.column, dataFrame.column1)
        plt.show()
        
        Two categorical variables
        
        side by side bar(prefer)
        
        sns.countplot(x='column', hue='column1', data=dataFrame)
        plt.show()
        
        stacked bar
        
        look at matplotlib mind map
      - Multivariate analysis
        
        two categorical variables and a quantitative variable.
        
        Grouped box plots
        
        sns.boxplot(x = "Education", y = "CompTotal", palette = "pastel", data = salary_data)
        plt.show()
        
        side by box plots
        
        sns.boxplot(x = "Education", y = "CompTotal", hue = "Gender", palette = "pastel", data = salary_data)
        plt.show()
        
        Multi-dimensional plots(scatter plot)
        
        Don't use a 3d plots if you need wrtie a report. 3D plots can be difficult to read in two dimensions.
        
        .# import library
        import plotly.express as px
        .# load in iris data
        df = px.data.iris()
        .# create 3D scatter plot
        fig = px.scatter_3d(df, x='sepal_length', y='sepal_width', z='petal_width', color='species')
        fig.show()
        
        Quantitative variables
        
        Scatter plot with additional:
        
        colors,
        
        shapes,
        
        and patterns
        
        sns.scatterplot(x = 'column', y = 'column1', hue = 'Status', palette = 'bright', data = dataFrame)
        plt.show()
        
        heat map of a correlation matrix
        
        ask chat gpt
      - Time Series Data
        
        We have a sales data(csv) from a small business over a five-year period. Let's visualize this data using different ways.
        
        Line plot
        
        .# convert string to datetime64
        sales_data["date"] = sales_data["date"].apply(pd.to_datetime)
        sales_data.set_index("date", inplace=True)
        plt.plot(sales_data["date"], sales_data["sales"])
        plt.xlabel("Date")
        plt.ylabel("Sales (USD)")
        plt.show()
        
        Box plot
        
        .# extract year from date column
        sales_data["year"] = sales_data["date"].dt.year
        .# box plot grouped by year
        sns.boxplot(data=sales_data, x="year", y="sales")
        plt.show()
        
        Heatmap
        
        .# calculate total sales for each month
        sales = sales_data.groupby(["year", "month"]).sum()
        .# re-format the data for the heat-map
        sales_month_year = sales.reset_index().pivot(index="year", columns="month", values="sales")
        .# create heatmap
        sns.heatmap(sales_month_year, cbar_kws={"label": "Total Sales"})
        plt.title("Sales Over Time")
        plt.xlabel("Month")
        plt.ylabel("Year")
        plt.show()
        
        Lag scatter plot
        
        interpretation of lag scatter plot(it's like in correlation):
        
        If the points move from the bottom left to the top right - positive correlation between observations and their lag 1 values.
        For example, high sales on one day are associated with high sales on the previous day.
        
        If the points move from the top left to the bottom right - negative correlation between observations and their lag 1 values.
        For example, high sales on one day are associated with low sales on the previous day and vice versa.
        
        there is no identifiable structure in the lag plot - Data is random
        
        .# import lag_plot function
        from pandas.plotting import lag_plot
        .# lag scatter plot
        lag_plot(dataFrame)
        plt.show()
        
        Autocorrelation plot
        
        If autocorrelation plot for the sales data forms waves, oscillating between strong negative and positive correlation - dataset seasonable.
        
        If autocorrelation decreases over time - it indicates a trend.
        
        Fast fading - no trend or seasonality
        
        White noise - data is random
        
        .# import autocorrelation function
        from pandas.plotting import autocorrelation_plot
        .# autocorrelation plot
        autocorrelation_plot(dataFrame)
        plt.show()
      - Data Visualizations for Messy Data
        
        Beforme plotting handle missing data
        
        Numerical data
        
        Scatter plot
        
        Just plot data
        
        housing = pd.read_csv('housing.csv')
        sns.scatterplot(housing['sqfeet'], housing['price'])
        
        1 more item...
        
        Too much outliers
        
        2 more items...
        
        log scale
        
        just plot data(histogram)
        displot - это функция Seaborn для создания гистограммы распределения.
        
        sns.displot(dataFrame.column)
        
        1 more item...
        
        Log transformation
        
        log_price = dataFrame.column[dataFrame.column>0]
        log_price = np.log(log_price)
        sns.displot(log_price)
        plt.xlabel('log price')
        
        1 more item...
        
        getting closer
        
        1 more item...
        
        Categorical data
        
        scatterplot
        
        sns.scatterplot('beds', 'baths', data = housing_sub)
        
        improve visualization
        
        data is perfectly overlapping and we don't see anything.
        Adding a jitter adjusts the spread of points along either (or both) axes in order to more easily see some many points there are in each group:
        
        sns.lmplot('beds', 'baths', data = housing_sub, x_jitter = .15, y_jitter = .15, fit_reg = False)
        
        1 more item...
        
        if want to see regressin line
        
        1 more item...
        
        sns.lmplot('colimn','colimn1', data=dataFrame, x_jitter = .25, scatter_kws={"s": 5, "alpha":0.15}, fit_reg=False)
      - Data Visualization Best Practices
        
        General
        
        Legends and colors should be used when plotting data across multiple groups.
        
        clearly labeled axes that are evenly spaced
        
        Example:
        
        Context metter
        
        What to avoid
        
        Pie charts
        
        Stacked bar charts
        
        Stacked area charts
      - Правило для выбора размера фигуры:
        Например, если для одного графика подходит размер (5, 3), то для сетки из 7 строк и 2 столбцов можно было бы начать с размера (10, 21) (умножили ширину на 2 и высоту на 7).