Project 3¶

Regression using Kaggle House Price dataset¶

1) The dataset¶

This Kaggle competition involves using the Ames Housing dataset to predict house prices using a large array of variables.

This is pre-split into a training set and testing set with the caveat being that the only way to score my predictions using the test set is to submit the predictions to the Kaggle competition.

According to Kaggle:

Submissions are evaluated on Root-Mean-Squared-Error (RMSE) between the logarithm of the predicted value and the logarithm of the observed sales price. (Taking logs means that errors in predicting expensive houses and cheap houses will affect the result equally.)

I intend to experient with 3 different methods:¶

• 1) Use all of the variables
  • Use the numerical data as is
  • Convert the ones I can into numerical variables and give each one a unique value
  • deal with missing values
  • remove collinearity
• 2) Trying stepwise regression and using only features with the highest impact
  • impute all using most frequent
  • one-hot encoding every feature
  • remove all remaining non numeric features
  • remove collinearity
• 3) Simply Factorizing (turning all categorical into numbers)

2) What is regression and how does it work?¶

Explain what regression is and how it works (specifically linear regression, as we have covered it in class). Bonus points for math (up to 5 bonus points for very good explanations).¶

Linear regression generalizes variables to calculate an output. It takes one or more variables with coefficients and constant(s) as a function and using this you can ‘predict.’ These are great when some factor has a relationship with another. For example, an easy visualization is with students: the more higher they score on tests, the higher their final exam grade will be. One could include many variables alongside this such as attendance, homework grades and quiz grades. However, this may be tricky when trying this with calculating final grades. There is linear dependence between test and quiz grades and the final grade of a class, at least when you include all grades.
There are 5 assumptions that must be made when performing linear regression:

• All variables are independent of one another (no multicollinearity)
• There is a linear relationship them (they are related)
• Resisudal errors should be normally distributed
• All observations are independent (the value of a single occurance should not depend on another)
• There are no missing values

There are 2 main statistics to look at after running this model:

• r², the coefficient of determination, represents the percentage of variation that your dependent variable can be explained by your independent variable(s). The closer this value is to 1 the better your data variation can explain your results.
• and RMSE, Root Mean Square Error, which represents how well a model is able to fit the dataset. The closer this value is to 0 the better.

Work in progress

3) Experiment 1: Data understanding¶

Before diving right into the work, what steps do you take to first gain an understanding of your data? For example, are any features correlated with each other? Are there any existing patterns? What visualizations can you make to help gain this initial understanding?¶

Package pre-reqs:

• Pandas for loading and manipulating the csv files
• sklearn for statistical analysis
• seaborn for fancy plots and charts

To get started, we first must load our dataset

1460 entries and 81 variables!¶

We really don’t want or need the Id column for predictions

I will not name all of the variables, I will however describe some that I believe will be the most important (we’ll find out soon!)

• MSZoning (categorical): Identifies the general zoning classification of the sale.
• LotArea (continuous): Lot size in square feet
• Neighborhood (categorical): Physical locations within Ames city limits
• OverallQual (ordinal): Rates the overall material and finish of the house
• 1stFlrSF: First Floor square feet

Let’s see if we can find some patterns! First a correlation heatmap

This is a bit much, either way we know that some variables are highly correlated with others. If we’re looking for relationships to the target, SalePrice then we know that the following are highly related:

• OverallQual
• TotalBsmtSF
• 1stFlrSF
• GrLivArea
• GarageCars
• GarageArea

And these are just the ones with more than a 0.6 correlation.
Now let’s explore what variables are highly related to other variables using the 0.60 cutoff again:

• GarageCars and OverallQual
• GarageCars and GarageArea
• GarageYrBlt and YearBuilt (duh?)
• GarageYrBlt and YearRemodAdd
• TotRmsAbvGrd and GrLivArea
• 1stFlrSF and TotalBsmtSF (makes sense, usually basements are around the same size as the ‘ground’ floor)

With these in mind…

4) Experiment 1: Pre-processing¶

After the initial understanding, what steps need to be taken? Do you need to deal with null values? What features are you going to try using for your first experiment and why? Do you need to transform any features (e.g., change from categorical to numerical)? Are you trying to create any new features to use?¶

Let’s check for null values.

In a dataset with 1460 entries, it is unlikely that these variables with large amounts of null values will make a huge difference.
Let’s verify this with the matrix above comparing these variables with SalePrice:

• MiscFeature: N/A (categorical) cannot alter for linear regression.
• PoolQC: N/A (categorical) could perhaps convert this to an ordinal value. 1453 nulls.
• Alley: N/A (categorical) cannot alter for linear regression.
• Fence: N/A (categorical) could perhaps convert this to an ordinal value. 1179 nulls.
• FireplaceQu: N/A (categorical) could perhaps convert this to an ordinal value. 690 nulls.
• LotFrontage: 0.4 with 259 nulls.

I am only willing to convert FirePlaceQu (Fire place quality) to a numerical variable because it has fewer than half the total values missing. Let’s try it out.
Let’s convert all ordinal variables into numeric values.
I will use 1 as the highest quality with the higher the value the lower the quality is.

We will also delete these fields with a large number of NA values

Let’s check our collinearity again. Also since we’re only interested in numeric fields here let us update our df to reflect this. This will make our job easier down the line.¶

NA values and non-numerical values will cause issue when we try to reduce collinear fields¶

Dropping these problematic rows yeilds a 364 entry loss, or 24.9% of the previous df

Okay so it worked, there are more fields… now what? Well let’s get rid of some of this collinearity. I found a cool package called ‘collinearity’ and we’re gonna put it to use.¶

We must create a nonCollinar object, and I’m going to set the threshold to 0.4. That should be good enough for our regression right?¶

Also, number of features before removing collinearity

And number of features after removing collinearity

Down to 25 features! Hopefully these are the most impactful.

Also we’re going to try a pairwise plot so we can try to identify trends

Truth be told, I wish that were useful but there are too many fields at this time.
What would be useful though is a list of final features.

4a) Experiment 1: Modeling¶

Create a linear regression model for your first experiment.¶

Documentation: https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html

Creating our first linear model object¶

Let’s jump straight into it. Let’s fit it and score it!
In order to do this, we’ll split this into a 20/80 test/train split

4b) Experiment 1: Evaluation¶

Evaluate your first model. For example, use root mean squared error (RMSE).¶

Note: if you would like to try submitting your work to the competition, their submissions are evaluated using RMSE. Submitting to the competition is not required, but it might be fun to experience! More information on how to submit is under the "Overview" tab, then "Evaluation."

An r² score of 0.73. Wow 73% is better than I expexted. That is how well our dependent variable can be explained by the model using our independent variables.

Let’s get the test csv set up to be scored by Kaggle¶

First, how many are we predicting?

1459!

Okay, so these missing values are going to mess with us and this time we can’t simply drop them because we need all 1459 to get the best score possible. So we’ll use sklearns imputing abilities and try out filling in the values with mean, median or most_frequent.¶

Okay…let’s predict then set up the csv.¶

It should be formatted: Id,SalePrice

We’ve done it! Let’s score this thing up.¶

Kaggle has given me a score of 0.98067
That’s good I think?
Welp, I’m in place 3882 out of ~4000 sooooo
Yeah, looks like a score closest to zero is best. Let’s work on that!

5) Experiment 2¶

For the next experiment, discuss what you try changing compared to the first experiment. For example, do you want to experiment with different features? Are you trying a different model? Then, talk about your results and how it changes from the previous experiment.¶

Preprocessing:¶

Reset back to our starting position

Step 1: impute all features using the most frequent value. AKA replace all missing values with the most common value¶

+ impute all using most frequent
+ one-hot encoding every feature
+ remove collinearity

No missing values now!¶

Welp, these aren’t the data types we want.¶

Let’s let pandas infer them to get the numeric types correct, then we’ll manually set the remaining object types to categories.
Afterwards, we’ll one-hot encode them

So we still have categorical fields, let’s make sure they’re typed. TODO: change some of these to have the ordered attribute and see if that matters.

Step 2) One-hot encoding¶

Let’s make sure that no column has more than 10 or so unique values before we one-hot-encode this

We’d better check these out, are these even categorical??

• Neighborhood: Checks out
• Exterior1st: Checks out
• Exterior2nd: Also…checks out.

81 features before adding dummies….
247 features after adding dummies. We made sure to remove the original as well. That pesky collinearity…

Step 3) Remove collinearity¶

For the first experiment, we created the rem_collinear_features() function to help remove collinearity. Let’s use that again. First, we must split this into x and y, then, in this order, pass: correlation_threshold, scoring, x_df, and y_df.

Let’s see what we’re left with afterwards… before our shape would’ve been 1460×246 (if we exclude the SalePrice)

Let’s experiment!¶

Our previous scores were…

0.73 r^2 score 44227 RMSE score

We’ve a higher r² score and a lower RMSE!
Now: 0.78 r² score 40278 RMSE score

Let’s Kaggle it!¶

To do this we must: impute, change the dtypes, one-hot-encode, and make sure we’ve the same columns

What in tarnation… This test dataset doesn’t have all of the same unique categories in some columns as the training set.
We can’t use our model to predict wit these NA columns… unless… let’s try some imputer magic.

BY GOLLY IT WORKED

Let’s score up experiment #2!¶

And we’ve done it with a score of Score: 0.47749
Even better!!!! Turns out scores closer to 0 are better.
I’m sure there is more improvement to be done. 3689 place now.

6) Experiment 3¶

Let’s just straight up factorize and impute it¶

For the next experiment, discuss what you try changing compared to the first experiment. For example, do you want to experiment with different features? Are you trying a different model? Then, talk about your results and how it changes from the previous experiment.

We’re gonna have to factorize the train and test set together…

One tiny note… let’s get rid of and seperate the ID columns before they get lost

Seperate them and carry onwards!

We’re gonna split x and y early here

Wow, that’s pretty nice.
Let’s get it on over to Kaggle

• Factorize (doen above)
• Impute
• Scale
• Predict!

Okay last one!

Score: 0.35858
3298 place
Better and better!

Experimental experiment using sklearn grid search¶

https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html

7) Impact Section¶

Discuss the possible impact of your project. This can be socially, ethically, etc. It cannot be something like “our project has no impact” or “our project has no negative impact.” Even the most well-intentioned projects could have a negative impact. We will not be checking for “right” or “wrong” answers, but showing your critical thinking.¶

Personally, I would hate for a perfect algorithm to be used to predict housing prices. I can only imagine that if there were more variables taken into account, including time and economic conditions, then this could get pretty darn close to accurate. If it could be used to predict future prices accurately, it could create a timeline in which people could invest in housing with 100% confidence. I personally think that this could reduce the overall populations home ownership proportion thus negatively impacting the overall proportion of families’ ability to aggregate generation wealth to pass on to their descendents.

On the flip side of this, if every ‘normal’ person could take advantage of this, the banks could be certain the loans they disperse would be safe. It all depends on the situation.

8) Conclusion¶

Discuss what you have learned from this project and through the different experiments. For example, did certain pre-processing steps help improve the model? Did you try using feature selection, and including/excluding certain features improve performance?¶

I’m thankful my results weren’t too accurate although it would’ve been very cool. I’ve learned I need to become more familiar with sklearns pipeline. I did end up finding a cool package called collinearity and becoming more familiar with sklearn. Pandas is really great for factorizing and adding dummies as well. Removing collinearity was a key component in some of my experiments, while standardizing data was key in my final one. I’d love to try our sklearns grid_search on this. I do intend to try out auto-sklearn on this as well. I’m currently trying to implement cuml (Nvidias drop-in replacement for sklearn) to run these faster and at parallel!

9) References¶

• https://www.kaggle.com/competitions/house-prices-advanced-regression-techniques/overview/evaluation
• http://www.amstat.org/publications/jse/v19n3/decock.pdf
• https://towardsdatascience.com/how-does-linear-regression-actually-work-3297021970dd
• https://towardsdatascience.com/introduction-to-machine-learning-algorithms-linear-regression-14c4e325882a
• https://github.com/gianlucamalato/collinearity

9) Code¶

Make sure to include your code somehow as well. You can do this in various ways. For example, you can include code snippets throughout your writing and/or include a link to your Jupyter notebook file.¶

While we’re here… let’s use this to make an image for the website!

Also to convert this to html: jupyter nbconvert –to html /path/to/example.ipynb –HTMLExporter.theme=dark from https://stackoverflow.com/questions/60894682/jupyter-notebook-exported-html-dark-color thanks nbconvert