From c799eaccbfd80476b2bd4ea75e5e55657ebb135d Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Tue, 3 Dec 2013 11:59:42 -0800 Subject: [PATCH 01/79] started conversion of slides --- slides/2013-01-25-Intro.html | 1214 ++++++++++------------------------ 1 file changed, 356 insertions(+), 858 deletions(-) diff --git a/slides/2013-01-25-Intro.html b/slides/2013-01-25-Intro.html index 11919c8..f710758 100644 --- a/slides/2013-01-25-Intro.html +++ b/slides/2013-01-25-Intro.html @@ -1,887 +1,385 @@ - - - - -2013-01-25-Intro - - - - - - - - - - - - - - - - - - - -
- -
- -
-

2013-01-25-Intro

- - - - -
-

1 Data Mining i290    slide

-
- -
    -
  • Jim Blomo & Shreyas -
    • -
- - -
- -
- -
-

2 Course Goals    slide

-
- -
    -
  • Extract information from data -
    • -
  • Understand techniques to find patterns -
    • -
  • Apply algorithms to real data sets -
    • -
- - -
- -
- -
-

3 We'll Do Stuff    slide

-
- -
    -
  • 30%: 10 Homework Assignments -
    • -
  • 30%: 1 Midterm -
    • -
  • 40%: 1 Project: Find, Mine, Report on Data -
    • -
- - -
- -
-

3.1 Homework Details    notes

-
- -
    -
  • HW due at midnight Thursday before class -
    • -
  • Each 24 hours late is 10% off -
    • -
  • HW will be turned in by GitHub pull request -
    • -
  • Project will be submitted by email & presentation -
    • -
- - -
-
- -
- -
-

4 But Don't Worry    slide

-
- -
    -
  • This isn't a programming class -
    • -
  • Grades are based on understanding of the concepts, not the craziest project -
    • -
  • Shreyas & I are here to help -
    • -
- - -
- -
-

4.1 Help    notes

-
- -
    -
  • We realize there's a wide range of technical skill -
    • -
  • We will help get anyone up to speed in these technical areas -
    • -
- - -
-
- -
- -
-

5 This is a Graduate class    slide

-
- -
    -
  • Perform well without supervision -
    • -
  • Readings from both book and online documentation -
    • -
  • TMTOWTDI -
    • -
  • Getting frameworks working on your computer -
    • -
- - -
- -
-

5.1 Style    notes

-
- -
    -
  • More firehouse than spoon feed, you'll need to follow up for - understanding -
    • -
  • Honor system: No copying code or answers. Helping each other with - concepts is encouraged, but document it. -
    • -
  • Everybody has a different workflow. We'll be covering the most basic. - Great if you want to do something different, but realize we may not be able - to help you as much. -
    • -
  • Non ISchool students should email student ID from EDU account to shreyas and - jblomo and we will get them ischool accounts. -
    • -
  • You may want to use other frameworks for your projects. Great! But again, - we may not be familiar with them -
    • -
- - -
-
- -
- -
-

6 Prerequisites    slide

-
- -
    -
  • Basic probability: P(A), P(A or B), P(A and B), P(A | B) -
    • -
  • Basic programming: Python -
    • -
  • Basic command line: SSH, downloading, copying large files, running programs + + + + Title + + + + + + + + From d1614a0246f8487226261bfaea0b70e5e7dca765 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Thu, 5 Dec 2013 21:23:05 -0800 Subject: [PATCH 02/79] start conversion to markdown --- slides/2013-01-25-Intro.html | 12 -- slides/2013-01-25-Intro.markdown | 241 ++++++++++++++++++++++++++++++ slides/Makefile | 9 ++ slides/presentation_template.html | 132 ++++++++++++++++ 4 files changed, 382 insertions(+), 12 deletions(-) create mode 100644 slides/2013-01-25-Intro.markdown create mode 100644 slides/Makefile create mode 100644 slides/presentation_template.html diff --git a/slides/2013-01-25-Intro.html b/slides/2013-01-25-Intro.html index f710758..1beaa61 100644 --- a/slides/2013-01-25-Intro.html +++ b/slides/2013-01-25-Intro.html @@ -121,8 +121,6 @@ diff --git a/slides/2013-01-25-Intro.markdown b/slides/2013-01-25-Intro.markdown new file mode 100644 index 0000000..a1d5fbc --- /dev/null +++ b/slides/2013-01-25-Intro.markdown @@ -0,0 +1,241 @@ +name: inverse +layout: true +class: left, top, inverse + +--- + +class: center + +# Data Mining i290 +## Jimmy Retzlaff & Shreyas + +--- + +# Course Goals + + Extract *information* from *data* + + Understand techniques to find patterns + + Apply algorithms to real data sets + +--- + +# We'll Do Stuff + + 30%: 10 Homework Assignments + + 30%: 1 Midterm + + 40%: 1 Project: Find, Mine, Report on Data + +??? + +## Homework Details + + Homework due at midnight Wednesday before class + + Each 24 hours late is 10% off + + Homework will be turned in by GitHub pull request + + Project will be submitted by email & presentation + +--- + +# But Don't Worry + + This isn't a programming class + + Grades are based on understanding of the concepts, not the craziest project + + Shreyas & I are here to help + +??? + +## Help + + We realize there's a wide range of technical skill + + We will help get anyone up to speed in these technical areas + +--- + +# This is a Graduate class + + Perform well without supervision + + Readings from both book and online documentation + + TMTOWTDI + + Getting frameworks working on your computer + +??? + +## Style + + More fire hose than spoon fed, you'll need to follow up for understanding + + Honor system: No copying code or answers. Helping each other with + concepts is encouraged, but document it. + + Everybody has a different work flow. We'll be covering the most basic. + Great if you want to do something different, but realize we may not be + able to help you as much. + + Non ISchool students should email student ID from EDU account to shreyas + and jretz and we will get you an ischool accounts. + + You may want to use other frameworks for your projects. Great! But again, + we may not be familiar with them. + +--- + +# Prerequisites + + Basic probability: P(A), P(A or B), P(A and B), P(A | B) + + Basic programming: Python + + Basic command line: SSH, downloading, copying large files, running programs + against data + + Textbook: Han, J., Kamber, M., & Pei, J. (2011). _Data Mining: Concepts and Techniques_, Third Edition *(3rd ed.)*. Morgan Kaufmann. + + Technology will be available on http://ischool.berkeley.edu + +??? + +## Basics + + "Probability of A", "Probability of A or B" "A and B" "A given B" + + Most assignments filling in algorithm code + + Project you may use any language, though we suggest Python. + + We'll introduce any specific frameworks + + Command line: cp, mv, less... Imagine you have a 10GB file, how are you + going to inspect the contents? + +--- + +# Material + + Process: from finding data to mining it to visualizing results + + Algorithms: all intuitively motivated, some rigorously studied + + Programming: using algorithms against data sets + + Discovery: finding information in self-defined project + +??? + +## What will we learn? + + Data mining is not just about algorithms. We'll learn how to obtain, clean, + and store data. + + In real life, this is 70% of the job! + + We'll cover many different algorithms, and dive deep on several of + them. But we're not going to get into any hairy math proofs. + + Programming is the best way to precisely describe an algorithm. It is also + the way data mining is used in the real world. + + Your own project should emphasize your passion. Again, the real world requires + you to grab data and squeeze information out of it without external help. + +--- + +# Lectures & Labs + + Start with Q&A for at least 10 minutes + + Expect to be asked a question + + Breaks + + Lab: Stick around and get the first question of homework done + + Slides on http://jretz.github.com/datamining290/ + +??? + +## Helpful tips + + Helpful to me if you say your name + + Sorry, I tend to forget names + + If I am not calling on you, check to make sure you are on the class list! + + I'm not taking attendance, but let me know if you can't make it so I + won't call on you + +--- + +Todo - setup piazza +# Office Hours + + We'll stay after class + + or schedule a Skype call + + [Piazza](https://piazza.com/class#spring2014/i290) for questions and + announcements + + Wait list will be processed normally until 3rd week... then I'll accept + everyone who's participated in class if we have physical room + +--- + +# *Questions?* + +--- + +Todo - use real dates and make it fit +# Schedule +Available at [GitHub Syllabus page](http://jretz.github.com/datamining290/) + + + Jan 25 Class Intro ; Tools Intro by /GUEST: Shreyas/ + + lab: Git Intro + + Feb 1 Case Studies ; Obtaining Data + + Feb 8 Probability ; Preprocessing + + Feb 15 MapReduce, Data Warehouse + + Feb 22 Decision Trees; Naive Bayes + + Mar 1 SVM ; Neural Networks + + Mar 8 Clustering ; Review + + lab: Project Proposal Due + + Mar 15 *Midterm* + + lab: - + + Mar 21 Dimensionality Curse ; Graph Mining + + Mar 29 HOLIDAY + + Apr 5 Pattern ; Evaluations + + Apr 12 Collaborative Filtering; PageRank + + Apr 19 Feature Extraction ; Evaluation + + Apr 26 Images ; Audio + + May 3 Visualization ; HTML + + May 10 In Real Life ; Review + + lab: - + + May 17 Final Presentation + + lab: Bye! + +--- + +# Hi, I'm Jimmy Retzlaff + + Yelp - ads engineer and now ads engineering manager + + Amazon / Lab126 - Kindle on-device content search + + Aver - sales visualizations for the investment industry + + Career Central - allow employers to search for job seekers + + Animatrix - executive information systems + + Xerox - printer drivers for high volume printers + + Harvey Mudd College, B.S. Math + +--- + +# Hi, I'm Shreyas + + First year Grad Student (MIMS '14) + + Also TA'd Analyzing Big Data class + + I can be reached at [seekshreyas@gmail.com](mailto:seekshreyas@gmail.com) + +--- + +# Data is Important + + Making decisions is a core part of humanity + + Data can help you make better decisions + + Challenge: extract information from data to improve decisions + +??? + +## Decisions + + From big to small; from planning to execution + + Business questions: what is the ROI of this feature? Where to concentrate + development? + + Personal questions: Where to eat dinner tonight? What movie to see? + + Improving decisions means improving quality of life + +--- + +class: center + +Todo - get iframe to work +# Data is Important + + +??? + +## Nice example of data mining + + Stop at 3:51 + + Had to work with external parties to get data (Yelp, city of Seattle) + + Had to clean data (literally, sometimes he was just handed paper receipts) + + Used regression analysis to discover patterns + + created follow up questions + + Used result to understand the meaning behind the data + +--- + +## Data Mining ecosystem + + Data mining is part of a process to make decisions from data + + Intersection between statistics, computer science, data management, and + machine learning + + Analysis & visualization often required + +??? + +## Ecosystem + + We'll talk about several ways to think about the process from data to + knowledge + + No universally agreed process, or black-and-white boundaries + + Analysis: used at the beginning of investigations to understand data + characteristics + + Visualization: better understanding of the results of analysis or data + mining diff --git a/slides/Makefile b/slides/Makefile new file mode 100644 index 0000000..325652c --- /dev/null +++ b/slides/Makefile @@ -0,0 +1,9 @@ +PLACE_HOLDER=<\!--markdown goes here--> +SOURCES=$(wildcard *.markdown) +PRESENTATIONS=$(SOURCES:.markdown=.html) +EXECUTABLE=hello + +all: $(PRESENTATIONS) + +%.html: %.markdown presentation_template.html + sed -e "/$(PLACE_HOLDER)/r $<" < presentation_template.html | sed -e "s/$(PLACE_HOLDER)//" > $@ diff --git a/slides/presentation_template.html b/slides/presentation_template.html new file mode 100644 index 0000000..1dc9166 --- /dev/null +++ b/slides/presentation_template.html @@ -0,0 +1,132 @@ + + + + Title + + + + + + + + + From b4205d1f1058c10e262ea5428e87be807ebb637d Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sat, 7 Dec 2013 17:36:51 -0800 Subject: [PATCH 03/79] finished converting 2013-01-25 to markdown --- slides/2013-01-25-Intro.html | 226 +++++++++++++++++++++--------- slides/2013-01-25-Intro.markdown | 211 ++++++++++++++++++++-------- slides/presentation_template.html | 15 +- 3 files changed, 314 insertions(+), 138 deletions(-) diff --git a/slides/2013-01-25-Intro.html b/slides/2013-01-25-Intro.html index 1beaa61..bbba2aa 100644 --- a/slides/2013-01-25-Intro.html +++ b/slides/2013-01-25-Intro.html @@ -1,7 +1,7 @@ - Title + Data Mining @@ -134,14 +129,14 @@ --- -# Course Goals +## Course Goals + Extract *information* from *data* + Understand techniques to find patterns + Apply algorithms to real data sets --- -# We'll Do Stuff +## We'll Do Stuff + 30%: 10 Homework Assignments + 30%: 1 Midterm + 40%: 1 Project: Find, Mine, Report on Data @@ -156,7 +151,7 @@ --- -# But Don't Worry +## But Don't Worry + This isn't a programming class + Grades are based on understanding of the concepts, not the craziest project + Shreyas & I are here to help @@ -164,12 +159,12 @@ ??? ## Help - + We realize there's a wide range of technical skill - + We will help get anyone up to speed in these technical areas + + We realize there's a wide range of technical skill + + We will help get anyone up to speed in these technical areas --- -# This is a Graduate class +## This is a Graduate Course + Perform well without supervision + Readings from both book and online documentation + TMTOWTDI @@ -191,7 +186,7 @@ --- -# Prerequisites +## Prerequisites + Basic probability: P(A), P(A or B), P(A and B), P(A | B) + Basic programming: Python + Basic command line: SSH, downloading, copying large files, running programs @@ -211,11 +206,11 @@ --- -# Material +## Material + Process: from finding data to mining it to visualizing results + Algorithms: all intuitively motivated, some rigorously studied + Programming: using algorithms against data sets - + Discovery: finding information in self-defined project + + Discovery: finding information in a self-defined project ??? @@ -232,12 +227,12 @@ --- -# Lectures & Labs +## Lectures & Labs + Start with Q&A for at least 10 minutes + Expect to be asked a question + Breaks + Lab: Stick around and get the first question of homework done - + Slides on http://jretz.github.com/datamining290/ + + Slides on http://github.com/jretz/datamining290/ ??? @@ -251,7 +246,7 @@ --- Todo - setup piazza -# Office Hours +## Office Hours + We'll stay after class + or schedule a Skype call + [Piazza](https://piazza.com/class#spring2014/i290) for questions and @@ -261,40 +256,39 @@ --- -# *Questions?* +## *Questions?* --- -Todo - use real dates and make it fit -# Schedule -Available at [GitHub Syllabus page](http://jretz.github.com/datamining290/) +## Schedule +Available at [GitHub Syllabus page](http://github.com/jretz/datamining290/) - + Jan 25 Class Intro ; Tools Intro by /GUEST: Shreyas/ + + Jan 23 Class Intro ; Tools Intro by /GUEST: Shreyas/ + lab: Git Intro - + Feb 1 Case Studies ; Obtaining Data - + Feb 8 Probability ; Preprocessing - + Feb 15 MapReduce, Data Warehouse - + Feb 22 Decision Trees; Naive Bayes - + Mar 1 SVM ; Neural Networks - + Mar 8 Clustering ; Review + + Jan 30 Case Studies ; Obtaining Data + + Feb 6 Probability ; Preprocessing + + Feb 13 MapReduce, Data Warehouse + + Feb 20 Decision Trees; Naive Bayes + + Feb 27 SVM ; Neural Networks + + Mar 6 Clustering ; Review + lab: Project Proposal Due - + Mar 15 *Midterm* + + Mar 13 *Midterm* + lab: - - + Mar 21 Dimensionality Curse ; Graph Mining - + Mar 29 HOLIDAY - + Apr 5 Pattern ; Evaluations - + Apr 12 Collaborative Filtering; PageRank - + Apr 19 Feature Extraction ; Evaluation - + Apr 26 Images ; Audio - + May 3 Visualization ; HTML - + May 10 In Real Life ; Review + + Mar 20 Dimensionality Curse ; Graph Mining + + Mar 27 HOLIDAY + + Apr 3 Pattern ; Evaluations + + Apr 10 Collaborative Filtering; PageRank + + Apr 17 Feature Extraction ; Evaluation + + Apr 24 Images ; Audio + + May 1 Visualization ; HTML + + May 8 In Real Life ; Review + lab: - - + May 17 Final Presentation + + May 15 Final Presentation + lab: Bye! --- -# Hi, I'm Jimmy Retzlaff +## Hi, I'm Jimmy Retzlaff + Yelp - ads engineer and now ads engineering manager + Amazon / Lab126 - Kindle on-device content search + Aver - sales visualizations for the investment industry @@ -305,14 +299,14 @@ --- -# Hi, I'm Shreyas +## Hi, I'm Shreyas + First year Grad Student (MIMS '14) + Also TA'd Analyzing Big Data class + I can be reached at [seekshreyas@gmail.com](mailto:seekshreyas@gmail.com) --- -# Data is Important +## Data is Important + Making decisions is a core part of humanity + Data can help you make better decisions + Challenge: extract information from data to improve decisions @@ -320,29 +314,27 @@ ??? ## Decisions - + From big to small; from planning to execution - + Business questions: what is the ROI of this feature? Where to concentrate - development? - + Personal questions: Where to eat dinner tonight? What movie to see? - + Improving decisions means improving quality of life + + From big to small; from planning to execution + + Business questions: what is the ROI of this feature? Where to concentrate + development? + + Personal questions: Where to eat dinner tonight? What movie to see? + + Improving decisions means improving quality of life --- -class: center - -Todo - get iframe to work -# Data is Important - +## Data is Important +[![Reviews, Reputation, and Revenue: The Case for Yelp.com](http://img.youtube.com/vi/y7een27u1GM/0.jpg)](http://www.youtube.com/embed/y7een27u1GM) ??? +[![Reviews, Reputation, and Revenue: The Case for Yelp.com](http://img.youtube.com/vi/y7een27u1GM/0.jpg)](http://www.youtube.com/watch?v=y7een27u1GM) ## Nice example of data mining - + Stop at 3:51 - + Had to work with external parties to get data (Yelp, city of Seattle) - + Had to clean data (literally, sometimes he was just handed paper receipts) - + Used regression analysis to discover patterns - + created follow up questions - + Used result to understand the meaning behind the data + + Stop at 3:51 + + Had to work with external parties to get data (Yelp, city of Seattle) + + Had to clean data (literally, sometimes he was just handed paper receipts) + + Used regression analysis to discover patterns + + created follow up questions + + Used result to understand the meaning behind the data --- @@ -355,13 +347,109 @@ ??? ## Ecosystem - + We'll talk about several ways to think about the process from data to - knowledge - + No universally agreed process, or black-and-white boundaries - + Analysis: used at the beginning of investigations to understand data - characteristics - + Visualization: better understanding of the results of analysis or data - mining + + We'll talk about several ways to think about the process from data to + knowledge + + No universally agreed process, or black-and-white boundaries + + Analysis: used at the beginning of investigations to understand data + characteristics + + Visualization: better understanding of the results of analysis or data + mining + +--- + +## Analysis vs. Data Mining + +### Analysis + + Manually investigating data. No algorithms. + + Statistical qualities: mean, median, standard deviation + + Histograms (manually set buckets) + + Counts / Percentages + +### Data Mining + + Discovering patterns though automated algorithms + + Regressions: fitting data to a model + + Clustering: grouping data without manually set descriptions + + Classification: identifying divisive features + +??? + +## Pedantic + + + Difference is subtle, but important for both the project and your resume + +--- + +## Machine Learning + + + Programs that can learn from data + + Focus on prediction, based on verified training data + + Used in two ways: during Data Mining and after Data Mining + +[![Robot reading](http://distillery.s3.amazonaws.com/media/2011/06/15/04cfeacce2f4404483d96a4428f9adbd_7.jpg)](http://photopile.me/user/hartanta/?max_id=100596871_2487764#) + +??? + +## Uses + + + During - assume we have training data, train on it, see how useful trained + program is or find outliers + + After - Discover clusters, verify and label clusters. Use labeled clusters + to train a program to recognize new data points + +--- + +## Probability & Statistics + + + Data describes real world events + + Probability can describe real world *expected* events + + Distributions can be used to summarize data, understand the factors behind + its creation + +
    + +??? + +## Uses + + + Can "fit" data to a distribution, find outliers that are unexpected + + An example: Poisson distribution describes the expectation of a particular + number of events occurring. + + e.g., pieces of mail. Average is 4, but it can vary. Is getting 7 or more + pieces of mail really an outlier? + +--- + +## Process + +### [Knowledge Discovery in Databases (KDD)](http://en.wikipedia.org/wiki/Data_mining#Process) + + Selection + + Pre-processing + + Transformation + + Data Mining + + Interpretation/Evaluation + +### [Cross Industry Standard Process for Data Mining](http://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining) + + Business Understanding + + Data Understanding + + Data Preparation + + Modeling + + Evaluation + + Deployment + +??? + +## Common Themes + + Figure out what you want to do + + Get the data + + Make sure it's OK + + Understanding + + Make a decision, test its effectiveness + + Reading will cover another process, aimed at "Data Science", but basically + applies to Data Mining + +--- + +## *Break* -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: From 0a2ccb3c10f4b99cc95f9cf4242bf2547c2f6b78 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sat, 7 Dec 2013 18:44:38 -0800 Subject: [PATCH 05/79] convert README to markdown --- README.markdown | 40 ++++++++++++++++++++++++++++++++++++++++ README.org | 48 ------------------------------------------------ 2 files changed, 40 insertions(+), 48 deletions(-) create mode 100644 README.markdown delete mode 100644 README.org diff --git a/README.markdown b/README.markdown new file mode 100644 index 0000000..6467fdd --- /dev/null +++ b/README.markdown @@ -0,0 +1,40 @@ +# Data Mining 290 + +### Description +Learn how to obtain, clean, visualize, understand, model, and +predict the world around you using data. Grading will consist of homework +(30%), midterm (30%), project (40%). + +### Instructor +Jimmy Retzlaff + +### GSI +Shreyas + +### Textbook +Han, J., Kamber, M., & Pei, J. (2011). _Data Mining: Concepts and Techniques_, Third Edition *(3rd ed.)*. Morgan Kaufmann. + +--- + +# Syllabus +DM[0-9]+ indicates chapters from the text, _Data Mining_. + +| Date | Readings | Slides | Homework / Project | +|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| +| Jan 23 | [Try Github](http://try.github.com) ; [A Taxonomy of Data Science](http://www.dataists.com/2010/09/a-taxonomy-of-data-science/) | [Class Intro](file:slides/2013-01-25-Intro.html) ; Tools Intro by /GUEST: Shreyas/ | [Git Intro](https://github.com/seekshreyas/Introduction-to-Git-Github) | +| Jan 30 | DM1 ; [The Yelp Factor: Are Consumer Reviews Good for Business?](http://hbswk.hbs.edu/item/6836.html) | [Case Studies](file:slides/2013-02-01-CaseStudies.html) ; [Obtaining Data](file:slides/2013-02-01-Obtaining-Data.html) | [Obtain & Explore Data](file:slides/2013-02-01-Lab.html) | +| Feb 6 | DM2, DM3 | [Probability](file:slides/2013-02-08-Probability.html) ; [Preprocessing](file:slides/2013-02-08-Preprocessing.html) | [Data Stats](file:slides/2013-02-08-Lab.html) | +| Feb 13 | DM4, [Apache Hadoop: Petabytes and Terawatts](http://www.youtube.com/watch?v=SS27F-hYWfU) ([slides](http://prezi.com/u0ukvqzpyh5p/apache-hadoop-petabytes-and-terawatts/)); [mrjob docs](http://packages.python.org/mrjob/) (for homework) | [Data Warehouse](file:slides/2013-02-15-Data-Warehouse.html) ; [MapReduce](file:slides/2013-02-15-MapReduce.html) | [Project Details](file:slides/2013-02-15-Project.html) ; [mrjob](file:slides/2013-02-15-mrjob.html) | +| Feb 20 | DM8 | [Decision Trees](file:slides/2013-02-22-Decision-Trees.html); [Naive Bayes](file:slides/2013-02-22-Bayes.html) | [Gini Index](file:slides/2013-02-22-Gini.html) | +| Feb 27 | DM[9.1-9.3], 9.5 ; [Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html) | [SVM](file:slides/2013-03-01-SVM.html) ; [Neural Networks](file:slides/2013-03-01-Neural-Network.html) | [Neural Network Back Propagation](file:slides/2013-03-01-Lab-NN.html) | +| Mar 6 | DM10 | [Agglomerative - Clustering](file:slides/2013-03-07-Clustering.html) ; [Hierarchical, Density - Clustering](file:slides/2013-03-07-Hierarchical.html) | [K-Means](file:slides/2013-03-07-k-means.html) | +| Mar 13 | DM11.1 | [Review](file:slides/2013-03-15-Review.html) | prepare 1 cheat sheet | +| Mar 20 | 1 cheat sheet | *Midterm* | | +| Mar 27 | HOLIDAY | | | +| Apr 3 | DM6 | [Advanced Clustering](file:slides/2013-03-15-Advanced-Cluster.html) ; [Frequent Pattern](file:slides/2013-04-05-Frequent-Pattern.html) | [AWS](file:slides/2013-04-05-AWS.html) ; Project Proposal Due | +| Apr 10 | DM11.3; [PageRank](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf); [Uncovering Social Network Sybils in the Wild](http://arxiv.org/pdf/1106.5321) | [Graphs](file:slides/2013-04-12-Graphs.html); [PageRank](file:slides/2013-04-12-PageRank.html) | [Adjacency Representations](file:slides/2013-04-12-AdjacencyRepresentations.html) | +| Apr 17 | [Non-linear regression](file:slides/2013-04-19-Nonlinear.pdf) | GUEST: Gene Lee Ceaser's [Pricing Strategy](file:slides/RM Pricing Strategy.ppt); [Ceaser's Recruiting](file:slides/Campus Recruiting Deck_2012_UC Berkeley.ppt) | [Price Elasticity](file:slides/2013-04-19-Elasticity.html) | +| Apr 24 | DM12; [Shazam Audio Search](http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf) | [Outliers](file:slides/2013-04-26-Outliers.html); [Images & Audio](file:slides/2013-04-26-Multimedia.html) | [Midterm Review](file:slides/2013-04-26-Midterm-HW.html) | +| May 1 | [Embedded Plots](https://groups.google.com/group/gsofgs/attach/2f1cdd7a999c3ad8/embedded-plots.pdf?part=2&authuser=0) ; [Data-Driven Documents](http://vis.stanford.edu/files/2011-D3-InfoVis.pdf) | [Visualization]file:slides/2013-05-03-Visualization.html) ; [Yelp's Visualizations](file:slides/2013-05-03-Yelp-Visualization.html) | [D3 Intro](http://vogievetsky.github.io/IntroD3/); [D3 Lab](file:slides/2013-05-03-D3.html) | +| May 18 | [A Few Useful Things to Know about Machine Learning](http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) ; [Top 10 Algorithms in Data Mining](http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf) | [In Real Life](file:slides/2013-05-10-Real-World.html) ; Presentations | May 14th: Project Papers Due | +| May 15 | - | Final Presentation | Bye! | diff --git a/README.org b/README.org deleted file mode 100644 index b501fb3..0000000 --- a/README.org +++ /dev/null @@ -1,48 +0,0 @@ -* Data Mining 290 :slide: - + Description :: Learn how to obtain, clean, visualize, understand, model, and - predict the world around you using data. Grading will consist of homework - (30%), midterm (30%), project (40%). - + Instructor :: Jim Blomo - + GSI :: Shreyas - + Textbook :: Han, J., Kamber, M., & Pei, J. (2011). _Data Mining: Concepts and Techniques_, Third Edition *(3rd ed.)*. Morgan Kaufmann. - - -* Syllabus :slide: -DM[0-9]+ indicates chapters from the text, _Data Mining_. - -| Date | Readings | Slides | Homework / Project | -|------+----------+--------+--------------------| -| Jan 25 | [[http://try.github.com][Try Github]] ; [[http://www.dataists.com/2010/09/a-taxonomy-of-data-science/][A Taxonomy of Data Science]] | [[file:slides/2013-01-25-Intro.html][Class Intro]] ; Tools Intro by /GUEST: Shreyas/ | [[ https://github.com/seekshreyas/Introduction-to-Git-Github][Git Intro]] | -| Feb 1 | DM1 ; [[http://hbswk.hbs.edu/item/6836.html][The Yelp Factor: Are Consumer Reviews Good for Business?]] | [[file:slides/2013-02-01-CaseStudies.html][Case Studies]] ; [[file:slides/2013-02-01-Obtaining-Data.html][Obtaining Data]] | [[file:slides/2013-02-01-Lab.html][Obtain & Explore Data]] | -| Feb 8 | DM2, DM3 | [[file:slides/2013-02-08-Probability.html][Probability]] ; [[file:slides/2013-02-08-Preprocessing.html][Preprocessing]] | [[file:slides/2013-02-08-Lab.html][Data Stats]] | -| Feb 15 | DM4, [[http://www.youtube.com/watch?v=SS27F-hYWfU][Apache Hadoop: Petabytes and Terawatts]] ([[http://prezi.com/u0ukvqzpyh5p/apache-hadoop-petabytes-and-terawatts/][slides]]); [[http://packages.python.org/mrjob/][mrjob docs]] (for homework) | [[file:slides/2013-02-15-Data-Warehouse.html][Data Warehouse]] ; [[file:slides/2013-02-15-MapReduce.html][MapReduce]] | [[file:slides/2013-02-15-Project.html][Project Details]] ; [[file:slides/2013-02-15-mrjob.html][mrjob]] | -| Feb 22 | DM8 | [[file:slides/2013-02-22-Decision-Trees.html][Decision Trees]]; [[file:slides/2013-02-22-Bayes.html][Naive Bayes]] | [[file:slides/2013-02-22-Gini.html][Gini Index]] | -| Mar 1 | DM[9.1-9.3], 9.5 ; [[http://scott.fortmann-roe.com/docs/BiasVariance.html][Understanding the Bias-Variance Tradeoff]] | [[file:slides/2013-03-01-SVM.html][SVM]] ; [[file:slides/2013-03-01-Neural-Network.html][Neural Networks]] | [[file:slides/2013-03-01-Lab-NN.html][Neural Network Back Propagation]] | -| Mar 8 | DM10 | [[file:slides/2013-03-07-Clustering.html][Agglomerative - Clustering]] ; [[file:slides/2013-03-07-Hierarchical.html][Hierarchical, Density - Clustering]] | [[file:slides/2013-03-07-k-means.html][K-Means]] | -| Mar 15 | DM11.1 | [[file:slides/2013-03-15-Review.html][Review]] | prepare 1 cheat sheet | -| Mar 22 | 1 cheat sheet | *Midterm* | - | -| Mar 29 | HOLIDAY -| Apr 5 | DM6 | [[file:slides/2013-03-15-Advanced-Cluster.html][Advanced Clustering]] ; [[file:slides/2013-04-05-Frequent-Pattern.html][Frequent Pattern]] | [[file:slides/2013-04-05-AWS.html][AWS]] ; Project Proposal Due | -| Apr 12 | DM11.3; [[http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf][PageRank]]; [[http://arxiv.org/pdf/1106.5321][Uncovering Social Network Sybils in the Wild]] | [[file:slides/2013-04-12-Graphs.html][Graphs]]; [[file:slides/2013-04-12-PageRank.html][PageRank]] | [[file:slides/2013-04-12-AdjacencyRepresentations.html][Adjacency Representations]] | -| Apr 19 | [[file:slides/2013-04-19-Nonlinear.pdf][Non-linear regression]] | GUEST: Gene Lee Ceaser's [[file:slides/RM Pricing Strategy.ppt][Pricing Strategy]]; [[file:slides/Campus Recruiting Deck_2012_UC Berkeley.ppt][Ceaser's Recruiting]]| [[file:slides/2013-04-19-Elasticity.html][Price Elasticity]] | -| Apr 26 | DM12; [[http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf][Shazam Audio Search]] | [[file:slides/2013-04-26-Outliers.html][Outliers]]; [[file:slides/2013-04-26-Multimedia.html][Images & Audio]] | [[file:slides/2013-04-26-Midterm-HW.html][Midterm Review]] | -| May 3 | [[https://groups.google.com/group/gsofgs/attach/2f1cdd7a999c3ad8/embedded-plots.pdf?part=2&authuser=0][Embedded Plots]] ; [[http://vis.stanford.edu/files/2011-D3-InfoVis.pdf][Data-Driven Documents]]| [[file:slides/2013-05-03-Visualization.html][Visualization]] ; [[file:slides/2013-05-03-Yelp-Visualization.html][Yelp's Visualizations]] | [[http://vogievetsky.github.io/IntroD3/][D3 Intro]] [[file:slides/2013-05-03-D3.html][D3 Lab]] | -| May 10 | [[http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf][A Few Useful Things to Know about Machine Learning]] ; [[http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf][Top 10 Algorithms in Data Mining]] | [[file:slides/2013-05-10-Real-World.html][In Real Life]] ; Presentations | May 16th: Project Papers Due | -| May 17 | - | Final Presentation | Bye! | - - -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: - -#+BEGIN_HTML - -Fork me on GitHub -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: From 1b0fc24e72e45824007458c95b398e4e696a3554 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sat, 7 Dec 2013 18:53:05 -0800 Subject: [PATCH 06/79] starting to update dates --- README.markdown | 28 +++++++++---------- ...01-23-Intro.html => 2014-01-23-Intro.html} | 0 ...tro.markdown => 2014-01-23-Intro.markdown} | 0 3 files changed, 14 insertions(+), 14 deletions(-) rename slides/{2013-01-23-Intro.html => 2014-01-23-Intro.html} (100%) rename slides/{2013-01-23-Intro.markdown => 2014-01-23-Intro.markdown} (100%) diff --git a/README.markdown b/README.markdown index 6467fdd..d457bed 100644 --- a/README.markdown +++ b/README.markdown @@ -21,20 +21,20 @@ DM[0-9]+ indicates chapters from the text, _Data Mining_. | Date | Readings | Slides | Homework / Project | |--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| -| Jan 23 | [Try Github](http://try.github.com) ; [A Taxonomy of Data Science](http://www.dataists.com/2010/09/a-taxonomy-of-data-science/) | [Class Intro](file:slides/2013-01-25-Intro.html) ; Tools Intro by /GUEST: Shreyas/ | [Git Intro](https://github.com/seekshreyas/Introduction-to-Git-Github) | -| Jan 30 | DM1 ; [The Yelp Factor: Are Consumer Reviews Good for Business?](http://hbswk.hbs.edu/item/6836.html) | [Case Studies](file:slides/2013-02-01-CaseStudies.html) ; [Obtaining Data](file:slides/2013-02-01-Obtaining-Data.html) | [Obtain & Explore Data](file:slides/2013-02-01-Lab.html) | -| Feb 6 | DM2, DM3 | [Probability](file:slides/2013-02-08-Probability.html) ; [Preprocessing](file:slides/2013-02-08-Preprocessing.html) | [Data Stats](file:slides/2013-02-08-Lab.html) | -| Feb 13 | DM4, [Apache Hadoop: Petabytes and Terawatts](http://www.youtube.com/watch?v=SS27F-hYWfU) ([slides](http://prezi.com/u0ukvqzpyh5p/apache-hadoop-petabytes-and-terawatts/)); [mrjob docs](http://packages.python.org/mrjob/) (for homework) | [Data Warehouse](file:slides/2013-02-15-Data-Warehouse.html) ; [MapReduce](file:slides/2013-02-15-MapReduce.html) | [Project Details](file:slides/2013-02-15-Project.html) ; [mrjob](file:slides/2013-02-15-mrjob.html) | -| Feb 20 | DM8 | [Decision Trees](file:slides/2013-02-22-Decision-Trees.html); [Naive Bayes](file:slides/2013-02-22-Bayes.html) | [Gini Index](file:slides/2013-02-22-Gini.html) | -| Feb 27 | DM[9.1-9.3], 9.5 ; [Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html) | [SVM](file:slides/2013-03-01-SVM.html) ; [Neural Networks](file:slides/2013-03-01-Neural-Network.html) | [Neural Network Back Propagation](file:slides/2013-03-01-Lab-NN.html) | -| Mar 6 | DM10 | [Agglomerative - Clustering](file:slides/2013-03-07-Clustering.html) ; [Hierarchical, Density - Clustering](file:slides/2013-03-07-Hierarchical.html) | [K-Means](file:slides/2013-03-07-k-means.html) | -| Mar 13 | DM11.1 | [Review](file:slides/2013-03-15-Review.html) | prepare 1 cheat sheet | +| Jan 23 | [Try Github](http://try.github.com) ; [A Taxonomy of Data Science](http://www.dataists.com/2010/09/a-taxonomy-of-data-science/) | [Class Intro](file:slides/2014-01-23-Intro.html) ; Tools Intro by /GUEST: Shreyas/ | [Git Intro](https://github.com/seekshreyas/Introduction-to-Git-Github) | +| Jan 30 | DM1 ; [The Yelp Factor: Are Consumer Reviews Good for Business?](http://hbswk.hbs.edu/item/6836.html) | [Case Studies](file:slides/2014-01-30-CaseStudies.html) ; [Obtaining Data](file:slides/2014-01-30-Obtaining-Data.html) | [Obtain & Explore Data](file:slides/2014-01-30-Lab.html) | +| Feb 6 | DM2, DM3 | [Probability](file:slides/2014-02-06-Probability.html) ; [Preprocessing](file:slides/2014-02-06-Preprocessing.html) | [Data Stats](file:slides/2014-02-06-Lab.html) | +| Feb 13 | DM4, [Apache Hadoop: Petabytes and Terawatts](http://www.youtube.com/watch?v=SS27F-hYWfU) ([slides](http://prezi.com/u0ukvqzpyh5p/apache-hadoop-petabytes-and-terawatts/)); [mrjob docs](http://packages.python.org/mrjob/) (for homework) | [Data Warehouse](file:slides/2014-02-13-Data-Warehouse.html) ; [MapReduce](file:slides/2014-02-13-MapReduce.html) | [Project Details](file:slides/2014-02-13-Project.html) ; [mrjob](file:slides/2014-02-13-mrjob.html) | +| Feb 20 | DM8 | [Decision Trees](file:slides/2014-02-20-Decision-Trees.html); [Naive Bayes](file:slides/2014-02-20-Bayes.html) | [Gini Index](file:slides/2014-02-20-Gini.html) | +| Feb 27 | DM[9.1-9.3], 9.5 ; [Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html) | [SVM](file:slides/2014-02-27-SVM.html) ; [Neural Networks](file:slides/2014-02-27-Neural-Network.html) | [Neural Network Back Propagation](file:slides/2014-02-27-Lab-NN.html) | +| Mar 6 | DM10 | [Agglomerative - Clustering](file:slides/2014-03-06-Clustering.html) ; [Hierarchical, Density - Clustering](file:slides/2014-03-06-Hierarchical.html) | [K-Means](file:slides/2014-03-06-k-means.html) | +| Mar 13 | DM11.1 | [Review](file:slides/2014-03-13-Review.html) | prepare 1 cheat sheet | | Mar 20 | 1 cheat sheet | *Midterm* | | | Mar 27 | HOLIDAY | | | -| Apr 3 | DM6 | [Advanced Clustering](file:slides/2013-03-15-Advanced-Cluster.html) ; [Frequent Pattern](file:slides/2013-04-05-Frequent-Pattern.html) | [AWS](file:slides/2013-04-05-AWS.html) ; Project Proposal Due | -| Apr 10 | DM11.3; [PageRank](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf); [Uncovering Social Network Sybils in the Wild](http://arxiv.org/pdf/1106.5321) | [Graphs](file:slides/2013-04-12-Graphs.html); [PageRank](file:slides/2013-04-12-PageRank.html) | [Adjacency Representations](file:slides/2013-04-12-AdjacencyRepresentations.html) | -| Apr 17 | [Non-linear regression](file:slides/2013-04-19-Nonlinear.pdf) | GUEST: Gene Lee Ceaser's [Pricing Strategy](file:slides/RM Pricing Strategy.ppt); [Ceaser's Recruiting](file:slides/Campus Recruiting Deck_2012_UC Berkeley.ppt) | [Price Elasticity](file:slides/2013-04-19-Elasticity.html) | -| Apr 24 | DM12; [Shazam Audio Search](http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf) | [Outliers](file:slides/2013-04-26-Outliers.html); [Images & Audio](file:slides/2013-04-26-Multimedia.html) | [Midterm Review](file:slides/2013-04-26-Midterm-HW.html) | -| May 1 | [Embedded Plots](https://groups.google.com/group/gsofgs/attach/2f1cdd7a999c3ad8/embedded-plots.pdf?part=2&authuser=0) ; [Data-Driven Documents](http://vis.stanford.edu/files/2011-D3-InfoVis.pdf) | [Visualization]file:slides/2013-05-03-Visualization.html) ; [Yelp's Visualizations](file:slides/2013-05-03-Yelp-Visualization.html) | [D3 Intro](http://vogievetsky.github.io/IntroD3/); [D3 Lab](file:slides/2013-05-03-D3.html) | -| May 18 | [A Few Useful Things to Know about Machine Learning](http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) ; [Top 10 Algorithms in Data Mining](http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf) | [In Real Life](file:slides/2013-05-10-Real-World.html) ; Presentations | May 14th: Project Papers Due | +| Apr 3 | DM6 | [Advanced Clustering](file:slides/2014-03-13-Advanced-Cluster.html) ; [Frequent Pattern](file:slides/2014-04-03-Frequent-Pattern.html) | [AWS](file:slides/2014-04-03-AWS.html) ; Project Proposal Due | +| Apr 10 | DM11.3; [PageRank](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf); [Uncovering Social Network Sybils in the Wild](http://arxiv.org/pdf/1106.5321) | [Graphs](file:slides/2014-04-10-Graphs.html); [PageRank](file:slides/2014-04-10-PageRank.html) | [Adjacency Representations](file:slides/2014-04-10-AdjacencyRepresentations.html) | +| Apr 17 | [Non-linear regression](file:slides/2014-04-17-Nonlinear.pdf) | GUEST: Gene Lee Ceaser's [Pricing Strategy](file:slides/RM Pricing Strategy.ppt); [Ceaser's Recruiting](file:slides/Campus Recruiting Deck_2012_UC Berkeley.ppt) | [Price Elasticity](file:slides/2014-04-17-Elasticity.html) | +| Apr 24 | DM12; [Shazam Audio Search](http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf) | [Outliers](file:slides/2014-04-24-Outliers.html); [Images & Audio](file:slides/2014-04-24-Multimedia.html) | [Midterm Review](file:slides/2014-04-24-Midterm-HW.html) | +| May 1 | [Embedded Plots](https://groups.google.com/group/gsofgs/attach/2f1cdd7a999c3ad8/embedded-plots.pdf?part=2&authuser=0) ; [Data-Driven Documents](http://vis.stanford.edu/files/2011-D3-InfoVis.pdf) | [Visualization]file:slides/2014-05-01-Visualization.html) ; [Yelp's Visualizations](file:slides/2014-05-01-Yelp-Visualization.html) | [D3 Intro](http://vogievetsky.github.io/IntroD3/); [D3 Lab](file:slides/2014-05-01-D3.html) | +| May 18 | [A Few Useful Things to Know about Machine Learning](http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) ; [Top 10 Algorithms in Data Mining](http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf) | [In Real Life](file:slides/2014-05-08-Real-World.html) ; Presentations | May 14th: Project Papers Due | | May 15 | - | Final Presentation | Bye! | diff --git a/slides/2013-01-23-Intro.html b/slides/2014-01-23-Intro.html similarity index 100% rename from slides/2013-01-23-Intro.html rename to slides/2014-01-23-Intro.html diff --git a/slides/2013-01-23-Intro.markdown b/slides/2014-01-23-Intro.markdown similarity index 100% rename from slides/2013-01-23-Intro.markdown rename to slides/2014-01-23-Intro.markdown From 4482e704b29fe75f235870201aa678c1a9268d1a Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sat, 7 Dec 2013 20:21:32 -0800 Subject: [PATCH 07/79] more README cleanup --- README.markdown | 10 ++-- index.html | 119 ------------------------------------------------ 2 files changed, 5 insertions(+), 124 deletions(-) delete mode 100644 index.html diff --git a/README.markdown b/README.markdown index d457bed..922bf8b 100644 --- a/README.markdown +++ b/README.markdown @@ -2,14 +2,14 @@ ### Description Learn how to obtain, clean, visualize, understand, model, and -predict the world around you using data. Grading will consist of homework -(30%), midterm (30%), project (40%). +predict the world around you using data. Grading will consist of homework +(30%), a midterm (30%), and a project (40%). ### Instructor -Jimmy Retzlaff +Jimmy Retzlaff ### GSI -Shreyas +Shreyas ### Textbook Han, J., Kamber, M., & Pei, J. (2011). _Data Mining: Concepts and Techniques_, Third Edition *(3rd ed.)*. Morgan Kaufmann. @@ -37,4 +37,4 @@ DM[0-9]+ indicates chapters from the text, _Data Mining_. | Apr 24 | DM12; [Shazam Audio Search](http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf) | [Outliers](file:slides/2014-04-24-Outliers.html); [Images & Audio](file:slides/2014-04-24-Multimedia.html) | [Midterm Review](file:slides/2014-04-24-Midterm-HW.html) | | May 1 | [Embedded Plots](https://groups.google.com/group/gsofgs/attach/2f1cdd7a999c3ad8/embedded-plots.pdf?part=2&authuser=0) ; [Data-Driven Documents](http://vis.stanford.edu/files/2011-D3-InfoVis.pdf) | [Visualization]file:slides/2014-05-01-Visualization.html) ; [Yelp's Visualizations](file:slides/2014-05-01-Yelp-Visualization.html) | [D3 Intro](http://vogievetsky.github.io/IntroD3/); [D3 Lab](file:slides/2014-05-01-D3.html) | | May 18 | [A Few Useful Things to Know about Machine Learning](http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) ; [Top 10 Algorithms in Data Mining](http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf) | [In Real Life](file:slides/2014-05-08-Real-World.html) ; Presentations | May 14th: Project Papers Due | -| May 15 | - | Final Presentation | Bye! | +| May 15 | | Final Presentation | Bye! | diff --git a/index.html b/index.html deleted file mode 100644 index e6694e2..0000000 --- a/index.html +++ /dev/null @@ -1,119 +0,0 @@ - - - - -README - - - - - - - - - - - - - - - - - -
    - -
    - -
    -

    README

    - - -
    -

    Table of Contents

    -
    - -
    -
    - -
    -

    1 Data Mining 290    slide

    -
    - -
    -
    Description
    Learn how to obtain, clean, visualize, understand, model, and - predict the world around you using data. Grading will consist of homework - (30%), midterm (30%), project (40%). -
    -
    Instructor
    Jim Blomo <jblomo@ischool> -
    -
    GSI
    Shreyas <shreyas@ischool> -
    -
    Textbook
    Han, J., Kamber, M., & Pei, J. (2011). Data Mining: Concepts and Techniques, Third Edition (3rd ed.). Morgan Kaufmann. -
    -
    - - - -
    - -
    - -
    -

    2 Syllabus    slide

    -
    - -

    DM[0-9]+ indicates chapters from the text, Data Mining. -

    - - -- - - - - - - - - - - - - - - - - - - - - - - -
    DateReadingsSlidesHomework / Project
    Jan 25Try Github ; A Taxonomy of Data ScienceClass Intro ; Tools Intro by GUEST: ShreyasGit Intro
    Feb 1DM1 ; The Yelp Factor: Are Consumer Reviews Good for Business?Case Studies ; Obtaining DataObtain & Explore Data
    Feb 8DM2, DM3Probability ; PreprocessingData Stats
    Feb 15DM4, Apache Hadoop: Petabytes and Terawatts (slides); mrjob docs (for homework)Data Warehouse ; MapReduceProject Details ; mrjob
    Feb 22DM8Decision Trees; Naive BayesGini Index
    Mar 1DM[9.1-9.3], 9.5 ; Understanding the Bias-Variance TradeoffSVM ; Neural NetworksNeural Network Back Propagation
    Mar 8DM10Agglomerative - Clustering ; Hierarchical, Density - ClusteringK-Means
    Mar 15DM11.1Reviewprepare 1 cheat sheet
    Mar 221 cheat sheetMidterm-
    Mar 29HOLIDAY
    Apr 5DM6Advanced Clustering ; Frequent PatternAWS ; Project Proposal Due
    Apr 12DM11.3; PageRank; Uncovering Social Network Sybils in the WildGraphs; PageRankAdjacency Representations
    Apr 19Non-linear regressionGUEST: Gene Lee Ceaser's Pricing Strategy; Ceaser's RecruitingPrice Elasticity
    Apr 26DM12; Shazam Audio SearchOutliers; Images & AudioMidterm Review
    May 3Embedded Plots ; Data-Driven DocumentsVisualization ; Yelp's VisualizationsD3 Intro D3 Lab
    May 10A Few Useful Things to Know about Machine Learning ; Top 10 Algorithms in Data MiningIn Real Life ; PresentationsMay 16th: Project Papers Due
    May 17-Final PresentationBye!
    - - - - - - - -Fork me on GitHub - -
    -
    -
    - -
    -

    Date: 2013-05-10 00:54:43 PDT

    -

    Author: Jim Blomo

    -

    Org version 7.8.02 with Emacs version 23

    -Validate XHTML 1.0 - -
    - - From 196a68d0a6c9e8f1fa7ae9a74ae0ac8b21159241 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sat, 7 Dec 2013 20:24:13 -0800 Subject: [PATCH 08/79] fix brackets --- README.markdown | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.markdown b/README.markdown index 922bf8b..e70200b 100644 --- a/README.markdown +++ b/README.markdown @@ -6,10 +6,10 @@ predict the world around you using data. Grading will consist of homework (30%), a midterm (30%), and a project (40%). ### Instructor -Jimmy Retzlaff +Jimmy Retzlaff <jretz@ischool> ### GSI -Shreyas +Shreyas <shreyas@ischool> ### Textbook Han, J., Kamber, M., & Pei, J. (2011). _Data Mining: Concepts and Techniques_, Third Edition *(3rd ed.)*. Morgan Kaufmann. From 43e5edbb3f255eff2cfdb81699c8e9e8b45c6160 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sat, 7 Dec 2013 21:31:34 -0800 Subject: [PATCH 09/79] converted/updated Cast Studies slides --- slides/2013-02-01-CaseStudies.html | 740 ------------------------- slides/2013-02-01-CaseStudies.org | 209 ------- slides/2014-02-01-CaseStudies.html | 483 ++++++++++++++++ slides/2014-02-01-CaseStudies.markdown | 356 ++++++++++++ 4 files changed, 839 insertions(+), 949 deletions(-) delete mode 100644 slides/2013-02-01-CaseStudies.html delete mode 100644 slides/2013-02-01-CaseStudies.org create mode 100644 slides/2014-02-01-CaseStudies.html create mode 100644 slides/2014-02-01-CaseStudies.markdown diff --git a/slides/2013-02-01-CaseStudies.html b/slides/2013-02-01-CaseStudies.html deleted file mode 100644 index 64f2a2e..0000000 --- a/slides/2013-02-01-CaseStudies.html +++ /dev/null @@ -1,740 +0,0 @@ - - - - -2013-02-01-CaseStudies - - - - - - - - - - - - - - - - - - - -
    - -
    - -
    -

    2013-02-01-CaseStudies

    - - - - -
    -

    1 Questions    slide

    -
    - - - - -
    - -
    -

    1.1 Memory    notes

    -
    - -
      -
    • Spaced intervals makes memorizing easiest -
      • -
    - - -
    -
    - -
    - -
    -

    2 Case Studies    slide

    -
    - - -
    - -
    - -
    -

    3 Process    slide two_col

    -
    - -
      -
    • Knowledge Discovery in Databases (KDD) -
      • -
    • Selection -
      • -
    • Pre-processing -
      • -
    • Transformation -
      • -
    • Data Mining -
      • -
    • Interpretation/Evaluation -
      • -
    - - - -
      -
    • Cross Industry Standard Process for Data Mining -
      • -
    • Business Understanding -
      • -
    • Data Understanding -
      • -
    • Data Preparation -
      • -
    • Modeling -
      • -
    • Evaluation -
      • -
    • Deployment -
      • -
    - - -
    - -
    -

    3.1 Data to Knowledge    notes

    -
    - -
      -
    • We learned last week that the goal of data mining is to turn raw data into - knowledge -
      • -
    - - -
    -
    - -
    - -
    -

    4 Search Engine Logs    slide

    -
    - - - - -
    193.139.1 jim [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288
-282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929
-345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232
-10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342
-10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342
-
    - - -
      -
    • what is a common theme in these queries? -
      • -
    - - -
    - -
    -

    4.1 Raw Data    notes

    -
    - -
      -
    • raw data comes in many forms -
      • -
    • often well use tech examples: eg search engine logs -
      • -
    • these have information like user, IP, date-time, HTTP version, query -
      • -
    • can we extract actionable information from it? -
      • -
    - - -
    -
    - -
    - -
    -

    5 Flu Trends    slide two_col

    -
    - -
      -
    • Use dates to plot trends over time -
      • -
    • Use IPs to show activity per state or city -
      • -
    • Other ideas? -
      • -
    - - -

    - img/flu-trends.png -

    -
    - -
    -

    5.1 Other ideas    notes

    -
    - -
      -
    • What other information could you extract from log data? -
      • -
    • Spread of flu over countries, cities? -
      • -
    • Time of day? Do people notice in the morning? -
      • -
    • correlated with any other activity? (eg. travel) -
      • -
    • best day of the week to call in sick (and get away with it)? -
      • -
    - - -
    -
    - -
    - -
    -

    6 Asking Questions    slide

    -
    - -
      -
    • Many potential discoveries within search logs -
      • -
    • Asking meaningful questions is a difficult but essential part of data - mining -
      • -
    • Algorithms can answer questions for you, but it can't ask them -
      • -
    - - -
    - -
    -

    6.1 No magic    notes

    -
    - -
      -
    • Data mining is not a magical machine into which one throws data and gets - out interesting facts -
      • -
    • Data + question + algorithm suited for question => potential insights -
      • -
    - - -
    -
    - -
    - -
    -

    7 Data Mining Process    slide animate

    -
    - -
      -
    • Data cleaning -
      • -
    • Data integration -
      • -
    • Data selection -
      • -
    • Data transformation -
      • -
    • Data mining* -
      • -
    • Pattern evaluation -
      • -
    • Knowledge presentation -
      • -
    - - -
    - -
    -

    7.1 We cover the full process    notes

    -
    - -
    -
    Cleaning
    remove abuse requests, "Estimates for Connecticut for weeks - 2012-12-16 to 2013-01-06 were affected by a software glitch" -
    -
    Integration
    Collecting logs from different data centers, maybe from - different formats (over the years) -
    -
    Selection
    IPs, dates, queries -
    -
    Transformation
    IP to location. Dates to local time. -
    -
    Mining
    what words are associated with the flu? cold? fever? other - languages? -
    -
    Evaluation
    This year worse than last, peaking later. -
    -
    Presentation
    plotting, cartograms -
    -
    - - -
    -
    - -
    - -
    -

    8 Data Preparation    slide

    -
    - -
      -
    • Collecting, cleaning, integrating takes > 50% of the time in real world - situations -
      • -
    • Explains difficulty in finding good candidates for Data Scientist roles -
      • -
    - - -
    - -
    -

    8.1 Data Scientist    notes

    -
    - -
      -
    • In industry, most companies are hiring engineers to interact with the full - stack, so that they can collect data -
      • -
    • If preperation is > 50% and they hire you just for algorithms, they need ot - hire > 1 other person just to support you -
      • -
    • How many of you like just preparing data? -
      • -
    - - -
    -
    - -
    - -
    -

    9 Transactional Data    slide

    -
    - -
      -
    • Discrete history of events, containing some minimum amount of data: -
      • -
    • Subject: Who initiated action? -
      • -
    • Verb: What was done? -
      • -
    • Object: What was it done to? -
      • -
    • Timestamp: When? -
      • -
    - - -
    - -
    -

    9.1 Storage    notes

    -
    - -
      -
    • Most common example is purchase history -
      • -
    • Subject: user ID, or name -
      • -
    • Verb: In logs, can vary. In databases, you'll have a purchases table, so - verb is assumed to be "purchased" -
      • -
    • Object: product IDs (or in web logs, web pages) -
      • -
    • Timestamp: Make sure you account for timezones -
      • -
    • Other Data: previous page, extra info about action (purchase with CC? - Cash?) -
      • -
    - - -
    -
    - -
    - -
    -

    10 Other Data    slide

    -
    - -
      -
    • Often does not contain timestamps -
      • -
    • Spatial Data -
      • -
    • Multimedia -
      • -
    - -

    img/moonlight_sonata.jpg -

    -
    - -
    -

    10.1 Data    notes

    -
    - -
      -
    • Maps in general can be used to find interesting information: where are - cities typically located? What are properties of well planned cities? -
      • -
    • Videos have a time component, but are not transactional. -
      • -
    • Music can be seen non-linearly and analyzed -
      • -
    • img: http://flyingpudding.com/projects/viz_music/ -
      • -
    - - -
    -
    - -
    - -
    -

    11 Purpose of Data Mining    slide

    -
    - -
    -
    Purpose
    Obtaining actionable knowledge -
    -
    Descriptive
    explains data already seen -
    -
    Predictive
    Immediately understand new data -
    -
    - - -
    - -
    -

    11.1 Tasks    notes

    -
    - -
      -
    • At Amazon, dashboards for different countries -
      • -
    • Americans shopped at work; Germans shopped early morning, early evening; Japanese shopped late at night -
      • -
    • Can help with capacity planning, ideas for discounts, warehouse staffing -
      • -
    • Predictive: at Yelp, what business are you most likely to want to review - next? As you have activity, instantly understand what is the best - recommendation -
      • -
    - - -
    -
    - -
    - -
    -

    12 Types of Models    slide animate

    -
    - -
      -
    • Classifiers -
      • -
    • Regressions -
      • -
    • Clustering -
      • -
    • Outlier -
      • -
    - - -
    - -
    -

    12.1 Details    notes

    -
    - -
    -
    Classifiers
    describes and distinguishes cases. Yelp may want to find a - category for a business based on the reviews and business description -
    -
    Regressions
    Predict a continuous value. Eg. predict a home's selling - price given sq footage, # of bedrooms -
    -
    Clustering
    find "natural" groups of data without labels -
    -
    Outlier
    find anomalous transactions, eg. finding fraud for credit cards -
    -
    - - -
    - -
    - -
    -

    12.2 Tip of the Iceburg    slide two_col

    -
    - -

    img/iceberg11.jpg -

      -
    • Thousands of ways to calculate a model -
      • -
    • Combinatorially more ways to combine them -
      • -
    • In technique, large amount of overlap between purpose -
      • -
    - -
    - -
    - -
    -

    12.3 Survey    notes

    -
    - -
      -
    • ML and DM fields churn these models out -
      • -
    • Newest methods combine multiple models (boosting & bagging) -
      • -
    • We're going to cover these in much greater detail in the course -
      • -
    - - -
    -
    - -
    - -
    -

    13 Your own examples    slide animate

    -
    - -
      -
    • Classifiers -
      • -
    • Regressions -
      • -
    • Clustering -
      • -
    • Outlier -
      • -
    - - -
    - -
    -

    13.1 Examples    notes

    -
    - -
    -
    Classifiers
    Newly opened business -
    -
    Regressions
    Revenue estimates for a franchise store -
    -
    Clustering
    Movie genres -
    -
    Outlier
    Bot vs human web traffic -
    -
    - - -
    -
    - -
    - -
    -

    14 Machine Learning    slide

    -
    - -
    -
    Supervised
    Given data with a label, predict data without a - label -
    -
    Unsupervised
    Given data without labels, group "similar" items - together -
    -
    Semi-supervised
    Mix of the above: eg. unsupervised to find groups, - supervised to label and distinguish borderline cases -
    -
    Active
    Starting with unlabeled data, select the most helpful cases for a - human to label -
    -
    - - -
    - -
    - -
    -

    15 Matching    slide

    -
    - -
      -
    • Categories for businesses, where some business have correct labels, but not sure how precise categories should be -
      • -
    • Comparing search results algorithms: some queries return the same results, some return very different businesses -
      • -
    • Spam filter with existing corpus -
      • -
    • Demographic information about customers -
      • -
    - - -
    - -
    -

    15.1 Details    notes

    -
    - -
      -
    • Matching with the type of learning -
      • -
    - - -
    -
    - -
    - -
    -

    16 Break    slide

    -
    - - - - - - - -
    -
    -
    - -
    -

    Date: 2013-02-01 13:38:13 PST

    -

    Author: Jim Blomo

    -

    Org version 7.8.02 with Emacs version 23

    -Validate XHTML 1.0 - -
    - - diff --git a/slides/2013-02-01-CaseStudies.org b/slides/2013-02-01-CaseStudies.org deleted file mode 100644 index f0f73dc..0000000 --- a/slides/2013-02-01-CaseStudies.org +++ /dev/null @@ -1,209 +0,0 @@ -* Questions :slide: - + Differences from [[http://bid.berkeley.edu/cs294-1-spring13/index.php/Main_Page][CS 294 Behavioral Data Mining]] - + Course Load - + Readings: after lecture -** Memory :notes: - + Spaced intervals makes memorizing easiest - -* Case Studies :slide: - -* Process :slide:two_col: - + *Knowledge Discovery in Databases (KDD)* - + Selection - + Pre-processing - + Transformation - + Data Mining - + Interpretation/Evaluation - - - + *Cross Industry Standard Process for Data Mining* - + Business Understanding - + Data Understanding - + Data Preparation - + Modeling - + Evaluation - + Deployment -** Data to Knowledge :notes: - + We learned last week that the goal of data mining is to turn raw data into - knowledge - -* Search Engine Logs :slide: -#+begin_src log -193.139.1 jim [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288 -282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929 -345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232 -10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342 -10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342 -#+end_src - - + what is a common theme in these queries? -** Raw Data :notes: - + raw data comes in many forms - + often well use tech examples: eg search engine logs - + these have information like user, IP, date-time, HTTP version, query - + can we extract actionable information from it? - -* Flu Trends :slide:two_col: - + Use dates to plot trends over time - + Use IPs to show activity per state or city - + Other ideas? - - [[file:img/flu-trends.png]] -** Other ideas :notes: - + What other information could you extract from log data? - + Spread of flu over countries, cities? - + Time of day? Do people notice in the morning? - + correlated with any other activity? (eg. travel) - + best day of the week to call in sick (and get away with it)? - -* Asking Questions :slide: - + Many potential discoveries within search logs - + Asking meaningful questions is a difficult but essential part of data - mining - + Algorithms can answer questions for you, but it can't ask them -** No magic :notes: - + Data mining is not a magical machine into which one throws data and gets - out interesting facts - + Data + question + algorithm suited for question => potential insights - -* Data Mining Process :slide:animate: - + Data cleaning - + Data integration - + Data selection - + Data transformation - + Data mining* - + Pattern evaluation - + Knowledge presentation -** We cover the full process :notes: - + Cleaning :: remove abuse requests, "Estimates for Connecticut for weeks - 2012-12-16 to 2013-01-06 were affected by a software glitch" - + Integration :: Collecting logs from different data centers, maybe from - different formats (over the years) - + Selection :: IPs, dates, queries - + Transformation :: IP to location. Dates to local time. - + Mining :: what words are associated with the flu? cold? fever? other - languages? - + Evaluation :: This year worse than last, peaking later. - + Presentation :: plotting, cartograms - -* Data Preparation :slide: - + Collecting, cleaning, integrating takes > 50% of the time in real world - situations - + Explains difficulty in finding good candidates for Data Scientist roles -** Data Scientist :notes: - + In industry, most companies are hiring engineers to interact with the full - stack, so that they can collect data - + If preperation is > 50% and they hire you just for algorithms, they need ot - hire > 1 other person just to support you - + How many of you like just preparing data? - -* Transactional Data :slide: - + Discrete history of events, containing some minimum amount of data: - + Subject: Who initiated action? - + Verb: What was done? - + Object: What was it done to? - + Timestamp: When? -** Storage :notes: - + Most common example is purchase history - + Subject: user ID, or name - + Verb: In logs, can vary. In databases, you'll have a purchases table, so - verb is assumed to be "purchased" - + Object: product IDs (or in web logs, web pages) - + Timestamp: Make sure you account for timezones - + Other Data: previous page, extra info about action (purchase with CC? - Cash?) - -* Other Data :slide: - + Often does not contain timestamps - + Spatial Data - + Multimedia -[[file:img/moonlight_sonata.jpg]] -** Data :notes: - + Maps in general can be used to find interesting information: where are - cities typically located? What are properties of well planned cities? - + Videos have a time component, but are not transactional. - + Music can be seen non-linearly and analyzed - + img: http://flyingpudding.com/projects/viz_music/ - -* Purpose of Data Mining :slide: - + Purpose :: Obtaining *actionable knowledge* - + Descriptive :: explains data already seen - + Predictive :: Immediately understand new data -** Tasks :notes: - + At Amazon, dashboards for different countries - + Americans shopped at work; Germans shopped early morning, early evening; Japanese shopped late at night - + Can help with capacity planning, ideas for discounts, warehouse staffing - + Predictive: at Yelp, what business are you most likely to want to review - next? As you have activity, instantly understand what is the best - recommendation - -* Types of Models :slide:animate: - + Classifiers - + Regressions - + Clustering - + Outlier -** Details :notes: - + Classifiers :: describes and distinguishes cases. Yelp may want to find a - category for a business based on the reviews and business description - + Regressions :: Predict a continuous value. Eg. predict a home's selling - price given sq footage, # of bedrooms - + Clustering :: find "natural" groups of data *without labels* - + Outlier :: find anomalous transactions, eg. finding fraud for credit cards - -** Tip of the Iceburg :slide:two_col: -[[file:img/iceberg11.jpg]] - + Thousands of ways to calculate a model - + Combinatorially more ways to combine them - + In technique, large amount of overlap between purpose -** Survey :notes: - + ML and DM fields churn these models out - + Newest methods combine multiple models (boosting & bagging) - + We're going to cover these in much greater detail in the course - -* Your own examples :slide:animate: - + Classifiers - + Regressions - + Clustering - + Outlier -** Examples :notes: - + Classifiers :: Newly opened business - + Regressions :: Revenue estimates for a franchise store - + Clustering :: Movie genres - + Outlier :: Bot vs human web traffic - -* Machine Learning :slide: - + Supervised :: Given data with a label, predict data without a - label - + Unsupervised :: Given data without labels, group "similar" items - together - + Semi-supervised :: Mix of the above: eg. unsupervised to find groups, - supervised to label and distinguish borderline cases - + Active :: Starting with unlabeled data, select the most helpful cases for a - human to label - -* Matching :slide: - + Categories for businesses, where some business have correct labels, but not sure how precise categories should be - + Comparing search results algorithms: some queries return the same results, some return very different businesses - + Spam filter with existing corpus - + Demographic information about customers -** Details :notes: - + Matching with the type of learning - -* *Break* :slide: - -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: - -#+BEGIN_HTML - -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: diff --git a/slides/2014-02-01-CaseStudies.html b/slides/2014-02-01-CaseStudies.html new file mode 100644 index 0000000..3c0b3ce --- /dev/null +++ b/slides/2014-02-01-CaseStudies.html @@ -0,0 +1,483 @@ + + + + Data Mining + + + + + + + + + diff --git a/slides/2014-02-01-CaseStudies.markdown b/slides/2014-02-01-CaseStudies.markdown new file mode 100644 index 0000000..938d03f --- /dev/null +++ b/slides/2014-02-01-CaseStudies.markdown @@ -0,0 +1,356 @@ +name: inverse +layout: true +class: left, top, inverse + +--- + +## Questions + + Differences from [CS 294 Behavioral Data Mining](http://bid.berkeley.edu/cs294-1-spring13/index.php/Main_Page) + + Course Load + + Readings: after lecture + +??? + +## Memory :notes: + + Spaced intervals makes memorizing easiest + +--- + +## Case Studies + +--- + +## Process + +### [Knowledge Discovery in Databases (KDD)](http://en.wikipedia.org/wiki/Data_mining#Process) + + Selection + + Pre-processing + + Transformation + + Data Mining + + Interpretation/Evaluation + +### [Cross Industry Standard Process for Data Mining](http://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining) + + Business Understanding + + Data Understanding + + Data Preparation + + Modeling + + Evaluation + + Deployment + +??? + +## Data to Knowledge + + + We learned last week that the goal of data mining is to turn raw data into + knowledge + +--- + +## Search Engine Logs + +```log +193.139.1 jimmy [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288 +282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929 +345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232 +10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342 +10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342 +``` + +What is a common theme in these queries? + +??? + +## Raw Data + + raw data comes in many forms + + often we'll use tech examples: e.g., search engine logs + + these have information like user, IP, date-time, HTTP version, query + + can we extract actionable information from it? + +--- + +## Flu Trends +.left-column[ + + + Use dates to plot trends over time + + Use IPs to show activity per state or city + + Other ideas? +] + +.right-column[ +![Flu Trends](file:img/flu-trends.png) +] + +??? + +## Other ideas + + + What other information could you extract from log data? + + Spread of flu over countries, cities? + + Time of day? Do people notice in the morning? + + correlated with any other activity? (e.g., travel) + + best day of the week to call in sick (and get away with it)? + +--- + +## Asking Questions + + + Many potential discoveries within search logs + + Asking meaningful questions is a difficult but essential part of data + mining + + Algorithms can answer questions for you, but it can't ask them + +??? + +## No magic + + + Data mining is not a magical machine into which one throws data and gets + out interesting facts + + Data + question + algorithm suited for question => potential insights + +--- + +## Data Mining Process + + + Data cleaning + + Data integration + + Data selection + + Data transformation + + Data mining* + + Pattern evaluation + + Knowledge presentation + +??? + +## We cover the full process + +### Cleaning + remove abuse requests, "Estimates for Connecticut for weeks + 2012-12-16 to 2013-01-06 were affected by a software glitch" + +### Integration + Collecting logs from different data centers, maybe from + different formats (over the years) + +### Selection + IPs, dates, queries + +### Transformation + IP to location. Dates to local time. + +### Mining + what words are associated with the flu? cold? fever? other + languages? + +### Evaluation + This year worse than last, peaking later. + +### Presentation + plotting, cartograms + +--- + +## Data Preparation + + + Collecting, cleaning, integrating takes > 50% of the time in real world + situations + + Explains difficulty in finding good candidates for Data Scientist roles + +??? + +## Data Scientist + + + In industry, most companies are hiring engineers to interact with the full + stack, so that they can collect data + + If preparation is > 50% and they hire you just for algorithms, they need to + hire > 1 other person just to support you + + How many of you like just preparing data? + +--- + +## Transactional Data + + + Discrete history of events, containing some minimum amount of data: + + Subject: Who initiated action? + + Verb: What was done? + + Object: What was it done to? + + Timestamp: When? + +??? + +## Storage + + + Most common example is purchase history + + Subject: user ID, or name + + Verb: In logs, can vary. In databases, you'll have a purchases table, so + verb is assumed to be "purchased" + + Object: product IDs (or in web logs, web pages) + + Timestamp: Make sure you account for timezones + + Other Data: previous page, extra info about action (purchase with CC? + Cash?) + +--- + +## Other Data + + + Often does not contain timestamps + + Spatial Data + + Multimedia + +![Moonlight Sonata](file:img/moonlight_sonata.jpg) + +??? + +## Data + + + Maps in general can be used to find interesting information: where are + cities typically located? What are properties of well planned cities? + + Videos have a time component, but are not transactional. + + Music can be seen non-linearly and analyzed + + image: http://flyingpudding.com/projects/viz_music/ + +--- + +## Purpose of Data Mining + +Purpose + + + Obtaining *actionable knowledge* + +Descriptive + + + Explains data already seen + +Predictive + + + Immediately understand new data + +??? + +## Tasks + + At Amazon, dashboards for different countries + + Americans shopped at work; Germans shopped early morning, early evening; Japanese shopped late at night + + Can help with capacity planning, ideas for discounts, warehouse staffing + + Predictive: at Yelp, what business are you most likely to want to review + next? As you have activity, instantly understand what is the best + recommendation + +--- + +## Types of Models + + + Classifiers + + Regressions + + Clustering + + Outlier + +??? + +## Details + +Classifiers + + + describes and distinguishes cases. Yelp may want to find a + category for a business based on the reviews and business description + +Regressions + + + Predict a continuous value. e.g., predict a home's selling + price given sq footage, # of bedrooms + +Clustering + + + find "natural" groups of data *without labels* + +Outlier + + + find anomalous transactions, e.g., finding fraud for credit cards + +--- + +## Tip of the Iceberg + + + Thousands of ways to calculate a model + + Combinatorially more ways to combine them + + In technique, large amount of overlap between purpose + + + +??? + +## Survey + + + Machine Learning and Data Mining fields churn these models out + + Newest methods combine multiple models (boosting & bagging) + + We're going to cover these in much greater detail in the course + +--- + +## Your own examples + + + Classifiers + + Regressions + + Clustering + + Outlier + +??? + +## Examples + +Classifiers + + + Newly opened business + +Regressions + + + Revenue estimates for a franchise store + +Clustering + + + Movie genres + +Outlier + + + Bot vs human web traffic + +--- + +## Machine Learning + +Supervised + + + Given data with a label, predict data without a + label + +Unsupervised + + + Given data without labels, group "similar" items + together + +Semi-supervised + + + Mix of the above: e.g., unsupervised to find groups, + supervised to label and distinguish borderline cases + +Active + + + Starting with unlabeled data, select the most helpful cases + for a human to label + +--- + +## Matching + + + Categories for businesses, where some business have correct labels, but not sure how precise categories should be + + Comparing search results algorithms: some queries return the same results, some return very different businesses + + Spam filter with existing corpus + + Demographic information about customers + +??? + +## Details + + + Matching with the type of learning + +--- + +## *Break* From 96a3601defabbec80637305344a0659d86b2d7e1 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sat, 7 Dec 2013 21:35:29 -0800 Subject: [PATCH 10/79] fix dates in filenames --- .../{2014-02-01-CaseStudies.html => 2014-01-30-CaseStudies.html} | 0 ...02-01-CaseStudies.markdown => 2014-01-30-CaseStudies.markdown} | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename slides/{2014-02-01-CaseStudies.html => 2014-01-30-CaseStudies.html} (100%) rename slides/{2014-02-01-CaseStudies.markdown => 2014-01-30-CaseStudies.markdown} (100%) diff --git a/slides/2014-02-01-CaseStudies.html b/slides/2014-01-30-CaseStudies.html similarity index 100% rename from slides/2014-02-01-CaseStudies.html rename to slides/2014-01-30-CaseStudies.html diff --git a/slides/2014-02-01-CaseStudies.markdown b/slides/2014-01-30-CaseStudies.markdown similarity index 100% rename from slides/2014-02-01-CaseStudies.markdown rename to slides/2014-01-30-CaseStudies.markdown From daac5de1d9f4175e8a0ac86adaf3f9c27a38fc93 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sat, 7 Dec 2013 22:05:50 -0800 Subject: [PATCH 11/79] converted git Lab slides to markdown and improved styling --- slides/2013-02-01-Lab.html | 312 ----------------------------- slides/2013-02-01-Lab.org | 93 --------- slides/2014-01-23-Intro.html | 9 +- slides/2014-01-30-CaseStudies.html | 9 +- slides/2014-01-30-Lab.html | 240 ++++++++++++++++++++++ slides/2014-01-30-Lab.markdown | 112 +++++++++++ slides/presentation_template.html | 9 +- 7 files changed, 367 insertions(+), 417 deletions(-) delete mode 100644 slides/2013-02-01-Lab.html delete mode 100644 slides/2013-02-01-Lab.org create mode 100644 slides/2014-01-30-Lab.html create mode 100644 slides/2014-01-30-Lab.markdown diff --git a/slides/2013-02-01-Lab.html b/slides/2013-02-01-Lab.html deleted file mode 100644 index d7e8537..0000000 --- a/slides/2013-02-01-Lab.html +++ /dev/null @@ -1,312 +0,0 @@ - - - - -2013-02-01-Lab - - - - - - - - - - - - - - - - - - - -
    - -
    - -
    -

    2013-02-01-Lab

    - - - - -
    -

    1 Lab: Obtain and Explore Data    slide

    -
    - -
      -
    • Setup GitHub account -
      • -
    • Find a data set or external API -
      • -
    • Superficially examine it -
      • -
    • Summarize findings -
      • -
    • Submit assignment via GitHub -
      • -
    - - -
    - -
    - -
    -

    2 Why GitHub?    slide

    -
    - -
      -
    • git tool is standard in industry -
      • -
    • GitHub provides best tools for sharing, commenting code -
      • -
    • This assignment will not have code, just practice submitting -
      • -
    - - -
    - -
    - -
    -

    3 Setup GitHub account    slide

    -
    - - - - -
    - -
    - -
    -

    4 Setup git repository on ischool server    slide

    -
    - -
      -
    • On the server ischool.berkeley.edu -
      • -
    - - - - -
    $ git clone git://github.com/jblomo/datamining290.git
-
    - -
      -
    • On the server, in the datamining290 directory run -
      • -
    - - - - -
    $ git remote rename origin jblomo
-
    - - -
    - -
    - -
    -

    5 Connect it to GitHub    slide

    -
    - -
      -
    • After you recieve your free micro account on GitHub, create a private repository called datamining290 -
      • -
    • It will provide you with an SSH git path, let's call it PATH -
      • -
    • You must use the SSH PATH starting with git:// -
      • -
    • On the server, in the datamining290 directory, run -
      • -
    - - - - -
    $ git remote add origin PATH
-$ git push origin master
-
    - - -
    - -
    - -
    -

    6 Share with us    slide

    -
    - -
      -
    • Hopefully you now have a private copy of my repository -
      • -
    • Add Shreyas and me (users: seekshreyas, jblomo) as a contributor to your private repository -
      • -
    - - -
    - -
    - -
    -

    7 Obtain Data    slide

    -
    - -
      -
    • Look through the links in slides for interesting data sets, or find your own -
      • -
    • Or find a service API, like NYTimes -
      • -
    • Explore the data available to answer the following questions -
      • -
    - - -
    - -
    - -
    -

    8 Questions    slide

    -
    - -
      -
    • What are the types of data available to you? -
      • -
    • For data sets: how many records are in the data set? -
      • -
    • For API: what are the limits on fetching data? -
      • -
    • Provide an "interesting" record, explain its properties and why it is - interesting -
      • -
    • What are 3 questions you could answer using your data? -
      • -
    - - -
    - -
    - -
    -

    9 Submit Homework    slide

    -
    - -
      -
    • On the ischool server, create a branch called hw-obtain-data -
      • -
    • Create a text file to write the solution, a simple editor to use is pico -
      • -
    • git add the file -
      • -
    • git commit the change -
      • -
    • git push origin hw-obtain-data to put it on GitHub -
      • -
    • on github, submit a "pull request" from the hw-obtain-data branch to your master branch -
      • -
    - - -
    - -
    -

    9.1 Pull Requests    notes

    -
    - -
      -
    • Pull requests are a way of showing updates in a way that lets me provide - comments, get notifications -
      • -
    • This is the first time I've tried it for class, so you're on the cutting - edge. Hopefully it will work, give me feedback if it is not -
      • -
    - - -
    -
    - -
    - -
    -

    10 Going Forward    slide

    -
    - -
      -
    • Other homework assignments will be completing code -
      • -
    • General work-flow: -
        -
      • Start a new branch -
        • -
      • Add required files -
        • -
      • push to GitHub -
        • -
      • Submit Pull Request -
        • -
      - -
      • -
    - - - - - - - -
    -
    -
    - -
    -

    Date: 2013-02-01 17:25:49 PST

    -

    Author: Jim Blomo

    -

    Org version 7.8.02 with Emacs version 23

    -Validate XHTML 1.0 - -
    - - diff --git a/slides/2013-02-01-Lab.org b/slides/2013-02-01-Lab.org deleted file mode 100644 index ffc2413..0000000 --- a/slides/2013-02-01-Lab.org +++ /dev/null @@ -1,93 +0,0 @@ -* Lab: Obtain and Explore Data :slide: - + Setup GitHub account - + Find a data set or external API - + Superficially examine it - + Summarize findings - + Submit assignment via GitHub - -* Why GitHub? :slide: - + =git= tool is standard in industry - + GitHub provides best tools for sharing, commenting code - + This assignment will not have code, just practice submitting - -* Setup GitHub account :slide: - + Create a [[https://github.com/signup/free][GitHub Account]], making sure to - use your .edu address - + Use [[https://github.com/edu][GitHub/Edu]] to request a free micro plan: - these let us use private accounts - + Setup a [[https://help.github.com/articles/generating-ssh-keys][GitHub SSH Key]] - -* Setup git repository on ischool server :slide: - + On the server ischool.berkeley.edu -#+begin_src bash -$ git clone git://github.com/jblomo/datamining290.git -#+end_src - + On the server, in the datamining290 directory run -#+begin_src bash -$ git remote rename origin jblomo -#+end_src - -* Connect it to GitHub :slide: - + After you recieve your free micro account on GitHub, create a private repository called datamining290 - + It will provide you with an SSH git path, let's call it PATH - + You must use the *SSH* PATH starting with =git://= - + On the server, in the datamining290 directory, run -#+begin_src html -$ git remote add origin PATH -$ git push origin master -#+end_src - -* Share with us :slide: - + Hopefully you now have a private copy of my repository - + Add Shreyas and me (users: seekshreyas, jblomo) as a contributor to your private repository - -* Obtain Data :slide: - + Look through the links in slides for interesting data sets, or find your own - + Or find a service API, like NYTimes - + Explore the data available to answer the following questions - -* Questions :slide: - + What are the types of data available to you? - + For data sets: how many records are in the data set? - + For API: what are the limits on fetching data? - + Provide an "interesting" record, explain its properties and why it is - interesting - + What are 3 questions you could answer using your data? - -* Submit Homework :slide: - + On the ischool server, create a branch called =hw-obtain-data= - + Create a text file to write the solution, a simple editor to use is =pico= - + =git add= the file - + =git commit= the change - + =git push origin hw-obtain-data= to put it on GitHub - + on github, submit a "pull request" from the =hw-obtain-data= branch to your master branch -** Pull Requests :notes: - + Pull requests are a way of showing updates in a way that lets me provide - comments, get notifications - + This is the first time I've tried it for class, so you're on the cutting - edge. Hopefully it will work, give me feedback if it is not - -* Going Forward :slide: - + Other homework assignments will be completing code - + General work-flow: - + Start a new branch - + Add required files - + push to GitHub - + Submit Pull Request - -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: - -#+BEGIN_HTML - -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index bbba2aa..c445e4c 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -31,11 +31,12 @@ text-decoration: none; } code { - -moz-border-radius: 5px; - -web-border-radius: 5px; + -moz-border-radius: 3px; + -web-border-radius: 3px; background: #e7e8e2; - border-radius: 5px; - font-size: 16px; + color: black; + border-radius: 3px; + /*font-size: 16px;*/ } em { color: #80cafa; } .pull-left { diff --git a/slides/2014-01-30-CaseStudies.html b/slides/2014-01-30-CaseStudies.html index 3c0b3ce..07d03c7 100644 --- a/slides/2014-01-30-CaseStudies.html +++ b/slides/2014-01-30-CaseStudies.html @@ -31,11 +31,12 @@ text-decoration: none; } code { - -moz-border-radius: 5px; - -web-border-radius: 5px; + -moz-border-radius: 3px; + -web-border-radius: 3px; background: #e7e8e2; - border-radius: 5px; - font-size: 16px; + color: black; + border-radius: 3px; + /*font-size: 16px;*/ } em { color: #80cafa; } .pull-left { diff --git a/slides/2014-01-30-Lab.html b/slides/2014-01-30-Lab.html new file mode 100644 index 0000000..3e4c725 --- /dev/null +++ b/slides/2014-01-30-Lab.html @@ -0,0 +1,240 @@ + + + + Data Mining + + + + + + + + + diff --git a/slides/2014-01-30-Lab.markdown b/slides/2014-01-30-Lab.markdown new file mode 100644 index 0000000..b61125e --- /dev/null +++ b/slides/2014-01-30-Lab.markdown @@ -0,0 +1,112 @@ +name: inverse +layout: true +class: left, top, inverse + +--- + +## Lab: Obtain and Explore Data + + + Setup GitHub account + + Find a data set or external API + + Superficially examine it + + Summarize findings + + Submit assignment via GitHub + +--- + +## Why GitHub? + + + ```git``` tool is standard in industry + + GitHub provides best tools for sharing, commenting code + + This assignment will not have code, just practice submitting + +--- + +## Setup GitHub account + + + Create a [GitHub Account](https://github.com/signup/free), making sure to + use your .edu address + + Use [GitHub/Edu](https://github.com/edu) to request a free micro plan: + these let us use private accounts + + Setup a [GitHub SSH Key](https://help.github.com/articles/generating-ssh-keys) + +--- + +## Setup git repository on ischool server + + + On the server ischool.berkeley.edu +```bash +$ git clone git://github.com/jretz/datamining290.git +``` + + On the server, in the datamining290 directory run +```bash +$ git remote rename origin jretz +``` + +--- + +## Connect it to GitHub + + + After you receive your free micro account on GitHub, create a private repository called datamining290 + + It will provide you with an SSH git path, let's call it PATH + + You must use the *SSH* PATH starting with ```git://``` + + On the server, in the datamining290 directory, run +```bash +$ git remote add origin PATH +$ git push origin master +``` + +--- + +## Share with us + + + Hopefully you now have a private copy of my repository + + Add Shreyas and me (users: seekshreyas, jretz) as a contributor to your private repository + +--- + +## Obtain Data + + + Look through the links in slides for interesting data sets, or find your own + + Or find a service API, like NYTimes + + Explore the data available to answer the following questions + +--- + +## Questions + + + What are the types of data available to you? + + For data sets: how many records are in the data set? + + For API: what are the limits on fetching data? + + Provide an "interesting" record, explain its properties and why it is + interesting + + What are 3 questions you could answer using your data? + +--- + +## Submit Homework + + + On the ischool server, create a branch called ```hw-obtain-data``` + + Create a text file to write the solution, a simple editor to use is ```pico``` + + ```git add``` the file + + ```git commit``` the change + + ```git push origin hw-obtain-data``` to put it on GitHub + + On github, submit a "pull request" from the ```hw-obtain-data``` branch to your master branch + +??? + +## Pull Requests + + + Pull requests are a way of showing updates in a way that lets me provide + comments, get notifications + +--- + +## Going Forward + + + Other homework assignments will be completing code + + General work-flow: + + Start a new branch + + Add required files + + Push to GitHub + + Submit Pull Request diff --git a/slides/presentation_template.html b/slides/presentation_template.html index d4d7af4..eaae6ac 100644 --- a/slides/presentation_template.html +++ b/slides/presentation_template.html @@ -31,11 +31,12 @@ text-decoration: none; } code { - -moz-border-radius: 5px; - -web-border-radius: 5px; + -moz-border-radius: 3px; + -web-border-radius: 3px; background: #e7e8e2; - border-radius: 5px; - font-size: 16px; + color: black; + border-radius: 3px; + /*font-size: 16px;*/ } em { color: #80cafa; } .pull-left { From 8b45666337aca0378db5a3e6c59d125eedbe7cdc Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sun, 8 Dec 2013 00:02:28 -0800 Subject: [PATCH 12/79] converted the rest of the second lecture to markdown --- slides/2013-02-01-Obtaining-Data.html | 944 ---------------------- slides/2013-02-01-Obtaining-Data.org | 279 ------- slides/2014-01-23-Intro.html | 13 + slides/2014-01-30-CaseStudies.html | 13 + slides/2014-01-30-Lab.html | 13 + slides/2014-01-30-Obtaining-Data.html | 576 +++++++++++++ slides/2014-01-30-Obtaining-Data.markdown | 435 ++++++++++ slides/presentation_template.html | 13 + 8 files changed, 1063 insertions(+), 1223 deletions(-) delete mode 100644 slides/2013-02-01-Obtaining-Data.html delete mode 100644 slides/2013-02-01-Obtaining-Data.org create mode 100644 slides/2014-01-30-Obtaining-Data.html create mode 100644 slides/2014-01-30-Obtaining-Data.markdown diff --git a/slides/2013-02-01-Obtaining-Data.html b/slides/2013-02-01-Obtaining-Data.html deleted file mode 100644 index 1376a1b..0000000 --- a/slides/2013-02-01-Obtaining-Data.html +++ /dev/null @@ -1,944 +0,0 @@ - - - - -2013-02-01-Obtaining-Data - - - - - - - - - - - - - - - - - - - -
    - -
    - -
    -

    2013-02-01-Obtaining-Data

    - - - - -
    -

    1 Obtaining Data    slide

    -
    - - -
    - -
    - -
    -

    2 Ways to Collect    slide two_col

    -
    - -
      -
    • Operational Data -
      • -
    • Data Warehouse -
      • -
    • Unstructured Data -
      • -
    • External API -
      • -
    • Data Sets -
      • -
    - - -

    -img/bottlecaps.jpg -

    - -
    - -
    -

    2.1 Operational Data    slide

    -
    - -
      -
    • Most frequent in industry -
      • -
    • Usually stored in databases best suited for transactional use -
      • -
    • Challenge is reorganizing data to suit question -
      • -
    - - -
    - -
    -

    2.1.1 Data from production    notes

    -
    - -
      -
    • Most frequently you'll have data that is being used by the application, - and you'll want to find insights in it -
      • -
    • We'll go into more detail in another class, but online use is - optimized for small queries and small updates -
      • -
    • Frequently just accessing the data in bulk is a software engineering - problem: -
        -
      • ensuring long queries don't hold up production usage -
        • -
      • joining across databases via software -
        • -
      • understanding esoteric columns, like "flags" -
        • -
      - -
      • -
    • Often will want to reorganize data to look like transactional -
      • -
    • img: http://woodwarddesign.ca/blog/2009/03/06/bottle-caps/ -
      • -
    - - -
    -
    - -
    - -
    -

    2.2 Example    slide

    -
    - -
      -
    • Find the user names with most "liked" reviews on Yelp -
      • -
    - - -

    -users -

    - -- - - - - -
    userIDnameflags
    25234Jim0x200
    - - -

    -reviews -

    - -- - - - - -
    reviewIDbusinessIDuserIDstarstextflags
    28252432252344great place!0x1
    - - - -

    -feedback -

    - -- - - - - -
    reviewIDsrcUserIDufcFlagsflags
    28282050x10x0
    - - - -
    - -
    -

    2.2.1 Distributed Data    notes

    -
    - -
      -
    • At Yelp we have a variety of database tables, and those tables can be - spread across different databases -
      • -
    • At a minimum we frequently need to JOIN across tables to answer queries -
        -
      • eg. matching up user names with reviews from separate tables -
        • -
      - -
      • -
    • It is possible the review table is only indexed on business ID, and so - finding all reviews by a user is really disk intensive: make sure you're - not slowing down the whole site! -
      • -
    • An additional challenge is when the "feedback" tables are in a separate - database: can no longer issue normal SQL queries -
      • -
    • What are these "flag" columns for? -
      • -
    • Exactly: no one knows. Often must look into code, or compare data to - production representation to guess meaning. In Yelp, 0x1 often means - "inactive", so we probably don't want to count that feedback -
      • -
    - - -
    -
    - -
    - -
    -

    2.3 Data Warehouse    slide two_col

    -
    - -
      -
    • Data located on same system -
      • -
    • Organized for analytics queries -
      • -
    • Requires extra maintenance and understanding of construction -
      • -
    - - -

    - img/Ikea-Warehouse.jpg -

    -
    - -
    -

    2.3.1 No free lunch    notes

    -
    - -
      -
    • A strong data warehouse can be a big improvement over operational data -
      • -
    • Hopefully, someone has already cleaned, joined data in a way that makes - sense! -
      • -
    • Optimized for long running queries: less fear of brining down website! -
      • -
    • But you must learn how that process was accomplished in order to understand - potential problems -
      • -
    • How to handle missing data? -
      • -
    • We'll go into more detail about how data warehouse schemas compare to - online ones later in the course -
      • -
    - - -
    -
    - -
    - -
    -

    2.4 Unstructured    slide

    -
    - -
      -
    • Haphazard collection of data -
      • -
    • Unclear what structure should be -
      • -
    • Examples: Web logs, text, multimedia -
      • -
    • Must extract structure eventually -
      • -
    - - -
    - -
    -

    2.4.1 Yelp JSON logs    notes

    -
    - -
      -
    • When developing a web application, new context or details become - important: how long did certain requests take? What link did a user follow - to a website? -
      • -
    • Relational Databases aren't well suited for these wide varieties of - potential attributes that don't apply to all items -
      • -
    • So the current work around is just to write all useful information down in - a log, and extract what is needed later -
      • -
    • Text, like business reviews, another example: desired structure changes - radically between questions: How many words? Characters? What is the sentiment? -
      • -
    • Pictures can contain attributes like color depth, length, width -
      • -
    • First step of data mining is often imposing structure on data: the data is - not inherently unstructured, it just is unclear what the structure should be - until query time -
      • -
    - - -
    -
    - -
    - -
    -

    2.5 Search Logs Example    slide

    -
    - - - - -
    193.139.1 jim [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288
-282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929
-345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232
-10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342
-10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342
-
    - - - - -- - - - - - - - - -
    userNamedatequery
    jim10/Oct/2013:13:55:36 -0700headache
    shreyas10/Oct/2013:13:56:36 -0700bananas
    steven10/Oct/2013:13:56:37 -0700cold
    anne10/Oct/2013:13:56:39 -0700flu shot
    lily10/Oct/2013:13:57:40 -0700i290
    - - - -
    - -
    -

    2.5.1 Imposing Structure    notes

    -
    - -
      -
    • Extract only the rows we know follow a format -
      • -
    • Format queries from some encoding (eg. URL) to standardized format -
      • -
    - - -
    -
    - -
    - -
    -

    2.6 External APIs    slide

    -
    - -
      -
    • Better documented than internal data! -
      • -
    • More limited in amount and detail -
      • -
    • Commonly HTTP/REST based -
      • -
    - - -
    - -
    -

    2.6.1 Motivation    notes

    -
    - -
      -
    • Companies are often searching for other ways to leverage their data -
      • -
    • Both for immediate business purposes, and for brand recognition -
      • -
    • Twitter more (in)famous example -
      • -
    • NYTimes another good option -
      • -
    - - -
    -
    - -
    - -
    -

    2.7 NYTimes API Example    slide

    -
    - - - - -
    - -
    -

    2.7.1 Accessing these    notes

    -
    - -
      -
    • More info on how to access these APIs is in the Web Architecture class, - but feel free to ask Shreyas or I about how best to access them -
      • -
    - - -
    -
    - -
    - -
    -

    2.8 Data Sets    slide

    -
    - -
      -
    • Download large, curated set of data all at once -
      • -
    • Formats vary, but usually documented -
      • -
    • Can be useful to combine with other datasets or APIs -
      • -
    - - -

    -img/kaggle-digits.png -

    -
    - -
    -

    2.8.1 Research    notes

    -
    - -
      -
    • Data sets commonly used in research: can compare different techniques on - same data to understand advantages -
      • -
    • Sizes can range to a few MB to GB -
      • -
    • JSON, CSV, XML all potential formats. Cleaning, organization for your - question again becomes an important aspect -
      • -
    - - -
    -
    - -
    - -
    -

    2.9 Data Set Example    slide two_col

    -
    - - - - -

    -img/video.jpg -

    -
    -
    - -
    - -
    -

    3 Exploring Data    slide

    -
    - -
      -
    • Data sets are frequently too large to fit in standard tools like Excel or - Word -
      • -
    • Simplest to explore on the command line -
      • -
    • Homework will be exploring a data set of your choice -
      • -
    - - -
    - -
    -

    3.1 Size    notes

    -
    - -
      -
    • Some formats will not be easily parsed into Excel: eg. JSON, XML -
      • -
    • Word will be slow, or unworkable for GB size data -
      • -
    • CLI provides many composable tools for text manipulation -
      • -
    - - -
    -
    - -
    - -
    -

    4 Yelp Academic Dataset    slide

    -
    - -
      -
    • Yelp Data Set covers reviews, users, - businesses -
      • -
    • To download, you'll need to sign up: process takes ~24 hours for approval -
      • -
    • Use .edu email -
      • -
    - - -
    - -
    -

    4.1 Example    notes

    -
    - -
      -
    • We'll use this as an example, you can use any data set of your choice -
      • -
    • Just for HW, don't need to use for project -
      • -
    - - -
    -
    - -
    - -
    -

    5 CLI introduction    slide

    -
    - -
      -
    • Standard commands available in Learn CLI the hard way -
      • -
    • All example will be run on ischool.berkeley.edu -
      • -
    • Sheyas and I available for more help -
      • -
    - - -
    - -
    -

    5.1 Help    notes

    -
    - -
      -
    • If you're new, don't be intimidated. -
      • -
    • Security policies ensure you can't break anything besides your own files -
      • -
    • Keep backups of important stuff anyway -
      • -
    - - -
    -
    - -
    - -
    -

    6 wget    slide

    -
    - -
      -
    • Used for downloading files -
      • -
    • Downloading with the browser is fine, but sometimes nice to use faster - connection, or download it directly to machine you're working on -
      • -
    - - - - -
    $ wget 'http://www.grouplens.org/system/files/ml-100k.zip'
-
    - - -
    - -
    -

    6.1 Command    notes

    -
    - -
      -
    • Just wget URL -
      • -
    • I like to use quotes in case there are special characters in the URL, eg - ? -
      • -
    • Will download to current directory, same name as remote file -
      • -
    - - -
    -
    - -
    - -
    -

    7 scp    slide

    -
    - -
      -
    • Copy a file to or from a remote machine -
      • -
    • Uses same connection as SSH, but copies data instead -
      • -
    • Example: Copy data you've downloaded in your browser -
      • -
    - - - - -
    $ scp ~/Downloads/ml-100k.zip jblomo@ischool.berkeley.edu:
-# or
-$ scp ~/Downloads/ml-100k.zip jblomo@ischool.berkeley.edu:i290/movielens-100k.zip
-
    - - -
    - -
    -

    7.1 Command    notes

    -
    - -
      -
    • Trailing : is important: signifies remote machine -
      • -
    • If you don't specify path or filename, will copy the file with the same - name into your home directory -
      • -
    - - -
    -
    - -
    - -
    -

    8 gunzip unzip    slide

    -
    - -
      -
    • Uncompress data sets for simpler, faster manipulation -
      • -
    - - - - -
    $ unzip ml-100k.zip
-# or
-$ gunzip yelp_academic_dataset.json.gz
-
    - - -
    - -
    -

    8.1 Commands    notes

    -
    - -
    -
    unzip
    expand potentially many file, leave original alone -
    -
    gunzip
    expand original file, leaving only the uncompressed version -
    -
    - - -
    -
    - -
    - -
    -

    9 less    slide

    -
    - -
      -
    • View a file -
      • -
    • History: original command was called more to see a file a page at a time -
      • -
    • "Less is more" -
      • -
    - - - - -
    less yelp_academic_dataset.json
-
    - - -
    - -
    - -
    -

    10 Searching in less    slide

    -
    - -
      -
    • / (forward slash) lets you input search text -
      • -
    • q will quit -
      • -
    - - - - -
    /type": "user"
-/type": "review"
-
    - - -
    - -
    -

    10.1 Command    notes

    -
    - -
      -
    • Useful for finding specific instances to investigate -
      • -
    - - -
    -
    - -
    - -
    -

    11 grep    slide

    -
    - -
      -
    • Find and print lines matching a "regular expression" -
      • -
    • Regular expressions are "find" on steroids, but you can use simple strings -
      • -
    - - - - -
    $ grep 'type": "review"' yelp_academic_dataset.json
-
    - - -
    - -
    - -
    -

    12 wc    slide

    -
    - -
      -
    • "wordcount" counts characters, words, lines -
      • -
    • Most useful in data sets for lines: -l -
      • -
    - - - - -
    $ wc -l yelp_academic_dataset.json
-474434 yelp_academic_dataset.json
-
    - - -
    - -
    - -
    -

    13 Composable    slide

    -
    - -
      -
    • Genius of Unix: do one thing well, compose commands to get what you want -
      • -
    • | pipe characters "sends" output from one program to the input of another -
      • -
    • How many reviews in the data set? -
      • -
    - - - - -
    $ grep 'type": "review"' yelp_academic_dataset.json | wc -l
-330071
-$ egrep -o 'business_id": "\w+"' yelp_academic_dataset.json  | sort -u | wc -l
-9592
-
    - - - - - -
    -
    -
    - -
    -

    Date: 2013-02-01 13:45:14 PST

    -

    Author: Jim Blomo

    -

    Org version 7.8.02 with Emacs version 23

    -Validate XHTML 1.0 - -
    - - diff --git a/slides/2013-02-01-Obtaining-Data.org b/slides/2013-02-01-Obtaining-Data.org deleted file mode 100644 index 6de7701..0000000 --- a/slides/2013-02-01-Obtaining-Data.org +++ /dev/null @@ -1,279 +0,0 @@ -* Obtaining Data :slide: - -* Ways to Collect :slide:two_col: - + Operational Data - + Data Warehouse - + Unstructured Data - + External API - + Data Sets - -[[file:img/bottlecaps.jpg]] - -** Operational Data :slide: - + Most frequent in industry - + Usually stored in databases best suited for transactional use - + Challenge is reorganizing data to suit question -*** Data from production :notes: - + Most frequently you'll have data that is being used by the application, - and you'll want to find insights in it - + We'll go into more detail in another class, but online use is - optimized for small queries and small updates - + Frequently just accessing the data in bulk is a software engineering - problem: - + ensuring long queries don't hold up production usage - + joining across databases via software - + understanding esoteric columns, like "flags" - + Often will want to reorganize data to look like transactional - + img: http://woodwarddesign.ca/blog/2009/03/06/bottle-caps/ - -** Example :slide: - + Find the user names with most "liked" reviews on Yelp - -users -| userID | name | flags | -| 25234 | Jim | 0x200 | - -reviews -| reviewID | businessID | userID | stars | text | flags | -| 282 | 52432 | 25234 | 4 | great place! | 0x1 | - - -feedback -| reviewID | srcUserID | ufcFlags | flags | -| 282 | 8205 | 0x1 | 0x0 | - -*** Distributed Data :notes: - + At Yelp we have a variety of database tables, and those tables can be - spread across different databases - + At a minimum we frequently need to =JOIN= across tables to answer queries - + eg. matching up user names with reviews from separate tables - + It is possible the review table is only indexed on business ID, and so - finding all reviews by a user is really disk intensive: make sure you're - not slowing down the whole site! - + An additional challenge is when the "feedback" tables are in a separate - database: can no longer issue normal SQL queries - + What are these "flag" columns for? - + Exactly: no one knows. Often must look into code, or compare data to - production representation to guess meaning. In Yelp, =0x1= often means - "inactive", so we probably don't want to count that feedback - -** Data Warehouse :slide:two_col: - + Data located on same system - + Organized for analytics queries - + Requires extra maintenance and understanding of construction - - [[file:img/Ikea-Warehouse.jpg]] -*** No free lunch :notes: - + A strong data warehouse can be a big improvement over operational data - + Hopefully, someone has already cleaned, joined data in a way that makes - sense! - + Optimized for long running queries: less fear of brining down website! - + But you must learn how that process was accomplished in order to understand - potential problems - + How to handle missing data? - + We'll go into more detail about how data warehouse schemas compare to - online ones later in the course - -** Unstructured :slide: - + Haphazard collection of data - + Unclear what structure should be - + Examples: Web logs, text, multimedia - + Must extract structure eventually -*** Yelp JSON logs :notes: - + When developing a web application, new context or details become - important: how long did certain requests take? What link did a user follow - to a website? - + Relational Databases aren't well suited for these wide varieties of - potential attributes that don't apply to all items - + So the current work around is just to write all useful information down in - a log, and extract what is needed later - + Text, like business reviews, another example: desired structure changes - radically between questions: How many words? Characters? What is the sentiment? - + Pictures can contain attributes like color depth, length, width - + First step of data mining is often imposing structure on data: the data is - not inherently unstructured, it just is unclear what the structure *should be* - until query time - -** Search Logs Example :slide: -#+begin_src log -193.139.1 jim [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288 -282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929 -345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232 -10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342 -10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342 -#+end_src - -| userName | date | query | -| jim | 10/Oct/2013:13:55:36 -0700 | headache | -| shreyas | 10/Oct/2013:13:56:36 -0700 | bananas | -| steven | 10/Oct/2013:13:56:37 -0700 | cold | -| anne | 10/Oct/2013:13:56:39 -0700 | flu shot | -| lily | 10/Oct/2013:13:57:40 -0700 | i290 | - -*** Imposing Structure :notes: - + Extract only the rows we know follow a format - + Format queries from some encoding (eg. URL) to standardized format - -** External APIs :slide: - + Better documented than internal data! - + More limited in amount and detail - + Commonly HTTP/REST based -*** Motivation :notes: - + Companies are often searching for other ways to leverage their data - + Both for immediate business purposes, and for brand recognition - + Twitter more (in)famous example - + NYTimes another good option - -** NYTimes API Example :slide: - + [[http://developer.nytimes.com/docs/read/article_search_api][Article Search API]] - + http://api.nytimes.com/svc/search/v1/article?format=json&query=ballot&api-key=6578bab7f8c3808ce4c392edc9a793f0:8:5717915 -*** Accessing these :notes: - + More info on how to access these APIs is in the Web Architecture class, - but feel free to ask Shreyas or I about how best to access them - -** Data Sets :slide: - + Download large, curated set of data all at once - + Formats vary, but usually documented - + Can be useful to combine with other datasets or APIs - -[[file:img/kaggle-digits.png]] -*** Research :notes: - + Data sets commonly used in research: can compare different techniques on - same data to understand advantages - + Sizes can range to a few MB to GB - + JSON, CSV, XML all potential formats. Cleaning, organization for your - question again becomes an important aspect - -** Data Set Example :slide:two_col: - + [[http://www.grouplens.org/node/73][MovieLens Data Sets]] - + [[https://www.kaggle.com/c/digit-recognizer][Kaggle Digit Recognizer]] - + [[https://bitly.com/bundles/hmason/1][Hilary Mason's Data Sets]] - -[[file:img/video.jpg]] - -* Exploring Data :slide: - + Data sets are frequently too large to fit in standard tools like Excel or - Word - + Simplest to explore on the command line - + Homework will be exploring a data set of your choice -** Size :notes: - + Some formats will not be easily parsed into Excel: eg. JSON, XML - + Word will be slow, or unworkable for GB size data - + CLI provides many composable tools for text manipulation - -* Yelp Academic Dataset :slide: - + [[http://yelp.com/academic_dataset][Yelp Data Set]] covers reviews, users, - businesses - + To download, you'll need to sign up: process takes ~24 hours for approval - + Use .edu email -** Example :notes: - + We'll use this as an example, you can use any data set of your choice - + Just for HW, don't need to use for project - -* CLI introduction :slide: - + Standard commands available in [[http://cli.learncodethehardway.com][Learn CLI the hard way]] - + All example will be run on =ischool.berkeley.edu= - + Sheyas and I available for more help -** Help :notes: - + If you're new, don't be intimidated. - + Security policies ensure you can't break anything besides your own files - + Keep backups of important stuff anyway - -* =wget= :slide: - + Used for downloading files - + Downloading with the browser is fine, but sometimes nice to use faster - connection, or download it directly to machine you're working on -#+begin_src bash -$ wget 'http://www.grouplens.org/system/files/ml-100k.zip' -#+end_src -** Command :notes: - + Just =wget URL= - + I like to use quotes in case there are special characters in the URL, eg - =?= - + Will download to current directory, same name as remote file - -* =scp= :slide: - + Copy a file to or from a remote machine - + Uses same connection as SSH, but copies data instead - + Example: Copy data you've downloaded in your browser -#+begin_src bash -$ scp ~/Downloads/ml-100k.zip jblomo@ischool.berkeley.edu: -# or -$ scp ~/Downloads/ml-100k.zip jblomo@ischool.berkeley.edu:i290/movielens-100k.zip -#+end_src -** Command :notes: - + Trailing =:= is important: signifies remote machine - + If you don't specify path or filename, will copy the file with the same - name into your home directory - -* =gunzip= =unzip= :slide: - + Uncompress data sets for simpler, faster manipulation -#+begin_src bash -$ unzip ml-100k.zip -# or -$ gunzip yelp_academic_dataset.json.gz -#+end_src -** Commands :notes: - + unzip :: expand potentially many file, leave original alone - + gunzip :: expand original file, leaving only the uncompressed version - -* =less= :slide: - + View a file - + History: original command was called =more= to see a file a page at a time - + "Less is more" -#+begin_src bash -less yelp_academic_dataset.json -#+end_src - -* Searching in =less= :slide: - + =/= (forward slash) lets you input search text - + =q= will quit -#+begin_src less -/type": "user" -/type": "review" -#+end_src -** Command :notes: - + Useful for finding specific instances to investigate - -* =grep= :slide: - + Find and print lines matching a "regular expression" - + [[http://www.regular-expressions.info/quickstart.html][Regular expressions]] are "find" on steroids, but you can use simple strings -#+begin_src bash -$ grep 'type": "review"' yelp_academic_dataset.json -#+end_src - -* =wc= :slide: - + "wordcount" counts characters, words, lines - + Most useful in data sets for lines: =-l= -#+begin_src bash -$ wc -l yelp_academic_dataset.json -474434 yelp_academic_dataset.json -#+end_src - -* Composable :slide: - + Genius of Unix: do one thing well, compose commands to get what you want - + =|= pipe characters "sends" output from one program to the input of another - + How many reviews in the data set? -#+begin_src bash -$ grep 'type": "review"' yelp_academic_dataset.json | wc -l -330071 -$ egrep -o 'business_id": "\w+"' yelp_academic_dataset.json | sort -u | wc -l -9592 -#+end_src - -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: - -#+BEGIN_HTML - -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index c445e4c..e93acbc 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -111,6 +111,19 @@ width: 48%; float: right; } + + /* Tables */ + table { + border-collapse: collapse; + margin: 0px; + } + table, th, td { + border: 1px solid white; + } + th, td { + padding: 7px; + } + diff --git a/slides/2014-01-30-CaseStudies.html b/slides/2014-01-30-CaseStudies.html index 07d03c7..d3f11eb 100644 --- a/slides/2014-01-30-CaseStudies.html +++ b/slides/2014-01-30-CaseStudies.html @@ -111,6 +111,19 @@ width: 48%; float: right; } + + /* Tables */ + table { + border-collapse: collapse; + margin: 0px; + } + table, th, td { + border: 1px solid white; + } + th, td { + padding: 7px; + } + diff --git a/slides/2014-01-30-Lab.html b/slides/2014-01-30-Lab.html index 3e4c725..6efca4f 100644 --- a/slides/2014-01-30-Lab.html +++ b/slides/2014-01-30-Lab.html @@ -111,6 +111,19 @@ width: 48%; float: right; } + + /* Tables */ + table { + border-collapse: collapse; + margin: 0px; + } + table, th, td { + border: 1px solid white; + } + th, td { + padding: 7px; + } + diff --git a/slides/2014-01-30-Obtaining-Data.html b/slides/2014-01-30-Obtaining-Data.html new file mode 100644 index 0000000..bb3db19 --- /dev/null +++ b/slides/2014-01-30-Obtaining-Data.html @@ -0,0 +1,576 @@ + + + + Data Mining + + + + + + + + + diff --git a/slides/2014-01-30-Obtaining-Data.markdown b/slides/2014-01-30-Obtaining-Data.markdown new file mode 100644 index 0000000..9e89145 --- /dev/null +++ b/slides/2014-01-30-Obtaining-Data.markdown @@ -0,0 +1,435 @@ +name: inverse +layout: true +class: left, top, inverse + +--- + +# Obtaining Data + +--- + +## Ways to Collect + +.left-column[ + + + Operational Data + + Data Warehouse + + Unstructured Data + + External API + + Data Sets +] + +.right-column[ + +] + +??? + +image: http://woodwarddesign.ca/blog/2009/03/06/bottle-caps/ + +--- + +## Operational Data + + + Most frequent in industry + + Usually stored in databases best suited for transactional use + + Challenge is reorganizing data to suit question + +??? + +## Data from production + + Most frequently you'll have data that is being used by the application, + and you'll want to find insights in it + + We'll go into more detail in another class, but online use is + optimized for small queries and small updates + + Frequently just accessing the data in bulk is a software engineering + problem: + + ensuring long queries don't hold up production usage + + joining across databases via software + + understanding esoteric columns, like "flags" + + Often will want to reorganize data to look like transactional + +--- + +## Example + +Find the user names with most "liked" reviews on Yelp + +Users + +| user_id | name | flags | +|---------|------|-------| +| 25234 | Bob | 0x200 | + +Reviews + +| review_id | business_id | user_id | stars | text | flags | +|-----------|-------------|---------|-------|--------------|-------| +| 282 | 52432 | 25234 | 4 | great place! | 0x1 | + +Feedback + +| review_id | source_user_id | ufc_flags | flags | +|-----------|----------------|-----------|-------| +| 282 | 8205 | 0x1 | 0x0 | + +??? + +## Distributed Data + + + At Yelp we have a variety of database tables, and those tables can be + spread across different databases + + At a minimum we frequently need to ```JOIN``` across tables to answer queries + + e.g., matching up user names with reviews from separate tables + + It is possible the review table is only indexed on business_id, and so + finding all reviews by a user is really disk intensive: make sure you're + not slowing down the whole site! + + An additional challenge is when the "feedback" tables are in a separate + database: can no longer issue normal SQL queries + + What are these "flag" columns for? + + Exactly: no one knows. Often must look into code, or compare data to + production representation to guess meaning. In Yelp, ```0x1``` often means + "inactive", so we probably don't want to count that feedback + +--- + +## Data Warehouse + +.left-column[ + + + Data located on same system + + Organized for analytics queries + + Requires extra maintenance and understanding of construction +] +.right-column[ + +] +??? + +## No free lunch + + + A strong data warehouse can be a big improvement over operational data + + Hopefully, someone has already cleaned, joined data in a way that makes + sense! + + Optimized for long running queries: less fear of brining down website! + + But you must learn how that process was accomplished in order to understand + potential problems + + How to handle missing data? + + We'll go into more detail about how data warehouse schemas compare to + online ones later in the course + +--- + +## Unstructured + + + Haphazard collection of data + + Unclear what structure should be + + Examples: Web logs, text, multimedia + + Must extract structure eventually + +??? + +## Yelp JSON logs + + + When developing a web application, new context or details become + important: how long did certain requests take? What link did a user follow + to a website? + + Relational Databases aren't well suited for these wide varieties of + potential attributes that don't apply to all items + + So the current work around is just to write all useful information down in + a log, and extract what is needed later + + Text, like business reviews, another example: desired structure changes + radically between questions: How many words? Characters? What is the sentiment? + + Pictures can contain attributes like color depth, length, width + + First step of data mining is often imposing structure on data: the data is + not inherently unstructured, it just is unclear what the structure *should be* + until query time + +--- + +## Search Logs Example + +``` +193.139.1 jimmy [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288 +282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929 +345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232 +10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342 +10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342 +``` + +| user_name | date | query | +|-----------|----------------------------|----------| +| jimmy | 10/Oct/2013:13:55:36 -0700 | headache | +| shreyas | 10/Oct/2013:13:56:36 -0700 | bananas | +| steven | 10/Oct/2013:13:56:37 -0700 | cold | +| anne | 10/Oct/2013:13:56:39 -0700 | flu shot | +| lily | 10/Oct/2013:13:57:40 -0700 | i290 | + +??? + +## Imposing Structure + + + Extract only the rows we know follow a format + + Format queries from some encoding (e.g., URL) to standardized format + +--- + +## External APIs + + + Better documented than internal data! + + More limited in amount and detail + + Commonly HTTP/REST based + +??? + +## Motivation + + + Companies are often searching for other ways to leverage their data + + Both for immediate business purposes, and for brand recognition + + Twitter more (in)famous example + + NYTimes another good option + +--- + +## NYTimes API Example + + + [Article Search API](http://developer.nytimes.com/docs/read/article_search_api) + + http://api.nytimes.com/svc/search/v1/article?format=json&query=ballot&api-key=6578bab7f8c3808ce4c392edc9a793f0:8:5717915 + +??? + +## Accessing these + + More info on how to access these APIs is in the Web Architecture class, + but feel free to ask Shreyas or I about how best to access them + +--- + +## Data Sets + + + Download large, curated set of data all at once + + Formats vary, but usually documented + + Can be useful to combine with other datasets or APIs + + + +??? + +## Research + + + Data sets commonly used in research: can compare different techniques on + same data to understand advantages + + Sizes can range to a few MB to GB + + JSON, CSV, XML all potential formats. Cleaning, organization for your + question again becomes an important aspect + +--- + +## Data Set Examples + +.left-column[ + + + [MovieLens Data Sets](http://www.grouplens.org/node/73) + + [Kaggle Digit Recognizer](https://www.kaggle.com/c/digit-recognizer) + + [Hilary Mason's Data Sets](https://bitly.com/bundles/hmason/1) +] +.right-column[ + +] + +--- + +## Exploring Data + + + Data sets are frequently too large to fit in standard tools like Excel + or Word + + Simplest to explore on the command line + + Homework will be exploring a data set of your choice + +??? + +## Size + + + Some formats will not be easily parsed into Excel: e.g., JSON, XML + + Word will be slow, or unworkable for GB size data + + CLI provides many composable tools for text manipulation + +--- + +## Yelp Academic Dataset + + + [Yelp Dataset Challenge](http://www.yelp.com/dataset_challenge/) data covers reviews, users, + businesses, and check-ins + + To download, you'll need to sign up, but it's instant + + Use .edu email + +??? + +## Example + + + We'll use this as an example, you can use any data set of your choice + + Just for homework, don't need to use for project + +--- + +## CLI introduction + + + Standard commands available in [Learn CLI the hard way](http://cli.learncodethehardway.com) + + All example will be run on ```ischool.berkeley.edu``` + + Shreyas and I are available for more help + +??? + +## Help + + + If you're new, don't be intimidated. + + Security policies ensure you can't break anything besides your own files + + Keep backups of important stuff anyway + +--- + +## ```wget``` + + + Used for downloading files + + Downloading with the browser is fine, but sometimes nice to use a faster + connection, or download it directly to the machine you're working on +```bash +$ wget 'http://www.grouplens.org/system/files/ml-100k.zip' +``` + +??? + +## Command + + + Just ```wget "URL"``` + + I like to use quotes in case there are special characters in the URL, eg + ```?``` + + Will download to current directory with the same name as the remote file + +--- + +## ```scp``` + + + Copy a file to or from a remote machine + + Uses same connection as SSH, but copies data instead + + Example: Copy data you've downloaded in your browser + +```bash +$ scp ~/Downloads/ml-100k.zip jretz@ischool.berkeley.edu: + +# or + +$ scp ~/Downloads/ml-100k.zip \ +jretz@ischool.berkeley.edu:i290/movielens-100k.zip +``` + +??? + +## Command + + + Trailing ```:``` is important: signifies remote machine + + If you don't specify path or filename, will copy the file with the same + name into your home directory + +--- + +## ```gunzip``` / ```unzip``` + + + Uncompress data sets for simpler, faster manipulation + ``` + $ unzip ml-100k.zip + + # or + + $ gunzip dataset.json.gz + ``` + +??? + +## Commands + +### unzip + expand potentially many files, leave original alone + +### gunzip + expand original file, leaving only the uncompressed version + +--- + +## ```less``` + + + View a file + + History: original command was called ```more``` to see a file a page at a time + + "Less is more" + ```bash + less yelp_academic_dataset_user.json + ``` + +--- + +## Searching in ```less``` + + + ```/``` (forward slash) lets you input search text, `````` performs the search + + After finding the first occurrence, ```/``` will find the next occurrence + + ```?``` will find the previous occurrence + + ```q``` will quit +``` +/"name"": "Bob" +/"name"": "Cindy" +``` + +??? + +## Command + + + Useful for finding specific instances to investigate + +--- + +## ```grep``` + + + Find and print lines matching a "regular expression" + + [Regular expressions](http://www.regular-expressions.info/quickstart.html) are "find" on steroids, but you can use simple strings +```bash +$ grep '"name": "Cindy"' yelp_academic_dataset_user.json +``` + +--- + +## ```wc``` + + + "wordcount" counts characters, words, lines + + Most useful in data sets for lines: ```-l``` +```bash +$ wc -l yelp_academic_dataset_user.json +43873 yelp_academic_dataset_user.json +``` + +--- + +## Composability + + + Genius of Unix: do one thing well, compose commands to get what you want + + ```|``` pipe characters "sends" output of one program to the input of another + + How many people named Cindy in the dataset? + +```bash +$ grep '"name": "Cindy"' yelp_academic_dataset_user.json | wc -l +91 +``` + ++ What are the most common names in the dataset? + +```bash +$ egrep -o '"name": "([^"]*)"' yelp_academic_dataset_user.json | \ + sort -n | uniq -c | sort -nr | head +465 "name": "David" +447 "name": "John" +418 "name": "Michael" +417 "name": "Chris" +383 "name": "Mike" +365 "name": "Jennifer" +298 "name": "Brian" +267 "name": "Scott" +265 "name": "Jason" +261 "name": "Mark" +``` diff --git a/slides/presentation_template.html b/slides/presentation_template.html index eaae6ac..94b31b3 100644 --- a/slides/presentation_template.html +++ b/slides/presentation_template.html @@ -111,6 +111,19 @@ width: 48%; float: right; } + + /* Tables */ + table { + border-collapse: collapse; + margin: 0px; + } + table, th, td { + border: 1px solid white; + } + th, td { + padding: 7px; + } + From f991adcee3b2edd53483ad80499f27414a4b359e Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Tue, 10 Dec 2013 07:11:46 -0800 Subject: [PATCH 13/79] make slides functional offline by including remark js --- slides/2014-01-23-Intro.html | 2 +- slides/2014-01-30-CaseStudies.html | 2 +- slides/2014-01-30-Lab.html | 2 +- slides/2014-01-30-Obtaining-Data.html | 2 +- slides/presentation_template.html | 2 +- slides/production/remark-0.5.9.min.js | 5 +++++ 6 files changed, 10 insertions(+), 5 deletions(-) create mode 100644 slides/production/remark-0.5.9.min.js diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index e93acbc..b1c53ca 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -466,7 +466,7 @@ ## *Break* - ",returnEnd:true,subLanguage:"javascript"}},{begin:"<%",end:"%>",subLanguage:"vbscript"},{className:"tag",begin:"",relevance:0,contains:[{className:"title",begin:"[^ /><]+"},TAG_INTERNALS]}]}}},{name:"css",create:function(hljs){var IDENT_RE="[a-zA-Z-][a-zA-Z0-9_-]*";var FUNCTION={className:"function",begin:IDENT_RE+"\\(",end:"\\)",contains:["self",hljs.NUMBER_MODE,hljs.APOS_STRING_MODE,hljs.QUOTE_STRING_MODE]};return{case_insensitive:true,illegal:"[=/|']",contains:[hljs.C_BLOCK_COMMENT_MODE,{className:"id",begin:"\\#[A-Za-z0-9_-]+"},{className:"class",begin:"\\.[A-Za-z0-9_-]+",relevance:0},{className:"attr_selector",begin:"\\[",end:"\\]",illegal:"$"},{className:"pseudo",begin:":(:)?[a-zA-Z0-9\\_\\-\\+\\(\\)\\\"\\']+"},{className:"at_rule",begin:"@(font-face|page)",lexems:"[a-z-]+",keywords:"font-face page"},{className:"at_rule",begin:"@",end:"[{;]",contains:[{className:"keyword",begin:/\S+/},{begin:/\s/,endsWithParent:true,excludeEnd:true,relevance:0,contains:[FUNCTION,hljs.APOS_STRING_MODE,hljs.QUOTE_STRING_MODE,hljs.NUMBER_MODE]}]},{className:"tag",begin:IDENT_RE,relevance:0},{className:"rules",begin:"{",end:"}",illegal:"[^\\s]",relevance:0,contains:[hljs.C_BLOCK_COMMENT_MODE,{className:"rule",begin:"[^\\s]",returnBegin:true,end:";",endsWithParent:true,contains:[{className:"attribute",begin:"[A-Z\\_\\.\\-]+",end:":",excludeEnd:true,illegal:"[^\\s]",starts:{className:"value",endsWithParent:true,excludeEnd:true,contains:[FUNCTION,hljs.NUMBER_MODE,hljs.QUOTE_STRING_MODE,hljs.APOS_STRING_MODE,hljs.C_BLOCK_COMMENT_MODE,{className:"hexcolor",begin:"#[0-9A-Fa-f]+"},{className:"important",begin:"!important"}]}}]}]}]}}},{name:"scala",create:function(hljs){var ANNOTATION={className:"annotation",begin:"@[A-Za-z]+"};var STRING={className:"string",begin:'u?r?"""',end:'"""',relevance:10};return{keywords:"type yield lazy override def with val var false true sealed abstract private trait "+"object null if for while throw finally protected extends import final return else "+"break new catch super class case package default try this match continue throws",contains:[{className:"javadoc",begin:"/\\*\\*",end:"\\*/",contains:[{className:"javadoctag",begin:"@[A-Za-z]+"}],relevance:10},hljs.C_LINE_COMMENT_MODE,hljs.C_BLOCK_COMMENT_MODE,STRING,hljs.APOS_STRING_MODE,hljs.QUOTE_STRING_MODE,{className:"class",begin:"((case )?class |object |trait )",end:"({|$)",illegal:":",keywords:"case class trait object",contains:[{beginWithKeyword:true,keywords:"extends with",relevance:10},{className:"title",begin:hljs.UNDERSCORE_IDENT_RE},{className:"params",begin:"\\(",end:"\\)",contains:[hljs.APOS_STRING_MODE,hljs.QUOTE_STRING_MODE,STRING,ANNOTATION]}]},hljs.C_NUMBER_MODE,ANNOTATION]}}},{name:"coffeescript",create:function(hljs){var KEYWORDS={keyword:"in if for while finally new do return else break catch instanceof throw try this "+"switch continue typeof delete debugger super "+"then unless until loop of by when and or is isnt not",literal:"true false null undefined "+"yes no on off",reserved:"case default function var void with const let enum export import native "+"__hasProp __extends __slice __bind __indexOf",built_in:"npm require console print module exports global window document"};var JS_IDENT_RE="[A-Za-z$_][0-9A-Za-z$_]*";var TITLE={className:"title",begin:JS_IDENT_RE};var SUBST={className:"subst",begin:"#\\{",end:"}",keywords:KEYWORDS};var EXPRESSIONS=[hljs.BINARY_NUMBER_MODE,hljs.inherit(hljs.C_NUMBER_MODE,{starts:{end:"(\\s*/)?",relevance:0}}),{className:"string",begin:"'''",end:"'''",contains:[hljs.BACKSLASH_ESCAPE]},{className:"string",begin:"'",end:"'",contains:[hljs.BACKSLASH_ESCAPE],relevance:0},{className:"string",begin:'"""',end:'"""',contains:[hljs.BACKSLASH_ESCAPE,SUBST]},{className:"string",begin:'"',end:'"',contains:[hljs.BACKSLASH_ESCAPE,SUBST],relevance:0},{className:"regexp",begin:"///",end:"///",contains:[hljs.HASH_COMMENT_MODE]},{className:"regexp",begin:"//[gim]*",relevance:0},{className:"regexp",begin:"/\\S(\\\\.|[^\\n])*?/[gim]*(?=\\s|\\W|$)"},{className:"property",begin:"@"+JS_IDENT_RE},{begin:"`",end:"`",excludeBegin:true,excludeEnd:true,subLanguage:"javascript"}];SUBST.contains=EXPRESSIONS;return{keywords:KEYWORDS,contains:EXPRESSIONS.concat([{className:"comment",begin:"###",end:"###"},hljs.HASH_COMMENT_MODE,{className:"function",begin:"("+JS_IDENT_RE+"\\s*=\\s*)?(\\(.*\\))?\\s*[-=]>",end:"[-=]>",returnBegin:true,contains:[TITLE,{className:"params",begin:"\\(",returnBegin:true,contains:[{begin:/\(/,end:/\)/,keywords:KEYWORDS,contains:["self"].concat(EXPRESSIONS)}]}]},{className:"class",beginWithKeyword:true,keywords:"class",end:"$",illegal:"[:\\[\\]]",contains:[{beginWithKeyword:true,keywords:"extends",endsWithParent:true,illegal:":",contains:[TITLE]},TITLE]},{className:"attribute",begin:JS_IDENT_RE+":",end:":",returnBegin:true,excludeEnd:true}])}}},{name:"lisp",create:function(hljs){var LISP_IDENT_RE="[a-zA-Z_\\-\\+\\*\\/\\<\\=\\>\\&\\#][a-zA-Z0-9_\\-\\+\\*\\/\\<\\=\\>\\&\\#!]*";var LISP_SIMPLE_NUMBER_RE="(\\-|\\+)?\\d+(\\.\\d+|\\/\\d+)?((d|e|f|l|s)(\\+|\\-)?\\d+)?";var SHEBANG={className:"shebang",begin:"^#!",end:"$"};var LITERAL={className:"literal",begin:"\\b(t{1}|nil)\\b"};var NUMBERS=[{className:"number",begin:LISP_SIMPLE_NUMBER_RE},{className:"number",begin:"#b[0-1]+(/[0-1]+)?"},{className:"number",begin:"#o[0-7]+(/[0-7]+)?"},{className:"number",begin:"#x[0-9a-f]+(/[0-9a-f]+)?"},{className:"number",begin:"#c\\("+LISP_SIMPLE_NUMBER_RE+" +"+LISP_SIMPLE_NUMBER_RE,end:"\\)"}];var STRING={className:"string",begin:'"',end:'"',contains:[hljs.BACKSLASH_ESCAPE],relevance:0};var COMMENT={className:"comment",begin:";",end:"$"};var VARIABLE={className:"variable",begin:"\\*",end:"\\*"};var KEYWORD={className:"keyword",begin:"[:&]"+LISP_IDENT_RE};var QUOTED_LIST={begin:"\\(",end:"\\)",contains:["self",LITERAL,STRING].concat(NUMBERS)};var QUOTED1={className:"quoted",begin:"['`]\\(",end:"\\)",contains:NUMBERS.concat([STRING,VARIABLE,KEYWORD,QUOTED_LIST])};var QUOTED2={className:"quoted",begin:"\\(quote ",end:"\\)",keywords:{title:"quote"},contains:NUMBERS.concat([STRING,VARIABLE,KEYWORD,QUOTED_LIST])};var LIST={className:"list",begin:"\\(",end:"\\)"};var BODY={endsWithParent:true,relevance:0};LIST.contains=[{className:"title",begin:LISP_IDENT_RE},BODY];BODY.contains=[QUOTED1,QUOTED2,LIST,LITERAL].concat(NUMBERS).concat([STRING,COMMENT,VARIABLE,KEYWORD]);return{illegal:"[^\\s]",contains:NUMBERS.concat([SHEBANG,LITERAL,STRING,COMMENT,QUOTED1,QUOTED2,LIST])}}},{name:"clojure",create:function(hljs){var keywords={built_in:"def cond apply if-not if-let if not not= = < < > <= <= >= == + / * - rem "+"quot neg? pos? delay? symbol? keyword? true? false? integer? empty? coll? list? "+"set? ifn? fn? associative? sequential? sorted? counted? reversible? number? decimal? "+"class? distinct? isa? float? rational? reduced? ratio? odd? even? char? seq? vector? "+"string? map? nil? contains? zero? instance? not-every? not-any? libspec? -> ->> .. . "+"inc compare do dotimes mapcat take remove take-while drop letfn drop-last take-last "+"drop-while while intern condp case reduced cycle split-at split-with repeat replicate "+"iterate range merge zipmap declare line-seq sort comparator sort-by dorun doall nthnext "+"nthrest partition eval doseq await await-for let agent atom send send-off release-pending-sends "+"add-watch mapv filterv remove-watch agent-error restart-agent set-error-handler error-handler "+"set-error-mode! error-mode shutdown-agents quote var fn loop recur throw try monitor-enter "+"monitor-exit defmacro defn defn- macroexpand macroexpand-1 for doseq dosync dotimes and or "+"when when-not when-let comp juxt partial sequence memoize constantly complement identity assert "+"peek pop doto proxy defstruct first rest cons defprotocol cast coll deftype defrecord last butlast "+"sigs reify second ffirst fnext nfirst nnext defmulti defmethod meta with-meta ns in-ns create-ns import "+"intern refer keys select-keys vals key val rseq name namespace promise into transient persistent! conj! "+"assoc! dissoc! pop! disj! import use class type num float double short byte boolean bigint biginteger "+"bigdec print-method print-dup throw-if throw printf format load compile get-in update-in pr pr-on newline "+"flush read slurp read-line subvec with-open memfn time ns assert re-find re-groups rand-int rand mod locking "+"assert-valid-fdecl alias namespace resolve ref deref refset swap! reset! set-validator! compare-and-set! alter-meta! "+"reset-meta! commute get-validator alter ref-set ref-history-count ref-min-history ref-max-history ensure sync io! "+"new next conj set! memfn to-array future future-call into-array aset gen-class reduce merge map filter find empty "+"hash-map hash-set sorted-map sorted-map-by sorted-set sorted-set-by vec vector seq flatten reverse assoc dissoc list "+"disj get union difference intersection extend extend-type extend-protocol int nth delay count concat chunk chunk-buffer "+"chunk-append chunk-first chunk-rest max min dec unchecked-inc-int unchecked-inc unchecked-dec-inc unchecked-dec unchecked-negate "+"unchecked-add-int unchecked-add unchecked-subtract-int unchecked-subtract chunk-next chunk-cons chunked-seq? prn vary-meta "+"lazy-seq spread list* str find-keyword keyword symbol gensym force rationalize"};var CLJ_IDENT_RE="[a-zA-Z_0-9\\!\\.\\?\\-\\+\\*\\/\\<\\=\\>\\&\\#\\$';]+";var SIMPLE_NUMBER_RE="[\\s:\\(\\{]+\\d+(\\.\\d+)?";var NUMBER={className:"number",begin:SIMPLE_NUMBER_RE,relevance:0};var STRING={className:"string",begin:'"',end:'"',contains:[hljs.BACKSLASH_ESCAPE],relevance:0};var COMMENT={className:"comment",begin:";",end:"$",relevance:0};var COLLECTION={className:"collection",begin:"[\\[\\{]",end:"[\\]\\}]"};var HINT={className:"comment",begin:"\\^"+CLJ_IDENT_RE};var HINT_COL={className:"comment",begin:"\\^\\{",end:"\\}"};var KEY={className:"attribute",begin:"[:]"+CLJ_IDENT_RE};var LIST={className:"list",begin:"\\(",end:"\\)"};var BODY={endsWithParent:true,keywords:{literal:"true false nil"},relevance:0};var TITLE={keywords:keywords,lexems:CLJ_IDENT_RE,className:"title",begin:CLJ_IDENT_RE,starts:BODY};LIST.contains=[{className:"comment",begin:"comment"},TITLE];BODY.contains=[LIST,STRING,HINT,HINT_COL,COMMENT,KEY,COLLECTION,NUMBER];COLLECTION.contains=[LIST,STRING,HINT,COMMENT,KEY,COLLECTION,NUMBER];return{illegal:"\\S",contains:[COMMENT,LIST]}}},{name:"http",create:function(hljs){return{illegal:"\\S",contains:[{className:"status",begin:"^HTTP/[0-9\\.]+",end:"$",contains:[{className:"number",begin:"\\b\\d{3}\\b"}]},{className:"request",begin:"^[A-Z]+ (.*?) HTTP/[0-9\\.]+$",returnBegin:true,end:"$",contains:[{className:"string",begin:" ",end:" ",excludeBegin:true,excludeEnd:true}]},{className:"attribute",begin:"^\\w",end:": ",excludeEnd:true,illegal:"\\n|\\s|=",starts:{className:"string",end:"$"}},{begin:"\\n\\n",starts:{subLanguage:"",endsWithParent:true}}]}}}];for(var i=0;i.left{text-align:left;}.remark-slide>.center{text-align:center;}.remark-slide>.right{text-align:right;}.remark-slide>.top{vertical-align:top;}.remark-slide>.middle{vertical-align:middle;}.remark-slide>.bottom{vertical-align:bottom;}.remark-slide .remark-slide-content{background-position:center;background-repeat:no-repeat;display:table-cell;padding:1em 4em 1em 4em;}.remark-slide .remark-slide-content .left{display:block;text-align:left;}.remark-slide .remark-slide-content .center{display:block;text-align:center;}.remark-slide .remark-slide-content .right{display:block;text-align:right;}.remark-slide .remark-slide-number{bottom:12px;opacity:0.5;position:absolute;right:20px;}.remark-visible{display:block;}.remark-backdrop{position:absolute;top:0;bottom:0;left:0;right:0;display:none;opacity:0.95;background:#000;}.remark-help{bottom:0;top:0;right:0;left:0;display:none;position:absolute;z-index:1000;-webkit-transform-origin:top left;-moz-transform-origin:top left;transform-origin:top-left;}.remark-help .remark-help-content{color:white;font-family:Helvetica,arial,freesans,clean,sans-serif;font-size:12pt;position:absolute;top:10%;bottom:10%;left:10%;height:10%;}.remark-help .remark-help-content td{color:white;font-size:12pt;padding:10px;}.remark-help .remark-help-content td:first-child{padding-left:0;}.remark-help .remark-help-content .key{background:white;color:black;min-width:1em;display:inline-block;padding:3px 6px;text-align:center;border-radius:4px;}.remark-help .dismiss{top:85%;}.remark-container.remark-help-mode .remark-help{display:block;}.remark-container.remark-help-mode .remark-backdrop{display:block;}.remark-preview-area{bottom:2%;left:2%;display:none;opacity:0.5;position:absolute;height:47.25%;width:48%;}.remark-preview-area .remark-slide-container{display:block;}.remark-notes-area{background:#e7e8e2;bottom:0;display:none;left:52%;overflow:hidden;padding:1.5em;position:absolute;right:0;top:0;}.remark-toolbar{color:#979892;padding-bottom:1em;}.remark-toolbar .remark-toolbar-link{border:2px solid #d7d8d2;color:#979892;display:inline-block;padding:2px 2px;text-decoration:none;text-align:center;min-width:20px;}.remark-toolbar .remark-toolbar-link:hover{border-color:#979892;color:#676862;}.remark-container.remark-presenter-mode .remark-slides-area{top:2%;left:2%;height:47.25%;width:48%;}.remark-container.remark-presenter-mode .remark-preview-area{display:block;}.remark-container.remark-presenter-mode .remark-notes-area{display:block;}@media print{.remark-container{overflow:visible;} .remark-slide-container{display:block;}}@page {size:908px 681px;margin:0;}",containerLayout:'
    \n
    \n +\n -\n
    \n
    \n
    \n
    \n\n
    \n
    \n
    \n
    \n
    \n
    \n

    Help

    \n

    Keyboard shortcuts

    \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n
    \n ↑,\n ←,\n Pg Up,\n K\n Go to previous slide
    \n ↓,\n →,\n Pg Dn,\n Space,\n J\n Go to next slide
    \n Home\n Go to first slide
    \n End\n Go to last slide
    \n F\n Toggle fullscreen mode
    \n C\n Clone slideshow
    \n P\n Toggle presenter mode
    \n ?\n Toggle this help
    \n
    \n
    \n \n \n \n \n \n
    \n Esc\n Back to slideshow
    \n
    \n
    \n'}},{}],5:[function(require,module,exports){var process=module.exports={};process.nextTick=function(){var canSetImmediate=typeof window!=="undefined"&&window.setImmediate;var canPost=typeof window!=="undefined"&&window.postMessage&&window.addEventListener;if(canSetImmediate){return function(f){return window.setImmediate(f)}}if(canPost){var queue=[];window.addEventListener("message",function(ev){if(ev.source===window&&ev.data==="process-tick"){ev.stopPropagation();if(queue.length>0){var fn=queue.shift();fn()}}},true);return function nextTick(fn){queue.push(fn);window.postMessage("process-tick","*")}}return function nextTick(fn){setTimeout(fn,0)}}();process.title="browser";process.browser=true;process.env={};process.argv=[];process.binding=function(name){throw new Error("process.binding is not supported")};process.cwd=function(){return"/"};process.chdir=function(dir){throw new Error("process.chdir is not supported")}},{}],6:[function(require,module,exports){!function(process){if(!process.EventEmitter)process.EventEmitter=function(){};var EventEmitter=exports.EventEmitter=process.EventEmitter;var isArray=typeof Array.isArray==="function"?Array.isArray:function(xs){return Object.prototype.toString.call(xs)==="[object Array]"};function indexOf(xs,x){if(xs.indexOf)return xs.indexOf(x);for(var i=0;i0&&this._events[type].length>m){this._events[type].warned=true;console.error("(node) warning: possible EventEmitter memory "+"leak detected. %d listeners added. "+"Use emitter.setMaxListeners() to increase limit.",this._events[type].length);console.trace()}}this._events[type].push(listener)}else{this._events[type]=[this._events[type],listener]}return this};EventEmitter.prototype.on=EventEmitter.prototype.addListener;EventEmitter.prototype.once=function(type,listener){var self=this;self.on(type,function g(){self.removeListener(type,g);listener.apply(this,arguments)});return this};EventEmitter.prototype.removeListener=function(type,listener){if("function"!==typeof listener){throw new Error("removeListener only takes instances of Function")}if(!this._events||!this._events[type])return this;var list=this._events[type];if(isArray(list)){var i=indexOf(list,listener);if(i<0)return this;list.splice(i,1);if(list.length==0)delete this._events[type]}else if(this._events[type]===listener){delete this._events[type]}return this};EventEmitter.prototype.removeAllListeners=function(type){if(arguments.length===0){this._events={};return this}if(type&&this._events&&this._events[type])this._events[type]=null;return this};EventEmitter.prototype.listeners=function(type){if(!this._events)this._events={};if(!this._events[type])this._events[type]=[];if(!isArray(this._events[type])){this._events[type]=[this._events[type]]}return this._events[type]}}(require("__browserify_process"))},{__browserify_process:5}],2:[function(require,module,exports){var EventEmitter=require("events").EventEmitter,highlighter=require("./highlighter"),Slideshow=require("./models/slideshow"),SlideshowView=require("./views/slideshowView"),Controller=require("./controller");module.exports.highlighter=highlighter;module.exports.create=function(options){var events,slideshow,slideshowView,controller;options=applyDefaults(options);events=new EventEmitter;events.setMaxListeners(0);slideshow=new Slideshow(events,options);slideshowView=new SlideshowView(events,options.container,slideshow);controller=new Controller(events,slideshowView);return slideshow};function applyDefaults(options){var sourceElement;options=options||{};if(!options.hasOwnProperty("source")){sourceElement=document.getElementById("source");if(sourceElement){options.source=sourceElement.innerHTML;sourceElement.style.display="none"}}if(!(options.container instanceof window.HTMLElement)){options.container=document.body}return options}},{events:6,"./highlighter":3,"./models/slideshow":7,"./views/slideshowView":8,"./controller":9}],9:[function(require,module,exports){module.exports=Controller;function Controller(events,slideshowView){addApiEventListeners(events,slideshowView);addNavigationEventListeners(events,slideshowView);addKeyboardEventListeners(events);addMouseEventListeners(events);addTouchEventListeners(events)}function addApiEventListeners(events,slideshowView){events.on("pause",function(event){removeKeyboardEventListeners(events);removeMouseEventListeners(events);removeTouchEventListeners(events)});events.on("resume",function(event){addKeyboardEventListeners(events);addMouseEventListeners(events);addTouchEventListeners(events)})}function addNavigationEventListeners(events,slideshowView){if(slideshowView.isEmbedded()){events.emit("gotoSlide",1)}else{events.on("hashchange",navigateByHash);events.on("slideChanged",updateHash);navigateByHash()}events.on("message",navigateByMessage);function navigateByHash(){var slideNoOrName=(window.location.hash||"").substr(1);events.emit("gotoSlide",slideNoOrName)}function updateHash(slideNoOrName){window.location.hash="#"+slideNoOrName}function navigateByMessage(message){var cap;if((cap=/^gotoSlide:(\d+)$/.exec(message.data))!==null){events.emit("gotoSlide",parseInt(cap[1],10))}}}function removeKeyboardEventListeners(events){events.removeAllListeners("keydown");events.removeAllListeners("keypress")}function addKeyboardEventListeners(events){events.on("keydown",function(event){switch(event.keyCode){case 33:case 37:case 38:events.emit("gotoPreviousSlide");break;case 32:case 34:case 39:case 40:events.emit("gotoNextSlide");break;case 36:events.emit("gotoFirstSlide");break;case 35:events.emit("gotoLastSlide");break;case 27:events.emit("hideOverlay");break}});events.on("keypress",function(event){if(event.metaKey||event.ctrlKey){return}switch(String.fromCharCode(event.which)){case"j":events.emit("gotoNextSlide");break;case"k":events.emit("gotoPreviousSlide");break;case"c":events.emit("createClone");break;case"p":events.emit("togglePresenterMode");break;case"f":events.emit("toggleFullscreen");break;case"?":events.emit("toggleHelp");break}})}function removeMouseEventListeners(events){events.removeAllListeners("mousewheel")}function addMouseEventListeners(events){events.on("mousewheel",function(event){if(event.wheelDeltaY>0){events.emit("gotoPreviousSlide")}else if(event.wheelDeltaY<0){events.emit("gotoNextSlide")}})}function removeTouchEventListeners(events){events.removeAllListeners("touchstart");events.removeAllListeners("touchend");events.removeAllListeners("touchmove")}function addTouchEventListeners(events){var touch,startX,endX;var isTap=function(){return Math.abs(startX-endX)<10};var handleTap=function(){events.emit("tap",endX)};var handleSwipe=function(){if(startX>endX){events.emit("gotoNextSlide")}else{events.emit("gotoPreviousSlide")}};events.on("touchstart",function(event){touch=event.touches[0];startX=touch.clientX});events.on("touchend",function(event){if(event.target.nodeName.toUpperCase()==="A"){return}touch=event.changedTouches[0];endX=touch.clientX;if(isTap()){handleTap()}else{handleSwipe()}});events.on("touchmove",function(event){event.preventDefault()})}},{}],7:[function(require,module,exports){var Navigation=require("./slideshow/navigation"),Events=require("./slideshow/events"),utils=require("../utils"),Slide=require("./slide"),Parser=require("../parser");module.exports=Slideshow;function Slideshow(events,options){var self=this,slides=[];options=options||{};Events.call(self,events);Navigation.call(self,events);self.loadFromString=loadFromString;self.getSlides=getSlides;self.getSlideCount=getSlideCount;self.getSlideByName=getSlideByName;self.getRatio=getOrDefault("ratio","4:3");self.getHighlightStyle=getOrDefault("highlightStyle","default");self.getHighlightLanguage=getOrDefault("highlightLanguage","");loadFromString(options.source);function loadFromString(source){source=source||"";slides=createSlides(source);expandVariables(slides);events.emit("slidesChanged")}function getSlides(){return slides.map(function(slide){return slide})}function getSlideCount(){return slides.length}function getSlideByName(name){return slides.byName[name]}function getOrDefault(key,defaultValue){return function(){if(options[key]===undefined){return defaultValue}return options[key]}}}function createSlides(slideshowSource){var parser=new Parser,parsedSlides=parser.parse(slideshowSource),slides=[],byName={},layoutSlide;slides.byName={};parsedSlides.forEach(function(slide,i){var template,slideViewModel;if(slide.properties.continued==="true"&&i>0){template=slides[slides.length-1]}else if(byName[slide.properties.template]){template=byName[slide.properties.template]}else if(slide.properties.layout==="false"){layoutSlide=undefined}else if(layoutSlide&&slide.properties.layout!=="true"){template=layoutSlide}slideViewModel=new Slide(slides.length+1,slide,template);if(slide.properties.layout==="true"){layoutSlide=slideViewModel}if(slide.properties.name){byName[slide.properties.name]=slideViewModel}if(slide.properties.layout!=="true"){slides.push(slideViewModel);if(slide.properties.name){slides.byName[slide.properties.name]=slideViewModel}}});return slides}function expandVariables(slides){slides.forEach(function(slide){slide.expandVariables()})}},{"./slideshow/navigation":10,"./slideshow/events":11,"../utils":12,"./slide":13,"../parser":14}],8:[function(require,module,exports){var SlideView=require("./slideView"),Scaler=require("../scaler"),resources=require("../resources"),addClass=require("../utils").addClass,toggleClass=require("../utils").toggleClass,getPrefixedProperty=require("../utils").getPrefixedProperty;module.exports=SlideshowView;function SlideshowView(events,containerElement,slideshow){var self=this;self.events=events;self.slideshow=slideshow;self.scaler=new Scaler(events,slideshow);self.slideViews=[];self.configureContainerElement(containerElement);self.configureChildElements();self.updateDimensions();self.scaleElements();self.updateSlideViews();events.on("slidesChanged",function(){self.updateSlideViews()});events.on("hideSlide",function(slideIndex){self.hideSlide(slideIndex)});events.on("showSlide",function(slideIndex){self.showSlide(slideIndex)});events.on("togglePresenterMode",function(){toggleClass(self.containerElement,"remark-presenter-mode");self.scaleElements()});events.on("toggleHelp",function(){toggleClass(self.containerElement,"remark-help-mode")});handleFullscreen(self)}function handleFullscreen(self){var requestFullscreen=getPrefixedProperty(self.containerElement,"requestFullScreen"),cancelFullscreen=getPrefixedProperty(document,"cancelFullScreen");self.events.on("toggleFullscreen",function(){var fullscreenElement=getPrefixedProperty(document,"fullscreenElement")||getPrefixedProperty(document,"fullScreenElement");if(!fullscreenElement&&requestFullscreen){requestFullscreen.call(self.containerElement,Element.ALLOW_KEYBOARD_INPUT)}else if(cancelFullscreen){cancelFullscreen.call(document)}self.scaleElements()})}SlideshowView.prototype.isEmbedded=function(){return this.containerElement!==document.body};SlideshowView.prototype.configureContainerElement=function(element){var self=this;self.containerElement=element;addClass(element,"remark-container");if(element===document.body){addClass(document.getElementsByTagName("html")[0],"remark-container");forwardEvents(self.events,window,["hashchange","resize","keydown","keypress","mousewheel","message"]);forwardEvents(self.events,document,["touchstart","touchmove","touchend"])}else{element.style.position="absolute";element.tabIndex=-1;forwardEvents(self.events,window,["resize"]);forwardEvents(self.events,element,["keydown","keypress","mousewheel","touchstart","touchmove","touchend"])}self.events.on("tap",function(endX){if(endX0){self.showSlide(self.slideshow.getCurrentSlideNo()-1)}};SlideshowView.prototype.scaleSlideBackgroundImages=function(dimensions){var self=this;self.slideViews.forEach(function(slideView){slideView.scaleBackgroundImage(dimensions)})};SlideshowView.prototype.showSlide=function(slideIndex){var self=this,slideView=self.slideViews[slideIndex],nextSlideView=self.slideViews[slideIndex+1];self.events.emit("beforeShowSlide",slideIndex);slideView.show();self.notesElement.innerHTML=slideView.notesMarkup;if(nextSlideView){self.previewArea.innerHTML=nextSlideView.containerElement.outerHTML}else{self.previewArea.innerHTML=""}self.events.emit("afterShowSlide",slideIndex)};SlideshowView.prototype.hideSlide=function(slideIndex){var self=this,slideView=self.slideViews[slideIndex];self.events.emit("beforeHideSlide",slideIndex);slideView.hide();self.events.emit("afterHideSlide",slideIndex)};SlideshowView.prototype.updateDimensions=function(){var self=this,dimensions=self.scaler.dimensions;self.helpElement.style.width=dimensions.width+"px";self.helpElement.style.height=dimensions.height+"px";self.scaleSlideBackgroundImages(dimensions);self.scaleElements()};SlideshowView.prototype.scaleElements=function(){var self=this;self.slideViews.forEach(function(slideView){slideView.scale(self.elementArea)});if(self.previewArea.children.length){self.scaler.scaleToFit(self.previewArea.children[0].children[0],self.previewArea)}self.scaler.scaleToFit(self.helpElement,self.containerElement)}},{"./slideView":15,"../scaler":16,"../resources":4,"../utils":12}],10:[function(require,module,exports){module.exports=Navigation;function Navigation(events){var self=this,currentSlideNo=0;self.getCurrentSlideNo=getCurrentSlideNo;self.gotoSlide=gotoSlide;self.gotoPreviousSlide=gotoPreviousSlide;self.gotoNextSlide=gotoNextSlide;self.gotoFirstSlide=gotoFirstSlide;self.gotoLastSlide=gotoLastSlide;self.pause=pause;self.resume=resume;events.on("gotoSlide",gotoSlide);events.on("gotoPreviousSlide",gotoPreviousSlide);events.on("gotoNextSlide",gotoNextSlide);events.on("gotoFirstSlide",gotoFirstSlide);events.on("gotoLastSlide",gotoLastSlide);events.on("slidesChanged",function(){if(currentSlideNo>self.getSlideCount()){currentSlideNo=self.getSlideCount()}});events.on("createClone",function(){if(!self.clone||self.clone.closed){self.clone=window.open(location.href,"_blank","location=no")}else{self.clone.focus()}});function pause(){events.emit("pause")}function resume(){events.emit("resume")}function getCurrentSlideNo(){return currentSlideNo}function gotoSlide(slideNoOrName){var slideNo=getSlideNo(slideNoOrName),alreadyOnSlide=slideNo===currentSlideNo,slideOutOfRange=slideNo<1||slideNo>self.getSlideCount();if(alreadyOnSlide||slideOutOfRange){return}if(currentSlideNo!==0){events.emit("hideSlide",currentSlideNo-1)}events.emit("showSlide",slideNo-1);currentSlideNo=slideNo;events.emit("slideChanged",slideNoOrName||slideNo);if(self.clone&&!self.clone.closed){self.clone.postMessage("gotoSlide:"+currentSlideNo,"*")}if(window.opener){window.opener.postMessage("gotoSlide:"+currentSlideNo,"*")}}function gotoPreviousSlide(){self.gotoSlide(currentSlideNo-1)}function gotoNextSlide(){self.gotoSlide(currentSlideNo+1)}function gotoFirstSlide(){self.gotoSlide(1)}function gotoLastSlide(){self.gotoSlide(self.getSlideCount())}function getSlideNo(slideNoOrName){var slideNo,slide;if(typeof slideNoOrName==="number"){return slideNoOrName}slideNo=parseInt(slideNoOrName,10);if(slideNo.toString()===slideNoOrName){return slideNo}slide=self.getSlideByName(slideNoOrName);if(slide){return slide.getSlideNo()}return 1}}},{}],11:[function(require,module,exports){var EventEmitter=require("events").EventEmitter;module.exports=Events;function Events(events){var self=this,externalEvents=new EventEmitter;externalEvents.setMaxListeners(0);self.on=function(){externalEvents.on.apply(externalEvents,arguments);return self};["showSlide","hideSlide","beforeShowSlide","afterShowSlide","beforeHideSlide","afterHideSlide"].map(function(eventName){events.on(eventName,function(slideIndex){var slide=self.getSlides()[slideIndex];externalEvents.emit(eventName,slide)})})}},{events:6}],12:[function(require,module,exports){exports.addClass=function(element,className){element.className=exports.getClasses(element).concat([className]).join(" ")};exports.removeClass=function(element,className){element.className=exports.getClasses(element).filter(function(klass){return klass!==className}).join(" ")};exports.toggleClass=function(element,className){var classes=exports.getClasses(element),index=classes.indexOf(className);if(index!==-1){classes.splice(index,1)}else{classes.push(className)}element.className=classes.join(" ")};exports.getClasses=function(element){return element.className.split(" ").filter(function(s){return s!==""})};exports.getPrefixedProperty=function(element,propertyName){var capitalizedPropertName=propertyName[0].toUpperCase()+propertyName.slice(1);return element[propertyName]||element["moz"+capitalizedPropertName]||element["webkit"+capitalizedPropertName]};forEach([Array,window.NodeList,window.HTMLCollection],extend);function extend(object){var prototype=object&&object.prototype;if(!prototype){return}prototype.forEach=prototype.forEach||function(f){forEach(this,f)};prototype.filter=prototype.filter||function(f){var result=[];this.forEach(function(element){if(f(element,result.length)){result.push(element)}});return result};prototype.map=prototype.map||function(f){var result=[];this.forEach(function(element){result.push(f(element,result.length))});return result}}function forEach(list,f){var i;for(i=0;icontainerHeight/ratio.height){scale=containerHeight/dimensions.height}else{scale=containerWidth/dimensions.width}scaledWidth=dimensions.width*scale;scaledHeight=dimensions.height*scale;left=(containerWidth-scaledWidth)/2;top=(containerHeight-scaledHeight)/2;element.style["-webkit-transform"]="scale("+scale+")";element.style.MozTransform="scale("+scale+")";element.style.left=Math.max(left,0)+"px";element.style.top=Math.max(top,0)+"px"};function getRatio(slideshow){var ratioComponents=slideshow.getRatio().split(":"),ratio;ratio={width:parseInt(ratioComponents[0],10),height:parseInt(ratioComponents[1],10)};ratio.ratio=ratio.width/ratio.height;return ratio}function getDimensions(ratio){return{width:Math.floor(referenceWidth/referenceRatio*ratio.ratio),height:referenceHeight}}},{}],14:[function(require,module,exports){var Lexer=require("./lexer"),converter=require("./converter");module.exports=Parser;function Parser(){}Parser.prototype.parse=function(src){var lexer=new Lexer,tokens=lexer.lex(src),slides=[],slide=createSlide(),tag,classes;tokens.forEach(function(token){switch(token.type){case"text":case"code":case"fences":appendTo(slide,token.text);break;case"content_start":tag=token.block?"div":"span";classes=token.classes.join(" ");appendTo(slide,"<"+tag+' class="'+classes+'">');break;case"content_end":tag=token.block?"div":"span";appendTo(slide,"</"+tag+">");break;case"separator":slides.push(slide);slide=createSlide();slide.properties.continued=(token.text==="--").toString();break;case"notes_separator":slide.notes="";break}});slides.push(slide);slides.forEach(function(slide){slide.source=extractProperties(slide.source,slide.properties)});return slides};function createSlide(){return{source:"",properties:{continued:"false"}}}function appendTo(slide,content){if(slide.notes!==undefined){slide.notes+=content}else{slide.source+=content}}function extractProperties(source,properties){var propertyFinder=/^\n*([-\w]+):([^$\n]*)/i,match;while((match=propertyFinder.exec(source))!==null){source=source.substr(0,match.index)+source.substr(match.index+match[0].length);properties[match[1].trim()]=match[2].trim();propertyFinder.lastIndex=match.index}return source}},{"./lexer":17,"./converter":18}],15:[function(require,module,exports){var converter=require("../converter"),highlighter=require("../highlighter"),utils=require("../utils");module.exports=SlideView;function SlideView(events,slideshow,scaler,slide){var self=this;self.events=events;self.slideshow=slideshow;self.scaler=scaler;self.slide=slide;self.notesMarkup=createNotesMarkup(slideshow,slide.notes);self.configureElements();self.updateDimensions();self.events.on("propertiesChanged",function(changes){if(changes.hasOwnProperty("ratio")){self.updateDimensions() +}})}SlideView.prototype.updateDimensions=function(){var self=this,dimensions=self.scaler.dimensions;self.scalingElement.style.width=dimensions.width+"px";self.scalingElement.style.height=dimensions.height+"px"};SlideView.prototype.scale=function(containerElement){var self=this;self.scaler.scaleToFit(self.scalingElement,containerElement)};SlideView.prototype.show=function(){utils.addClass(this.containerElement,"remark-visible")};SlideView.prototype.hide=function(){utils.removeClass(this.containerElement,"remark-visible")};SlideView.prototype.configureElements=function(){var self=this;self.containerElement=document.createElement("div");self.containerElement.className="remark-slide-container";self.scalingElement=document.createElement("div");self.scalingElement.className="remark-slide-scaler";self.element=document.createElement("div");self.element.className="remark-slide";self.contentElement=createContentElement(self.events,self.slideshow,self.slide);self.numberElement=document.createElement("div");self.numberElement.className="remark-slide-number";self.numberElement.innerHTML=self.slide.number+" / "+self.slideshow.getSlides().length;self.contentElement.appendChild(self.numberElement);self.element.appendChild(self.contentElement);self.scalingElement.appendChild(self.element);self.containerElement.appendChild(self.scalingElement)};SlideView.prototype.scaleBackgroundImage=function(dimensions){var self=this,styles=window.getComputedStyle(this.contentElement),backgroundImage=styles.backgroundImage,match,image;if((match=/^url\(("?)([^\)]+?)\1\)/.exec(backgroundImage))!==null){image=new Image;image.onload=function(){if(image.width>dimensions.width||image.height>dimensions.height){if(!self.originalBackgroundSize){self.originalBackgroundSize=self.contentElement.style.backgroundSize;self.backgroundSizeSet=true;self.contentElement.style.backgroundSize="contain"}}else{if(self.backgroundSizeSet){self.contentElement.style.backgroundSize=self.originalBackgroundSize;self.backgroundSizeSet=false}}};image.src=match[2]}};function createContentElement(events,slideshow,slide){var element=document.createElement("div");if(slide.properties.name){element.id="slide-"+slide.properties.name}styleContentElement(slideshow,element,slide.properties);element.innerHTML=converter.convertMarkdown(slide.source);element.innerHTML=element.innerHTML.replace(/

    \s*<\/p>/g,"");highlightCodeBlocks(element,slideshow);return element}function styleContentElement(slideshow,element,properties){element.className="";setClassFromProperties(element,properties);setHighlightStyleFromProperties(element,properties,slideshow);setBackgroundFromProperties(element,properties)}function createNotesMarkup(slideshow,notes){var element=document.createElement("div");element.innerHTML=converter.convertMarkdown(notes);element.innerHTML=element.innerHTML.replace(/

    \s*<\/p>/g,"");highlightCodeBlocks(element,slideshow);return element.innerHTML}function setBackgroundFromProperties(element,properties){var backgroundImage=properties["background-image"];if(backgroundImage){element.style.backgroundImage=backgroundImage}}function setHighlightStyleFromProperties(element,properties,slideshow){var highlightStyle=properties["highlight-style"]||slideshow.getHighlightStyle();if(highlightStyle){utils.addClass(element,"hljs-"+highlightStyle)}}function setClassFromProperties(element,properties){utils.addClass(element,"remark-slide-content");(properties["class"]||"").split(/,| /).filter(function(s){return s!==""}).forEach(function(c){utils.addClass(element,c)})}function highlightCodeBlocks(content,slideshow){var codeBlocks=content.getElementsByTagName("code");codeBlocks.forEach(function(block){if(block.className===""){block.className=slideshow.getHighlightLanguage()}if(block.className!==""){highlighter.engine.highlightBlock(block," ")}utils.addClass(block,"remark-code")})}},{"../converter":18,"../highlighter":3,"../utils":12}],17:[function(require,module,exports){module.exports=Lexer;var CODE=1,CONTENT=2,FENCES=3,SEPARATOR=4,NOTES_SEPARATOR=5;var regexByName={CODE:/(?:^|\n)( {4}[^\n]+\n*)+/,CONTENT:/(?:\\)?((?:\.[a-zA-Z_\-][a-zA-Z\-_0-9]*)+)\[/,FENCES:/(?:^|\n) *(`{3,}|~{3,}) *(?:\S+)? *\n(?:[\s\S]+?)\s*\3 *(?:\n+|$)/,SEPARATOR:/(?:^|\n)(---?)(?:\n|$)/,NOTES_SEPARATOR:/(?:^|\n)(\?{3})(?:\n|$)/};var block=replace(/CODE|CONTENT|FENCES|SEPARATOR|NOTES_SEPARATOR/,regexByName),inline=replace(/CODE|CONTENT|FENCES/,regexByName);function Lexer(){}Lexer.prototype.lex=function(src){var tokens=lex(src,block),i;for(i=tokens.length-2;i>=0;i--){if(tokens[i].type==="text"&&tokens[i+1].type==="text"){tokens[i].text+=tokens[i+1].text;tokens.splice(i+1,1)}}return tokens};function lex(src,regex,tokens){var cap,text;tokens=tokens||[];while((cap=regex.exec(src))!==null){if(cap.index>0){tokens.push({type:"text",text:src.substring(0,cap.index)})}if(cap[CODE]){tokens.push({type:"code",text:cap[0]})}else if(cap[FENCES]){tokens.push({type:"fences",text:cap[0]})}else if(cap[SEPARATOR]){tokens.push({type:"separator",text:cap[SEPARATOR]})}else if(cap[NOTES_SEPARATOR]){tokens.push({type:"notes_separator",text:cap[NOTES_SEPARATOR]})}else if(cap[CONTENT]){text=getTextInBrackets(src,cap.index+cap[0].length);if(text!==undefined){src=src.substring(text.length+1);tokens.push({type:"content_start",classes:cap[CONTENT].substring(1).split("."),block:text.indexOf("\n")!==-1});lex(text,inline,tokens);tokens.push({type:"content_end",block:text.indexOf("\n")!==-1})}else{tokens.push({type:"text",text:cap[0]})}}src=src.substring(cap.index+cap[0].length)}if(src||!src&&tokens.length===0){tokens.push({type:"text",text:src})}return tokens}function replace(regex,replacements){return new RegExp(regex.source.replace(/\w{2,}/g,function(key){return replacements[key].source}))}function getTextInBrackets(src,offset){var depth=1,pos=offset,chr;while(depth>0&&pos");source=marked(source.replace(/^\s+/,""));source=source.replace(/&[l|g]t;/g,function(match){return match==="<"?"<":">"});source=source.replace(/&/g,"&");source=source.replace(/"/g,'"');return source}},{marked:19}],19:[function(require,module,exports){!function(global){!function(){var block={newline:/^\n+/,code:/^( {4}[^\n]+\n*)+/,fences:noop,hr:/^( *[-*_]){3,} *(?:\n+|$)/,heading:/^ *(#{1,6}) *([^\n]+?) *#* *(?:\n+|$)/,nptable:noop,lheading:/^([^\n]+)\n *(=|-){3,} *\n*/,blockquote:/^( *>[^\n]+(\n[^\n]+)*\n*)+/,list:/^( *)(bull) [\s\S]+?(?:hr|\n{2,}(?! )(?!\1bull )\n*|\s*$)/,html:/^ *(?:comment|closed|closing) *(?:\n{2,}|\s*$)/,def:/^ *\[([^\]]+)\]: *]+)>?(?: +["(]([^\n]+)[")])? *(?:\n+|$)/,table:noop,paragraph:/^((?:[^\n]+\n?(?!hr|heading|lheading|blockquote|tag|def))+)\n*/,text:/^[^\n]+/};block.bullet=/(?:[*+-]|\d+\.)/;block.item=/^( *)(bull) [^\n]*(?:\n(?!\1bull )[^\n]*)*/;block.item=replace(block.item,"gm")(/bull/g,block.bullet)();block.list=replace(block.list)(/bull/g,block.bullet)("hr",/\n+(?=(?: *[-*_]){3,} *(?:\n+|$))/)();block._tag="(?!(?:"+"a|em|strong|small|s|cite|q|dfn|abbr|data|time|code"+"|var|samp|kbd|sub|sup|i|b|u|mark|ruby|rt|rp|bdi|bdo"+"|span|br|wbr|ins|del|img)\\b)\\w+(?!:/|@)\\b";block.html=replace(block.html)("comment",//)("closed",/<(tag)[\s\S]+?<\/\1>/)("closing",/])*?>/)(/tag/g,block._tag)();block.paragraph=replace(block.paragraph)("hr",block.hr)("heading",block.heading)("lheading",block.lheading)("blockquote",block.blockquote)("tag","<"+block._tag)("def",block.def)();block.normal=merge({},block);block.gfm=merge({},block.normal,{fences:/^ *(`{3,}|~{3,}) *(\w+)? *\n([\s\S]+?)\s*\1 *(?:\n+|$)/,paragraph:/^/});block.gfm.paragraph=replace(block.paragraph)("(?!","(?!"+block.gfm.fences.source.replace("\\1","\\2")+"|")();block.tables=merge({},block.gfm,{nptable:/^ *(\S.*\|.*)\n *([-:]+ *\|[-| :]*)\n((?:.*\|.*(?:\n|$))*)\n*/,table:/^ *\|(.+)\n *\|( *[-:]+[-| :]*)\n((?: *\|.*(?:\n|$))*)\n*/});function Lexer(options){this.tokens=[];this.tokens.links={};this.options=options||marked.defaults;this.rules=block.normal;if(this.options.gfm){if(this.options.tables){this.rules=block.tables}else{this.rules=block.gfm}}}Lexer.rules=block;Lexer.lex=function(src,options){var lexer=new Lexer(options);return lexer.lex(src)};Lexer.prototype.lex=function(src){src=src.replace(/\r\n|\r/g,"\n").replace(/\t/g," ").replace(/\u00a0/g," ").replace(/\u2424/g,"\n");return this.token(src,true)};Lexer.prototype.token=function(src,top){var src=src.replace(/^ +$/gm,""),next,loose,cap,bull,b,item,space,i,l;while(src){if(cap=this.rules.newline.exec(src)){src=src.substring(cap[0].length);if(cap[0].length>1){this.tokens.push({type:"space"})}}if(cap=this.rules.code.exec(src)){src=src.substring(cap[0].length);cap=cap[0].replace(/^ {4}/gm,"");this.tokens.push({type:"code",text:!this.options.pedantic?cap.replace(/\n+$/,""):cap});continue}if(cap=this.rules.fences.exec(src)){src=src.substring(cap[0].length);this.tokens.push({type:"code",lang:cap[2],text:cap[3]});continue}if(cap=this.rules.heading.exec(src)){src=src.substring(cap[0].length);this.tokens.push({type:"heading",depth:cap[1].length,text:cap[2]});continue}if(top&&(cap=this.rules.nptable.exec(src))){src=src.substring(cap[0].length);item={type:"table",header:cap[1].replace(/^ *| *\| *$/g,"").split(/ *\| */),align:cap[2].replace(/^ *|\| *$/g,"").split(/ *\| */),cells:cap[3].replace(/\n$/,"").split("\n")};for(i=0;i ?/gm,"");this.token(cap,top);this.tokens.push({type:"blockquote_end"});continue}if(cap=this.rules.list.exec(src)){src=src.substring(cap[0].length);this.tokens.push({type:"list_start",ordered:isFinite(cap[2])});cap=cap[0].match(this.rules.item);if(this.options.smartLists){bull=block.bullet.exec(cap[0])[0]}next=false;l=cap.length;i=0;for(;i])/,autolink:/^<([^ >]+(@|:\/)[^ >]+)>/,url:noop,tag:/^|^<\/?\w+(?:"[^"]*"|'[^']*'|[^'">])*?>/,link:/^!?\[(inside)\]\(href\)/,reflink:/^!?\[(inside)\]\s*\[([^\]]*)\]/,nolink:/^!?\[((?:\[[^\]]*\]|[^\[\]])*)\]/,strong:/^__([\s\S]+?)__(?!_)|^\*\*([\s\S]+?)\*\*(?!\*)/,em:/^\b_((?:__|[\s\S])+?)_\b|^\*((?:\*\*|[\s\S])+?)\*(?!\*)/,code:/^(`+)\s*([\s\S]*?[^`])\s*\1(?!`)/,br:/^ {2,}\n(?!\s*$)/,del:noop,text:/^[\s\S]+?(?=[\\?(?:\s+['"]([\s\S]*?)['"])?\s*/;inline.link=replace(inline.link)("inside",inline._inside)("href",inline._href)();inline.reflink=replace(inline.reflink)("inside",inline._inside)();inline.normal=merge({},inline);inline.pedantic=merge({},inline.normal,{strong:/^__(?=\S)([\s\S]*?\S)__(?!_)|^\*\*(?=\S)([\s\S]*?\S)\*\*(?!\*)/,em:/^_(?=\S)([\s\S]*?\S)_(?!_)|^\*(?=\S)([\s\S]*?\S)\*(?!\*)/});inline.gfm=merge({},inline.normal,{escape:replace(inline.escape)("])","~|])")(),url:/^(https?:\/\/[^\s<]+[^<.,:;"')\]\s])/,del:/^~~(?=\S)([\s\S]*?\S)~~/,text:replace(inline.text)("]|","~]|")("|","|https?://|")()});inline.breaks=merge({},inline.gfm,{br:replace(inline.br)("{2,}","*")(),text:replace(inline.gfm.text)("{2,}","*")()});function InlineLexer(links,options){this.options=options||marked.defaults;this.links=links;this.rules=inline.normal;if(!this.links){throw new Error("Tokens array requires a `links` property.")}if(this.options.gfm){if(this.options.breaks){this.rules=inline.breaks}else{this.rules=inline.gfm}}else if(this.options.pedantic){this.rules=inline.pedantic}}InlineLexer.rules=inline;InlineLexer.output=function(src,links,options){var inline=new InlineLexer(links,options);return inline.output(src)};InlineLexer.prototype.output=function(src){var out="",link,text,href,cap;while(src){if(cap=this.rules.escape.exec(src)){src=src.substring(cap[0].length);out+=cap[1];continue}if(cap=this.rules.autolink.exec(src)){src=src.substring(cap[0].length);if(cap[2]==="@"){text=cap[1][6]===":"?this.mangle(cap[1].substring(7)):this.mangle(cap[1]);href=this.mangle("mailto:")+text}else{text=escape(cap[1]);href=text}out+=''+text+"";continue}if(cap=this.rules.url.exec(src)){src=src.substring(cap[0].length);text=escape(cap[1]);href=text;out+=''+text+"";continue}if(cap=this.rules.tag.exec(src)){src=src.substring(cap[0].length);out+=this.options.sanitize?escape(cap[0]):cap[0];continue}if(cap=this.rules.link.exec(src)){src=src.substring(cap[0].length);out+=this.outputLink(cap,{href:cap[2],title:cap[3]});continue}if((cap=this.rules.reflink.exec(src))||(cap=this.rules.nolink.exec(src))){src=src.substring(cap[0].length);link=(cap[2]||cap[1]).replace(/\s+/g," ");link=this.links[link.toLowerCase()];if(!link||!link.href){out+=cap[0][0];src=cap[0].substring(1)+src;continue}out+=this.outputLink(cap,link);continue}if(cap=this.rules.strong.exec(src)){src=src.substring(cap[0].length);out+=""+this.output(cap[2]||cap[1])+"";continue}if(cap=this.rules.em.exec(src)){src=src.substring(cap[0].length);out+=""+this.output(cap[2]||cap[1])+"";continue}if(cap=this.rules.code.exec(src)){src=src.substring(cap[0].length);out+=""+escape(cap[2],true)+"";continue}if(cap=this.rules.br.exec(src)){src=src.substring(cap[0].length);out+="
    ";continue}if(cap=this.rules.del.exec(src)){src=src.substring(cap[0].length);out+=""+this.output(cap[1])+"";continue}if(cap=this.rules.text.exec(src)){src=src.substring(cap[0].length);out+=escape(cap[0]);continue}if(src){throw new Error("Infinite loop on byte: "+src.charCodeAt(0))}}return out};InlineLexer.prototype.outputLink=function(cap,link){if(cap[0][0]!=="!"){return'"+this.output(cap[1])+""}else{return''+escape(cap[1])+'"}};InlineLexer.prototype.mangle=function(text){var out="",l=text.length,i=0,ch;for(;i.5){ch="x"+ch.toString(16)}out+="&#"+ch+";"}return out};function Parser(options){this.tokens=[];this.token=null;this.options=options||marked.defaults}Parser.parse=function(src,options){var parser=new Parser(options);return parser.parse(src)};Parser.prototype.parse=function(src){this.inline=new InlineLexer(src.links,this.options);this.tokens=src.reverse();var out="";while(this.next()){out+=this.tok()}return out};Parser.prototype.next=function(){return this.token=this.tokens.pop()};Parser.prototype.peek=function(){return this.tokens[this.tokens.length-1]||0};Parser.prototype.parseText=function(){var body=this.token.text;while(this.peek().type==="text"){body+="\n"+this.next().text}return this.inline.output(body)};Parser.prototype.tok=function(){switch(this.token.type){case"space":{return""}case"hr":{return"

    \n"}case"heading":{return""+this.inline.output(this.token.text)+"\n"}case"code":{if(this.options.highlight){var code=this.options.highlight(this.token.text,this.token.lang);if(code!=null&&code!==this.token.text){this.token.escaped=true;this.token.text=code}}if(!this.token.escaped){this.token.text=escape(this.token.text,true)}return"
    "+this.token.text+"
    \n"}case"table":{var body="",heading,i,row,cell,j;body+="\n\n";for(i=0;i'+heading+"\n":""+heading+"\n"}body+="\n\n";body+="\n";for(i=0;i\n";for(j=0;j'+cell+"\n":""+cell+"\n"}body+="\n"}body+="\n";return"\n"+body+"
    \n"}case"blockquote_start":{var body="";while(this.next().type!=="blockquote_end"){body+=this.tok()}return"
    \n"+body+"
    \n"}case"list_start":{var type=this.token.ordered?"ol":"ul",body="";while(this.next().type!=="list_end"){body+=this.tok()}return"<"+type+">\n"+body+"\n"}case"list_item_start":{var body="";while(this.next().type!=="list_item_end"){body+=this.token.type==="text"?this.parseText():this.tok()}return"
  • "+body+"
    • \n"}case"loose_item_start":{var body="";while(this.next().type!=="list_item_end"){body+=this.tok()}return"
  • "+body+"
    • \n"}case"html":{return!this.token.pre&&!this.options.pedantic?this.inline.output(this.token.text):this.token.text}case"paragraph":{return"

    "+this.inline.output(this.token.text)+"

    \n"}case"text":{return"

    "+this.parseText()+"

    \n"}}};function escape(html,encode){return html.replace(!encode?/&(?!#?\w+;)/g:/&/g,"&").replace(//g,">").replace(/"/g,""").replace(/'/g,"'")}function replace(regex,opt){regex=regex.source;opt=opt||"";return function self(name,val){if(!name)return new RegExp(regex,opt);val=val.source||val;val=val.replace(/(^|[^\[])\^/g,"$1");regex=regex.replace(name,val);return self}}function noop(){}noop.exec=noop;function merge(obj){var i=1,target,key;for(;iAn error occured:

    "+escape(e.message+"",true)+"
    "}throw e}}marked.options=marked.setOptions=function(opt){merge(marked.defaults,opt);return marked};marked.defaults={gfm:true,tables:true,breaks:false,pedantic:false,sanitize:false,smartLists:false,silent:false,highlight:null,langPrefix:"lang-"};marked.Parser=Parser;marked.parser=Parser.parse;marked.Lexer=Lexer;marked.lexer=Lexer.lex;marked.InlineLexer=InlineLexer;marked.inlineLexer=InlineLexer.output;marked.parse=marked;if(typeof exports==="object"){module.exports=marked}else if(typeof define==="function"&&define.amd){define(function(){return marked})}else{this.marked=marked}}.call(function(){return this||(typeof window!=="undefined"?window:global)}())}(window)},{}]},{},[1]); + From b6895a2bc82df32a9231e1418305a7b9ec91c21e Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Tue, 10 Dec 2013 07:43:12 -0800 Subject: [PATCH 14/79] converted 3rd lecture lab --- slides/2013-02-08-Lab.html | 288 --------------------------------- slides/2013-02-08-Lab.org | 78 --------- slides/2014-02-06-Lab.html | 245 ++++++++++++++++++++++++++++ slides/2014-02-06-Lab.markdown | 104 ++++++++++++ 4 files changed, 349 insertions(+), 366 deletions(-) delete mode 100644 slides/2013-02-08-Lab.html delete mode 100644 slides/2013-02-08-Lab.org create mode 100644 slides/2014-02-06-Lab.html create mode 100644 slides/2014-02-06-Lab.markdown diff --git a/slides/2013-02-08-Lab.html b/slides/2013-02-08-Lab.html deleted file mode 100644 index 12967f0..0000000 --- a/slides/2013-02-08-Lab.html +++ /dev/null @@ -1,288 +0,0 @@ - - - - -2013-02-08-Lab - - - - - - - - - - - - - - - - - - - -
    - -
    - -
    -

    2013-02-08-Lab

    - - - - -
    -

    1 Lab: Data Stats    slide

    -
    - -
      -
    • Obtain California (CA) campaign finance data -
      • -
    • Decompress -
      • -
    • Manually check -
      • -
    • Run code/stats.py -
      • -
    • Edit code/stats.py to add functionality -
      • -
    • Pull Request submission -
      • -
    - - -
    - -
    - -
    -

    2 Data    slide

    -
    - -
      -
    • campaign finance data -
      • -
    • We'll just be using CA data -
      • -
    • Decompress and investigate using the tools we discussed -
      • -
    - - -
    - -
    - -
    -

    3 Code    slide

    -
    - - - - -
    $ git checkout master
-$ git pull jblomo master
-$ git checkout -b hw-stats
-
    - -
      -
    • Run and edit code/stats.py -
      • -
    - - -
    - -
    - -
    -

    4 Stats    slide

    -
    - -
      -
    • Minimum -
      • -
    • Maxiumum -
      • -
    • Mean -
      • -
    • Median -
      • -
    • Standard Deviation -
      • -
    • Candidates -
      • -
    • Normalized sample contributions -
      • -
    - - -
    - -
    - -
    -

    5 Extra Credit    slide

    -
    - -
      -
    • Extra credit is used to get you up to 100% -
      • -
    • On the current assignment -
      • -
    • Also helpful for learning topics more in depth -
      • -
    • You may do partial extra credit -
      • -
    - - -
    - -
    -

    5.1 Overall Extra Credit    notes

    -
    - -
      -
    • EC that applies to overall grade will not be assigned -
      • -
    - - -
    -
    - -
    - -
    -

    6 Extra Credit    slide

    -
    - -
      -
    • Stats per candidate -
      • -
    • z-score -
      • -
    - - -
    - -
    - -
    -

    7 Git usage    slide

    -
    - -
      -
    • All edits, commits, pushes should happen on a hw- or project branch -
      • -
    • git status -
      • -
    • All pulls (typically from jblomo) should happen on master branch -
        -
      • If you use an editor connected to ischool server, make sure you are - either editing or using git -
        • -
      - -
      • -
    - - -
    - -
    -

    7.1 Exceptions    notes

    -
    - -
      -
    • There are exceptions but know what you're trying to do -
      • -
    • External editors can write back files after you've changed git branches -
      • -
    - - -
    -
    - -
    - -
    -

    8 Submission    slide

    -
    - -
      -
    • GitHub pull request -
      • -
    • If something is going wrong, submit by email: jblomo@ischool, - shreyas@ischool -
      • -
    • We'll help you submit the pull request, but HW will be full credit -
      • -
    - - -
    - -
    -

    8.1 Submission    notes

    -
    - -
      -
    • The pull request is a way for Shreyas and I to easily see changes, grade -
      • -
    • It'll give you good experience, but it is not a fundemental skill of the - class, so I'm not too worried about it -
      • -
    - - - - - - - -
    -
    -
    -
    - -
    -

    Date: 2013-02-08 13:51:10 PST

    -

    Author: Jim Blomo

    -

    Org version 7.8.02 with Emacs version 23

    -Validate XHTML 1.0 - -
    - - diff --git a/slides/2013-02-08-Lab.org b/slides/2013-02-08-Lab.org deleted file mode 100644 index a693ef0..0000000 --- a/slides/2013-02-08-Lab.org +++ /dev/null @@ -1,78 +0,0 @@ -* Lab: Data Stats :slide: - + Obtain California (CA) [[http://www.fec.gov/disclosurep/PDownload.do][campaign finance data]] - + Decompress - + Manually check - + Run =code/stats.py= - + Edit =code/stats.py= to add functionality - + Pull Request submission - -* Data :slide: - + [[http://www.fec.gov/disclosurep/PDownload.do][campaign finance data]] - + We'll just be using CA data - + Decompress and investigate using the tools we discussed - -* Code :slide: -#+begin_src bash -$ git checkout master -$ git pull jblomo master -$ git checkout -b hw-stats -#+end_src - + Run and edit =code/stats.py= - -* Stats :slide: - + Minimum - + Maximum - + Mean - + Median - + Standard Deviation - + Candidates - + Normalized sample contributions - -* Extra Credit :slide: - + Extra credit is used to get you *up to 100%* - + On the *current assignment* - + Also helpful for learning topics more in depth - + You may do partial extra credit -** Overall Extra Credit :notes: - + EC that applies to overall grade will not be assigned - -* Extra Credit :slide: - + Stats per candidate - + z-score - -* Git usage :slide: - + All edits, commits, pushes should happen on a =hw-= or =project= branch - + =git status= - + All pulls (typically from =jblomo=) should happen on =master= branch - + If you use an editor connected to ischool server, make sure you are - *either* editing *or* using git -** Exceptions :notes: - + There are exceptions but know what you're trying to do - + External editors can write back files *after* you've changed git branches - -* Submission :slide: - + GitHub pull request - + If something is going wrong, submit by email: jblomo@ischool, - shreyas@ischool - + We'll help you submit the pull request, but HW will be full credit -** Submission :notes: - + The pull request is a way for Shreyas and I to easily see changes, grade - + It'll give you good experience, but it is not a fundamental skill of the - class, so I'm not too worried about it - -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: - -#+BEGIN_HTML - -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: diff --git a/slides/2014-02-06-Lab.html b/slides/2014-02-06-Lab.html new file mode 100644 index 0000000..cc7c10f --- /dev/null +++ b/slides/2014-02-06-Lab.html @@ -0,0 +1,245 @@ + + + + Data Mining + + + + + + + + + diff --git a/slides/2014-02-06-Lab.markdown b/slides/2014-02-06-Lab.markdown new file mode 100644 index 0000000..26d934d --- /dev/null +++ b/slides/2014-02-06-Lab.markdown @@ -0,0 +1,104 @@ +name: inverse +layout: true +class: left, top, inverse + +--- + +*Todo: test!* + +## Lab: Data Stats + + + Obtain California (CA) [campaign finance data](http://www.fec.gov/disclosurep/PDownload.do) + + Decompress + + Manually check + + Run ```code/stats.py``` + + Edit ```code/stats.py``` to add functionality + + Pull Request submission + +--- + +*Todo: test!* + +## Data + + + [campaign finance data](http://www.fec.gov/disclosurep/PDownload.do) + + We'll just be using CA data + + Decompress and investigate using the tools we discussed + +--- + +## Code + +```bash +$ git checkout master +$ git pull jretz master +$ git checkout -b hw-stats +``` + + Run and edit ```code/stats.py``` + +--- + +## Stats + + + Minimum + + Maximum + + Mean + + Median + + Standard Deviation + + Candidates + + Normalized sample contributions + +--- + +## Extra credit + + + Extra credit is used to get you *up to 100%* + + On the *current assignment* + + Also helpful for learning topics more in depth + + You may do partial extra credit + +??? + +## Overall Extra Credit + + + Extra credit that applies to overall grade will not be assigned + +--- + +## Extra Credit + + Stats per candidate + + z-score + +--- + +## Git usage + + + All edits, commits, pushes should happen on a ```hw-``` or ```project``` branch + + ```git status``` + + All pulls (typically from ```jretz```) should happen on ```master``` branch + + If you use an editor connected to ischool server, make sure you are + *either* editing *or* using git + +??? + +## Exceptions + + + There are exceptions but know what you're trying to do + + External editors can write back files *after* you've changed git branches + +--- + +## Submission + + + GitHub pull request + + If something is going wrong, submit by email: jretz@ischool, + shreyas@ischool + + We'll help you submit the pull request, but you'll get full credit + +??? + +## Submission + + + The pull request is a way for Shreyas and I to easily see changes, grade + + It'll give you good experience, but it is not a fundamental skill of the + class, so I'm not too worried about it From 56c682cda1713aa35cf7c6bbbc176ed9a76f7539 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Tue, 10 Dec 2013 10:15:08 -0800 Subject: [PATCH 15/79] converted Preprocessing deck --- slides/2013-02-08-Preprocessing.html | 940 ----------------------- slides/2013-02-08-Preprocessing.org | 271 ------- slides/2014-02-06-Preprocessing.html | 674 ++++++++++++++++ slides/2014-02-06-Preprocessing.markdown | 533 +++++++++++++ 4 files changed, 1207 insertions(+), 1211 deletions(-) delete mode 100644 slides/2013-02-08-Preprocessing.html delete mode 100644 slides/2013-02-08-Preprocessing.org create mode 100644 slides/2014-02-06-Preprocessing.html create mode 100644 slides/2014-02-06-Preprocessing.markdown diff --git a/slides/2013-02-08-Preprocessing.html b/slides/2013-02-08-Preprocessing.html deleted file mode 100644 index 4342f83..0000000 --- a/slides/2013-02-08-Preprocessing.html +++ /dev/null @@ -1,940 +0,0 @@ - - - - -2013-02-08-Preprocessing - - - - - - - - - - - - - - - - - - - -
    - -
    - -
    -

    2013-02-08-Preprocessing

    - - - - -
    -

    1 Preprocessing    slide

    -
    - - -
    - -
    - -
    -

    2 Real World is Dirty    slide

    -
    - -
    -
    Incomplete
    missing timestamps for actions -
    -
    Noisy
    salary = -10 -
    -
    Inconsistent
    age: 42, birthday: 1997-03-07 -
    -
    - - -
    - -
    -

    2.1 Types of dirty    notes

    -
    - -
    -
    Incomplete
    lacking some attribute values, containing only aggregate - data. Eg. We often regret not including timestamps on different actions - like UFCing, instead of tracking total votes (aggregation) -
    -
    Noisy
    Containing errors, like impossible salary data, or decimals in the - wrong place -
    -
    Inconsistent
    If there's every two fields that depend on each other, in a - large dataset you'll find them disagreeing. Errors ofen come from failures: - processes failing halfway into updating -
    -
    - - -
    -
    - -
    - -
    -

    3 Causes of Problems    slide

    -
    - -
      -
    • Humans -
      • -
    • Software -
      • -
    • Hardware -
      • -
    - - -
    - -
    -

    3.1 Problems    notes

    -
    - -
      -
    • Berkeley experiment to measure temperature across campus -
      • -
    • Turned out average on campus much warmer than external weather services - predicted -
      • -
    • But sample data looked in line with predictions -
      • -
    • Problem: one monitoring station right next to air conditioning unit! -
      • -
    • Hardware failure rare, but with large numbers of machines, probable. Eg. - RAM can suffer ~1 bit/hour/gigabyte (ECC can help) -
      • -
    - - -
    - -
    - -
    -

    3.2 Inconsistent Different Sources    slide

    -
    - -
      -
    • Great value in combining data sources -
      • -
    • Challenge is merging them together, removing duplicates -
      • -
    • Example: Business names -
      • -
    - -
    - -
    - -
    -

    3.3 Business names    notes

    -
    - -
      -
    • Starbucks vs. Starbucks Coffee Shop -
      • -
    • Buck's vs Bucks -
      • -
    • Trying to use address? Stackbucks vs. Starbucks across the street -
      • -
    • Best strategy here is to use DM/ML techniques on the combination of - features to determine likelihood of match. We'll discuss specific - algorithms later in the course -
      • -
    - - -
    -
    - -
    - -
    -

    4 Preprocessing    slide

    -
    - -
    -
    Cleaning
    fill missing values, smooth noisy data, identify or remove - outliers, resolve inconsistencies -
    -
    Integration
    merging data from multiple sources -
    -
    Reduction
    obtain a smaller data set that can sufficiently answer - important questions -
    -
    Transformation
    change data to a form that is easier to mine or analyze -
    -
    - - -
    - -
    -

    4.1 Flu Trend Problems (Questions)    notes

    -
    - -
      -
    • We have millisecond search resolution, but will only be plotting on a per day basis -
      • -
    • We have the exact text of each query, but just care if it is about the flu or not -
      • -
    • Flu Trends, we sometimes see out of control search bots doing 100,000s of searches per day -
      • -
    • Mobile phone searches and web searches hit different machines, software, logs -
      • -
    • We have IPs in the logs, but will by plotting against geographical areas -
      • -
    - - -
    -
    - -
    - -
    -

    5 Missing Values    slide two_col

    -
    - - - -- - - - - - - - - -
    PersonHeight
    Jim6'0
    Ashley-
    Sam5'11
    Alice5'9
    Kate-
    - - -

    -img/tallest-shortest-man.jpg -

    -
    - -
    -

    5.1 What to do?    notes

    -
    - -
      -
    • (Heights are made up) -
      • -
    • We want to get an average class height -
      • -
    • Q: What to do with missing rows? -
      • -
    • ignore, fill, constant, average, average wrt gender -
      • -
    - - -
    -
    - -
    - -
    -

    6 Fill Missing Values    slide animate

    -
    - -
      -
    • Ignore the record -
      • -
    • Find value manually -
      • -
    • Global constant -
      • -
    • Average -
      • -
    • Average with respect to class -
      • -
    • "Most probable" -
      • -
    - - -
    - -
    -

    6.1 Details    notes

    -
    - -
    -
    Trade-offs
    core to engineering -
    -
    Ignore
    simply drop from data set. Hope there are not too many to affect - answer. Drawbacks? When missing values are all same class (skew data) -
    -
    Find value manually
    Even for a small class, might be difficult. Get - ruler, measure them. For historical data, impossible. -
    -
    Global constant
    replace with "N/A" or "6 foot". Can skew data, or cause - data to pop in other analysis (all grouped together) -
    -
    Average
    Mean or median. Either one has potential problems. -
    -
    Average with respect to class
    gender. Average female/male height to fill - in values -
    -
    "Most probable"
    Think of as another step from avg -> class avg. Now - throw in other details: age, family history, shoe size. Then weight - depending on how much those factors are correlated. Pretty soon you have a - regression or Bayesian model, which will cover later -
    -
    - - -
    -
    - -
    - -
    -

    7 Normalization    slide

    -
    - -
      -
    • Type of data transformation to make reasoning and comparison easier -
      • -
    • Is 6' tall? -
      • -
    • Coefficients on attributes in regressions understandable -
      • -
    - - -
    - -
    -

    7.1 Context, Comparison    notes

    -
    - -
      -
    • 6' Might be tall for this class, but not on a basketball team -
      • -
    • How to know when a data point "average" or towards the top of a range? -
      • -
    • For our housing model, we wanted to use sq. footage and # of bedrooms. But - the sq. footage number is huge compared to bedrooms. If we didn't - normalize, a formula for determine house price might seem to indicate that - # of bedrooms was way more important -
      • -
    - - -
    -
    - -
    - -
    -

    8 Min-max    slide

    -
    - -

    img/min-max.gif -

    -
    - -
    -

    8.1 New Range    notes

    -
    - -
      -
    • Typically new range is -
        -
      • [0-1] (thought of as %) -
        • -
      • [-1-1] (though of as bad->good -
        • -
      - -
      • -
    - - -
    -
    - -
    - -
    -

    9 Z-score    slide

    -
    - -

    img/z-score.gif -

    -
    - -
    -

    9.1 Uses    notes

    -
    - -
      -
    • When you want a relative measure of deviation -
      • -
    • When you have a distribution estimate, but are unsure of absolute min-max -
      • -
    - - -
    -
    - -
    - -
    -

    10 Comparison    slide

    -
    - -

    img/outliers.png -img/outliers-minmax-zscore.png -

    -
    - -
    -

    10.1 Min-max vs Z-score    notes

    -
    - -
      -
    • Min-max: Known range -
      • -
    • Z-score: more expressive range -
      • -
    • Min-max: requires knowing min-max -
      • -
    • Z-score: can estimate with sampling or informed guess -
      • -
    - - -
    -
    - -
    - -
    -

    11 Removing Noise    slide

    -
    - -
    -
    Binning
    create B bins << N data samples, use aggregate statistic of bin - for value -
    -
    Regression
    fit data to a function, use function value -
    -
    Outlier analysis
    find outlying points, understand and/or ignore them -
    -
    - - -
    - -
    -

    11.1 Monitoring Problem    notes

    -
    - -
      -
    • For the problem encountered in temperature monitoring, which makes the most - sense? -
      • -
    - - -
    - -
    - -
    -

    11.2 Trade-offs    slide

    -
    - -
    -
    Binning
    Simple way to remove outliers, but difficult to pick buckets - correctly -
    -
    Regression
    If one metric is a direct function of another, what extra - information does the value provide? -
    -
    Outlier analysis
    Manual process of understanding outliers, ignoring them - can obscure some analysis (eg. income disparity) -
    -
    - - -
    - -
    -

    11.2.1 Trade-offs again    notes

    -
    - -
      -
    • Remember: this class is exposing you to potential tools, up to you to be - asking the right questions, selecting the appropriate algorithms, - interpreting results -
      • -
    - - -
    -
    -
    - -
    - -
    -

    12 Data integration    slide

    -
    - -
      -
    • Merging two data sources -
      • -
    • Problem: uniquely identify a concept in both sources -
      • -
    • Find data points that are very "close" to each other, call them the same - with some probability -
      • -
    • Example: Yelp Menu Data -
      • -
    - - -
    - -
    -

    12.1 Yelp Menu Data    notes

    -
    - -
      -
    • Recently launched menu data -
      • -
    • Takes data about the restaurant menu, find reviews & pictures referring to - the menu item -
      • -
    • Joins them together -
      • -
    • Many different metrics for "close": remember them? -
      • -
    - - -
    -
    - -
    - -
    -

    13 Other measures of "close"    slide

    -
    - -

    Are A and B close? -

    - -- - - - - - - - - -
    AB
    260
    5150
    6180
    10300
    13390
    - - -
    - -
    -

    13.1 Correlation    notes

    -
    - -
      -
    • Imagine A and B have several different dimensions, maybe things like - length, height, width, radius -
      • -
    • Are they similar? -
      • -
    • On one hand no: clearly different order of magnitude -
      • -
    • Another way to think about similarity is correlation -
      • -
    • All of B dimensions are 30x of A -
      • -
    • Maybe just using different units! -
      • -
    • If I plotted A and B and x,y, what would the result look like? -
      • -
    - - -
    -
    - -
    - -
    -

    14 Χ2 Correlation Test    slide

    -
    - -

    img/correlation.png - img/chiequation.jpg -

    -
    - -
    -

    14.1 Motivation    notes

    -
    - -
      -
    • Answer: a straight line -
      • -
    • So a correlation coefficient gives a sense of how closely linearly - related two data sets are -
      • -
    • Note, besides positive & negative, the slop does not affect the correlation - score, just how well fit the data is -
      • -
    • Also note I said linear: patterns may still be exhibited, but they are not - linearly related, eg 30x -
      • -
    • Details of test are in book, you are expected to understand it -
      • -
    • Motivation: how different are the observed values from the expected? -
      • -
    • Expected is calculated using probability with the assumptions that the sets - are independent -
      • -
    - - -
    -
    - -
    - -
    -

    15 Covariance & Correlation    slide

    -
    - -
      -
    • Correlation is "normalized" covariance -
      • -
    • Covariance describes the degree to which two data sets track each other in - units of of the two data sets -
      • -
    • Correlations describes the degree of similarity without units -
      • -
    - - -
    - -
    -

    15.1 Use in industry    notes

    -
    - -
      -
    • Χ2 used most commonly, handy to have an expected [0-1] range -
      • -
    • "Correlation does not imply causation" -
      • -
    • A->B, B->A, C->A,B, A->B->A…, coincidence -
      • -
    - - -
    -
    - -
    - -
    -

    16 Data Reduction    slide two_col

    -
    - -
    -
    Dimensionality
    remove attributes that are the same or similar to other - attributes -
    -
    Numerosity
    represent or aggregate the data, sometimes with precision loss -
    -
    Compression
    generalized techniques to decrease the number of bytes needed - to store data -
    -
    - -

    img/compress-car.jpg -

    -
    - -
    -

    16.1 Deep Dive    notes

    -
    - -
      -
    • We're only going to cover selected topics in these areas. -
      • -
    • When reading, make sure to understand the intuition behind the other - techniques, but if we don't cover it in lecture, you won't need to - calculate it in midterm -
      • -
    • Ask questions about the concepts you don't understand! That's what - separates this class from a book :) -
      • -
    • But still potentially useful for your projects! -
      • -
    • img: http://www.flickr.com/photos/marcovdz/4520986339/sizes/o/in/photostream/ -
      • -
    - - -
    -
    - -
    - -
    -

    17 Subset Selection    slide

    -
    - -
      -
    • Two many attributes? -
      • -
    • Ignore some -
      • -
    • Tricky part: which to ignore? -
      • -
    • height x width = area -
      • -
    - - -
    - -
    -

    17.1 Simple to Sophisticated    notes

    -
    - -
      -
    • Ignore the ones that are not helpful -
      • -
    • Ignore an attribute highly correlated with another (cm, in) -
      • -
    • Ignore an attribute that can be built from others -
      • -
    - - -
    -
    - -
    - -
    -

    18 Principal Component Analysis    slide

    -
    - -

    img/GaussianScatterPCA.png -

      -
    • Map data to a locatoin along a few vectors -
      • -
    - - -
    - -
    -

    18.1 Higher dimensions    notes

    -
    - -
      -
    • Remember, 2 dimensions might not make much sense, but becomes useful in - higher number of dimensions -
      • -
    • These points described by two attributes, <x,y> -
      • -
    • What if we wanted to describe them in just 1 dimension? -
      • -
    • Pick some good vectors (in our case 1) -
      • -
    • Describe where a point is located using only those vectors -
      • -
    - - -
    -
    - -
    - -
    -

    19 Netflix and PCA    slide

    -
    - -
      -
    • A user may have many preferences: Mission Impossible, Love Actually, Man - from Nowhere, … -
      • -
    • Instead of keeping track of every preference, we can summarize -
      • -
    • Action, RomCom, Foreign -
      • -
    - - -
    - -
    -

    19.1 Summarize in discovered dimensions    notes

    -
    - -
      -
    • With 3 or more "categories", we can reconstruct the user's likely - preferences -
      • -
    • Dimensions don't necessarily fit into human notions: probably is not an - "foreign" dimension, but a subtle combination of other aspects -
      • -
    - - - - - - - - -
    -
    -
    -
    - -
    -

    Date: 2013-02-08 13:49:51 PST

    -

    Author: Jim Blomo

    -

    Org version 7.8.02 with Emacs version 23

    -Validate XHTML 1.0 - -
    - - diff --git a/slides/2013-02-08-Preprocessing.org b/slides/2013-02-08-Preprocessing.org deleted file mode 100644 index c229b9f..0000000 --- a/slides/2013-02-08-Preprocessing.org +++ /dev/null @@ -1,271 +0,0 @@ -* Preprocessing :slide: - -* Real World is Dirty :slide: - + Incomplete :: missing timestamps for actions - + Noisy :: salary = -10 - + Inconsistent :: age: 42, birthday: 1997-03-07 -** Types of dirty :notes: - + Incomplete :: lacking some attribute values, containing only aggregate - data. Eg. We often regret not including timestamps on different actions - like UFCing, instead of tracking total votes (aggregation) - + Noisy :: Containing errors, like impossible salary data, or decimals in the - wrong place - + Inconsistent :: If there's every two fields that depend on each other, in a - large dataset you'll find them disagreeing. Errors ofen come from failures: - processes failing halfway into updating - -* Causes of Problems :slide: - + Humans - + Software - + Hardware -** Problems :notes: - + Berkeley experiment to measure temperature across campus - + Turned out average on campus much warmer than external weather services - predicted - + But sample data looked in line with predictions - + Problem: one monitoring station right next to air conditioning unit! - + Hardware failure rare, but with large numbers of machines, probable. Eg. - RAM can suffer ~1 bit/hour/gigabyte (ECC can help) - -** Inconsistent Different Sources :slide: - + Great value in combining data sources - + Challenge is merging them together, removing duplicates - + Example: Business names -** Business names :notes: - + Starbucks vs. Starbucks Coffee Shop - + Buck's vs Bucks - + Trying to use address? Stackbucks vs. Starbucks across the street - + Best strategy here is to use DM/ML techniques on the *combination* of - features to determine likelihood of match. We'll discuss specific - algorithms later in the course - -* Preprocessing :slide: - + Cleaning :: fill missing values, smooth noisy data, identify or remove - outliers, resolve inconsistencies - + Integration :: merging data from multiple sources - + Reduction :: obtain a smaller data set that can sufficiently answer - important questions - + Transformation :: change data to a form that is easier to mine or analyze -** Flu Trend Problems (Questions) :notes: - + We have millisecond search resolution, but will only be plotting on a per day basis - + We have the exact text of each query, but just care if it is about the flu or not - + Flu Trends, we sometimes see out of control search bots doing 100,000s of searches per day - + Mobile phone searches and web searches hit different machines, software, logs - + We have IPs in the logs, but will by plotting against geographical areas - -* Missing Values :slide:two_col: -| Person | Height | -| Jim | 6'0 | -| Ashley | - | -| Sam | 5'11 | -| Alice | 5'9 | -| Kate | - | - -[[file:img/tallest-shortest-man.jpg]] -** What to do? :notes: - + (Heights are made up) - + We want to get an average class height - + Q: What to do with missing rows? - + ignore, fill, constant, average, average wrt gender - -* Fill Missing Values :slide:animate: - + Ignore the record - + Find value manually - + Global constant - + Average - + Average with respect to class - + "Most probable" -** Details :notes: - + Trade-offs :: core to engineering - + Ignore :: simply drop from data set. Hope there are not too many to affect - answer. Drawbacks? When missing values are all same class (skew data) - + Find value manually :: Even for a small class, might be difficult. Get - ruler, measure them. For historical data, impossible. - + Global constant :: replace with "N/A" or "6 foot". Can skew data, or cause - data to pop in other analysis (all grouped together) - + Average :: Mean or median. Either one has potential problems. - + Average with respect to class :: gender. Average female/male height to fill - in values - + "Most probable" :: Think of as another step from avg -> class avg. Now - throw in other details: age, family history, shoe size. Then weight - depending on how much those factors are correlated. Pretty soon you have a - regression or Bayesian model, which will cover later - -* Normalization :slide: - + Type of data transformation to make reasoning and comparison easier - + Is 6' tall? - + Coefficients on attributes in regressions understandable -** Context, Comparison :notes: - + 6' Might be tall for this class, but not on a basketball team - + How to know when a data point "average" or towards the top of a range? - + For our housing model, we wanted to use sq. footage and # of bedrooms. But - the sq. footage number is huge compared to bedrooms. If we didn't - normalize, a formula for determine house price might seem to indicate that - # of bedrooms was way more important - -* Min-max :slide: -[[file:img/min-max.gif]] -** New Range :notes: - + Typically new range is - + [0-1] (thought of as %) - + [-1-1] (though of as bad->good - -* Z-score :slide: -[[file:img/z-score.gif]] -** Uses :notes: - + When you want a relative measure of deviation - + When you have a distribution estimate, but are unsure of absolute min-max - -* Comparison :slide: -[[file:img/outliers.png]] -[[file:img/outliers-minmax-zscore.png]] -** Min-max vs Z-score :notes: - + Min-max: Known range - + Z-score: more expressive range - + Min-max: requires knowing min-max - + Z-score: can estimate with sampling or informed guess - -* Removing Noise :slide: - + Binning :: create B bins << N data samples, use aggregate statistic of bin - for value - + Regression :: fit data to a function, use function value - + Outlier analysis :: find outlying points, understand and/or ignore them -** Monitoring Problem :notes: - + For the problem encountered in temperature monitoring, which makes the most - sense? - -** Trade-offs :slide: - + Binning :: Simple way to remove outliers, but difficult to pick buckets - correctly - + Regression :: If one metric is a direct function of another, what extra - information does the value provide? - + Outlier analysis :: Manual process of understanding outliers, ignoring them - can obscure some analysis (eg. income disparity) -*** Trade-offs again :notes: - + Remember: this class is exposing you to potential tools, up to you to be - asking the right questions, selecting the appropriate algorithms, - interpreting results - -* Data integration :slide: - + Merging two data sources - + Problem: uniquely identify a concept in both sources - + Find data points that are very "close" to each other, call them the same - with some probability - + Example: [[http://www.yelp.com/menu/tartine-bakery-san-francisco][Yelp Menu Data]] -** Yelp Menu Data :notes: - + Recently launched menu data - + Takes data about the restaurant menu, find reviews & pictures referring to - the menu item - + Joins them together - + Many different metrics for "close": remember them? - -* Other measures of "close" :slide: -Are =A= and =B= close? -| A | B | -| 2 | 60 | -| 5 | 150 | -| 6 | 180 | -| 10 | 300 | -| 13 | 390 | -** Correlation :notes: - + Imagine =A= and =B= have several different dimensions, maybe things like - length, height, width, radius - + Are they similar? - + On one hand no: clearly different order of magnitude - + Another way to think about similarity is correlation - + All of =B= dimensions are 30x of =A= - + Maybe just using different units! - + If I plotted =A= and =B= and x,y, what would the result look like? - -* Χ^2 Correlation Test :slide: - [[file:img/correlation.png]] - [[file:img/chiequation.jpg]] -** Motivation :notes: - + Answer: a straight line - + So a correlation coefficient gives a sense of how closely *linearly* - related two data sets are - + Note, besides positive & negative, the slop does not affect the correlation - score, just how well fit the data is - + Also note I said linear: patterns may still be exhibited, but they are not - linearly related, eg 30x - + Details of test are in book, you are expected to understand it - + Motivation: how different are the observed values from the expected? - + Expected is calculated using probability with the assumptions that the sets - are *independent* - -* Covariance & Correlation :slide: - + Correlation is "normalized" covariance - + Covariance describes the degree to which two data sets track each other in - units of of the two data sets - + Correlations describes the degree of similarity without units -** Use in industry :notes: - + Χ^2 used most commonly, handy to have an expected [0-1] range - + "Correlation does not imply causation" - + A->B, B->A, C->A,B, A->B->A..., coincidence - -* Data Reduction :slide:two_col: - + Dimensionality :: remove attributes that are the same or similar to other - attributes - + Numerosity :: represent or aggregate the data, sometimes with precision loss - + Compression :: generalized techniques to decrease the number of bytes needed - to store data -[[file:img/compress-car.jpg]] -** Deep Dive :notes: - + We're only going to cover selected topics in these areas. - + When reading, make sure to understand the intuition behind the other - techniques, but if we don't cover it in lecture, you won't need to - calculate it in midterm - + Ask questions about the concepts you don't understand! That's what - separates this class from a book :) - + But still potentially useful for your projects! - + img: http://www.flickr.com/photos/marcovdz/4520986339/sizes/o/in/photostream/ - -* Subset Selection :slide: - + Two many attributes? - + *Ignore some* - + Tricky part: which to ignore? - + height x width = area -** Simple to Sophisticated :notes: - + Ignore the ones that are not helpful - + Ignore an attribute highly correlated with another (cm, in) - + Ignore an attribute that can be built from others - -* Principal Component Analysis :slide: -[[file:img/GaussianScatterPCA.png]] - + Map data to a locatoin along a few vectors -** Higher dimensions :notes: - + Remember, 2 dimensions might not make much sense, but becomes useful in - higher number of dimensions - + These points described by two attributes, - + What if we wanted to describe them in just 1 dimension? - + Pick some good vectors (in our case 1) - + Describe where a point is located using only those vectors - -* Netflix and PCA :slide: - + A user may have many preferences: Mission Impossible, Love Actually, Man - from Nowhere, ... - + Instead of keeping track of every preference, we can summarize - + Action, RomCom, Foreign -** Summarize in discovered dimensions :notes: - + With 3 or more "categories", we can reconstruct the user's likely - preferences - + Dimensions don't necessarily fit into human notions: probably is not an - "foreign" dimension, but a subtle combination of other aspects - - -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: - -#+BEGIN_HTML - -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: diff --git a/slides/2014-02-06-Preprocessing.html b/slides/2014-02-06-Preprocessing.html new file mode 100644 index 0000000..f34d410 --- /dev/null +++ b/slides/2014-02-06-Preprocessing.html @@ -0,0 +1,674 @@ + + + + Data Mining + + + + + + + + + diff --git a/slides/2014-02-06-Preprocessing.markdown b/slides/2014-02-06-Preprocessing.markdown new file mode 100644 index 0000000..ff9d416 --- /dev/null +++ b/slides/2014-02-06-Preprocessing.markdown @@ -0,0 +1,533 @@ +name: inverse +layout: true +class: left, top, inverse + +--- + +# Preprocessing + +--- + +## Real World is Dirty + +### Incomplete +missing timestamps for actions + +### Noisy +salary = -10 + +### Inconsistent +age: 42, birthday: 1997-03-07 + +??? + +## Types of dirty + +### Incomplete +lacking some attribute values, containing only aggregate data. +e.g., We often regret not including timestamps on different actions +like UFCing, instead of tracking total votes (aggregation) + +### Noisy +Containing errors, like impossible salary data, or decimals in the +wrong place + +### Inconsistent +If two fields depend on each other in a large dataset, you'll find them +disagreeing. Errors often come from failures: processes failing halfway +into updating + +--- + +## Causes of Problems + + + Humans + + Software + + Hardware + +??? + +## Problems + + + Berkeley experiment to measure temperature across campus + + Turned out average on campus much warmer than external weather services + predicted + + But sample data looked in line with predictions + + Problem: one monitoring station right next to air conditioning unit! + + Hardware failure rare, but with large numbers of machines, probable. e.g., + RAM can suffer ~1 bit/hour/gigabyte (ECC can help) + +--- + +## Inconsistent Different Sources + + + Great value in combining data sources + + Challenge is merging them together, removing duplicates + + Example: Business names + +??? + +## Business names + + + Starbucks vs. Starbucks Coffee Shop + + Buck's vs Bucks + + Trying to use address? Stackbucks vs. Starbucks across the street + + Best strategy here is to use DM/ML techniques on the *combination* of + features to determine likelihood of match. We'll discuss specific + algorithms later in the course + +--- + +## Preprocessing + +### Cleaning +fill missing values, smooth noisy data, identify or remove +outliers, resolve inconsistencies + +### Integration +merging data from multiple sources + +### Reduction +obtain a smaller data set that can sufficiently answer +important questions + +### Transformation +change data to a form that is easier to mine or analyze + +??? + +## Flu Trend Problems (Questions) + + + We have millisecond search resolution, but will only be plotting on a per day basis + + We have the exact text of each query, but just care if it is about the flu or not + + Flu Trends, we sometimes see out of control search bots doing 100,000s of searches per day + + Mobile phone searches and web searches hit different machines, software, logs + + We have IPs in the logs, but will be plotting against geographical areas + +--- + +## Missing Values + +.left-column[ +  + +| Person | Height | +|--------|--------| +| Bob | 6'0 | +| Ashley | - | +| Sam | 5'11 | +| Alice | 5'9 | +| Kate | - | +] +.right-column[ + +] + +??? + +## What to do? + + (Heights are made up) + + We want to get an average class height + + Q: What to do with missing rows? + + ignore, fill, constant, average, average wrt gender + +--- + +## Fill Missing Values + +??? + +## Details + + + Trade-offs + + core to engineering + +--- + +## Fill Missing Values + + + Ignore the record + +??? + +## Details + + + Ignore + + simply drop from data set. Hope there are not too many to affect + answer. Drawbacks? When missing values are all same class (skew data) + +--- + +## Fill Missing Values + + + Ignore the record + + Find value manually + +??? + +## Details + + + Find value manually + + Even for a small class, might be difficult. Get + ruler, measure them. For historical data, impossible. + +--- + +## Fill Missing Values + + + Ignore the record + + Find value manually + + Global constant + +??? + +## Details + + + Global constant + + replace with "N/A" or "6 foot". Can skew data, or cause + data to pop in other analysis (all grouped together) + +--- + +## Fill Missing Values + + + Ignore the record + + Find value manually + + Global constant + + Average + +??? + +## Details + + + Average + + Mean or median. Either one has potential problems. + +--- + +## Fill Missing Values + + + Ignore the record + + Find value manually + + Global constant + + Average + + Average with respect to class + +??? + +## Details + + + Average with respect to class + + gender. Average female/male height to fill + in values + +--- + +## Fill Missing Values + + + Ignore the record + + Find value manually + + Global constant + + Average + + Average with respect to class + + "Most probable" + +??? + +## Details + + + "Most probable" + + Think of as another step from avg -> class avg. Now + throw in other details: age, family history, shoe size. Then weight + depending on how much those factors are correlated. Pretty soon you have a + regression or Bayesian model, which will cover later + +--- + +## Normalization + + + Type of data transformation to make reasoning and comparison easier + + Is 6' tall? + + Coefficients on attributes in regressions understandable + +??? + +## Context, Comparison + + + 6' Might be tall for this class, but not on a basketball team + + How to know when a data point is "average" or towards the top of a range? + + For our housing model, we wanted to use sq. footage and # of bedrooms. But + the sq. footage number is huge compared to bedrooms. If we didn't + normalize, a formula for determine house price might seem to indicate that # + of bedrooms was way more important + +--- + +*Todo: image is transparent and assumes light background* + +## Min-max + + + +??? + +## New Range + + Typically new range is + + [0-1] (thought of as %) + + [-1-1] (though of as bad->good + +--- + +*Todo: image is transparent and assumes light background* + +## Z-score + + + +??? + +## Uses + + + When you want a relative measure of deviation + + When you have a distribution estimate, but are unsure of absolute min-max + +--- + +*Todo: image scale* + +## Comparison + + + + +??? + +## Min-max vs Z-score + + + Min-max: Known range + + Z-score: more expressive range + + Min-max: requires knowing min-max + + Z-score: can estimate with sampling or informed guess + +--- + +## Removing Noise + +### Binning + +create B bins << N data samples, use aggregate statistic of bin +for value + +### Regression + +fit data to a function, use function value + +### Outlier analysis + +find outlying points, understand and/or ignore them + +??? + +## Monitoring Problem + + + For the problem encountered in temperature monitoring, which makes the most + sense? + +--- + +## Trade-offs + +### Binning + +Simple way to remove outliers, but difficult to pick buckets +correctly + +### Regression + +If one metric is a direct function of another, what extra +information does the value provide? + +### Outlier analysis + +Manual process of understanding outliers, ignoring them +can obscure some analysis (e.g., income disparity) + +??? + +## Trade-offs again + + + Remember: this class is exposing you to potential tools, it's up to you + to be asking the right questions, selecting the appropriate algorithms, + interpreting results + +--- + +## Data integration + + + Merging two data sources + + Problem: uniquely identify a concept in both sources + + Find data points that are very "close" to each other, call them the same + with some probability + + Example: [Yelp Menu Data](http://www.yelp.com/menu/tartine-bakery-san-francisco) + +??? + +## Yelp Menu Data + + + Launched menu data in 2012 + + Takes data about the restaurant menu, find reviews & pictures referring to + the menu item + + Joins them together + + Many different metrics for "close": remember them? + +--- + +## Other measures of "close" + +Are ```A``` and ```B``` close? + +| A | B | +|----|-----| +| 2 | 60 | +| 5 | 150 | +| 6 | 180 | +| 10 | 300 | +| 13 | 390 | + +??? + +## Correlation + + + Imagine ```A``` and ```B``` have several different dimensions, maybe things like + length, height, width, radius + + Are they similar? + + On one hand no: clearly different order of magnitude + + Another way to think about similarity is correlation + + All of ```B``` dimensions are 30x of ```A``` + + Maybe just using different units! + + If I plotted ```A``` and ```B``` and x,y, what would the result look like? + +--- + +* Todo: chi? * + +## Χ^2 Correlation Test + + + + +??? + +## Motivation + + + Answer: a straight line + + So a correlation coefficient gives a sense of how closely *linearly* + related two data sets are + + Note, besides positive & negative, the slope does not affect the correlation + score, just how well fit the data is + + Also note I said linear: patterns may still be exhibited, but they are not + linearly related, eg 30x + + Details of test are in book, you are expected to understand it + + Motivation: how different are the observed values from the expected? + + Expected is calculated using probability with the assumptions that the sets + are *independent* + +--- + +## Covariance & Correlation + + + Correlation is "normalized" covariance + + Covariance describes the degree to which two data sets track each other in + units of the two data sets + + Correlations describes the degree of similarity without units + +??? + +## Use in industry + + + Χ^2 used most commonly, handy to have an expected [0-1] range + + "Correlation does not imply causation" + + A->B, B->A, C->A,B, A->B->A..., coincidence + +--- + +## Data Reduction + +### Dimensionality + +remove attributes that are the same or similar to other attributes + +### Numerosity + +represent or aggregate the data, sometimes with precision loss + +### Compression + +generalized techniques to decrease the number of bytes needed +to store data + +??? + +## Deep Dive + + + We're only going to cover selected topics in these areas. + + When reading, make sure to understand the intuition behind the other + techniques, but if we don't cover it in lecture, you won't need to + calculate it in midterm + + Ask questions about the concepts you don't understand! That's what + separates this class from a book :) + + But still potentially useful for your projects! + +--- + +## Subset Selection + + + Two many attributes? + + *Ignore some* + + Tricky part: which to ignore? + + height x width = area + +??? + +## Simple to Sophisticated + + + Ignore the ones that are not helpful + + Ignore an attribute highly correlated with another (cm, in) + + Ignore an attribute that can be built from others + +--- + +## Principal Component Analysis + + + Map data to a location along a few vectors + + + +??? + +## Higher dimensions + + + Remember, 2 dimensions might not make much sense, but becomes useful in + higher number of dimensions + + These points described by two attributes, + + What if we wanted to describe them in just 1 dimension? + + Pick some good vectors (in our case 1) + + Describe where a point is located using only those vectors + +--- + +## Netflix and PCA + + + A user may have many preferences: Mission Impossible, Love Actually, Man + from Nowhere, ... + + Instead of keeping track of every preference, we can summarize + + Action, RomCom, Foreign + +??? + +## Summarize in discovered dimensions + + + With 3 or more "categories", we can reconstruct the user's likely + preferences + + Dimensions don't necessarily fit into human notions: probably is not a + "foreign" dimension, but a subtle combination of other aspects From 149d58ab3abb4917e1e68c32f0e22d90d39b62c9 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Tue, 10 Dec 2013 20:07:53 -0800 Subject: [PATCH 16/79] converted Probability deck --- slides/2013-02-08-Probability.html | 855 ------------------------- slides/2013-02-08-Probability.org | 214 ------- slides/2014-02-06-Probability.html | 514 +++++++++++++++ slides/2014-02-06-Probability.markdown | 373 +++++++++++ 4 files changed, 887 insertions(+), 1069 deletions(-) delete mode 100644 slides/2013-02-08-Probability.html delete mode 100644 slides/2013-02-08-Probability.org create mode 100644 slides/2014-02-06-Probability.html create mode 100644 slides/2014-02-06-Probability.markdown diff --git a/slides/2013-02-08-Probability.html b/slides/2013-02-08-Probability.html deleted file mode 100644 index 42b74bc..0000000 --- a/slides/2013-02-08-Probability.html +++ /dev/null @@ -1,855 +0,0 @@ - - - - -2013-02-08-Probability - - - - - - - - - - - - - - - - - - - -
    - -
    - -
    -

    2013-02-08-Probability

    - - - - -
    -

    1 Probability    slide

    -
    - - -
    - -
    - -
    -

    2 Nomenclature    slide

    -
    - -
    -
    Record
    a single entity or concept. Also: data object, sample, example, - instance, data point -
    -
    Feature
    a characteristic or way of describing a record. Also: attribute, - dimension, variable, signal -
    -
    - - -
    - -
    -

    2.1 Slightly different from book    notes

    -
    - -
      -
    • The meanings do carry different connotations, but are generally - transferable -
      • -
    • Eg. dimensions is usually used in the math domain -
      • -
    • Feature is usually used in the ML domain -
      • -
    - - -
    - -
    - -
    -

    2.2 Feautre Types    slide

    -
    - -
    -
    Binary
    True/False. Also: 0/1 -
    -
    Numeric
    Involving numbers. Also: integer, float, double -
    -
    Ordinal
    Feature with sortable values. -
    -
    Discrete
    countable, finite set. Also: classes -
    -
    Continuous
    unbounded numeric number. Also: integer, float, double -
    -
    Enumerated
    feature named, discrete values. Also: nominal, classed -
    -
    - - -
    - -
    -

    2.2.1 Rain data set    notes

    -
    - -
      -
    • Stored did/dot not rain -
      • -
    • Stored how many inches it rained -
      • -
    • Stored the day as an integer offset from Jan 1 1970 -
      • -
    • Stored weather information: Sunny, Partly Sunny, Cloudy, Rainy -
      • -
    • Stored barometer reading -
      • -
    • Stored day of the week -
      • -
    - - -
    -
    -
    - -
    - -
    -

    3 Central Tendency    slide

    -
    - -

    img/skew-normal.png -

    -
    - -
    -

    3.1 Define    notes

    -
    - -
    -
    Mean
    "average" all data points divided by size of set -
    -
    Median
    middle value -
    -
    Mode
    The value most likely to be picked -
    -
    discrete
    most common value -
    -
    continuous
    max probability density function -
    -
    - -
    -
    - - -
    - -
    - -
    -

    3.2 Skew Positive    slide

    -
    - -

    img/skew-positive.png -

    - -
    - -
    -

    3.3 Skew    notes

    -
    - -
      -
    • Think about mean - mode -
      • -
    • Or think about where the "tail" is -
      • -
    - - -
    - -
    - -
    -

    3.4 Skew Negative    slide

    -
    - -

    img/skew-negative.png -

    -
    -
    - -
    - -
    -

    4 The Long Tail    slide two_col

    -
    - -

    img/Long_tail.svg.png -

      -
    • Most popular are very popular -
      • -
    • Everything else, not so much -
      • -
    • But there's a lot of everything else -
      • -
    - - -
    - -
    -

    4.1 Movies    notes

    -
    - -
      -
    • Current releases: millions of people watching -
      • -
    • Older movies are rented by < 1 person a week -
      • -
    • What is the skew? -
      • -
    • Power law distribution (please follow up on Wikipedia or a stats class) -
      • -
    • Distributions are important, but will only be covered as necessary -
      • -
    - - -
    -
    - -
    - -
    -

    5 Dispersion    slide

    -
    - -

    img/dispersion.png -

      -
    • Centrality not the whole story -
      • -
    - - -
    - -
    -

    5.1 Differences    notes

    -
    - -
      -
    • Wildly different data sets can still share many of these characteristics -
      • -
    - - -
    -
    - -
    - -
    -

    6 Quartiles    slide

    -
    - -

    img/quartiles.png -

    -
    - -
    -

    6.1 Parts    notes

    -
    - -
      -
    • Go back to our unskewed normal distribution -
      • -
    • Quartiles divide the data into quarters -
      • -
    • InterQuartile Range is the distance of the middle two quartiles -
      • -
    • BoxPlot is one of the most useful tools for data. For public results, I - almost never want to see scatter plot or bar charts. I want to see box - plots. -
      • -
    • Bottom, we spit it up into standard deviations -
      • -
    • Variance measures, on average, how far points are away from the mean -
      • -
    • Standard deviation is the square root of the variance -
      • -
    - - -
    -
    - -
    - -
    -

    7 Standard Deviation    slide

    -
    - -

    img/stddev.png -

      -
    • Within 1: 68% -
      • -
    • Within 2: 95% -
      • -
    • Within 3: 99.7% -
      • -
    - - -
    - -
    -

    7.1 Standard Deviation    notes

    -
    - -
      -
    • Useful for thinking about what % of outliers you'd like to catch -
      • -
    • We use it for alerting: let us know when we're 2 stddev away from the - median, there's a very small likelihood of that happening -
      • -
    - - -
    -
    - -
    - -
    -

    8 Visualization Tools    slide

    -
    - -
      -
    • Python: Matplotlib -
      • -
    • R: builtin -
      • -
    • Matlab: builtin -
      • -
    • Octave: builtin (gnuplot) -
      • -
    • HTML: D3.js -
      • -
    - - -
    - -
    -

    8.1 Covered later    notes

    -
    - -
      -
    • Chapter 2 is going to cover some visualization stuff -
      • -
    • We're going to cover visualization a bit later in the course, and more of a - "how its done in industry" -
      • -
    • There is another class on visualization in general -
      • -
    - - -
    -
    - -
    - -
    -

    9 Mathmatical Representation    slide

    -
    - - - -- - - - - - - -
    Bad BoysRobin HoodWaterworld
    Prabha132
    AJ543
    Victor441
    - - - - -
    [ 1 3 2
-  5 4 3
-  4 4 1 ]
-
    - - -
    - -
    -

    9.1 Matrix    notes

    -
    - -
      -
    • Matrix representations very powerful, as we'll see later in class -
      • -
    • Usually rows are records, columns are attributes -
      • -
    • Sometimes you can think of data in different ways, can take the transpose - of the matrix to get attributes about movies -
      • -
    - - -
    -
    - -
    - -
    -

    10 Waterworld    slide

    -
    - -

    img/waterworld.jpg -

    -
    - -
    - -
    -

    11 Similarity | Distance    slide

    -
    - -
      -
    • Two sides of the same coin -
      • -
    • similarity = 1 - distance -
      • -
    • We'll use these metrics for many other algorithms -
      • -
    - - -
    - -
    -

    11.1 Core Concept    notes

    -
    - -
      -
    • Many data mining techniques rely on finding a way to quantify similarity -
      • -
    • When you think about questions like "how similar are two users?" "is this - text plagiarism?" "are these products likely to be purchased together?" -
      • -
    • All are ways of thinking about similarity -
      • -
    - - -
    -
    - -
    - -
    -

    12 Nominal Distance    slide

    -
    - -
      -
    • Ratio of mismatches to potential matches -
      • -
    • Why can't we take the sum of the mismatches? -
      • -
    - - -
    - -
    -

    12.1 Nominal    notes

    -
    - -
      -
    • Nominal means we can't compare two values: there is no ordering -
      • -
    • All we can do is take ratio of the ones that are exactly the same -
      • -
    • The book describes how to think about this in terms of matrices -
      • -
    - - -
    -
    - -
    - -
    -

    13 Binary Distance    slide

    -
    - -
      -
    • Could use Nominal Distance: count all exact matches or mismatches -
      • -
    • Could use Numeric Distance: just treat values as 0/1 -
      • -
    • asymmetric binary dissimilarity: don't care about negative matches -
        -
      • mismatches / (positive_matches + mismatches) -
        • -
      - -
      • -
    • asymmetric binary similarity: care more about positive matches than mismatches -
        -
      • positive_matches / (positive_matches + mismatches) -
        • -
      - -
      • -
    - - -
    - -
    -

    13.1 Binary    notes

    -
    - -
      -
    • Nominal problem: for rare attributes, like a disease, two people who - don't have the disease, aren't necessarily very similar -
      • -
    - - -
    -
    - -
    - -
    -

    14 Jaccard Coefficient    slide

    -
    - -
      -
    • Asymmetric binary similarity -
      • -
    • More commonly used for calculating set similarity -
      • -
    • |intersection| / |union| -
      • -
    • "Jim likes pizza" | "Shreyas likes pizza" -
      • -
    - - -
    - -
    -

    14.1 Jaccard    notes

    -
    - -
      -
    1. Break up into a set -
      2. -
    2. calculate # in intersection -
      3. -
    3. calculate # in union -
      4. -
    4. divide -
      5. -
    - - -
    -
    - -
    - -
    -

    15 Euclidean distance    slide

    -
    - -
      -
    • Straight line between two points -
      • -
    • Again: usually considered with just (x,y), but can calculate for any number - of dimensions -
      • -
    - -

    img/euclidean.png -

    -
    - -
    -

    15.1 Ordinary    notes

    -
    - -
      -
    • Distance as you probably learned in grade school -
      • -
    - - -
    -
    - -
    - -
    -

    16 Manhattan distance    slide

    -
    - -
      -
    • How many blocks would you need to walk between two points? -
      • -
    - -

    img/manhattan.png -

    -
    - -
    -

    16.1 Usefulness    notes

    -
    - -
      -
    • Obviously useful for maps/directions -
      • -
    • But haven't seen it used much beyond that -
      • -
    - - -
    -
    - -
    - -
    -

    17 Lp norm    slide

    -
    - -
      -
    • Euclidean distance and Manhattan can be generalized -
      • -
    • Euclidean distance referred to as L2 norm -
      • -
    • Chebyshev distance is L∞ -
      • -
    - -

    img/lp-norm.png -

    -
    - -
    -

    17.1 Lp space    notes

    -
    - -
      -
    • Important for signal processing, math, other applications -
      • -
    • You may want to study these distances for comparing wave forms, like audio -
      • -
    - - -
    -
    - -
    - -
    -

    18 Ordinal Distance    slide

    -
    - -
      -
    • Normalize the ordinal rankings -
      • -
    • Use a numerical distance metric -
      • -
    - - -
    - -
    - -
    -

    19 Cosine Similarity    slide

    -
    - -
      -
    • Jaccard similarity can work well for sets of roughly equal size -
      • -
    • How to compare sets with a large difference in magnitude? -
      • -
    • Model them as vectors, take the cosign of the angle between -
      • -
    - -

    img/cosine-similarity.png -

    -
    - -
    -

    19.1 Cosign    notes

    -
    - - - - -
    -
    - -
    - -
    -

    20 Cosine Example    slide

    -
    - -
      -
    • "Jim likes pizza" | "Shreyas likes pizza" -
      • -
    - - - - - - - -
    -
    -
    - -
    -

    Date: 2013-02-08 13:46:10 PST

    -

    Author: Jim Blomo

    -

    Org version 7.8.02 with Emacs version 23

    -Validate XHTML 1.0 - -
    - - diff --git a/slides/2013-02-08-Probability.org b/slides/2013-02-08-Probability.org deleted file mode 100644 index 7a77857..0000000 --- a/slides/2013-02-08-Probability.org +++ /dev/null @@ -1,214 +0,0 @@ -* Probability :slide: - -* Nomenclature :slide: - + Record :: a single entity or concept. Also: data object, sample, example, - instance, data point - + Feature :: a characteristic or way of describing a record. Also: attribute, - dimension, variable, signal -** Slightly different from book :notes: - + The meanings do carry different connotations, but are generally - transferable - + Eg. dimensions is usually used in the math domain - + Feature is usually used in the ML domain - -** Feautre Types :slide: - + Binary :: True/False. Also: 0/1 - + Numeric :: Involving numbers. Also: integer, float, double - + Ordinal :: Feature with sortable values. - + Discrete :: countable, finite set. Also: classes - + Continuous :: unbounded numeric number. Also: integer, float, double - + Enumerated :: feature named, discrete values. Also: nominal, classed -*** Rain data set :notes: - + Stored did/dot not rain - + Stored how many inches it rained - + Stored the day as an integer offset from Jan 1 1970 - + Stored weather information: Sunny, Partly Sunny, Cloudy, Rainy - + Stored barometer reading - + Stored day of the week - -* Central Tendency :slide: -[[file:img/skew-normal.png]] -** Define :notes: - + Mean :: "average" all data points divided by size of set - + Median :: middle value - + Mode :: The value most likely to be picked - + discrete :: most common value - + continuous :: max probability density function - -** Skew Positive :slide: -[[file:img/skew-positive.png]] -** Skew :notes: - + Think about =mean - mode= - + Or think about where the "tail" is - -** Skew Negative :slide: -[[file:img/skew-negative.png]] - -* The Long Tail :slide:two_col: -[[file:img/Long_tail.svg.png]] - + Most popular are *very* popular - + Everything else, not so much - + But there's a lot of everything else -** Movies :notes: - + Current releases: millions of people watching - + Older movies are rented by < 1 person a week - + What is the skew? - + Power law distribution (please follow up on Wikipedia or a stats class) - + Distributions are important, but will only be covered as necessary - -* Dispersion :slide: -[[file:img/dispersion.png]] - + Centrality not the whole story -** Differences :notes: - + Wildly different data sets can still share many of these characteristics - -* Quartiles :slide: -[[file:img/quartiles.png]] -** Parts :notes: - + Go back to our unskewed normal distribution - + Quartiles divide the data into quarters - + InterQuartile Range is the distance of the middle two quartiles - + BoxPlot is one of the most useful tools for data. For public results, I - almost never want to see scatter plot or bar charts. I want to see box - plots. - + Bottom, we spit it up into standard deviations - + Variance measures, on average, how far points are away from the mean - + Standard deviation is the square root of the variance - -* Standard Deviation :slide: - [[file:img/stddev.png]] - + Within 1: 68% - + Within 2: 95% - + Within 3: 99.7% -** Standard Deviation :notes: - + Useful for thinking about what % of outliers you'd like to catch - + We use it for alerting: let us know when we're 2 stddev away from the - median, there's a very small likelihood of that happening - -* Visualization Tools :slide: - + Python: Matplotlib - + R: builtin - + Matlab: builtin - + Octave: builtin (gnuplot) - + HTML: D3.js -** Covered later :notes: - + Chapter 2 is going to cover some visualization stuff - + We're going to cover visualization a bit later in the course, and more of a - "how its done in industry" - + There is another class on visualization in general - -* Mathmatical Representation :slide: -| | Bad Boys | Robin Hood | Waterworld | -| Prabha | 1 | 3 | 2 | -| AJ | 5 | 4 | 3 | -| Victor | 4 | 4 | 1 | -#+begin_src octave -[ 1 3 2 - 5 4 3 - 4 4 1 ] -#+end_src -** Matrix :notes: - + Matrix representations very powerful, as we'll see later in class - + Usually rows are records, columns are attributes - + Sometimes you can think of data in different ways, can take the transpose - of the matrix to get attributes about movies - -* Waterworld :slide: - [[file:img/waterworld.jpg]] - -* Similarity | Distance :slide: - + Two sides of the same coin - + =similarity = 1 - distance= - + We'll use these metrics for many other algorithms -** Core Concept :notes: - + Many data mining techniques rely on finding a way to quantify similarity - + When you think about questions like "how similar are two users?" "is this - text plagiarism?" "are these products likely to be purchased together?" - + All are ways of thinking about similarity - -* Nominal Distance :slide: - + Ratio of mismatches to potential matches - + Why can't we take the sum of the mismatches? -** Nominal :notes: - + Nominal means we can't compare two values: there is no ordering - + All we can do is take ratio of the ones that are exactly the same - + The book describes how to think about this in terms of matrices - -* Binary Distance :slide: - + Could use Nominal Distance: count all exact matches or mismatches - + Could use Numeric Distance: just treat values as 0/1 - + asymmetric binary dissimilarity: don't care about *negative matches* - + =mismatches / (positive_matches + mismatches)= - + asymmetric binary similarity: care more about *positive matches* than mismatches - + =positive_matches / (positive_matches + mismatches)= -** Binary :notes: - + Nominal problem: for rare attributes, like a disease, two people who - *don't* have the disease, aren't necessarily very similar - -* Jaccard Coefficient :slide: - + Asymmetric binary similarity - + More commonly used for calculating set similarity - + =|intersection| / |union|= - + "Jim likes pizza" | "Shreyas likes pizza" -** Jaccard :notes: - 1. Break up into a set - 1. calculate # in intersection - 1. calculate # in union - 1. divide - -* Euclidean distance :slide: - + Straight line between two points - + Again: usually considered with just (x,y), but can calculate for any number - of dimensions - [[file:img/euclidean.png]] -** Ordinary :notes: - + Distance as you probably learned in grade school - -* Manhattan distance :slide: - + How many blocks would you need to walk between two points? - [[file:img/manhattan.png]] -** Usefulness :notes: - + Obviously useful for maps/directions - + But haven't seen it used much beyond that - -* L_p norm :slide: - + Euclidean distance and Manhattan can be generalized - + Euclidean distance referred to as L_2 norm - + Chebyshev distance is L_∞ - [[file:img/lp-norm.png]] -** L_p space :notes: - + Important for signal processing, math, other applications - + You may want to study these distances for comparing wave forms, like audio - -* Ordinal Distance :slide: - + Normalize the ordinal rankings - + Use a numerical distance metric - -* Cosine Similarity :slide: - + Jaccard similarity can work well for sets of roughly equal size - + How to compare sets with a large difference in magnitude? - + Model them as vectors, take the cosign of the angle between - [[file:img/cosine-similarity.png]] -** Cosign :notes: - + Why cosine? Hint: nomalization - + img: http://cs.carleton.edu/cs_comps/0910/netflixprize/final_results/knn/index.html - -* Cosine Example :slide: - + "Jim likes pizza" | "Shreyas likes pizza" - -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: - -#+BEGIN_HTML - -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: diff --git a/slides/2014-02-06-Probability.html b/slides/2014-02-06-Probability.html new file mode 100644 index 0000000..65d67d2 --- /dev/null +++ b/slides/2014-02-06-Probability.html @@ -0,0 +1,514 @@ + + + + Data Mining + + + + + + + + + diff --git a/slides/2014-02-06-Probability.markdown b/slides/2014-02-06-Probability.markdown new file mode 100644 index 0000000..788c82f --- /dev/null +++ b/slides/2014-02-06-Probability.markdown @@ -0,0 +1,373 @@ +name: inverse +layout: true +class: left, top, inverse + +--- + +# Probability + +--- + +## Nomenclature + +### Record + +a single entity or concept. Also: data object, sample, example, +instance, data point + +### Feature + +a characteristic or way of describing a record. Also: attribute, +dimension, variable, signal + +??? + +## Slightly different from book + + + The meanings do carry different connotations, but are generally + transferable + + e.g., dimensions is usually used in the math domain + + Feature is usually used in the ML domain + +--- + +## Feautre Types + + + Binary: True/False. Also: 0/1 + + Numeric: Involving numbers. Also: integer, float, double + + Ordinal: Feature with sortable values. + + Discrete: countable, finite set. Also: classes + + Continuous: unbounded numeric number. Also: integer, float, double + + Enumerated: feature named, discrete values. Also: nominal, classed + +??? + +## Rain data set + + + Stored did/dot not rain + + Stored how many inches it rained + + Stored the day as an integer offset from Jan 1 1970 + + Stored weather information: Sunny, Partly Sunny, Cloudy, Rainy + + Stored barometer reading + + Stored day of the week + +--- + +## Central Tendency + + + +??? + +## Define + + + Mean: "average" all data points divided by size of set + + Median: middle value + + Mode: The value most likely to be picked + + discrete: most common value + + continuous: max probability density function + +--- + +## Skew Positive + + + +??? + +## Skew + + + Think about ```mean - mode``` + + Or think about where the "tail" is + +--- + +## Skew Negative + + + +--- + +## The Long Tail + + + Most popular are *very* popular + + Everything else, not so much + + But there's a lot of everything else + + + +??? + +## Movies + + + Current releases: millions of people watching + + Older movies are rented by < 1 person a week + + What is the skew? + + Power law distribution (please follow up on Wikipedia or a stats class) + + Distributions are important, but will only be covered as necessary + +--- + +*Todo: image assumes light background* + +## Dispersion + + + Centrality not the whole story + + + +??? + +## Differences + + + Wildly different data sets can still share many of these characteristics + +--- + +*Todo: image assumes light background* + +## Quartiles + + + +??? + +## Parts + + + Go back to our unskewed normal distribution + + Quartiles divide the data into quarters + + InterQuartile Range is the distance of the middle two quartiles + + BoxPlot is one of the most useful tools for data. For public results, I + almost never want to see scatter plot or bar charts. I want to see box + plots. + + Bottom, we spit it up into standard deviations + + Variance measures, on average, how far points are away from the mean + + Standard deviation is the square root of the variance + +--- + +*Todo: image assumes light background* + +## Standard Deviation + + + Within 1: 68% + + Within 2: 95% + + Within 3: 99.7% + + + +??? + +## Standard Deviation + + + Useful for thinking about what % of outliers you'd like to catch + + We use it for alerting: let us know when we're 2 stddev away from the + median, there's a very small likelihood of that happening + +--- + +## Visualization Tools + + + Python: Matplotlib + + R: builtin + + Matlab: builtin + + Octave: builtin (gnuplot) + + HTML: D3.js + +??? + +## Covered later + + + Chapter 2 is going to cover some visualization stuff + + We're going to cover visualization a bit later in the course, and more of a + "how its done in industry" + + There is another class on visualization in general + +--- + +## Mathmatical Representation +  + +| | Bad Boys | Robin Hood | Waterworld | +|--------|----------|------------|------------| +| Prabha | 1 | 3 | 2 | +| AJ | 5 | 4 | 3 | +| Victor | 4 | 4 | 1 | + +  + +```octave +[ 1 3 2 + 5 4 3 + 4 4 1 ] +``` + +??? + +## Matrix + + + Matrix representations very powerful, as we'll see later in class + + Usually rows are records, columns are attributes + + Sometimes you can think of data in different ways, can take the transpose + of the matrix to get attributes about movies + +--- + +*Todo: waterworld?* + +## Waterworld + + + +--- + +## Similarity | Distance + + + Two sides of the same coin + + ```similarity = 1 - distance``` + + We'll use these metrics for many other algorithms + +??? + +## Core Concept + + + Many data mining techniques rely on finding a way to quantify similarity + + When you think about questions like "how similar are two users?" "is this + text plagiarism?" "are these products likely to be purchased together?" + + All are ways of thinking about similarity + +--- + +## Nominal Distance + + + Ratio of mismatches to potential matches + + Why can't we take the sum of the mismatches? + +??? + +## Nominal + + + Nominal means we can't compare two values: there is no ordering + + All we can do is take ratio of the ones that are exactly the same + + The book describes how to think about this in terms of matrices + +--- + +## Binary Distance + + + Could use Nominal Distance: count all exact matches or mismatches + + Could use Numeric Distance: just treat values as 0/1 + + asymmetric binary dissimilarity: don't care about *negative matches* + + ```mismatches / (positive_matches + mismatches)``` + + asymmetric binary similarity: care more about *positive matches* than mismatches + + ```positive_matches / (positive_matches + mismatches)``` + +??? + +## Binary + + + Nominal problem: for rare attributes, like a disease, two people who + *don't* have the disease, aren't necessarily very similar + +--- + +## Jaccard Coefficient + + + Asymmetric binary similarity + + More commonly used for calculating set similarity + + ```|intersection| / |union|``` + + "Jimmy likes pizza" | "Shreyas likes pizza" + +??? + +## Jaccard + + 1. Break up into a set + 1. calculate # in intersection + 1. calculate # in union + 1. divide + +--- + +*Todo: image assumes light background* + +## Euclidean distance + + + Straight line between two points + + Again: usually considered with just (x,y), but can calculate for any number + of dimensions + + + +??? + +## Ordinary + + + Distance as you probably learned in grade school + +--- + +*Todo: image assumes light background* + +## Manhattan distance + + + How many blocks would you need to walk between two points? + + + +??? + +## Usefulness + + Obviously useful for maps/directions + + But haven't seen it used much beyond that + +--- + +*Todo: image assumes light background* + +*Todo: math symbols* + +## L_p norm + + + Euclidean distance and Manhattan can be generalized + + Euclidean distance referred to as L_2 norm + + Chebyshev distance is L_∞ + + + +??? + +## L_p space + + + Important for signal processing, math, other applications + + You may want to study these distances for comparing wave forms, like audio + +--- + +## Ordinal Distance + + + Normalize the ordinal rankings + + Use a numerical distance metric + +--- + +## Cosine Similarity + + + Jaccard similarity can work well for sets of roughly equal size + + How to compare sets with a large difference in magnitude? + + Model them as vectors, take the cosign of the angle between + + + +??? + +## Cosine + + + Why cosine? Hint: normalization + + img: http://cs.carleton.edu/cs_comps/0910/netflixprize/final_results/knn/index.html + +--- + +## Cosine Example + + + "Jimmy likes pizza" | "Shreyas likes pizza" From 82ef493470804288407d45103dd29d9f1a19e906 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Mon, 20 Jan 2014 14:19:20 -0800 Subject: [PATCH 17/79] moved old decks to uncoverted directory --- .../2013-02-15-Data-Warehouse.html | 0 .../{ => unconverted}/2013-02-15-Data-Warehouse.org | 0 slides/{ => unconverted}/2013-02-15-MapReduce.html | 0 slides/{ => unconverted}/2013-02-15-MapReduce.org | 0 slides/{ => unconverted}/2013-02-15-Project.html | 0 slides/{ => unconverted}/2013-02-15-Project.org | 0 slides/{ => unconverted}/2013-02-15-mrjob.html | 0 slides/{ => unconverted}/2013-02-15-mrjob.org | 0 slides/{ => unconverted}/2013-02-22-Bayes.html | 0 slides/{ => unconverted}/2013-02-22-Bayes.org | 0 .../2013-02-22-Decision-Trees.html | 0 .../{ => unconverted}/2013-02-22-Decision-Trees.org | 0 slides/{ => unconverted}/2013-02-22-Gini.html | 0 slides/{ => unconverted}/2013-02-22-Gini.org | 0 slides/{ => unconverted}/2013-03-01-Lab-NN.html | 0 slides/{ => unconverted}/2013-03-01-Lab-NN.org | 0 .../2013-03-01-Neural-Network.html | 0 .../{ => unconverted}/2013-03-01-Neural-Network.org | 0 slides/{ => unconverted}/2013-03-01-SVM.html | 0 slides/{ => unconverted}/2013-03-01-SVM.org | 0 slides/{ => unconverted}/2013-03-07-Clustering.html | 0 slides/{ => unconverted}/2013-03-07-Clustering.org | 0 .../{ => unconverted}/2013-03-07-Hierarchical.html | 0 .../{ => unconverted}/2013-03-07-Hierarchical.org | 0 slides/{ => unconverted}/2013-03-07-k-means.html | 0 slides/{ => unconverted}/2013-03-07-k-means.org | 0 .../2013-03-15-Advanced-Cluster.html | 0 .../2013-03-15-Advanced-Cluster.org | 0 slides/{ => unconverted}/2013-03-15-Review.html | 0 slides/{ => unconverted}/2013-03-15-Review.org | 0 slides/{ => unconverted}/2013-04-05-AWS.html | 0 slides/{ => unconverted}/2013-04-05-AWS.org | 0 .../2013-04-05-Frequent-Pattern.html | 0 .../2013-04-05-Frequent-Pattern.org | 0 .../2013-04-12-AdjacencyRepresentations.html | 0 .../2013-04-12-AdjacencyRepresentations.org | 0 slides/{ => unconverted}/2013-04-12-Graphs.html | 0 slides/{ => unconverted}/2013-04-12-Graphs.org | 0 slides/{ => unconverted}/2013-04-12-PageRank.html | 0 slides/{ => unconverted}/2013-04-12-PageRank.org | 0 slides/{ => unconverted}/2013-04-19-Elasticity.html | 0 slides/{ => unconverted}/2013-04-19-Elasticity.org | 0 slides/{ => unconverted}/2013-04-19-Nonlinear.pdf | Bin slides/{ => unconverted}/2013-04-26-Midterm-HW.html | 0 slides/{ => unconverted}/2013-04-26-Midterm-HW.org | 0 slides/{ => unconverted}/2013-04-26-Multimedia.html | 0 slides/{ => unconverted}/2013-04-26-Multimedia.org | 0 slides/{ => unconverted}/2013-04-26-Outliers.html | 0 slides/{ => unconverted}/2013-04-26-Outliers.org | 0 slides/{ => unconverted}/2013-05-03-D3.html | 0 slides/{ => unconverted}/2013-05-03-D3.org | 0 .../{ => unconverted}/2013-05-03-Visualization.html | 0 .../{ => unconverted}/2013-05-03-Visualization.org | 0 .../2013-05-03-Yelp-Visualization.html | 0 .../2013-05-03-Yelp-Visualization.org | 0 slides/{ => unconverted}/2013-05-10-Real-World.html | 0 slides/{ => unconverted}/2013-05-10-Real-World.org | 0 57 files changed, 0 insertions(+), 0 deletions(-) rename slides/{ => unconverted}/2013-02-15-Data-Warehouse.html (100%) rename slides/{ => unconverted}/2013-02-15-Data-Warehouse.org (100%) rename slides/{ => unconverted}/2013-02-15-MapReduce.html (100%) rename slides/{ => unconverted}/2013-02-15-MapReduce.org (100%) rename slides/{ => unconverted}/2013-02-15-Project.html (100%) rename slides/{ => unconverted}/2013-02-15-Project.org (100%) rename slides/{ => unconverted}/2013-02-15-mrjob.html (100%) rename slides/{ => unconverted}/2013-02-15-mrjob.org (100%) rename slides/{ => unconverted}/2013-02-22-Bayes.html (100%) rename slides/{ => unconverted}/2013-02-22-Bayes.org (100%) rename slides/{ => unconverted}/2013-02-22-Decision-Trees.html (100%) rename slides/{ => unconverted}/2013-02-22-Decision-Trees.org (100%) rename slides/{ => unconverted}/2013-02-22-Gini.html (100%) rename slides/{ => unconverted}/2013-02-22-Gini.org (100%) rename slides/{ => unconverted}/2013-03-01-Lab-NN.html (100%) rename slides/{ => unconverted}/2013-03-01-Lab-NN.org (100%) rename slides/{ => unconverted}/2013-03-01-Neural-Network.html (100%) rename slides/{ => unconverted}/2013-03-01-Neural-Network.org (100%) rename slides/{ => unconverted}/2013-03-01-SVM.html (100%) rename slides/{ => unconverted}/2013-03-01-SVM.org (100%) rename slides/{ => unconverted}/2013-03-07-Clustering.html (100%) rename slides/{ => unconverted}/2013-03-07-Clustering.org (100%) rename slides/{ => unconverted}/2013-03-07-Hierarchical.html (100%) rename slides/{ => unconverted}/2013-03-07-Hierarchical.org (100%) rename slides/{ => unconverted}/2013-03-07-k-means.html (100%) rename slides/{ => unconverted}/2013-03-07-k-means.org (100%) rename slides/{ => unconverted}/2013-03-15-Advanced-Cluster.html (100%) rename slides/{ => unconverted}/2013-03-15-Advanced-Cluster.org (100%) rename slides/{ => unconverted}/2013-03-15-Review.html (100%) rename slides/{ => unconverted}/2013-03-15-Review.org (100%) rename slides/{ => unconverted}/2013-04-05-AWS.html (100%) rename slides/{ => unconverted}/2013-04-05-AWS.org (100%) rename slides/{ => unconverted}/2013-04-05-Frequent-Pattern.html (100%) rename slides/{ => unconverted}/2013-04-05-Frequent-Pattern.org (100%) rename slides/{ => unconverted}/2013-04-12-AdjacencyRepresentations.html (100%) rename slides/{ => unconverted}/2013-04-12-AdjacencyRepresentations.org (100%) rename slides/{ => unconverted}/2013-04-12-Graphs.html (100%) rename slides/{ => unconverted}/2013-04-12-Graphs.org (100%) rename slides/{ => unconverted}/2013-04-12-PageRank.html (100%) rename slides/{ => unconverted}/2013-04-12-PageRank.org (100%) rename slides/{ => unconverted}/2013-04-19-Elasticity.html (100%) rename slides/{ => unconverted}/2013-04-19-Elasticity.org (100%) rename slides/{ => unconverted}/2013-04-19-Nonlinear.pdf (100%) rename slides/{ => unconverted}/2013-04-26-Midterm-HW.html (100%) rename slides/{ => unconverted}/2013-04-26-Midterm-HW.org (100%) rename slides/{ => unconverted}/2013-04-26-Multimedia.html (100%) rename slides/{ => unconverted}/2013-04-26-Multimedia.org (100%) rename slides/{ => unconverted}/2013-04-26-Outliers.html (100%) rename slides/{ => unconverted}/2013-04-26-Outliers.org (100%) rename slides/{ => unconverted}/2013-05-03-D3.html (100%) rename slides/{ => unconverted}/2013-05-03-D3.org (100%) rename slides/{ => unconverted}/2013-05-03-Visualization.html (100%) rename slides/{ => unconverted}/2013-05-03-Visualization.org (100%) rename slides/{ => unconverted}/2013-05-03-Yelp-Visualization.html (100%) rename slides/{ => unconverted}/2013-05-03-Yelp-Visualization.org (100%) rename slides/{ => unconverted}/2013-05-10-Real-World.html (100%) rename slides/{ => unconverted}/2013-05-10-Real-World.org (100%) diff --git a/slides/2013-02-15-Data-Warehouse.html b/slides/unconverted/2013-02-15-Data-Warehouse.html similarity index 100% rename from slides/2013-02-15-Data-Warehouse.html rename to slides/unconverted/2013-02-15-Data-Warehouse.html diff --git a/slides/2013-02-15-Data-Warehouse.org b/slides/unconverted/2013-02-15-Data-Warehouse.org similarity index 100% rename from slides/2013-02-15-Data-Warehouse.org rename to slides/unconverted/2013-02-15-Data-Warehouse.org diff --git a/slides/2013-02-15-MapReduce.html b/slides/unconverted/2013-02-15-MapReduce.html similarity index 100% rename from slides/2013-02-15-MapReduce.html rename to slides/unconverted/2013-02-15-MapReduce.html diff --git a/slides/2013-02-15-MapReduce.org b/slides/unconverted/2013-02-15-MapReduce.org similarity index 100% rename from slides/2013-02-15-MapReduce.org rename to slides/unconverted/2013-02-15-MapReduce.org diff --git a/slides/2013-02-15-Project.html b/slides/unconverted/2013-02-15-Project.html similarity index 100% rename from slides/2013-02-15-Project.html rename to slides/unconverted/2013-02-15-Project.html diff --git a/slides/2013-02-15-Project.org b/slides/unconverted/2013-02-15-Project.org similarity index 100% rename from slides/2013-02-15-Project.org rename to slides/unconverted/2013-02-15-Project.org diff --git a/slides/2013-02-15-mrjob.html b/slides/unconverted/2013-02-15-mrjob.html similarity index 100% rename from slides/2013-02-15-mrjob.html rename to slides/unconverted/2013-02-15-mrjob.html diff --git a/slides/2013-02-15-mrjob.org b/slides/unconverted/2013-02-15-mrjob.org similarity index 100% rename from slides/2013-02-15-mrjob.org rename to slides/unconverted/2013-02-15-mrjob.org diff --git a/slides/2013-02-22-Bayes.html b/slides/unconverted/2013-02-22-Bayes.html similarity index 100% rename from slides/2013-02-22-Bayes.html rename to slides/unconverted/2013-02-22-Bayes.html diff --git a/slides/2013-02-22-Bayes.org b/slides/unconverted/2013-02-22-Bayes.org similarity index 100% rename from slides/2013-02-22-Bayes.org rename to slides/unconverted/2013-02-22-Bayes.org diff --git a/slides/2013-02-22-Decision-Trees.html b/slides/unconverted/2013-02-22-Decision-Trees.html similarity index 100% rename from slides/2013-02-22-Decision-Trees.html rename to slides/unconverted/2013-02-22-Decision-Trees.html diff --git a/slides/2013-02-22-Decision-Trees.org b/slides/unconverted/2013-02-22-Decision-Trees.org similarity index 100% rename from slides/2013-02-22-Decision-Trees.org rename to slides/unconverted/2013-02-22-Decision-Trees.org diff --git a/slides/2013-02-22-Gini.html b/slides/unconverted/2013-02-22-Gini.html similarity index 100% rename from slides/2013-02-22-Gini.html rename to slides/unconverted/2013-02-22-Gini.html diff --git a/slides/2013-02-22-Gini.org b/slides/unconverted/2013-02-22-Gini.org similarity index 100% rename from slides/2013-02-22-Gini.org rename to slides/unconverted/2013-02-22-Gini.org diff --git a/slides/2013-03-01-Lab-NN.html b/slides/unconverted/2013-03-01-Lab-NN.html similarity index 100% rename from slides/2013-03-01-Lab-NN.html rename to slides/unconverted/2013-03-01-Lab-NN.html diff --git a/slides/2013-03-01-Lab-NN.org b/slides/unconverted/2013-03-01-Lab-NN.org similarity index 100% rename from slides/2013-03-01-Lab-NN.org rename to slides/unconverted/2013-03-01-Lab-NN.org diff --git a/slides/2013-03-01-Neural-Network.html b/slides/unconverted/2013-03-01-Neural-Network.html similarity index 100% rename from slides/2013-03-01-Neural-Network.html rename to slides/unconverted/2013-03-01-Neural-Network.html diff --git a/slides/2013-03-01-Neural-Network.org b/slides/unconverted/2013-03-01-Neural-Network.org similarity index 100% rename from slides/2013-03-01-Neural-Network.org rename to slides/unconverted/2013-03-01-Neural-Network.org diff --git a/slides/2013-03-01-SVM.html b/slides/unconverted/2013-03-01-SVM.html similarity index 100% rename from slides/2013-03-01-SVM.html rename to slides/unconverted/2013-03-01-SVM.html diff --git a/slides/2013-03-01-SVM.org b/slides/unconverted/2013-03-01-SVM.org similarity index 100% rename from slides/2013-03-01-SVM.org rename to slides/unconverted/2013-03-01-SVM.org diff --git a/slides/2013-03-07-Clustering.html b/slides/unconverted/2013-03-07-Clustering.html similarity index 100% rename from slides/2013-03-07-Clustering.html rename to slides/unconverted/2013-03-07-Clustering.html diff --git a/slides/2013-03-07-Clustering.org b/slides/unconverted/2013-03-07-Clustering.org similarity index 100% rename from slides/2013-03-07-Clustering.org rename to slides/unconverted/2013-03-07-Clustering.org diff --git a/slides/2013-03-07-Hierarchical.html b/slides/unconverted/2013-03-07-Hierarchical.html similarity index 100% rename from slides/2013-03-07-Hierarchical.html rename to slides/unconverted/2013-03-07-Hierarchical.html diff --git a/slides/2013-03-07-Hierarchical.org b/slides/unconverted/2013-03-07-Hierarchical.org similarity index 100% rename from slides/2013-03-07-Hierarchical.org rename to slides/unconverted/2013-03-07-Hierarchical.org diff --git a/slides/2013-03-07-k-means.html b/slides/unconverted/2013-03-07-k-means.html similarity index 100% rename from slides/2013-03-07-k-means.html rename to slides/unconverted/2013-03-07-k-means.html diff --git a/slides/2013-03-07-k-means.org b/slides/unconverted/2013-03-07-k-means.org similarity index 100% rename from slides/2013-03-07-k-means.org rename to slides/unconverted/2013-03-07-k-means.org diff --git a/slides/2013-03-15-Advanced-Cluster.html b/slides/unconverted/2013-03-15-Advanced-Cluster.html similarity index 100% rename from slides/2013-03-15-Advanced-Cluster.html rename to slides/unconverted/2013-03-15-Advanced-Cluster.html diff --git a/slides/2013-03-15-Advanced-Cluster.org b/slides/unconverted/2013-03-15-Advanced-Cluster.org similarity index 100% rename from slides/2013-03-15-Advanced-Cluster.org rename to slides/unconverted/2013-03-15-Advanced-Cluster.org diff --git a/slides/2013-03-15-Review.html b/slides/unconverted/2013-03-15-Review.html similarity index 100% rename from slides/2013-03-15-Review.html rename to slides/unconverted/2013-03-15-Review.html diff --git a/slides/2013-03-15-Review.org b/slides/unconverted/2013-03-15-Review.org similarity index 100% rename from slides/2013-03-15-Review.org rename to slides/unconverted/2013-03-15-Review.org diff --git a/slides/2013-04-05-AWS.html b/slides/unconverted/2013-04-05-AWS.html similarity index 100% rename from slides/2013-04-05-AWS.html rename to slides/unconverted/2013-04-05-AWS.html diff --git a/slides/2013-04-05-AWS.org b/slides/unconverted/2013-04-05-AWS.org similarity index 100% rename from slides/2013-04-05-AWS.org rename to slides/unconverted/2013-04-05-AWS.org diff --git a/slides/2013-04-05-Frequent-Pattern.html b/slides/unconverted/2013-04-05-Frequent-Pattern.html similarity index 100% rename from slides/2013-04-05-Frequent-Pattern.html rename to slides/unconverted/2013-04-05-Frequent-Pattern.html diff --git a/slides/2013-04-05-Frequent-Pattern.org b/slides/unconverted/2013-04-05-Frequent-Pattern.org similarity index 100% rename from slides/2013-04-05-Frequent-Pattern.org rename to slides/unconverted/2013-04-05-Frequent-Pattern.org diff --git a/slides/2013-04-12-AdjacencyRepresentations.html b/slides/unconverted/2013-04-12-AdjacencyRepresentations.html similarity index 100% rename from slides/2013-04-12-AdjacencyRepresentations.html rename to slides/unconverted/2013-04-12-AdjacencyRepresentations.html diff --git a/slides/2013-04-12-AdjacencyRepresentations.org b/slides/unconverted/2013-04-12-AdjacencyRepresentations.org similarity index 100% rename from slides/2013-04-12-AdjacencyRepresentations.org rename to slides/unconverted/2013-04-12-AdjacencyRepresentations.org diff --git a/slides/2013-04-12-Graphs.html b/slides/unconverted/2013-04-12-Graphs.html similarity index 100% rename from slides/2013-04-12-Graphs.html rename to slides/unconverted/2013-04-12-Graphs.html diff --git a/slides/2013-04-12-Graphs.org b/slides/unconverted/2013-04-12-Graphs.org similarity index 100% rename from slides/2013-04-12-Graphs.org rename to slides/unconverted/2013-04-12-Graphs.org diff --git a/slides/2013-04-12-PageRank.html b/slides/unconverted/2013-04-12-PageRank.html similarity index 100% rename from slides/2013-04-12-PageRank.html rename to slides/unconverted/2013-04-12-PageRank.html diff --git a/slides/2013-04-12-PageRank.org b/slides/unconverted/2013-04-12-PageRank.org similarity index 100% rename from slides/2013-04-12-PageRank.org rename to slides/unconverted/2013-04-12-PageRank.org diff --git a/slides/2013-04-19-Elasticity.html b/slides/unconverted/2013-04-19-Elasticity.html similarity index 100% rename from slides/2013-04-19-Elasticity.html rename to slides/unconverted/2013-04-19-Elasticity.html diff --git a/slides/2013-04-19-Elasticity.org b/slides/unconverted/2013-04-19-Elasticity.org similarity index 100% rename from slides/2013-04-19-Elasticity.org rename to slides/unconverted/2013-04-19-Elasticity.org diff --git a/slides/2013-04-19-Nonlinear.pdf b/slides/unconverted/2013-04-19-Nonlinear.pdf similarity index 100% rename from slides/2013-04-19-Nonlinear.pdf rename to slides/unconverted/2013-04-19-Nonlinear.pdf diff --git a/slides/2013-04-26-Midterm-HW.html b/slides/unconverted/2013-04-26-Midterm-HW.html similarity index 100% rename from slides/2013-04-26-Midterm-HW.html rename to slides/unconverted/2013-04-26-Midterm-HW.html diff --git a/slides/2013-04-26-Midterm-HW.org b/slides/unconverted/2013-04-26-Midterm-HW.org similarity index 100% rename from slides/2013-04-26-Midterm-HW.org rename to slides/unconverted/2013-04-26-Midterm-HW.org diff --git a/slides/2013-04-26-Multimedia.html b/slides/unconverted/2013-04-26-Multimedia.html similarity index 100% rename from slides/2013-04-26-Multimedia.html rename to slides/unconverted/2013-04-26-Multimedia.html diff --git a/slides/2013-04-26-Multimedia.org b/slides/unconverted/2013-04-26-Multimedia.org similarity index 100% rename from slides/2013-04-26-Multimedia.org rename to slides/unconverted/2013-04-26-Multimedia.org diff --git a/slides/2013-04-26-Outliers.html b/slides/unconverted/2013-04-26-Outliers.html similarity index 100% rename from slides/2013-04-26-Outliers.html rename to slides/unconverted/2013-04-26-Outliers.html diff --git a/slides/2013-04-26-Outliers.org b/slides/unconverted/2013-04-26-Outliers.org similarity index 100% rename from slides/2013-04-26-Outliers.org rename to slides/unconverted/2013-04-26-Outliers.org diff --git a/slides/2013-05-03-D3.html b/slides/unconverted/2013-05-03-D3.html similarity index 100% rename from slides/2013-05-03-D3.html rename to slides/unconverted/2013-05-03-D3.html diff --git a/slides/2013-05-03-D3.org b/slides/unconverted/2013-05-03-D3.org similarity index 100% rename from slides/2013-05-03-D3.org rename to slides/unconverted/2013-05-03-D3.org diff --git a/slides/2013-05-03-Visualization.html b/slides/unconverted/2013-05-03-Visualization.html similarity index 100% rename from slides/2013-05-03-Visualization.html rename to slides/unconverted/2013-05-03-Visualization.html diff --git a/slides/2013-05-03-Visualization.org b/slides/unconverted/2013-05-03-Visualization.org similarity index 100% rename from slides/2013-05-03-Visualization.org rename to slides/unconverted/2013-05-03-Visualization.org diff --git a/slides/2013-05-03-Yelp-Visualization.html b/slides/unconverted/2013-05-03-Yelp-Visualization.html similarity index 100% rename from slides/2013-05-03-Yelp-Visualization.html rename to slides/unconverted/2013-05-03-Yelp-Visualization.html diff --git a/slides/2013-05-03-Yelp-Visualization.org b/slides/unconverted/2013-05-03-Yelp-Visualization.org similarity index 100% rename from slides/2013-05-03-Yelp-Visualization.org rename to slides/unconverted/2013-05-03-Yelp-Visualization.org diff --git a/slides/2013-05-10-Real-World.html b/slides/unconverted/2013-05-10-Real-World.html similarity index 100% rename from slides/2013-05-10-Real-World.html rename to slides/unconverted/2013-05-10-Real-World.html diff --git a/slides/2013-05-10-Real-World.org b/slides/unconverted/2013-05-10-Real-World.org similarity index 100% rename from slides/2013-05-10-Real-World.org rename to slides/unconverted/2013-05-10-Real-World.org From 8547f414f00e970b2304e7ab24ed3ada06280be8 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Wed, 22 Jan 2014 09:34:22 -0800 Subject: [PATCH 18/79] cleaned up readme links --- README.markdown | 41 ++++++++++++++++++++++------------------- 1 file changed, 22 insertions(+), 19 deletions(-) diff --git a/README.markdown b/README.markdown index e70200b..438367c 100644 --- a/README.markdown +++ b/README.markdown @@ -14,27 +14,30 @@ Shreyas <shreyas@ischool> ### Textbook Han, J., Kamber, M., & Pei, J. (2011). _Data Mining: Concepts and Techniques_, Third Edition *(3rd ed.)*. Morgan Kaufmann. +### Course Discussion +[Info 290T: Data Mining on Piazza](https://piazza.com/berkeley/spring2014/info290t03) + --- # Syllabus DM[0-9]+ indicates chapters from the text, _Data Mining_. -| Date | Readings | Slides | Homework / Project | -|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------| -| Jan 23 | [Try Github](http://try.github.com) ; [A Taxonomy of Data Science](http://www.dataists.com/2010/09/a-taxonomy-of-data-science/) | [Class Intro](file:slides/2014-01-23-Intro.html) ; Tools Intro by /GUEST: Shreyas/ | [Git Intro](https://github.com/seekshreyas/Introduction-to-Git-Github) | -| Jan 30 | DM1 ; [The Yelp Factor: Are Consumer Reviews Good for Business?](http://hbswk.hbs.edu/item/6836.html) | [Case Studies](file:slides/2014-01-30-CaseStudies.html) ; [Obtaining Data](file:slides/2014-01-30-Obtaining-Data.html) | [Obtain & Explore Data](file:slides/2014-01-30-Lab.html) | -| Feb 6 | DM2, DM3 | [Probability](file:slides/2014-02-06-Probability.html) ; [Preprocessing](file:slides/2014-02-06-Preprocessing.html) | [Data Stats](file:slides/2014-02-06-Lab.html) | -| Feb 13 | DM4, [Apache Hadoop: Petabytes and Terawatts](http://www.youtube.com/watch?v=SS27F-hYWfU) ([slides](http://prezi.com/u0ukvqzpyh5p/apache-hadoop-petabytes-and-terawatts/)); [mrjob docs](http://packages.python.org/mrjob/) (for homework) | [Data Warehouse](file:slides/2014-02-13-Data-Warehouse.html) ; [MapReduce](file:slides/2014-02-13-MapReduce.html) | [Project Details](file:slides/2014-02-13-Project.html) ; [mrjob](file:slides/2014-02-13-mrjob.html) | -| Feb 20 | DM8 | [Decision Trees](file:slides/2014-02-20-Decision-Trees.html); [Naive Bayes](file:slides/2014-02-20-Bayes.html) | [Gini Index](file:slides/2014-02-20-Gini.html) | -| Feb 27 | DM[9.1-9.3], 9.5 ; [Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html) | [SVM](file:slides/2014-02-27-SVM.html) ; [Neural Networks](file:slides/2014-02-27-Neural-Network.html) | [Neural Network Back Propagation](file:slides/2014-02-27-Lab-NN.html) | -| Mar 6 | DM10 | [Agglomerative - Clustering](file:slides/2014-03-06-Clustering.html) ; [Hierarchical, Density - Clustering](file:slides/2014-03-06-Hierarchical.html) | [K-Means](file:slides/2014-03-06-k-means.html) | -| Mar 13 | DM11.1 | [Review](file:slides/2014-03-13-Review.html) | prepare 1 cheat sheet | -| Mar 20 | 1 cheat sheet | *Midterm* | | -| Mar 27 | HOLIDAY | | | -| Apr 3 | DM6 | [Advanced Clustering](file:slides/2014-03-13-Advanced-Cluster.html) ; [Frequent Pattern](file:slides/2014-04-03-Frequent-Pattern.html) | [AWS](file:slides/2014-04-03-AWS.html) ; Project Proposal Due | -| Apr 10 | DM11.3; [PageRank](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf); [Uncovering Social Network Sybils in the Wild](http://arxiv.org/pdf/1106.5321) | [Graphs](file:slides/2014-04-10-Graphs.html); [PageRank](file:slides/2014-04-10-PageRank.html) | [Adjacency Representations](file:slides/2014-04-10-AdjacencyRepresentations.html) | -| Apr 17 | [Non-linear regression](file:slides/2014-04-17-Nonlinear.pdf) | GUEST: Gene Lee Ceaser's [Pricing Strategy](file:slides/RM Pricing Strategy.ppt); [Ceaser's Recruiting](file:slides/Campus Recruiting Deck_2012_UC Berkeley.ppt) | [Price Elasticity](file:slides/2014-04-17-Elasticity.html) | -| Apr 24 | DM12; [Shazam Audio Search](http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf) | [Outliers](file:slides/2014-04-24-Outliers.html); [Images & Audio](file:slides/2014-04-24-Multimedia.html) | [Midterm Review](file:slides/2014-04-24-Midterm-HW.html) | -| May 1 | [Embedded Plots](https://groups.google.com/group/gsofgs/attach/2f1cdd7a999c3ad8/embedded-plots.pdf?part=2&authuser=0) ; [Data-Driven Documents](http://vis.stanford.edu/files/2011-D3-InfoVis.pdf) | [Visualization]file:slides/2014-05-01-Visualization.html) ; [Yelp's Visualizations](file:slides/2014-05-01-Yelp-Visualization.html) | [D3 Intro](http://vogievetsky.github.io/IntroD3/); [D3 Lab](file:slides/2014-05-01-D3.html) | -| May 18 | [A Few Useful Things to Know about Machine Learning](http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) ; [Top 10 Algorithms in Data Mining](http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf) | [In Real Life](file:slides/2014-05-08-Real-World.html) ; Presentations | May 14th: Project Papers Due | -| May 15 | | Final Presentation | Bye! | +| Date | Readings | Slides | Homework / Project | +|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------| +| Jan 23 | [Try Github](http://try.github.com) ; [A Taxonomy of Data Science](http://www.dataists.com/2010/09/a-taxonomy-of-data-science/) | [Class Intro](slides/2014-01-23-Intro.html) ; Tools Intro by GUEST: Shreyas | [Git Intro](https://github.com/seekshreyas/Introduction-to-Git-Github) | +| Jan 30 | DM1 ; [The Yelp Factor: Are Consumer Reviews Good for Business?](http://hbswk.hbs.edu/item/6836.html) | [Case Studies](slides/2014-01-30-CaseStudies.html) ; [Obtaining Data](slides/2014-01-30-Obtaining-Data.html) | [Obtain & Explore Data](slides/2014-01-30-Lab.html) | +| Feb 6 | DM2, DM3 | [Probability](slides/2014-02-06-Probability.html) ; [Preprocessing](slides/2014-02-06-Preprocessing.html) | [Data Stats](slides/2014-02-06-Lab.html) | +| Feb 13 | DM4, [Apache Hadoop: Petabytes and Terawatts](http://www.youtube.com/watch?v=SS27F-hYWfU) ([slides](http://prezi.com/u0ukvqzpyh5p/apache-hadoop-petabytes-and-terawatts/)); [mrjob docs](http://packages.python.org/mrjob/) (for homework) | [Data Warehouse](slides/2014-02-13-Data-Warehouse.html) ; [MapReduce](slides/2014-02-13-MapReduce.html) | [Project Details](slides/2014-02-13-Project.html) ; [mrjob](slides/2014-02-13-mrjob.html) | +| Feb 20 | DM8 | [Decision Trees](slides/2014-02-20-Decision-Trees.html); [Naive Bayes](slides/2014-02-20-Bayes.html) | [Gini Index](slides/2014-02-20-Gini.html) | +| Feb 27 | DM[9.1-9.3], 9.5 ; [Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html) | [SVM](slides/2014-02-27-SVM.html) ; [Neural Networks](slides/2014-02-27-Neural-Network.html) | [Neural Network Back Propagation](slides/2014-02-27-Lab-NN.html) | +| Mar 6 | DM10 | [Agglomerative - Clustering](slides/2014-03-06-Clustering.html) ; [Hierarchical, Density - Clustering](slides/2014-03-06-Hierarchical.html) | [K-Means](slides/2014-03-06-k-means.html) | +| Mar 13 | DM11.1 | [Review](slides/2014-03-13-Review.html) | prepare 1 cheat sheet | +| Mar 20 | 1 cheat sheet | *Midterm* | | +| Mar 27 | HOLIDAY | | | +| Apr 3 | DM6 | [Advanced Clustering](slides/2014-03-13-Advanced-Cluster.html) ; [Frequent Pattern](slides/2014-04-03-Frequent-Pattern.html) | [AWS](slides/2014-04-03-AWS.html) ; Project Proposal Due | +| Apr 10 | DM11.3; [PageRank](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf); [Uncovering Social Network Sybils in the Wild](http://arxiv.org/pdf/1106.5321) | [Graphs](slides/2014-04-10-Graphs.html); [PageRank](slides/2014-04-10-PageRank.html) | [Adjacency Representations](slides/2014-04-10-AdjacencyRepresentations.html) | +| Apr 17 | [Non-linear regression](slides/2014-04-17-Nonlinear.pdf) | GUEST: Gene Lee Ceaser's [Pricing Strategy](slides/RM Pricing Strategy.ppt); [Ceaser's Recruiting](slides/Campus Recruiting Deck_2012_UC Berkeley.ppt) | [Price Elasticity](slides/2014-04-17-Elasticity.html) | +| Apr 24 | DM12; [Shazam Audio Search](http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf) | [Outliers](slides/2014-04-24-Outliers.html); [Images & Audio](slides/2014-04-24-Multimedia.html) | [Midterm Review](slides/2014-04-24-Midterm-HW.html) | +| May 1 | [Embedded Plots](https://groups.google.com/group/gsofgs/attach/2f1cdd7a999c3ad8/embedded-plots.pdf?part=2&authuser=0) ; [Data-Driven Documents](http://vis.stanford.edu/files/2011-D3-InfoVis.pdf) | [Visualization]slides/2014-05-01-Visualization.html) ; [Yelp's Visualizations](slides/2014-05-01-Yelp-Visualization.html) | [D3 Intro](http://vogievetsky.github.io/IntroD3/); [D3 Lab](slides/2014-05-01-D3.html) | +| May 18 | [A Few Useful Things to Know about Machine Learning](http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) ; [Top 10 Algorithms in Data Mining](http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf) | [In Real Life](slides/2014-05-08-Real-World.html) ; Presentations | May 14th: Project Papers Due | +| May 15 | | Final Presentation | Bye! | From 4a2eb2827a94a6f9a8e76e19d26ee10980a10ccc Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Wed, 22 Jan 2014 09:50:14 -0800 Subject: [PATCH 19/79] Update index.html --- index.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/index.html b/index.html index e6694e2..b0ae879 100644 --- a/index.html +++ b/index.html @@ -107,7 +107,7 @@

    2 Syllabus   &nbs

- +

Date: 2013-05-10 00:54:43 PDT

Author: Jim Blomo

From 7b80a5302bba4596cc411eeba276b2425bb84fa5 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Wed, 22 Jan 2014 13:22:38 -0800 Subject: [PATCH 20/79] making github.io site functional --- Makefile | 11 ++ README.markdown | 4 +- index.html | 143 ++++++++++++++++++ slides/Makefile | 9 -- ...tion_template.html => slide_template.html} | 0 5 files changed, 157 insertions(+), 10 deletions(-) create mode 100644 Makefile create mode 100644 index.html delete mode 100644 slides/Makefile rename slides/{presentation_template.html => slide_template.html} (100%) diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..cb841c2 --- /dev/null +++ b/Makefile @@ -0,0 +1,11 @@ +PLACE_HOLDER=<\!--markdown goes here--> +SLIDE_SOURCES=$(wildcard slides/*.markdown) +SLIDES=$(SLIDE_SOURCES:.markdown=.html) + +all: index.html $(SLIDES) + +index.html: README.markdown + pandoc README.markdown -t html -c slides/production/common.css > index.html + +%.html: %.markdown slides/slide_template.html + sed -e "/$(PLACE_HOLDER)/r $<" < slides/slide_template.html | sed -e "s/$(PLACE_HOLDER)//" > $@ diff --git a/README.markdown b/README.markdown index 438367c..07f0579 100644 --- a/README.markdown +++ b/README.markdown @@ -1,4 +1,6 @@ -# Data Mining 290 +--- +title: Data Mining 290 +--- ### Description Learn how to obtain, clean, visualize, understand, model, and diff --git a/index.html b/index.html new file mode 100644 index 0000000..0e5b61b --- /dev/null +++ b/index.html @@ -0,0 +1,143 @@ + + + + + + + Data Mining 290 + + + + + +

Description

+

Learn how to obtain, clean, visualize, understand, model, and predict the world around you using data. Grading will consist of homework (30%), a midterm (30%), and a project (40%).

+

Instructor

+

Jimmy Retzlaff <jretz@ischool>

+

GSI

+

Shreyas <shreyas@ischool>

+

Textbook

+

Han, J., Kamber, M., & Pei, J. (2011). Data Mining: Concepts and Techniques, Third Edition (3rd ed.). Morgan Kaufmann.

+

Course Discussion

+

Info 290T: Data Mining on Piazza

+
+

Syllabus

+

DM[0-9]+ indicates chapters from the text, Data Mining.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
DateReadingsSlidesHomework / Project
Jan 23Try Github ; A Taxonomy of Data ScienceClass Intro ; Tools Intro by GUEST: ShreyasGit Intro
Jan 30DM1 ; The Yelp Factor: Are Consumer Reviews Good for Business?Case Studies ; Obtaining DataObtain & Explore Data
Feb 6DM2, DM3Probability ; PreprocessingData Stats
Feb 13DM4, Apache Hadoop: Petabytes and Terawatts (slides); mrjob docs (for homework)Data Warehouse ; MapReduceProject Details ; mrjob
Feb 20DM8Decision Trees; Naive BayesGini Index
Feb 27DM[9.1-9.3], 9.5 ; Understanding the Bias-Variance TradeoffSVM ; Neural NetworksNeural Network Back Propagation
Mar 6DM10Agglomerative - Clustering ; Hierarchical, Density - ClusteringK-Means
Mar 13DM11.1Reviewprepare 1 cheat sheet
Mar 201 cheat sheetMidterm
Mar 27HOLIDAY
Apr 3DM6Advanced Clustering ; Frequent PatternAWS ; Project Proposal Due
Apr 10DM11.3; PageRank; Uncovering Social Network Sybils in the WildGraphs; PageRankAdjacency Representations
Apr 17Non-linear regressionGUEST: Gene Lee Ceaser's Pricing Strategy; Ceaser's RecruitingPrice Elasticity
Apr 24DM12; Shazam Audio SearchOutliers; Images & AudioMidterm Review
May 1Embedded Plots ; Data-Driven Documents[Visualization]slides/2014-05-01-Visualization.html) ; Yelp's VisualizationsD3 Intro; D3 Lab
May 18A Few Useful Things to Know about Machine Learning ; Top 10 Algorithms in Data MiningIn Real Life ; PresentationsMay 14th: Project Papers Due
May 15Final PresentationBye!
+ + diff --git a/slides/Makefile b/slides/Makefile deleted file mode 100644 index 325652c..0000000 --- a/slides/Makefile +++ /dev/null @@ -1,9 +0,0 @@ -PLACE_HOLDER=<\!--markdown goes here--> -SOURCES=$(wildcard *.markdown) -PRESENTATIONS=$(SOURCES:.markdown=.html) -EXECUTABLE=hello - -all: $(PRESENTATIONS) - -%.html: %.markdown presentation_template.html - sed -e "/$(PLACE_HOLDER)/r $<" < presentation_template.html | sed -e "s/$(PLACE_HOLDER)//" > $@ diff --git a/slides/presentation_template.html b/slides/slide_template.html similarity index 100% rename from slides/presentation_template.html rename to slides/slide_template.html From 493e49a41641cdd9a81c1e1fe8a2fb919667a154 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Wed, 22 Jan 2014 13:30:48 -0800 Subject: [PATCH 21/79] restore title --- README.markdown | 4 +--- index.html | 6 ++---- 2 files changed, 3 insertions(+), 7 deletions(-) diff --git a/README.markdown b/README.markdown index 07f0579..438367c 100644 --- a/README.markdown +++ b/README.markdown @@ -1,6 +1,4 @@ ---- -title: Data Mining 290 ---- +# Data Mining 290 ### Description Learn how to obtain, clean, visualize, understand, model, and diff --git a/index.html b/index.html index 0e5b61b..19e9345 100644 --- a/index.html +++ b/index.html @@ -4,14 +4,12 @@ - Data Mining 290 + - +

Data Mining 290

Description

Learn how to obtain, clean, visualize, understand, model, and predict the world around you using data. Grading will consist of homework (30%), a midterm (30%), and a project (40%).

Instructor

From 406dd830afa0407c821aeb0b0d0e319f8f6d75f7 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Wed, 22 Jan 2014 22:10:32 -0800 Subject: [PATCH 22/79] fix URLs in intro slides --- slides/2014-01-23-Intro.html | 12 ++++++------ slides/2014-01-23-Intro.markdown | 12 ++++++------ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index b1c53ca..d64d5f8 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -138,7 +138,7 @@ class: center -# Data Mining i290 +# Data Mining 290T-03 ## Jimmy Retzlaff & Shreyas --- @@ -246,7 +246,8 @@ + Expect to be asked a question + Breaks + Lab: Stick around and get the first question of homework done - + Slides on http://github.com/jretz/datamining290/ + + Slides on http://jretz.github.io/datamining290/ + + Source for everything on https://github.com/jretz/datamining290/ ??? @@ -259,12 +260,11 @@ --- -Todo - setup piazza ## Office Hours + We'll stay after class + or schedule a Skype call - + [Piazza](https://piazza.com/class#spring2014/i290) for questions and - announcements + + [Piazza](https://piazza.com/berkeley/spring2014/info290t03/home) for + questions and announcements + Wait list will be processed normally until 3rd week... then I'll accept everyone who's participated in class if we have physical room @@ -277,7 +277,7 @@ ## Schedule Available at [GitHub Syllabus page](http://github.com/jretz/datamining290/) - + Jan 23 Class Intro ; Tools Intro by /GUEST: Shreyas/ + + Jan 23 Class Intro ; Tools Intro by GUEST: Shreyas + lab: Git Intro + Jan 30 Case Studies ; Obtaining Data + Feb 6 Probability ; Preprocessing diff --git a/slides/2014-01-23-Intro.markdown b/slides/2014-01-23-Intro.markdown index d919c38..b189b1d 100644 --- a/slides/2014-01-23-Intro.markdown +++ b/slides/2014-01-23-Intro.markdown @@ -6,7 +6,7 @@ class: left, top, inverse class: center -# Data Mining i290 +# Data Mining 290T-03 ## Jimmy Retzlaff & Shreyas --- @@ -114,7 +114,8 @@ class: center + Expect to be asked a question + Breaks + Lab: Stick around and get the first question of homework done - + Slides on http://github.com/jretz/datamining290/ + + Slides on http://jretz.github.io/datamining290/ + + Source for everything on https://github.com/jretz/datamining290/ ??? @@ -127,12 +128,11 @@ class: center --- -Todo - setup piazza ## Office Hours + We'll stay after class + or schedule a Skype call - + [Piazza](https://piazza.com/class#spring2014/i290) for questions and - announcements + + [Piazza](https://piazza.com/berkeley/spring2014/info290t03/home) for + questions and announcements + Wait list will be processed normally until 3rd week... then I'll accept everyone who's participated in class if we have physical room @@ -145,7 +145,7 @@ Todo - setup piazza ## Schedule Available at [GitHub Syllabus page](http://github.com/jretz/datamining290/) - + Jan 23 Class Intro ; Tools Intro by /GUEST: Shreyas/ + + Jan 23 Class Intro ; Tools Intro by GUEST: Shreyas + lab: Git Intro + Jan 30 Case Studies ; Obtaining Data + Feb 6 Probability ; Preprocessing From 1c32843b01d11292ca2f78a6fe8a6b1377964197 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Wed, 22 Jan 2014 22:33:49 -0800 Subject: [PATCH 23/79] updated Shreyas' intro --- slides/2014-01-23-Intro.html | 6 +++--- slides/2014-01-23-Intro.markdown | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index d64d5f8..d2782ce 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -314,9 +314,9 @@ --- ## Hi, I'm Shreyas - + First year Grad Student (MIMS '14) - + Also TA'd Analyzing Big Data class - + I can be reached at [seekshreyas@gmail.com](mailto:seekshreyas@gmail.com) + + Second year graduate student at ISchool + + Also TA'd Analyzing Big Data class and this class last year + + I can be reached at [shreyas@ischool](mailto:shreyas@ischool) --- diff --git a/slides/2014-01-23-Intro.markdown b/slides/2014-01-23-Intro.markdown index b189b1d..94d61c7 100644 --- a/slides/2014-01-23-Intro.markdown +++ b/slides/2014-01-23-Intro.markdown @@ -182,9 +182,9 @@ Available at [GitHub Syllabus page](http://github.com/jretz/datamining290/) --- ## Hi, I'm Shreyas - + First year Grad Student (MIMS '14) - + Also TA'd Analyzing Big Data class - + I can be reached at [seekshreyas@gmail.com](mailto:seekshreyas@gmail.com) + + Second year graduate student at ISchool + + Also TA'd Analyzing Big Data class and this class last year + + I can be reached at [shreyas@ischool](mailto:shreyas@ischool) --- From 07932da9f2d26d274bb8bf9a7c3f6ce6cf9eae6d Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Thu, 23 Jan 2014 15:40:41 -0800 Subject: [PATCH 24/79] update tech location --- slides/2014-01-23-Intro.html | 2 +- slides/2014-01-23-Intro.markdown | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index d2782ce..9d0110f 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -206,7 +206,7 @@ + Basic command line: SSH, downloading, copying large files, running programs against data + Textbook: Han, J., Kamber, M., & Pei, J. (2011). _Data Mining: Concepts and Techniques_, Third Edition *(3rd ed.)*. Morgan Kaufmann. - + Technology will be available on http://ischool.berkeley.edu + + Technology will be available on ```ischool.berkeley.edu``` ??? diff --git a/slides/2014-01-23-Intro.markdown b/slides/2014-01-23-Intro.markdown index 94d61c7..8e3766d 100644 --- a/slides/2014-01-23-Intro.markdown +++ b/slides/2014-01-23-Intro.markdown @@ -74,7 +74,7 @@ class: center + Basic command line: SSH, downloading, copying large files, running programs against data + Textbook: Han, J., Kamber, M., & Pei, J. (2011). _Data Mining: Concepts and Techniques_, Third Edition *(3rd ed.)*. Morgan Kaufmann. - + Technology will be available on http://ischool.berkeley.edu + + Technology will be available on ```ischool.berkeley.edu``` ??? From 89c31ec704d506b9ab3827b4cc2b40cff485c86d Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Mon, 27 Jan 2014 08:34:21 -0800 Subject: [PATCH 25/79] brought schedules in sync --- README.markdown | 6 +++--- index.html | 6 +++--- slides/2014-01-23-Intro.html | 14 +++++++------- slides/2014-01-23-Intro.markdown | 14 +++++++------- 4 files changed, 20 insertions(+), 20 deletions(-) diff --git a/README.markdown b/README.markdown index 438367c..5f321f6 100644 --- a/README.markdown +++ b/README.markdown @@ -31,13 +31,13 @@ DM[0-9]+ indicates chapters from the text, _Data Mining_. | Feb 20 | DM8 | [Decision Trees](slides/2014-02-20-Decision-Trees.html); [Naive Bayes](slides/2014-02-20-Bayes.html) | [Gini Index](slides/2014-02-20-Gini.html) | | Feb 27 | DM[9.1-9.3], 9.5 ; [Understanding the Bias-Variance Tradeoff](http://scott.fortmann-roe.com/docs/BiasVariance.html) | [SVM](slides/2014-02-27-SVM.html) ; [Neural Networks](slides/2014-02-27-Neural-Network.html) | [Neural Network Back Propagation](slides/2014-02-27-Lab-NN.html) | | Mar 6 | DM10 | [Agglomerative - Clustering](slides/2014-03-06-Clustering.html) ; [Hierarchical, Density - Clustering](slides/2014-03-06-Hierarchical.html) | [K-Means](slides/2014-03-06-k-means.html) | -| Mar 13 | DM11.1 | [Review](slides/2014-03-13-Review.html) | prepare 1 cheat sheet | +| Mar 13 | DM11.1 | [Advanced Clustering](slides/2014-03-13-Advanced-Cluster.html) ; [Review](slides/2014-03-13-Review.html) | prepare 1 cheat sheet | | Mar 20 | 1 cheat sheet | *Midterm* | | | Mar 27 | HOLIDAY | | | -| Apr 3 | DM6 | [Advanced Clustering](slides/2014-03-13-Advanced-Cluster.html) ; [Frequent Pattern](slides/2014-04-03-Frequent-Pattern.html) | [AWS](slides/2014-04-03-AWS.html) ; Project Proposal Due | +| Apr 3 | DM6 | [Frequent Patterns](slides/2014-04-03-Frequent-Pattern.html) | [AWS](slides/2014-04-03-AWS.html) ; Project Proposal Due | | Apr 10 | DM11.3; [PageRank](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf); [Uncovering Social Network Sybils in the Wild](http://arxiv.org/pdf/1106.5321) | [Graphs](slides/2014-04-10-Graphs.html); [PageRank](slides/2014-04-10-PageRank.html) | [Adjacency Representations](slides/2014-04-10-AdjacencyRepresentations.html) | | Apr 17 | [Non-linear regression](slides/2014-04-17-Nonlinear.pdf) | GUEST: Gene Lee Ceaser's [Pricing Strategy](slides/RM Pricing Strategy.ppt); [Ceaser's Recruiting](slides/Campus Recruiting Deck_2012_UC Berkeley.ppt) | [Price Elasticity](slides/2014-04-17-Elasticity.html) | | Apr 24 | DM12; [Shazam Audio Search](http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf) | [Outliers](slides/2014-04-24-Outliers.html); [Images & Audio](slides/2014-04-24-Multimedia.html) | [Midterm Review](slides/2014-04-24-Midterm-HW.html) | | May 1 | [Embedded Plots](https://groups.google.com/group/gsofgs/attach/2f1cdd7a999c3ad8/embedded-plots.pdf?part=2&authuser=0) ; [Data-Driven Documents](http://vis.stanford.edu/files/2011-D3-InfoVis.pdf) | [Visualization]slides/2014-05-01-Visualization.html) ; [Yelp's Visualizations](slides/2014-05-01-Yelp-Visualization.html) | [D3 Intro](http://vogievetsky.github.io/IntroD3/); [D3 Lab](slides/2014-05-01-D3.html) | -| May 18 | [A Few Useful Things to Know about Machine Learning](http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) ; [Top 10 Algorithms in Data Mining](http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf) | [In Real Life](slides/2014-05-08-Real-World.html) ; Presentations | May 14th: Project Papers Due | +| May 8 | [A Few Useful Things to Know about Machine Learning](http://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) ; [Top 10 Algorithms in Data Mining](http://www.cs.uvm.edu/~icdm/algorithms/10Algorithms-08.pdf) | [In Real Life](slides/2014-05-08-Real-World.html) ; Presentations | May 14th: Project Papers Due | | May 15 | | Final Presentation | Bye! | diff --git a/index.html b/index.html index 19e9345..86b771e 100644 --- a/index.html +++ b/index.html @@ -78,7 +78,7 @@

Syllabus

Mar 13 DM11.1 -Review +Advanced Clustering ; Review prepare 1 cheat sheet @@ -96,7 +96,7 @@

Syllabus

Apr 3 DM6 -Advanced Clustering ; Frequent Pattern +Frequent Patterns AWS ; Project Proposal Due @@ -124,7 +124,7 @@

Syllabus

D3 Intro; D3 Lab -May 18 +May 8 A Few Useful Things to Know about Machine Learning ; Top 10 Algorithms in Data Mining In Real Life ; Presentations May 14th: Project Papers Due diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index 9d0110f..84a4a13 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -284,16 +284,16 @@ + Feb 13 MapReduce, Data Warehouse + Feb 20 Decision Trees; Naive Bayes + Feb 27 SVM ; Neural Networks - + Mar 6 Clustering ; Review - + lab: Project Proposal Due - + Mar 13 *Midterm* + + Mar 6 Clustering + + Mar 13 Advanced Clustering ; Review + + Mar 20 *Midterm* + lab: - - + Mar 20 Dimensionality Curse ; Graph Mining + Mar 27 HOLIDAY - + Apr 3 Pattern ; Evaluations - + Apr 10 Collaborative Filtering; PageRank + + Apr 3 Patterns ; Evaluations + + lab: Project Proposal Due + + Apr 10 Graphs; PageRank + Apr 17 Feature Extraction ; Evaluation - + Apr 24 Images ; Audio + + Apr 24 Outliers ; Images ; Audio + May 1 Visualization ; HTML + May 8 In Real Life ; Review + lab: - diff --git a/slides/2014-01-23-Intro.markdown b/slides/2014-01-23-Intro.markdown index 8e3766d..5eb1276 100644 --- a/slides/2014-01-23-Intro.markdown +++ b/slides/2014-01-23-Intro.markdown @@ -152,16 +152,16 @@ Available at [GitHub Syllabus page](http://github.com/jretz/datamining290/) + Feb 13 MapReduce, Data Warehouse + Feb 20 Decision Trees; Naive Bayes + Feb 27 SVM ; Neural Networks - + Mar 6 Clustering ; Review - + lab: Project Proposal Due - + Mar 13 *Midterm* + + Mar 6 Clustering + + Mar 13 Advanced Clustering ; Review + + Mar 20 *Midterm* + lab: - - + Mar 20 Dimensionality Curse ; Graph Mining + Mar 27 HOLIDAY - + Apr 3 Pattern ; Evaluations - + Apr 10 Collaborative Filtering; PageRank + + Apr 3 Patterns ; Evaluations + + lab: Project Proposal Due + + Apr 10 Graphs; PageRank + Apr 17 Feature Extraction ; Evaluation - + Apr 24 Images ; Audio + + Apr 24 Outliers ; Images ; Audio + May 1 Visualization ; HTML + May 8 In Real Life ; Review + lab: - From 79e987beb1aeda5af4ebbbabcfef04202a347335 Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Mon, 27 Jan 2014 08:49:47 -0800 Subject: [PATCH 26/79] improved hw instructions --- README.markdown | 4 +- index.html | 4 +- slides/2014-01-23-Intro.html | 2 +- slides/2014-01-23-Intro.markdown | 2 +- slides/2014-01-23-Lab.html | 232 +++++++++++++++++++++++++++++++ slides/2014-01-23-Lab.markdown | 91 ++++++++++++ 6 files changed, 329 insertions(+), 6 deletions(-) create mode 100644 slides/2014-01-23-Lab.html create mode 100644 slides/2014-01-23-Lab.markdown diff --git a/README.markdown b/README.markdown index 5f321f6..6d93607 100644 --- a/README.markdown +++ b/README.markdown @@ -24,7 +24,7 @@ DM[0-9]+ indicates chapters from the text, _Data Mining_. | Date | Readings | Slides | Homework / Project | |--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------| -| Jan 23 | [Try Github](http://try.github.com) ; [A Taxonomy of Data Science](http://www.dataists.com/2010/09/a-taxonomy-of-data-science/) | [Class Intro](slides/2014-01-23-Intro.html) ; Tools Intro by GUEST: Shreyas | [Git Intro](https://github.com/seekshreyas/Introduction-to-Git-Github) | +| Jan 23 | [Try Github](http://try.github.com) ; [A Taxonomy of Data Science](http://www.dataists.com/2010/09/a-taxonomy-of-data-science/) | [Class Intro](slides/2014-01-23-Intro.html) ; [Tools Intro](https://speakerdeck.com/seekshreyas/introduction-to-git-and-github) by GUEST: Shreyas | [Git Intro](slides/2014-01-23-Lab.html) | | Jan 30 | DM1 ; [The Yelp Factor: Are Consumer Reviews Good for Business?](http://hbswk.hbs.edu/item/6836.html) | [Case Studies](slides/2014-01-30-CaseStudies.html) ; [Obtaining Data](slides/2014-01-30-Obtaining-Data.html) | [Obtain & Explore Data](slides/2014-01-30-Lab.html) | | Feb 6 | DM2, DM3 | [Probability](slides/2014-02-06-Probability.html) ; [Preprocessing](slides/2014-02-06-Preprocessing.html) | [Data Stats](slides/2014-02-06-Lab.html) | | Feb 13 | DM4, [Apache Hadoop: Petabytes and Terawatts](http://www.youtube.com/watch?v=SS27F-hYWfU) ([slides](http://prezi.com/u0ukvqzpyh5p/apache-hadoop-petabytes-and-terawatts/)); [mrjob docs](http://packages.python.org/mrjob/) (for homework) | [Data Warehouse](slides/2014-02-13-Data-Warehouse.html) ; [MapReduce](slides/2014-02-13-MapReduce.html) | [Project Details](slides/2014-02-13-Project.html) ; [mrjob](slides/2014-02-13-mrjob.html) | @@ -34,7 +34,7 @@ DM[0-9]+ indicates chapters from the text, _Data Mining_. | Mar 13 | DM11.1 | [Advanced Clustering](slides/2014-03-13-Advanced-Cluster.html) ; [Review](slides/2014-03-13-Review.html) | prepare 1 cheat sheet | | Mar 20 | 1 cheat sheet | *Midterm* | | | Mar 27 | HOLIDAY | | | -| Apr 3 | DM6 | [Frequent Patterns](slides/2014-04-03-Frequent-Pattern.html) | [AWS](slides/2014-04-03-AWS.html) ; Project Proposal Due | +| Apr 3 | DM6 | [Frequent Patterns](slides/2014-04-03-Frequent-Pattern.html) | [AWS](slides/2014-04-03-AWS.html) ; Project Proposal Due | | Apr 10 | DM11.3; [PageRank](http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf); [Uncovering Social Network Sybils in the Wild](http://arxiv.org/pdf/1106.5321) | [Graphs](slides/2014-04-10-Graphs.html); [PageRank](slides/2014-04-10-PageRank.html) | [Adjacency Representations](slides/2014-04-10-AdjacencyRepresentations.html) | | Apr 17 | [Non-linear regression](slides/2014-04-17-Nonlinear.pdf) | GUEST: Gene Lee Ceaser's [Pricing Strategy](slides/RM Pricing Strategy.ppt); [Ceaser's Recruiting](slides/Campus Recruiting Deck_2012_UC Berkeley.ppt) | [Price Elasticity](slides/2014-04-17-Elasticity.html) | | Apr 24 | DM12; [Shazam Audio Search](http://www.ee.columbia.edu/~dpwe/papers/Wang03-shazam.pdf) | [Outliers](slides/2014-04-24-Outliers.html); [Images & Audio](slides/2014-04-24-Multimedia.html) | [Midterm Review](slides/2014-04-24-Midterm-HW.html) | diff --git a/index.html b/index.html index 86b771e..5733f0d 100644 --- a/index.html +++ b/index.html @@ -36,8 +36,8 @@

Syllabus

Jan 23 Try Github ; A Taxonomy of Data Science -Class Intro ; Tools Intro by GUEST: Shreyas -Git Intro +Class Intro ; Tools Intro by GUEST: Shreyas +Git Intro Jan 30 diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index 84a4a13..06ea883 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -275,7 +275,7 @@ --- ## Schedule -Available at [GitHub Syllabus page](http://github.com/jretz/datamining290/) +Available at [GitHub Syllabus page](http://jretz.github.io/datamining290/) + Jan 23 Class Intro ; Tools Intro by GUEST: Shreyas + lab: Git Intro diff --git a/slides/2014-01-23-Intro.markdown b/slides/2014-01-23-Intro.markdown index 5eb1276..1931bab 100644 --- a/slides/2014-01-23-Intro.markdown +++ b/slides/2014-01-23-Intro.markdown @@ -143,7 +143,7 @@ class: center --- ## Schedule -Available at [GitHub Syllabus page](http://github.com/jretz/datamining290/) +Available at [GitHub Syllabus page](http://jretz.github.io/datamining290/) + Jan 23 Class Intro ; Tools Intro by GUEST: Shreyas + lab: Git Intro diff --git a/slides/2014-01-23-Lab.html b/slides/2014-01-23-Lab.html new file mode 100644 index 0000000..5164384 --- /dev/null +++ b/slides/2014-01-23-Lab.html @@ -0,0 +1,232 @@ + + + + Data Mining + + + + + + + + + diff --git a/slides/2014-01-23-Lab.markdown b/slides/2014-01-23-Lab.markdown new file mode 100644 index 0000000..150c5a9 --- /dev/null +++ b/slides/2014-01-23-Lab.markdown @@ -0,0 +1,91 @@ +name: inverse +layout: true +class: left, top, inverse + +--- + +## Lab: Github + + + Setup GitHub account + + Submit assignment via GitHub + +--- + +## Why GitHub? + + + ```git``` tool is standard in industry + + GitHub provides best tools for sharing, commenting code + + This assignment will not have code, just practice submitting + +--- + +## Setup GitHub account + + + Create a [GitHub Account](https://github.com/signup/free), making sure to + use your .edu address + + Use [GitHub/Edu](https://github.com/edu) to request a free micro plan: + these let us use private accounts + + Setup a [GitHub SSH Key](https://help.github.com/articles/generating-ssh-keys) + +--- + +## Setup git repository on ischool server (can also use your own laptop) + + + On the server ischool.berkeley.edu +```bash +$ git clone git://github.com/jretz/datamining290.git +``` + + On the server, in the datamining290 directory run +```bash +$ git remote rename origin jretz +``` + +--- + +## Connect it to GitHub + + + After you receive your free micro account on GitHub, create a private repository called datamining290 + + It will provide you with an SSH git path, let's call it PATH + + You must use the *SSH* PATH starting with ```git://``` + + On the server, in the datamining290 directory, run +```bash +$ git remote add origin PATH +$ git push origin master +``` + +--- + +## Share with us + + + Hopefully you now have a private copy of my repository + + Add Shreyas and me (users: seekshreyas, jretz) as a contributor to your private repository + +--- + +## Submit Homework + + + On the ischool server, create a branch called ```hw1``` + + Create a new text file with your favorite editor (a simple one is + ```pico```) telling us what you hope to get out of the course + + ```git add``` the file + + ```git commit``` the change + + ```git push origin hw1``` to put it on GitHub + + On github, submit a "pull request" from the ```hw1``` branch to your master branch + +??? + +## Pull Requests + + + Pull requests are a way of showing updates in a way that lets me provide + comments, get notifications + +--- + +## Going Forward + + + Other homework assignments will be completing code + + General work-flow: + + Start a new branch + + Add required files + + Push to GitHub + + Submit Pull Request From fd79c00290d33f2c925c56eca7c4082d3dca8f0d Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Tue, 28 Jan 2014 22:18:01 -0800 Subject: [PATCH 27/79] Enlarge fonts and make things fit --- slides/2014-01-23-Intro.html | 29 +++++--- slides/2014-01-23-Lab.html | 29 +++++--- slides/2014-01-30-CaseStudies.html | 61 ++++++++-------- slides/2014-01-30-CaseStudies.markdown | 32 +++------ slides/2014-01-30-Lab.html | 81 ++++++--------------- slides/2014-01-30-Lab.markdown | 52 -------------- slides/2014-01-30-Obtaining-Data.html | 85 ++++++++++++++++------- slides/2014-01-30-Obtaining-Data.markdown | 56 ++++++++++----- slides/2014-02-06-Lab.html | 29 +++++--- slides/2014-02-06-Preprocessing.html | 29 +++++--- slides/2014-02-06-Probability.html | 29 +++++--- slides/slide_template.html | 29 +++++--- 12 files changed, 278 insertions(+), 263 deletions(-) diff --git a/slides/2014-01-23-Intro.html b/slides/2014-01-23-Intro.html index 06ea883..66e311f 100644 --- a/slides/2014-01-23-Intro.html +++ b/slides/2014-01-23-Intro.html @@ -9,20 +9,25 @@ body { font-family: 'Droid Serif'; - font-size: 20px; + font-size: 25px; + } + .remark-slide-content { + padding: 1em 2em 1em 2em; } h1, h2, h3 { font-family: 'Yanone Kaffeesatz'; font-weight: 400; + margin-top: 0; margin-bottom: 0; } - h1 { font-size: 4em; } - h2 { font-size: 2em; } - h3 { font-size: 1.6em; } + h1 { font-size: 3em; } + h2 { font-size: 1.8em; } + h3 { font-size: 1.4em; } .footnote { position: absolute; bottom: 3em; } + ul { margin: 8px;} li p { line-height: 1.25em; } .red { color: #fa0000; } .large { font-size: 2em; } @@ -36,7 +41,9 @@ background: #e7e8e2; color: black; border-radius: 3px; - /*font-size: 16px;*/ + } + .tight-code { + font-size: 20px; } em { color: #80cafa; } .pull-left { @@ -51,10 +58,10 @@ clear: both; } #slideshow .slide .content code { - font-size: 0.8em; + font-size: 1.6em; } #slideshow .slide .content pre code { - font-size: 0.9em; + font-size: 1.6em; padding: 15px; } .inverse { @@ -64,7 +71,7 @@ } .inverse h1, .inverse h2 { color: #f3f3f3; - line-height: 0.8em; + line-height: 1.6em; } /* Slide-specific styling */ @@ -73,7 +80,7 @@ left: 20px; } #slide-how .slides { - font-size: 0.9em; + font-size: 1.6em; position: absolute; top: 151px; right: 140px; @@ -111,6 +118,10 @@ width: 48%; float: right; } + .right-column img { + max-width: 120%; + max-height: 120%; + } /* Tables */ table { diff --git a/slides/2014-01-23-Lab.html b/slides/2014-01-23-Lab.html index 5164384..5977d7b 100644 --- a/slides/2014-01-23-Lab.html +++ b/slides/2014-01-23-Lab.html @@ -9,20 +9,25 @@ body { font-family: 'Droid Serif'; - font-size: 20px; + font-size: 25px; + } + .remark-slide-content { + padding: 1em 2em 1em 2em; } h1, h2, h3 { font-family: 'Yanone Kaffeesatz'; font-weight: 400; + margin-top: 0; margin-bottom: 0; } - h1 { font-size: 4em; } - h2 { font-size: 2em; } - h3 { font-size: 1.6em; } + h1 { font-size: 3em; } + h2 { font-size: 1.8em; } + h3 { font-size: 1.4em; } .footnote { position: absolute; bottom: 3em; } + ul { margin: 8px;} li p { line-height: 1.25em; } .red { color: #fa0000; } .large { font-size: 2em; } @@ -36,7 +41,9 @@ background: #e7e8e2; color: black; border-radius: 3px; - /*font-size: 16px;*/ + } + .tight-code { + font-size: 20px; } em { color: #80cafa; } .pull-left { @@ -51,10 +58,10 @@ clear: both; } #slideshow .slide .content code { - font-size: 0.8em; + font-size: 1.6em; } #slideshow .slide .content pre code { - font-size: 0.9em; + font-size: 1.6em; padding: 15px; } .inverse { @@ -64,7 +71,7 @@ } .inverse h1, .inverse h2 { color: #f3f3f3; - line-height: 0.8em; + line-height: 1.6em; } /* Slide-specific styling */ @@ -73,7 +80,7 @@ left: 20px; } #slide-how .slides { - font-size: 0.9em; + font-size: 1.6em; position: absolute; top: 151px; right: 140px; @@ -111,6 +118,10 @@ width: 48%; float: right; } + .right-column img { + max-width: 120%; + max-height: 120%; + } /* Tables */ table { diff --git a/slides/2014-01-30-CaseStudies.html b/slides/2014-01-30-CaseStudies.html index ef72fd5..dee9110 100644 --- a/slides/2014-01-30-CaseStudies.html +++ b/slides/2014-01-30-CaseStudies.html @@ -9,20 +9,25 @@ body { font-family: 'Droid Serif'; - font-size: 20px; + font-size: 25px; + } + .remark-slide-content { + padding: 1em 2em 1em 2em; } h1, h2, h3 { font-family: 'Yanone Kaffeesatz'; font-weight: 400; + margin-top: 0; margin-bottom: 0; } - h1 { font-size: 4em; } - h2 { font-size: 2em; } - h3 { font-size: 1.6em; } + h1 { font-size: 3em; } + h2 { font-size: 1.8em; } + h3 { font-size: 1.4em; } .footnote { position: absolute; bottom: 3em; } + ul { margin: 8px;} li p { line-height: 1.25em; } .red { color: #fa0000; } .large { font-size: 2em; } @@ -36,7 +41,9 @@ background: #e7e8e2; color: black; border-radius: 3px; - /*font-size: 16px;*/ + } + .tight-code { + font-size: 20px; } em { color: #80cafa; } .pull-left { @@ -51,10 +58,10 @@ clear: both; } #slideshow .slide .content code { - font-size: 0.8em; + font-size: 1.6em; } #slideshow .slide .content pre code { - font-size: 0.9em; + font-size: 1.6em; padding: 15px; } .inverse { @@ -64,7 +71,7 @@ } .inverse h1, .inverse h2 { color: #f3f3f3; - line-height: 0.8em; + line-height: 1.6em; } /* Slide-specific styling */ @@ -73,7 +80,7 @@ left: 20px; } #slide-how .slides { - font-size: 0.9em; + font-size: 1.6em; position: absolute; top: 151px; right: 140px; @@ -111,6 +118,10 @@ width: 48%; float: right; } + .right-column img { + max-width: 120%; + max-height: 120%; + } /* Tables */ table { @@ -136,19 +147,7 @@ --- -## Questions - + Differences from [CS 294 Behavioral Data Mining](http://bid.berkeley.edu/cs294-1-spring13/index.php/Main_Page) - + Course Load - + Readings: after lecture - -??? - -## Memory :notes: - + Spaced intervals makes memorizing easiest - ---- - -## Case Studies +# Case Studies --- @@ -180,13 +179,15 @@ ## Search Engine Logs +.tight-code[ ```log -193.139.1 jimmy [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288 -282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929 -345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232 -10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342 -10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342 +193.139.1 jimmy [10/Oct/2013:13:55:36] "GET /search?q=headache HTTP/1.1" 200 9288 +282.482.3 shreyas [10/Oct/2013:13:56:36] "GET /search?q=bananas HTTP/1.1" 200 2929 +345.114.1 steven [10/Oct/2013:13:56:37] "GET /search?q=cold HTTP/1.1" 200 8232 +10.328.52 anne [10/Oct/2013:13:56:39] "GET /search?q=flu+shot HTTP/1.1" 200 2342 +10.328.52 lily [10/Oct/2013:13:57:40] "GET /search?q=i290 HTTP/1.1" 200 2342 ``` +] What is a common theme in these queries? @@ -209,7 +210,7 @@ ] .right-column[ -![Flu Trends](file:img/flu-trends.png) + ![Flu Trends](file:img/flu-trends.png) ] ??? @@ -219,8 +220,8 @@ + What other information could you extract from log data? + Spread of flu over countries, cities? + Time of day? Do people notice in the morning? - + correlated with any other activity? (e.g., travel) - + best day of the week to call in sick (and get away with it)? + + Correlated with any other activity? (e.g., travel) + + Best day of the week to call in sick (and get away with it)? --- diff --git a/slides/2014-01-30-CaseStudies.markdown b/slides/2014-01-30-CaseStudies.markdown index 938d03f..16be672 100644 --- a/slides/2014-01-30-CaseStudies.markdown +++ b/slides/2014-01-30-CaseStudies.markdown @@ -4,19 +4,7 @@ class: left, top, inverse --- -## Questions - + Differences from [CS 294 Behavioral Data Mining](http://bid.berkeley.edu/cs294-1-spring13/index.php/Main_Page) - + Course Load - + Readings: after lecture - -??? - -## Memory :notes: - + Spaced intervals makes memorizing easiest - ---- - -## Case Studies +# Case Studies --- @@ -48,13 +36,15 @@ class: left, top, inverse ## Search Engine Logs +.tight-code[ ```log -193.139.1 jimmy [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288 -282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929 -345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232 -10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342 -10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342 +193.139.1 jimmy [10/Oct/2013:13:55:36] "GET /search?q=headache HTTP/1.1" 200 9288 +282.482.3 shreyas [10/Oct/2013:13:56:36] "GET /search?q=bananas HTTP/1.1" 200 2929 +345.114.1 steven [10/Oct/2013:13:56:37] "GET /search?q=cold HTTP/1.1" 200 8232 +10.328.52 anne [10/Oct/2013:13:56:39] "GET /search?q=flu+shot HTTP/1.1" 200 2342 +10.328.52 lily [10/Oct/2013:13:57:40] "GET /search?q=i290 HTTP/1.1" 200 2342 ``` +] What is a common theme in these queries? @@ -77,7 +67,7 @@ What is a common theme in these queries? ] .right-column[ -![Flu Trends](file:img/flu-trends.png) + ![Flu Trends](file:img/flu-trends.png) ] ??? @@ -87,8 +77,8 @@ What is a common theme in these queries? + What other information could you extract from log data? + Spread of flu over countries, cities? + Time of day? Do people notice in the morning? - + correlated with any other activity? (e.g., travel) - + best day of the week to call in sick (and get away with it)? + + Correlated with any other activity? (e.g., travel) + + Best day of the week to call in sick (and get away with it)? --- diff --git a/slides/2014-01-30-Lab.html b/slides/2014-01-30-Lab.html index 8c8e936..bc833ef 100644 --- a/slides/2014-01-30-Lab.html +++ b/slides/2014-01-30-Lab.html @@ -9,20 +9,25 @@ body { font-family: 'Droid Serif'; - font-size: 20px; + font-size: 25px; + } + .remark-slide-content { + padding: 1em 2em 1em 2em; } h1, h2, h3 { font-family: 'Yanone Kaffeesatz'; font-weight: 400; + margin-top: 0; margin-bottom: 0; } - h1 { font-size: 4em; } - h2 { font-size: 2em; } - h3 { font-size: 1.6em; } + h1 { font-size: 3em; } + h2 { font-size: 1.8em; } + h3 { font-size: 1.4em; } .footnote { position: absolute; bottom: 3em; } + ul { margin: 8px;} li p { line-height: 1.25em; } .red { color: #fa0000; } .large { font-size: 2em; } @@ -36,7 +41,9 @@ background: #e7e8e2; color: black; border-radius: 3px; - /*font-size: 16px;*/ + } + .tight-code { + font-size: 20px; } em { color: #80cafa; } .pull-left { @@ -51,10 +58,10 @@ clear: both; } #slideshow .slide .content code { - font-size: 0.8em; + font-size: 1.6em; } #slideshow .slide .content pre code { - font-size: 0.9em; + font-size: 1.6em; padding: 15px; } .inverse { @@ -64,7 +71,7 @@ } .inverse h1, .inverse h2 { color: #f3f3f3; - line-height: 0.8em; + line-height: 1.6em; } /* Slide-specific styling */ @@ -73,7 +80,7 @@ left: 20px; } #slide-how .slides { - font-size: 0.9em; + font-size: 1.6em; position: absolute; top: 151px; right: 140px; @@ -111,6 +118,10 @@ width: 48%; float: right; } + .right-column img { + max-width: 120%; + max-height: 120%; + } /* Tables */ table { @@ -138,7 +149,6 @@ ## Lab: Obtain and Explore Data - + Setup GitHub account + Find a data set or external API + Superficially examine it + Summarize findings @@ -146,57 +156,6 @@ --- -## Why GitHub? - - + ```git``` tool is standard in industry - + GitHub provides best tools for sharing, commenting code - + This assignment will not have code, just practice submitting - ---- - -## Setup GitHub account - - + Create a [GitHub Account](https://github.com/signup/free), making sure to - use your .edu address - + Use [GitHub/Edu](https://github.com/edu) to request a free micro plan: - these let us use private accounts - + Setup a [GitHub SSH Key](https://help.github.com/articles/generating-ssh-keys) - ---- - -## Setup git repository on ischool server - - + On the server ischool.berkeley.edu -```bash -$ git clone git://github.com/jretz/datamining290.git -``` - + On the server, in the datamining290 directory run -```bash -$ git remote rename origin jretz -``` - ---- - -## Connect it to GitHub - - + After you receive your free micro account on GitHub, create a private repository called datamining290 - + It will provide you with an SSH git path, let's call it PATH - + You must use the *SSH* PATH starting with ```git://``` - + On the server, in the datamining290 directory, run -```bash -$ git remote add origin PATH -$ git push origin master -``` - ---- - -## Share with us - - + Hopefully you now have a private copy of my repository - + Add Shreyas and me (users: seekshreyas, jretz) as a contributor to your private repository - ---- - ## Obtain Data + Look through the links in slides for interesting data sets, or find your own diff --git a/slides/2014-01-30-Lab.markdown b/slides/2014-01-30-Lab.markdown index b61125e..d7261f6 100644 --- a/slides/2014-01-30-Lab.markdown +++ b/slides/2014-01-30-Lab.markdown @@ -6,7 +6,6 @@ class: left, top, inverse ## Lab: Obtain and Explore Data - + Setup GitHub account + Find a data set or external API + Superficially examine it + Summarize findings @@ -14,57 +13,6 @@ class: left, top, inverse --- -## Why GitHub? - - + ```git``` tool is standard in industry - + GitHub provides best tools for sharing, commenting code - + This assignment will not have code, just practice submitting - ---- - -## Setup GitHub account - - + Create a [GitHub Account](https://github.com/signup/free), making sure to - use your .edu address - + Use [GitHub/Edu](https://github.com/edu) to request a free micro plan: - these let us use private accounts - + Setup a [GitHub SSH Key](https://help.github.com/articles/generating-ssh-keys) - ---- - -## Setup git repository on ischool server - - + On the server ischool.berkeley.edu -```bash -$ git clone git://github.com/jretz/datamining290.git -``` - + On the server, in the datamining290 directory run -```bash -$ git remote rename origin jretz -``` - ---- - -## Connect it to GitHub - - + After you receive your free micro account on GitHub, create a private repository called datamining290 - + It will provide you with an SSH git path, let's call it PATH - + You must use the *SSH* PATH starting with ```git://``` - + On the server, in the datamining290 directory, run -```bash -$ git remote add origin PATH -$ git push origin master -``` - ---- - -## Share with us - - + Hopefully you now have a private copy of my repository - + Add Shreyas and me (users: seekshreyas, jretz) as a contributor to your private repository - ---- - ## Obtain Data + Look through the links in slides for interesting data sets, or find your own diff --git a/slides/2014-01-30-Obtaining-Data.html b/slides/2014-01-30-Obtaining-Data.html index ec3357c..c6fd901 100644 --- a/slides/2014-01-30-Obtaining-Data.html +++ b/slides/2014-01-30-Obtaining-Data.html @@ -9,20 +9,25 @@ body { font-family: 'Droid Serif'; - font-size: 20px; + font-size: 25px; + } + .remark-slide-content { + padding: 1em 2em 1em 2em; } h1, h2, h3 { font-family: 'Yanone Kaffeesatz'; font-weight: 400; + margin-top: 0; margin-bottom: 0; } - h1 { font-size: 4em; } - h2 { font-size: 2em; } - h3 { font-size: 1.6em; } + h1 { font-size: 3em; } + h2 { font-size: 1.8em; } + h3 { font-size: 1.4em; } .footnote { position: absolute; bottom: 3em; } + ul { margin: 8px;} li p { line-height: 1.25em; } .red { color: #fa0000; } .large { font-size: 2em; } @@ -36,7 +41,9 @@ background: #e7e8e2; color: black; border-radius: 3px; - /*font-size: 16px;*/ + } + .tight-code { + font-size: 20px; } em { color: #80cafa; } .pull-left { @@ -51,10 +58,10 @@ clear: both; } #slideshow .slide .content code { - font-size: 0.8em; + font-size: 1.6em; } #slideshow .slide .content pre code { - font-size: 0.9em; + font-size: 1.6em; padding: 15px; } .inverse { @@ -64,7 +71,7 @@ } .inverse h1, .inverse h2 { color: #f3f3f3; - line-height: 0.8em; + line-height: 1.6em; } /* Slide-specific styling */ @@ -73,7 +80,7 @@ left: 20px; } #slide-how .slides { - font-size: 0.9em; + font-size: 1.6em; position: absolute; top: 151px; right: 140px; @@ -111,6 +118,10 @@ width: 48%; float: right; } + .right-column img { + max-width: 120%; + max-height: 120%; + } /* Tables */ table { @@ -281,21 +292,23 @@ ## Search Logs Example +.tight-code[ ``` -193.139.1 jimmy [10/Oct/2013:13:55:36 -0700] "GET /search?query=headache HTTP/1.1" 200 9288 -282.482.3 shreyas [10/Oct/2013:13:56:36 -0700] "GET /search?query=bananas HTTP/1.1" 200 2929 -345.114.1 steven [10/Oct/2013:13:56:37 -0700] "GET /search?query=cold HTTP/1.1" 200 8232 -10.328.52 anne [10/Oct/2013:13:56:39 -0700] "GET /search?query=flu+shot HTTP/1.1" 200 2342 -10.328.52 lily [10/Oct/2013:13:57:40 -0700] "GET /search?query=i290 HTTP/1.1" 200 2342 +193.139.1 jimmy [10/Oct/2013:13:55:36] "GET /search?q=headache HTTP/1.1" 200 9288 +282.482.3 shreyas [10/Oct/2013:13:56:36] "GET /search?q=bananas HTTP/1.1" 200 2929 +345.114.1 steven [10/Oct/2013:13:56:37] "GET /search?q=cold HTTP/1.1" 200 8232 +10.328.52 anne [10/Oct/2013:13:56:39] "GET /search?q=flu+shot HTTP/1.1" 200 2342 +10.328.52 lily [10/Oct/2013:13:57:40] "GET /search?q=i290 HTTP/1.1" 200 2342 ``` +] -| user_name | date | query | -|-----------|----------------------------|----------| -| jimmy | 10/Oct/2013:13:55:36 -0700 | headache | -| shreyas | 10/Oct/2013:13:56:36 -0700 | bananas | -| steven | 10/Oct/2013:13:56:37 -0700 | cold | -| anne | 10/Oct/2013:13:56:39 -0700 | flu shot | -| lily | 10/Oct/2013:13:57:40 -0700 | i290 | +| user_name | date | query | +|-----------|----------------------|----------| +| jimmy | 10/Oct/2013:13:55:36 | headache | +| shreyas | 10/Oct/2013:13:56:36 | bananas | +| steven | 10/Oct/2013:13:56:37 | cold | +| anne | 10/Oct/2013:13:56:39 | flu shot | +| lily | 10/Oct/2013:13:57:40 | i290 | ??? @@ -325,8 +338,8 @@ ## NYTimes API Example - + [Article Search API](http://developer.nytimes.com/docs/read/article_search_api) - + http://api.nytimes.com/svc/search/v1/article?format=json&query=ballot&api-key=6578bab7f8c3808ce4c392edc9a793f0:8:5717915 + + [Article Search API](http://developer.nytimes.com/docs/read/article_search_api_v2) + + http://api.nytimes.com/svc/search/v2/articlesearch.json?fq=berkeley&begin_date=20140101&end_date=20140131&api-key=d394cd6a13605351d187e3864dfcea30:8:68746734 ??? @@ -424,9 +437,11 @@ + Used for downloading files + Downloading with the browser is fine, but sometimes nice to use a faster connection, or download it directly to the machine you're working on +.tight-code[ ```bash $ wget 'http://www.grouplens.org/system/files/ml-100k.zip' ``` +] ??? @@ -445,14 +460,20 @@ + Uses same connection as SSH, but copies data instead + Example: Copy data you've downloaded in your browser +.tight-code[ ```bash $ scp ~/Downloads/ml-100k.zip jretz@ischool.berkeley.edu: +``` +] -# or +OR +.tight-code[ +``` $ scp ~/Downloads/ml-100k.zip \ jretz@ischool.berkeley.edu:i290/movielens-100k.zip ``` +] ??? @@ -469,9 +490,11 @@ + Uncompress data sets for simpler, faster manipulation ``` $ unzip ml-100k.zip + ``` - # or + OR + ``` $ gunzip dataset.json.gz ``` @@ -520,10 +543,14 @@ ## ```grep``` + Find and print lines matching a "regular expression" - + [Regular expressions](http://www.regular-expressions.info/quickstart.html) are "find" on steroids, but you can use simple strings + + [Regular expressions](http://www.regular-expressions.info/quickstart.html) + are "find" on steroids, but you can use simple strings + +.tight-code[ ```bash $ grep '"name": "Cindy"' yelp_academic_dataset_user.json ``` +] --- @@ -544,16 +571,19 @@ + ```|``` pipe characters "sends" output of one program to the input of another + How many people named Cindy in the dataset? +.tight-code[ ```bash $ grep '"name": "Cindy"' yelp_academic_dataset_user.json | wc -l 91 ``` +] + What are the most common names in the dataset? +.tight-code[ ```bash $ egrep -o '"name": "([^"]*)"' yelp_academic_dataset_user.json | \ - sort -n | uniq -c | sort -nr | head + sort | uniq -c | sort -nr | head 465 "name": "David" 447 "name": "John" 418 "name": "Michael" @@ -565,6 +595,7 @@ 265 "name": "Jason" 261 "name": "Mark" ``` +] ''', + r'''#+END_HTML''', + r'''# Local Variables:''', + r'''# org-export-html-style-include-default: nil''', + r'''# org-export-html-style-include-scripts: nil''', + r'''# buffer-file-coding-system: utf-8-unix''', + r'''# End:''', +]) + + +def count_of_leading_char(s, leading_char): + index = 0 + while index < len(s) and s[index] == leading_char: + index += 1 + return index + + +def main(input_filename, output_filename): + todo = [] + slide_number = 0 + under_heading = False + + with open(output_filename, 'wb') as output_file: + output_file.write(HEADER) + with open(input_filename, 'rb') as input_file: + for line in input_file: + line = line[:-1] # remove \n + + if line in LINES_TO_ELIMINATE: + continue + + if under_heading and line.strip(): + output_file.write('\n') + + # Checks for upcoming slide + if ':animate:' in line: + todo.append('next slide is animated') + if ':two_col:' in line: + todo.append('next slide is two column') + + # Is this a new slide? + if ':slide:' in line: + slide_number += 1 + todo.append('Slide %d' % slide_number) + todo.append(' main') + output_file.write('---\n\n') + line = line.replace(':slide:', '') + + # Is this notes? + if ':notes:' in line: + todo.append(' notes') + output_file.write('\n???\n\n') + line = line.replace(':notes:', '') + + # Is this a heading? + if line.startswith('*'): + level = count_of_leading_char(line, '*') + line = (level * '#') + line[level:] + under_heading = True + else: + under_heading = False + + # Normalize indentation (should be an even number of spaces) + if count_of_leading_char(line, ' ') % 2: + line = line[1:] + + # Definitions + if ' :: ' in line: + todo.append(' a series of sections will work better for some definitions') + line = line.replace(' :: ', ': ') + + # Images + old_line = line + line = re.sub( + r'''\[\[(file:)?([^\]]+)\]\]''', + r'''''', + line + ) + if old_line != line: + todo.append(' does image fit? is it OK on a dark background?') + + # Links + old_line = line + line = re.sub( + r'''\[\[([^]]+)\]\[([^]]+)\]\]''', + r'''[\2](\1)''', + line + ) + if old_line != line: + todo.append(' check link') + + # Block code + line = re.sub( + r'''#\+begin_src *''', + '```', + line + ) + line = line.replace('#+end_src', '```') + + # Inline code + line = re.sub( + r''' =([^ =].*?[^ ])=''', + r''' ```\1```''', + line + ) + + # Jim --> Jimmy + line = line.replace('Jim', 'Jimmy') + line = line.replace('jim', 'jimmy') + line = line.replace('Blomo', 'Retzlaff') + line = line.replace('jblomo', 'jretz') + + # Table + first_bar = line.find('|') + if first_bar >= 0: + if line.find('|', first_bar + 1): + todo.append(' table') + + # Unknown construct + if line.startswith('#+'): + print 'Unknown construct:', line + + output_file.write(line.rstrip()) + output_file.write('\n') + + todo.append('Headings are the right level?') + output_file.write('\n---\n\n') + for line in todo: + output_file.write(line) + output_file.write('\n') + + +if __name__ == '__main__': + main(*sys.argv[1:]) From d0596cde4b464c4cd74a5424023a8af145489a1a Mon Sep 17 00:00:00 2001 From: Jimmy Retzlaff Date: Sun, 9 Feb 2014 18:27:01 -0800 Subject: [PATCH 36/79] mechanically translated Feb 13 lecture --- slides/2014-02-13-Data-Warehouse.html | 1250 +++++++--------- slides/2014-02-13-Data-Warehouse.markdown | 652 ++++++--- slides/2014-02-13-MapReduce.html | 1598 ++++++++------------- slides/2014-02-13-MapReduce.markdown | 781 ++++++---- slides/2014-02-13-Project.html | 546 +++---- slides/2014-02-13-Project.markdown | 140 +- slides/2014-02-13-mrjob.html | 467 +++--- slides/2014-02-13-mrjob.markdown | 132 +- 8 files changed, 2847 insertions(+), 2719 deletions(-) diff --git a/slides/2014-02-13-Data-Warehouse.html b/slides/2014-02-13-Data-Warehouse.html index 28e56fc..0371f79 100644 --- a/slides/2014-02-13-Data-Warehouse.html +++ b/slides/2014-02-13-Data-Warehouse.html @@ -1,818 +1,650 @@ - - - - -2013-02-15-Data-Warehouse - - - - - - - - - - - - - - - - - - - -
- -
- -
-

2013-02-15-Data-Warehouse

- - - - -
-

1 Data Warehouse    slide

-
- - -
- -
- -
-

2 Database Types    slide

-
- -
-
Data Warehouse
Database designed for using data to make decisions -
-
OLAP
OnLine Analytical Processing -
-
OLTP
OnLine Transactional Processing -
-
- - -
- -
-

2.1 Data Mining    notes

-
- -
    -
  • These databases are often the starting point for data mining in companies -
    • -
  • Most of the data sets from companies typical come from exporting some - portion of their data warehouse -
    • -
- - -
-
- -
- -
-

3 Properties    slide

-
- -
-
Subject Oriented
Focus on core business objects -
-
Integrated
Access to as much data as possible -
-
Time Variant
Contains historical data with time parameter -
-
Non-volatile
Updated (relatively) infrequently, in bulk -
-
- - -
- -
-

3.1 Examples    notes

-
- -
    -
  • Yelp users can be directed to a datacenter depending on conditions. This - data probably doesn't need to be in the DW -
    • -
  • Yelp has severl databases: log summaries, user info, salesforce. Most - useful if they are all in the same place -
    • -
  • Operationally, when someone changes their address, we just overwrite it in - the OLTP DB. But DW potentially cares about the old value -
    • -
  • OLTP writes to rows every time someone updates profile, review, etc. Lots of - simultaneous updates. DW: typically once a day, in bulk -
    • -
- - -
-
- -
- -
-

4 OLAP or OLTP?    slide animate

-
- -
    -
  • Transactional Focus vs. Analytic Focus -
    • -
  • Used by Managers, Executives vs. DBAs, programmers -
    • -
  • Contains current information vs. Historical -
    • -
  • Variety of differently summarized data vs normalized -
    • -
  • Short transactions vs. Long queries -
    • -
  • Full table scans vs. Indexes on for fast lookups -
    • -
  • Simultaneous queries: 100s-1000s vs 1-100 -
    • -
  • Simple updates vs Complex queries -
    • -
  • Guarenteed high performance vs Flexibility & Customization -
    • -
- - -
- -
- -
-

5 Overview    slide

-
- -

img/olap-overview.png -

-
- -
-

5.1 From the front    notes

-
- -
    -
  • Analytics team uses charts, reports, et -
    • -
  • Generated from an OLAP server -
    • -
  • Which uses data from a data warehouse (often DW and OLAP server are - integrated) -
    • -
  • Uses a process (ETL) to move the data from other source into DW -
    • -
- - -
-
- -
- -
-

6 Types of Data Warehouses    slide

-
- -
-
Enterprise
turnkey solution, often expensive, sophisticated but complex - ingestion, integration, security features -
-
Data Mart
Smaller, limited in scope. Designed for specific team or - department -
-
Virtual
OLAP built on top of an OLTP database -
-
Cloud
Google BigQuery, Amazon RedShift -
-
- - -
- -
-

6.1 Vendors    notes

-
- -
-
Enterprise
Oracle, Greenplum, AsterData -
-
Data Mart
MySQL, PostgreSQL -
-
Virtual
MySQL, PostgreSQL views or admin interface -
-
- - -
-
- -
- -
-

7 Metadata    slide

-
- -
    -
  • Data about the data being stored -
    • -
  • Overview: schema, languages -
    • -
  • Operational: last update, query latency -
    • -
  • Algorithms: normalization, transformation -
    • -
  • Performance: job dependencies -
    • -
  • Business: ownership, permissions -
    • -
- + + + + Data Mining + + + + + + + + diff --git a/slides/2014-02-13-Data-Warehouse.markdown b/slides/2014-02-13-Data-Warehouse.markdown index 995fc5c..01b25b0 100644 --- a/slides/2014-02-13-Data-Warehouse.markdown +++ b/slides/2014-02-13-Data-Warehouse.markdown @@ -1,30 +1,53 @@ -* Data Warehouse :slide: - -* Database Types :slide: - + Data Warehouse :: Database designed for using data to make decisions - + OLAP :: OnLine Analytical Processing - + OLTP :: OnLine Transactional Processing -** Data Mining :notes: - + These databases are often the starting point for data mining in companies - + Most of the data sets from companies typical come from exporting some - portion of their data warehouse - -* Properties :slide: - + Subject Oriented :: Focus on core business objects - + Integrated :: Access to as much data as possible - + Time Variant :: Contains historical data with time parameter - + Non-volatile :: Updated (relatively) infrequently, in bulk -** Examples :notes: - + Yelp users can be directed to a datacenter depending on conditions. This - data probably doesn't need to be in the DW - + Yelp has severl databases: log summaries, user info, salesforce. Most - useful if they are all in the same place - + Operationally, when someone changes their address, we just overwrite it in - the OLTP DB. But DW potentially cares about the old value - + OLTP writes to rows every time someone updates profile, review, etc. Lots of - simultaneous updates. DW: typically once a day, in bulk - -* OLAP or OLTP? :slide:animate: +name: inverse +layout: true +class: left, top, inverse + +--- + +# Data Warehouse + +--- + +# Database Types + + + Data Warehouse: Database designed for using data to make decisions + + OLAP: OnLine Analytical Processing + + OLTP: OnLine Transactional Processing + +??? + +## Data Mining + + + These databases are often the starting point for data mining in companies + + Most of the data sets from companies typical come from exporting some + portion of their data warehouse + +--- + +# Properties + + + Subject Oriented: Focus on core business objects + + Integrated: Access to as much data as possible + + Time Variant: Contains historical data with time parameter + + Non-volatile: Updated (relatively) infrequently, in bulk + +??? + +## Examples + + + Yelp users can be directed to a datacenter depending on conditions. This + data probably doesn't need to be in the DW + + Yelp has severl databases: log summaries, user info, salesforce. Most + useful if they are all in the same place + + Operationally, when someone changes their address, we just overwrite it in + the OLTP DB. But DW potentially cares about the old value + + OLTP writes to rows every time someone updates profile, review, etc. Lots of + simultaneous updates. DW: typically once a day, in bulk + +--- + +# OLAP or OLTP? animate: + + Transactional Focus vs. Analytic Focus + Used by Managers, Executives vs. DBAs, programmers + Contains current information vs. Historical @@ -35,177 +58,428 @@ + Simple updates vs Complex queries + Guarenteed high performance vs Flexibility & Customization -* Overview :slide: - [[file:img/olap-overview.png]] -** From the front :notes: - + Analytics team uses charts, reports, et - + Generated from an OLAP server - + Which uses data from a data warehouse (often DW and OLAP server are - integrated) - + Uses a process (ETL) to move the data from other source into DW - -* Types of Data Warehouses :slide: - + Enterprise :: turnkey solution, often expensive, sophisticated but complex +--- + +# Overview + + + +??? + +## From the front + + + Analytics team uses charts, reports, et + + Generated from an OLAP server + + Which uses data from a data warehouse (often DW and OLAP server are + integrated) + + Uses a process (ETL) to move the data from other source into DW + +--- + +# Types of Data Warehouses + + + Enterprise: turnkey solution, often expensive, sophisticated but complex ingestion, integration, security features - + Data Mart :: Smaller, limited in scope. Designed for specific team or + + Data Mart: Smaller, limited in scope. Designed for specific team or department - + Virtual :: OLAP built on top of an OLTP database - + Cloud :: Google BigQuery, Amazon RedShift -** Vendors :notes: - + Enterprise :: Oracle, Greenplum, AsterData - + Data Mart :: MySQL, PostgreSQL - + Virtual :: MySQL, PostgreSQL views or admin interface - -* Metadata :slide: + + Virtual: OLAP built on top of an OLTP database + + Cloud: Google BigQuery, Amazon RedShift + +??? + +## Vendors + + + Enterprise: Oracle, Greenplum, AsterData + + Data Mart: MySQL, PostgreSQL + + Virtual: MySQL, PostgreSQL views or admin interface + +--- + +# Metadata + + Data about the data being stored + Overview: schema, languages + Operational: last update, query latency + Algorithms: normalization, transformation + Performance: job dependencies + Business: ownership, permissions -** Considerations :notes: - + As soon as several people start using the DW, they'll need to know about - how it is put together - + Metadata often comes as an after thought but is an important part of - scaling - -* Overview :slide: - [[file:img/olap-overview.png]] -** Data Cubes :notes: - + Why are there cubes in the OLAP area? - -* Datacube :slide:two_col: + +??? + +## Considerations + + + As soon as several people start using the DW, they'll need to know about + how it is put together + + Metadata often comes as an after thought but is an important part of + scaling + +--- + +# Overview + + + +??? + +## Data Cubes + + + Why are there cubes in the OLAP area? + +--- + +# Datacube two_col: + + Way of thinking about multi dimensional data + Useful metaphor because one can reason about ways to satisfy a query - [[file:img/BorgFirstContact.jpg]] - -* Dimensions :slide: - | | Day 1 | Day 2 | Day 3 | - | Region 1 | $200 | $80 | $600 | - | Region 2 | $300 | $90 | $650 | - | Region 2 | $400 | $100 | $700 | -** Data... Square :notes: - + More of a data square: only 2 dimensions - + Advertising on Yelp - + Now we want to know Product TYpe of things sold (CPC, CPM, National) - -** Cube: 3rd Dimension :slide: -[[file:img/cube-3d.gif]] -*** More :notes: - + Now we want to know Page Type (Business, Search, Home) - + Hard to draw 4 dimensions, so instead... - -** Multi-Cube :slide: -[[file:img/cube-4d.png]] -*** More :notes: - + Keep adding dimension as necessary - -** Lattice :slide: - [[file:img/cube-lattice.jpg]] -*** Moving :notes: - + Move back and forth from our 2d table - + To our 3d cube, to our 4d multi-cube - + The lower dimensions summarize table - + At the extreme is just the total (ie all money made) - -* Schemas :slide:two_col: + + +--- + +# Dimensions + +| | Day 1 | Day 2 | Day 3 | +| Region 1 | $200 | $80 | $600 | +| Region 2 | $300 | $90 | $650 | +| Region 2 | $400 | $100 | $700 | + +??? + +## Data... Square + + + More of a data square: only 2 dimensions + + Advertising on Yelp + + Now we want to know Product TYpe of things sold (CPC, CPM, National) + +--- + +## Cube: 3rd Dimension + + + +??? + +### More + + + Now we want to know Page Type (Business, Search, Home) + + Hard to draw 4 dimensions, so instead... + +--- + +## Multi-Cube + + + +??? + +### More + + + Keep adding dimension as necessary + +--- + +## Lattice + + + +??? + +### Moving + + + Move back and forth from our 2d table + + To our 3d cube, to our 4d multi-cube + + The lower dimensions summarize table + + At the extreme is just the total (ie all money made) + +--- + +# Schemas two_col: + + Data cube a way of visualizing multi dimensional data + Star schema is a way store the data in a database - [[file:img/sun.jpg]] - -** Fact table :slide: - [[file:img/star-1.png]] - -** Dimension table :slide: - [[file:img/star-2.png]] - -** Dimension tables :slide: - [[file:img/star-3.png]] - -** Dimension tables :slide: - [[file:img/star-4.png]] - -** Dimension tables :slide: - [[file:img/star-5.png]] - -** Star Schema :slide: - [[file:img/star-schema.jpg]] - -** Dimensions of Dimensions :slide: - [[file:img/star-6.png]] - -** Dimensions of Dimensions :slide: - [[file:img/star-7.png]] - -** Dimensions of Dimensions :slide: - [[file:img/star-8.png]] - -** Dimensions of Dimensions :slide: - [[file:img/star-9.png]] -*** Schema Name? :notes: - + Any guesses what this fractal looking schema is called? - -** Snowflake Schema :slide: - + Schema with radiating dimension tables - [[file:img/star-snowflake.jpg]] - -** Constellation Schema :slide: - + Schema with several fact tables and related dimensions - [[file:img/star-constilation.jpg]] - -* Data Warehouse Operations :slide: - + Rollup :: Summarize data along fewer dimensions - + Drill-down :: Get details within a particular dimension - + Slice :: Select a particular value in a dimension - + Dice :: Consider a subset of the values in a dimension - + Pivot :: Swap, or rotate dimensions -** Examples :notes: - + Rollup :: What countries are selling the most ads? - + Drill-down :: Spike in Q1 ad views. Which month most responsible? - + Slice :: Chart sales only for CPC - + Dice :: Only look at sales in US, IT, DE - + Pivot :: Swap axis on a chart - -* Materialized Views :slide: - + View :: virtual table defined by a query - + Full :: Pre-compute and store - + None :: Calculate summaries on the fly - + Partial :: Variety of strategies: eg. cache results after calculating -** Usefulness :notes: - + In DW, often storing different cubes in the lattice - + For the country sample, do we have those summaries stored in another DB - table? On disk? By month? Year? - + Storing all possible summarize expensive when loading data, and requires a - lot more storage - -* Architecture :slide: - + ROLAP :: Relational. Implement OLAP on top of a relational database - + MOLAP :: Multidimensional. Implements data cube as storage paradigm - + HOLAP :: Hybrid. Data in ROLAP, rollups in MOLAP - + Specialized :: Often distributed storage, parallel DB technology - + NoSQL :: Store data as key-value pairs, optimized in different ways -** Details :notes: - + ROLAP :: MySQL, PostgreSQL - + MOLAP :: Oracle, Palo - + HOLAP :: MS SQL - + Specialized: AsterData, Greenplumb - + NoSQL: Hive, BigTable, Cassandra - -* *Break* :slide: - -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: -#+STYLE: - -#+BEGIN_HTML - -#+END_HTML - -# Local Variables: -# org-export-html-style-include-default: nil -# org-export-html-style-include-scripts: nil -# buffer-file-coding-system: utf-8-unix -# End: + + +--- + +## Fact table + + + +--- + +## Dimension table + + + +--- + +## Dimension tables + + + +--- + +## Dimension tables + + + +--- + +## Dimension tables + + + +--- + +## Star Schema + + + +--- + +## Dimensions of Dimensions + + + +--- + +## Dimensions of Dimensions + + + +--- + +## Dimensions of Dimensions + + + +--- + +## Dimensions of Dimensions + + + +??? + +### Schema Name? + + + Any guesses what this fractal looking schema is called? + +--- + +## Snowflake Schema + + + Schema with radiating dimension tables + + +--- + +## Constellation Schema + + + Schema with several fact tables and related dimensions + + +--- + +# Data Warehouse Operations + + + Rollup: Summarize data along fewer dimensions + + Drill-down: Get details within a particular dimension + + Slice: Select a particular value in a dimension + + Dice: Consider a subset of the values in a dimension + + Pivot: Swap, or rotate dimensions + +??? + +## Examples + + + Rollup: What countries are selling the most ads? + + Drill-down: Spike in Q1 ad views. Which month most responsible? + + Slice: Chart sales only for CPC + + Dice: Only look at sales in US, IT, DE + + Pivot: Swap axis on a chart + +--- + +# Materialized Views + + + View: virtual table defined by a query + + Full: Pre-compute and store + + None: Calculate summaries on the fly + + Partial: Variety of strategies: eg. cache results after calculating + +??? + +## Usefulness + + + In DW, often storing different cubes in the lattice + + For the country sample, do we have those summaries stored in another DB + table? On disk? By month? Year? + + Storing all possible summarize expensive when loading data, and requires a + lot more storage + +--- + +# Architecture + + + ROLAP: Relational. Implement OLAP on top of a relational database + + MOLAP: Multidimensional. Implements data cube as storage paradigm + + HOLAP: Hybrid. Data in ROLAP, rollups in MOLAP + + Specialized: Often distributed storage, parallel DB technology + + NoSQL: Store data as key-value pairs, optimized in different ways + +??? + +## Details + + + ROLAP: MySQL, PostgreSQL + + MOLAP: Oracle, Palo + + HOLAP: MS SQL + + Specialized: AsterData, Greenplumb + + NoSQL: Hive, BigTable, Cassandra + +--- + +# *Break* + + + + +--- + +Slide 1 + main +Slide 2 + main + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + notes +Slide 3 + main + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + notes +next slide is animated +Slide 4 + main +Slide 5 + main + does image fit? is it OK on a dark background? + notes +Slide 6 + main + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + notes + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions +Slide 7 + main + notes +Slide 8 + main + does image fit? is it OK on a dark background? + notes +next slide is two column +Slide 9 + main + does image fit? is it OK on a dark background? +Slide 10 + main + table + table + table + table + notes +Slide 11 + main + does image fit? is it OK on a dark background? + notes +Slide 12 + main + does image fit? is it OK on a dark background? + notes +Slide 13 + main + does image fit? is it OK on a dark background? + notes +next slide is two column +Slide 14 + main + does image fit? is it OK on a dark background? +Slide 15 + main + does image fit? is it OK on a dark background? +Slide 16 + main + does image fit? is it OK on a dark background? +Slide 17 + main + does image fit? is it OK on a dark background? +Slide 18 + main + does image fit? is it OK on a dark background? +Slide 19 + main + does image fit? is it OK on a dark background? +Slide 20 + main + does image fit? is it OK on a dark background? +Slide 21 + main + does image fit? is it OK on a dark background? +Slide 22 + main + does image fit? is it OK on a dark background? +Slide 23 + main + does image fit? is it OK on a dark background? +Slide 24 + main + does image fit? is it OK on a dark background? + notes +Slide 25 + main + does image fit? is it OK on a dark background? +Slide 26 + main + does image fit? is it OK on a dark background? +Slide 27 + main + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + notes + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions +Slide 28 + main + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + notes +Slide 29 + main + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions + notes + a series of sections will work better for some definitions + a series of sections will work better for some definitions + a series of sections will work better for some definitions +Slide 30 + main +Headings are the right level? diff --git a/slides/2014-02-13-MapReduce.html b/slides/2014-02-13-MapReduce.html index 58b216c..f3769ae 100644 --- a/slides/2014-02-13-MapReduce.html +++ b/slides/2014-02-13-MapReduce.html @@ -1,1129 +1,749 @@ - - - - -2013-02-15-MapReduce - - - - - - - - - - - - - - - - - - - -
- -
- -
-

2013-02-15-MapReduce

- - - - -
-

1 MapReduce    slide

-
- - -
- -
-

1.1 Spoilers    notes

-
- -
    -
  • Don't look ahead in the slides -
    • -
  • If you know MapReduce, try to let others answer and genuinely think about - how you would solve the problem. -
    • -
- - -
-
- -
- -
-

2 Yelp has a problem    slide

-
- -
    -
  • 250+ GB of logs per day -
    • -
  • Each GB takes 10 minutes to process -
    • -
  • How long to handle a day's logs? -
    • -
- -

img/yelp-growth.png -

-
- -
-

2.1 Too long    notes

-
- -
    -
  • On a single machine 40+ hours! -
    • -
  • If we really had only a single machine, we wouldn't be able to keep up! -
    • -
  • Mistake can't be fixed in a day (billing especially important) -
    • -
- - -
-
- -
- -
-

3 Solution?    slide animate

-
- -
    -
  • Don't use one machine! -
    • -
  • What are the new challenges? -
    • -
  • Distributing data -
    • -
  • Calculating overall statistics -
    • -
  • Failures -
    • -
- - -
- -
-

3.1 New Challenges    notes

-
- -