Data Mining & Engineering
Background
Due to advances in information technology and high performance computing, and an expansion in the number of sensors integrated into manufacturing systems and products in the world, very large data sets are becoming available to engineers; often in real-time form. The rate of production of such data far outstrips our ability to analyze them manually. For example, a computational simulation can generate terabytes of data within a few hours, whereas human analysts may take several weeks to analyze these data sets. Other examples include several digital sky surveys, and data sets from the fields of medical imaging, bioinformatics, and remote sensing. As a result, there is an increasing interest in various scientific communities to explore the use of emerging data mining techniques for the analysis of these large data sets.
Data mining is the semi-automatic discovery of patterns, associations, changes, anomalies, and statistically significant structures and events in data. Traditional data analysis is assumption driven as a hypothesis is formed and validated against the data. Data mining, in contrast, is discovery driven as the patterns are automatically extracted from data.
Goals
Your task in this project is to leverage existing datasets provided by Dr.s Hallinan or Reissman or datasets emerging form any other source to create data-mining models to derive useful information about the data. In the process, you will (or may):
• Gain experience in managing large or pretty large datasets
• Evaluate the ‘design space’ of the dataset – as the models developed can only work in the ‘design space’ covered by the data
• Eliminate outliers in the dataset, e.g., points that don’t well represent the normal data points however that is defined
• Identify the input factors (predictors) that contribute most strongly to the prediction of target factors (response)
• Develop data-mining based models that enable prediction of the target factors, working to minimize the error between the predicted and actual target factors.
Datasets
University of California Irvine Data-Mining Datasets
The UC Irvine Machine Learning Repository currently maintain 351 data sets as a service to the machine learning community. You can use any of the datasets hosted on this site.
Were you to use one of the datasets, you need to provide Dr. Hallinan a description of the dataset you will be using and your goal(s) for information creation from the dataset.
Deliverables
Each project will be somewhat different. However, the expected deliverables should minimally include the following. First, submit a Word file that has the following sections.
I. Known: Describe what you know about your (in a bulleted format). This includes the dataset used.
II. Visualize: Key to any project is a visualization of the provided data. The visualization should be an aid to how you look at the data and should potentially inform what your goals are.
III. Goals: Describe the end goal(s) in a bulleted format. Please be sure to justify the specific human thermal comfort conditions you will be seeking in your design.
IV. Factors: Define factors (input and target) considered in your analysis.
V. Design Space: Establish the design space for your model. Ideally this will include histograms of the input or predictor variables to establish the valid space for models created.
VI. Outliers (if applicable): Identify outlier data points. These data points should be moved from your model data. An outlier plot is essential.
VII. Factor Importance: Present results ranking the input factors in terms of the influence on the final model. It is important for your final model to include only input factors that actually influence the goodness of the model.
VIII. Model: Present model results; ideally showing the quality of your model predictions (both visually and through statistical metrics, e.g, r2). You should present validation results that show how your models have improved through use of different data-mining approaches or variants of one approach. For example, if outliers are removed, you should show the value of doing so in terms of the model results.
IX. Conclusions: Summarize information created from your data mining.
X. Individual Hot-Wash: Describe what you learned from this project. Describe also if you will consider using R-based data-mining in your future career.
Submission
In one zip file, include the following:
• Word File addressing the deliverables.
• R-script file(s) used in your analysis. Please be sure to comment your script.
• R-data files with saved input and processed data.