Data Analytic – Decision And Modeling Research Paper

DAT 520 Final Project Guidelines and Rubric

Overview You must complete a decision analysis research project as your final project for this course. Your research project will focus on a real-world topic of your choice, as approved by your instructor. You will pick a topic from the list provided or with approval from your instructor, and creat e a data analysis plan and decision tree model based on a real-world scenario. This assessment will provide you with the opportunity to employ highly valued decision support skills and concepts for data within a real-world context. You can use the Final Project Notes document, found in the Assignment Guidelines and Rubrics section of the course. The project is divided into three milestones, which will be submitted at various points throughout the course to scaffold learning and ensure quality final submissions. These milestones will be submitted in Modules Two, Five, and Seven. The final submission will occur in Module Nine. This project will address the following course outcomes:

 Appraise data in context according to industry-standard methods and techniques for its utility in supporting decision making

 Determine suitable data manipulation and mode ling methods for decision support  Articulate data frameworks for organizational decision support by applying data manipulation, modeling, and management concep ts

 Evaluate the ethical issues surrounding organizational use of decision-oriented data based on industry standards and one’s personal ethical criteria

 Create and assess the agility of solutions through application of data-mining procedures for decision support in various industries

Prompt Your decision analysis model and report should answer the following prompt: How does your model and evaluation resolve uncertainty in making a decision? In order to produce your analytic report, you will need to choose and investigate a data set using the decision analysis techniques you learned in class. Then you will formulate a research question, write an analytic plan, and implement it. Your report should not solely consist of descriptions of what you did. It should also contain detailed explorations into the meaning behind your model and the implications of its results. You will also be testing your model’s fitness and evalu ating its strengths and weaknesses. The project in a nutshell:

1. Choose a data set (get ideas from the source list in the spreadsheet Final Project Topics and Sources.xls) 2. Formulate your decision analysis research question 3. Write an analytic plan 4. Perform the top-down or bottom-up modeling 5. Perform model diagnostics 6. Evaluate

These activities are broken up into milestones so that the work is spread throughout the term and you can get early assistance with any obstacles. A decision analysis report is similar to any other analytic report. These reports introduce a problem, state a line of inquir y, explain a model that the author developed, discuss results and limitations, and then make conclusions and recommendations. Some decision models seek the best expected value among a discrete set of choices. Other decision analyses might seek the threshold values at which the model changes from o ne recommendation to another, describe the implications, and leave it to the reader to decide what to do. Still other decision models might look for the likeliest path to explain pat terns that are already present in a data set. In all cases, they have some thing in common: They are trying to help resolve uncertainty. Your job is to bring clarity to the decision being made. Decision analysis seeks less to produce a definitive result, and more to accurately explain the combinations of possibilities that can lead decision makers to clearer choices. This is the modeling aspect. If you model the weather but never take into account barometric pressure, your model would fail if trying to determine the worst hurricane trajectories. These are the kinds of things you will be looking at in your decision models: searching for ways to explain the conditions that produce outcomes and to evaluate the strengths and weaknesses of the models you produce. The three main ideas that your report should encompass are your ability to formulate a decision analysis research question based on an appropriate data set, develop your model, and finally evaluate the model’s utilities, results, strengths , and weaknesses. In short, if your report fully encompasses these three concepts, you will produce an authentic document that would stand on its own in a professional setting. Data sources to choose from: The included spreadsheet lists data sets used in previous sessions of DAT 520. Students found these data sets, prepared them for modeling on their own, and wrote excellent papers on the topics. Remember that your data set needs to be appropriate for modeling a discrete set of choices. Either those choices are built into the model as categorical variables, or you will need to do some legwork by converting continuous variables into rational categorical groups. This activity would be part of the data preparation and documented in your data appraisal section. Your final project must include the following sections:

 Title Page

 Abstract: 300 words or less  Table of Contents

 Introduction, with research question: Up to two pages

 Data Appraisal: Up to two pages  Techniques (a.k.a. Methods): Up to two pages

 Evaluation: Two to four pages

 Model, including optimizations Up to one page for graphic(s) Up to two pages for model explanation

 Results: Up to two pages

 Limitations: Up to two pages

 Conclusion: Up to two pages Sources: Note that the core elements add up to about 15–20 pages, double-spaced. The overall target for the core elements is still 15–20 pages, so that you have room to adjust each section according to the needs of the project. Everything you need to say in the report should fit w ithin 15–20 double-spaced, 12-point font pages with one-inch margins. To see some good final projects, consult the exemplars. Not all of them are 100% perfect papers, but they do embody the level of complex thinking that characterizes an interesting project. The idea behind the page limit is to explore the con cept of “less is more.” If you add up the text, graphics, sources, and supporting material from all the milestones, you end up with 15 to 20 pages. For the final, that means some compression needs to occur. This means finding the most important information from what you have previously written and leaving room for the new parts that you need to write. Follow the list of required elements for the final to guide how to structure your research paper. Specifically, the following critical elements must be met in your final submission:

I. Introduction: Analyze the purpose, type, intended populations, and uses of the analysis to establish an appropriate context for the data-mining and analysis plan.

II. Data Appraisal A. Characterize the data set. For example, what is the purpose such data are generally used for? B. Appraise the data within the context of the problem to be solved and industry standards. How will you use the data? For example, expound

upon the limitations of the data set in the context of your needs. C. Explain the utilities that you will be using and how the data supports that choice.

III. Select Appropriate Techniques A. Determine and explain the appropriate steps for preparation of the data sets into a usable form: what steps were taken to make data

descriptions clear, how extreme or missing values were addressed, and how data quality was improved. B. Determine the appropriate steps (including: risk assessment, probability calculations, and modeling techniques) for data manipulation and in-

depth analysis to support organizational decision-making. C. Models and checkpoints: How will you optimize the models, what will you test for, and how will you build in checks to d etermine a successful

analysis? D. Defend the ethicality and legality of the analytic selections made for use, interpretation, and manipulation of the data based on in dustry

standards for legal compliance, policies, and social responsibility. If there are no potential ethical and legal compliance issues, explain how your prep and use of this data are both ethical and legal.

IV. Defend and Evaluate Choices A. Why are these choices the best for the data and problem at hand? What research or industry standards are supportive of your choices of

methods? Explain how the methods chosen will support organizational decision-making.

B. Determine the agility of these choices for decision support based on research and relevant examples: how can they be adapted to alternative needs or reapplied to future analysis?

C. Address ethical and legal issues that might arise from the use and interpretation of the data, based on industry standards, policies, and social responsibility. How can you ensure that your selected procedures, use of data, and results will be socially r esponsible and in line with your own ethical standards?

D. Implement your plan: Perform data preparation, mining and modeling procedures, and create your decision support solution. V. Decision Tree Model (bottom-up, top-down): Include the detailed process and programming steps necessary to complete the analysis. Be sure to:

A. Defend the overall structure and purpose of the tree model in organizational decision support. B. Develop process-documentation that addresses potential complications. This piece should resemble a recipe/outline that provides enough

information for addressing potential implementation issues. C. Evaluate the results of your decision tree model. At minimum, attend to the following:

1. Are the results reasonable? 2. How accurate is your model? 3. Are there missing or extraneous elements that could have influenced your results? 4. What common errors are made during creation of the model you chose? How did you ensure that you did not make these errors?

VI. Articulation of Response/Final Report: Utilizes visualization options that effectively address the needs of the audience. Options may include annotated shell tables, visualizations, and a compositional structure.

To guide you in writing your final paper, follow the Final Project Rubric. The rubric is less about format and more about thought. Specifically, you should write sections that detail the limitations and justification for your analysis. You should also take the time to address any ethica l or legal issues that connect with your results or decisions being analyzed. You should annotate and caption your graphics. You could include a table that characteri zes the data set. You should address what your model does to assist decision makers. You should defend your choices of variables and groupings. Lastly, you should address the agility of your analysis and how it might be applied to future uses.

Milestones Milestone One: Choose a Data Set and Formulate Decision Analysis Research Question In Module Two, you will choose a data set from the curated list of sources (Final Project Topics and Sources.xls), or you may submit your proposal for a different data source than those listed. Then you will write a decision analysis research question, which should be two to three pages in length and framed as a discrete set of choices to be analyzed. This milestone is graded with the Milestone One Rubric. Milestone Two: Develop Decision Analysis Model In Module Five, you will draft your decision tree. This task presupposes a data set, a viable decision analysis research question, and the necessary data prep. To complete this milestone, you may have to experiment with different modeling styles. The main objective is t o draft your model, explain what you did, and explain why it is the best model for your research question. This milestone is graded with the Milestone Two Rubric.

Milestone Three: Revise and Evaluate Decision Analysis Model In Module Seven, you will revise and evaluate your decision model based on the feedback you received from the instructor for the previous milestone. Evaluation in this case could mean a few different things. If you are performing a bottom -up style recursive partitioning analysis, you should report on the error rate and variable selection. You might also consider alternative variable categorizations to improve your model. If you are p erforming a top-down decision tree modeling exercise, what are the threshold values that cause the tree to flip? You should perform sensitivity analysis on the critical variables in your tree and report what those sensitivity analyses are telling you. For either style of modeling, what makes your tree stronger? What bre aks the model? This milestone is graded with the Milestone Three Rubric.

Final Submission: Decision Analysis Model and Report In Module Nine, you will submit your decision analysis model and report, compiling all the components used to develop the model and produce the report, as well as a leading abstract, table of contents, and in a format that addresses all of the critical elements in the instructions. The project should include sections that detail the limitations and justification for your analysis. You will probably be compressing what you wrote for your introduction to make it fit within the eight-page limit. You should also take the time to address any ethical or legal issues that connect with your results or decisions being analyzed. Lastly, you should address the agility of your analysis and how it might be applied to future uses. This assignment is graded with the Final Project Rubric.

Deliverables

Milestone Deliverables Module Due Grading

One Research Question Two Graded separately; Milestone One Rubric

Two Develop Decision Analysis Model Five Graded separately; Milestone Two Rubric

Three Revise and Evaluate Model Seven Graded separately; Milestone Three Rubric

Decision Analysis Model and Report Nine Graded separately; Final Project Rubric

Final Project Rubric Guidelines for Submission: The final report will be a 15–20 page research paper, double-spaced, in 12-point Times New Roman font with one-inch margins all around and APA citations. Title page, abstract, appendices and bibliography of sources are extra beyond the 15–20 pages of the report. You may include one page or less of annotated/captioned graphics as part of the report. The purpose of the limits is to keep the discussions compact and to maintain the integrity o f publication-quality research.

Critical Elements Exemplary (100%) Proficient (90%) Needs Improvement (70%) Not Evident (0%) Value

Introduction Meets “Profi ci ent” cri teri a and ci tes s peci fi c, rel evant exampl es to es tabl i s h a robus t context for the data-mi ni ng anal ys is pl an

The purpos e, type, i ntended popul ati ons , and us es of the anal ys is report are anal yzed to es tabl i s h an appropriate

context for the data -mi ni ng anal ys is pl an

The purpos e, type, i ntended popul ati ons , and us es of the anal ys is report are not s uffi ci entl y analyzed to

es tabl i s h an appropriate context for the data -mi ni ng anal ys is pl an

Ei ther the purpos e, type, i ntended popul ati ons , or us es of the anal ys is report are not anal yzed

6.25

Data Appraisal: Characterize

Meets “Profi ci ent” cri teri a and cl ai ms are qual ified wi th s ource

evi dence or exampl es

Makes accurate cl ai ms about the general us e of the

datas et(s ) and the i ntended purpos e of the data

Not al l cl aims about the general us e of the datas et(s ) and the

i ntended purpos e of the data i s accurate gi ven the avai l able evi dence

Does not make cl ai ms about the general us e of the datas et(s )

and the i ntended purpos e of the data

6.25

Data Appraisal: Context

Meets “Profi ci ent” cri teri a and qual i fi es claims s pecific to

di s crete needs of the organi zati on

Makes accurate cl ai ms about the data wi thi n i ndus try

s tandards and the context of the probl em to be s ol ved

Not al l cl aims about the data are accurate bas ed on i ndus try

s tandards and the context of the probl em to be s ol ved

Does not make cl ai ms about the data bas ed on the context of

the probl em to be s ol ved and i ndus try s tandards

6.25

Data Appraisal: Measurable Utilities

Meets “Profi ci ent” cri teri a and s upporti ng expl anati on i s qual i fi ed wi th exampl es or

res earch evi dence

Makes accurate determi nati on and thoroughl y expl ai ns the meas urabl e uti l i ti es and how

the data s upports that choi ce

Determi nati on of uni t of anal ys is i s not enti rel y accurate or expl anati on does not

thoroughl y expl ai n how the data s upports meas urabl e uti l i ti es determi nati on

Does not determi ne a meas urabl e uti l i ti es

6.25

Select Appropriate

Techniques: Preparation

Meets “Profi ci ent” cri teri a and

qual i ty of expl anati on al l ows for a s eaml es s del i very of the i ni ti al mol di ng proces s

Makes appropri ate anal ysis s tep

s el ecti ons and expl ai ns the proces s for prepari ng the raw data

Not al l anal ysis s tep s el ecti ons

are appropri ate for prepari ng the raw data, or not al l s tep proces s es are s uffi ci entl y expl ai ned

Does not s el ect and expl ai n

anal ys is s teps for prepari ng raw data i nto a us eabl e form

6.25

Select Appropriate Techniques:

Manipulation

Meets “Profi ci ent” cri teri a and s tep s el ecti on and expl anati ons

are s eaml es s l y i ntegrated i nto a cl ear proces s

Makes appropri ate s tep s el ecti ons and expl ai ns the

proces s of s teps for i n-depth anal ys is and mani pul ati on of the data to s upport organi zati onal deci s ion maki ng

Not al l s teps are appropri ate for i n-depth anal ys is and

mani pul ati on i n s upport of organi zati onal deci s ion maki ng or not al l s teps are expl ai ned i n terms of proces s

Does not s el ect and expl ai n i n- depth anal ys is and

mani pul ati on s teps for deci s i on s upport

6.25

Select Appropriate

Techniques: Checkpoints

Meets “Profi ci ent” cri teri a and

the expl anati ons of the s el ecti ons provi de cl ear and s eaml es s i ntegrati on of s teps i nto the overal l mani pul ati on

proces s

Makes appropri ate al gori thm

s el ecti ons , and expl ai ns the proces s of the s el ecti ons , for the opti mi zati on, ri s k as s es s ment, and bui l t-i n check

poi nts to ens ure the s ucces s of data anal ys is and mani pulation

Not al l al gorithm s el ecti ons and

expl anati ons of proces s for opti mi zati on, ri s k as s es sment, and bui l t-i n check poi nts are appropri ate to ens ure

s ucces s ful data analysis and mani pul ati on, or key val uabl e methods are mi s s ed

Does not s el ect and expl ai n the

proces s of al gori thm s el ecti ons for opti mi zati on, ri s k as s es s ment, and bui l t-i n checkpoi nts

6.25

Select Appropriate Techniques: Defend

Meets “Profi ci ent” cri teri a and s ubs tanti ates cl aims wi th

s chol arly res earch evi denci ng cons i derati ons of s oci al res pons i bi lity

Makes and jus ti fi es cl aims about the ethi cal and l egal

i s s ues rel ated to the us e, i nterpretati on, and mani pul ati on of the data for the

deci s i ons bei ng made, bas ed on i ndus try s tandards , l aws, and organi zati onal pol icies

Not al l cl aims about the ethi cal and l egal i s s ues rel ated to the

us e, i nterpretati on, and mani pul ati on of the data for the deci s i ons bei ng made are

jus ti fi abl e bas ed on i ndus try s tandards , l aws , and organi zati onal pol icies

Does not make cl ai ms about the ethi cal and l egal i s s ues rel ated

to the us e, i nterpretati on, and mani pul ati on of the data for the deci s i ons bei ng made

6.25

Defend and Evaluate Choices: Best

Meets “Profi ci ent” cri teri a and s ubs tanti ates cl aims wi th

res earch i n s peci fi c s upport of the deci s i ons /problem at hand

Makes and jus ti fi es cl aims about the appropri atenes s of

the methods for mani pul ati on and al gori thm s el ecti ons made for deci s i on s upport bas ed on anal ys is of i ndus try s tandards

and val i d res earch

Not al l cl aims about the appropri atenes s of the methods

for mani pul ati on and al gorithm s el ecti ons made are jus ti fi able bas ed on anal ys i s of i ndus try s tandards and val id res ea rch

Does not make and jus ti fy cl ai ms about the

appropri atenes s of the methods for mani pul ati on and al gorithm s el ecti ons made

6.25

Defend and Evaluate Choices: Agility

Meets “Profi ci ent” cri teri a and s ubs tanti ates cl aims wi th s chol arly res earch and real

worl d exampl es

Makes and jus ti fi es cl aims about the agi l i ty of the choi ces made for deci s i on s upport i n

vari ous i ndus tries , projects , and organi zati ons wi th res earch and rel evant exampl es

Not al l cl aims about the agi l i ty of the choi ces made for deci s i on s upport i n vari ous

i ndus tri es , projects , and organi zati ons are jus ti fiable bas ed on the provi ded res earch and exampl es

Does not make cl ai ms about the agi l i ty of the choi ces made for deci s i on s upport i n vari ous

i ndus tri es , projects , and organi zati ons

6.25

Defend and Evaluate Choices: Address

Issues

Meets “Profi ci ent” cri teri a and the detai l s of the expl anati on

expound upon s oci al res pons i bi lity and i ndus try s tandards

Detai l s the ethi cal cons i derati ons that s houl d be

made about us e of the res ul ts of the s ol uti on and how ethi cal us e can be ens ured

Expl ai ns ethi cal cons iderati ons for us e of the res ul ts of the

s ol uti on, but l acks detai l or does not expl ai n how ethi cal us e can be ens ured

Does not expl ai n ethi cal cons i derati ons for us e of

s ol uti on res ul ts

6.25

Decision Tree Model: Implement

Meets “Profi ci ent” cri teri a and performance of mi ni ng proces s

and accuracy of deci s i on s ol uti on evi dence appropri ate pl anni ng and i mpl ementati on of pl an wi thi n the context of the

s el ected topi c

Correctl y performs the data mi ni ng proces s and creates an

accurate deci s i on s upport s ol uti on

Performs the data mi ni ng proces s and creates a deci s i on

s upport s ol uti on, but s ol uti on i s not accurate

Does not perform the data mi ni ng proces s and create a

deci s i on s upport s ol uti on

6.25

Decision Tree Model: Structure

Meets “Profi ci ent” cri teri a and s ubs tanti ates cl aims wi th s chol arly evi dence and real worl d exampl es

Makes and jus ti fi es cl aims about the overal l s tructure and purpos e of model for organi zati onal deci s ion s upport

bas ed on s peci fi c exampl es and res earch

Not al l cl aims about the overal l s tructure and purpos e of model for deci s i ons s upport are jus ti fi abl e

Does not make cl ai ms about the overal l s tructure and purpos e of model for organi zati onal deci s i on s upport

6.25

Decision Tree Model:

Documentation

Meets “Profi ci ent” cri teri a and model i s of qual i ty to al l ow

others to devel op further, more detai l ed model s to addres s pos s i bl e i s sues

Outl i ne effecti vel y acts as proces s documentati on for

addres s i ng potenti al compl i cati ons duri ng i mpl ementati on of the anal ys i s pl an

Not al l as pects of outl i ne woul d be effecti ve i n addres s i ng the

potenti al compl i cati ons of i mpl ementati on, or common major i s s ues are not addres s ed

Does not i ncl ude an outl i ne for addres s i ng potenti al

compl i cati ons duri ng i mpl ementati on

6.25

Decision Tree

Model: Results

Meets “Profi ci ent” cri teri a and

comprehens i vel y eval uates agai ns t cri teri a above the gi ven cri teri a and s peci fi cally rel evant to the context of the s el ected

topi c

Accuratel y eval uates the res ul ts

of the deci s i on tree model agai ns t the gi ven cri teri a

Eval uates the res ul ts agai nst the

gi ven cri teri a, but wi th gaps i n accuracy

Does not eval uate the res ul ts

agai ns t the gi ven cri teri a

6.25

Articulation of Response

Submi s s i on i s free of errors rel ated to ci tati ons , grammar, s pel l i ng, s yntax, and organi zati on and i s pres ented i n

a profes s i onal and eas y to read format

Submi s s i on uti l izes vi s ualization opti ons that effecti vel y addres s the needs of the audi ence and has no major errors rel ated to

ci tati ons , gra mmar, s pel l i ng, s yntax, or organi zati on

Submi s s i on uti l izes various vi s ual izati on opti ons that don’t effecti vel y addres s the needs of the audi ence or has major

errors rel ated to ci tati ons , grammar, s pel l i ng, s yntax, or organi zati on that negati vel y

i mpact readabi l i ty and arti cul ation of mai n i deas

Submi s s i on does not uti l i ze vi s ual izati on opti ons for the audi ence or has cri ti cal errors rel ated to ci tati ons , grammar,

s pel l i ng, s yntax, or organi zati on that prevent unders tandi ng of i deas

6.25

Earned Total 100%