Data Cleaning and Preparation

Data Cleaning & Preparation | Kaggle Survey 2020

Tools: Python, Pandas, Object-Oriented Programming (OOP)

Cleaned and restructured the Kaggle Machine Learning & Data Science Survey 2020 dataset into analysis-ready datasets using a reusable, object-oriented Python workflow.

Before & After

- Raw dataset: 20,036 rows × 355 columns.

- Cleaned main dataset: 20,017 rows × 51 columns.

- Long-format dataset: 560,323 rows × 5 columns.

Key Tasks

- Removed the extra question-text header row, 19 exact duplicate responses, and fully empty columns.

- Added a unique "respondent_id" to link records across datasets.

- Restructured 29 multiple-choice questions by consolidating their answers into single columns, with multiple answers separated by "|".

- Transformed multiple-choice responses into a long-format dataset containing "respondent_id", "question_id", "question", "kind", and "choice", supporting accurate counting and visualization.

- Used the "kind" column to distinguish between technologies respondents currently use and those they want to learn.

- Converted salary ranges into numeric midpoint values for quantitative analysis.

- Converted survey completion times to "HH:MM:SS" and flagged potential time outliers, including durations under one minute and unusually long responses.

- Standardized the logical ordering of education and experience categories.

- Preserved missing values where they represented unanswered or inapplicable survey questions.

Deliverables

- "survey_clean.csv" — cleaned main dataset, with one row per respondent.

- "survey_long.csv" — long-format dataset, with one row per multiple-choice answer.

- Python cleaning script — reusable, object-oriented implementation of the cleaning workflow.

- Data notes — documentation describing the output columns.

Outcome

Produced two structured, analysis-ready datasets suitable for further exploration, reporting, and visualization in Excel, Power BI, or Python.