• Understand the concept of “raw” vs “cleaned” data

• Understand the different forms of data

• Understand basic tenets for organizing data

Data are a set of values of qualitative or quantitative variables collected through observations

Raw data have not been “cleaned” to remove outliers, instrument/observation errors, or data entry errors

Data may be raw to you, but they may have been pre-processed by someone prior to you receiving them

Data chain of custody refers to a process whereby every analyst should

1. keep a copy of the raw data
   
2. record all of the operations used to generate the clean data
   
3. document the contents of the clean data

This process insures reproducibility

There are many different data formats

• flat files (.csv, .txt)

• spreadsheets (Excel, Google)

• JavaScript Object Notation (JSON)

• databases (SQL, Access)

Be consistent

• variable names (female and not Female or F)

• missing values (NA and not N/A or -999)

• date formats (2021-01-22 and not January 22, 2021)

Choose good names

No empty cells – use NA or some other placeholder

One thing per cell, for example

• change lat-lon to lat and lon

• no notes or comments like “0 (below detection limit)”

• don’t merge cells

Use a tidy format where

• the file is rectangular

• one variable per column

• one observation (record) per row

• one table for each kind of data

• linked IDs for columns

Create a data dictionary, which includes info like

• exact variable name as in the data file

• [optional] short variable name that might be used in plots

• A detailed definition of what the variable is

• measurement units

• summary statistics (eg, min & max values)

Be very cautious of Excel workbooks and Google sheets

• they allow for calculations within cells (C1=A1+B1)

• they allow for fonts, highlighting, shading, in-cell comments, etc

For data entry in Excel, consider data validation options

(see how to do this here)

Save the data in plain text

• .csv files are generally preferred

• .txt with tabs or space delimiters are more popular in languages where a “,” indicates a decimal

A reproducible analysis should include step-by-step instructions for moving from raw to clean data

Items to include:

• explicit instructions

• versions of software/packages

• values for non-constants

We’ll learn about reading data from different sources