Editing and Organizing Raw Data

In the previous chapter, "Collecting Primary and Secondary Data", we explored the various ways an investigator goes out into the field to gather information. But imagine you have just returned to your office after a month-long survey. Your desk is buried under a mountain of filled-out questionnaires, interview transcripts, and printed spreadsheets.

Can you immediately start drawing graphs, calculating averages, or making official reports from this pile?

Absolutely not. If you try to analyze this data exactly as it arrived from the field, your final results will be riddled with errors. Before data can be classified, tabulated, or analyzed, it must go through a rigorous quality-control checkpoint. This chapter is all about that checkpoint: the crucial phase where we take messy, real-world information and clean it up so it is actually useful.

The Messy Reality of Raw Data

When data is first collected, it is in its most primitive state. We call this raw data.

Think of raw data like freshly harvested wheat straight from the farm. It contains the grain you want, but it is also mixed with dirt, stones, insects, and chaff. You cannot bake bread with it until it has been sifted, washed, and milled. Similarly, data straight from the field contains human errors, blank spaces, misunderstandings, and contradictions.

Why is raw data so messy? Because human beings are involved.

• A respondent might have skipped a question because they were in a hurry.
• An enumerator (the person asking the questions) might have misheard an answer and written down the wrong number.
• A respondent might have written their income in rupees, while another wrote it in thousands of rupees.
• Someone might have lied, or given a joke answer.

If you feed this uncleaned data into your statistical formulas, you will fall victim to a famous rule in statistics and computer science: GIGO (Garbage In, Garbage Out). If the input is garbage, the output will be garbage, no matter how advanced your mathematical formulas are.

What is Data Editing?

To prevent "Garbage Out," we must perform Data Editing.

Editing is the process of scrutinizing the raw data to detect errors and omissions, and then correcting them wherever possible. It is the bridge between collecting the data and classifying the data.

The person who does this is often called the editor. The editor's job is to act like a detective. They look at every single questionnaire or data row and ask: Does this make sense? Is anything missing? Is the math right?

Here is where editing fits into the overall lifecycle of a statistical inquiry:

Notice that editing and coding happen before classification and tabulation. You cannot group things into categories if you don't even know if the data is correct.

The Four Pillars of Data Editing

When an editor sits down with a stack of raw data, they are looking for four specific things. Every piece of data must be checked against these four pillars: Completeness, Consistency, Accuracy, and Uniformity.

Let's break down exactly what each of these means, how errors happen, and how an editor fixes them.

1. Editing for Completeness

The very first thing an editor checks is whether all the questions have been answered. Blank spaces are the enemy of statistics.

The Problem: Imagine a survey asking for a person's Age, Occupation, and Monthly Income. A respondent fills in their Age and Occupation but leaves the Monthly Income blank. Why? Maybe they felt it was too personal. Maybe they simply forgot.

The Solution: When an editor finds incomplete data, they have a few choices, depending on how severe the omission is:

• Drop the question: If only one minor question is left blank across a few forms, the editor might just ignore that specific question for those respondents.
• Drop the whole form: If a questionnaire is half-empty or missing critical information (like the main subject of the survey), the editor must discard the entire questionnaire. It is better to have a slightly smaller sample size than to include useless, incomplete forms.
• Contact the respondent: If the data is crucial and the respondent's contact info is available, the editor might call them or send the enumerator back to get the missing answer. This is the best solution, but it is expensive and time-consuming.
• Infer the answer: Sometimes, the missing answer can be logically guessed from other answers. For example, if a respondent leaves their "Gender" blank, but their name is "Ramesh" and their relationship to the head of the household is listed as "Son," the editor can safely fill in "Male."

2. Editing for Consistency

Consistency means that the answers within a single questionnaire do not contradict each other. The data must make logical sense.

The Problem: People make mistakes when filling out forms, leading to impossible combinations of facts.

• A respondent lists their Age as 14, but under "Marital Status" they check "Married for 10 years."
• A person lists their "Highest Education" as "Primary School," but lists their "Occupation" as "Neurosurgeon."
• A business reports that its "Total Sales" were ₹50,000, but its "Taxes Paid on Sales" were ₹80,000.

The Solution: The editor must spot these logical clashes. When a contradiction is found, the editor must figure out which answer is the mistake. If the editor can clearly see it was a typo (e.g., the age was probably meant to be 41, not 14, given they have a 10-year marriage and a 20-year-old child listed elsewhere), they can correct it. If the contradiction cannot be resolved logically, the editor might have to discard those specific answers or the whole form, because contradictory data destroys the reliability of the survey.

3. Editing for Accuracy

Accuracy is about factual and mathematical correctness. While consistency checks for logical clashes, accuracy checks for outright errors in measurement or calculation.

The Problem: This often happens when enumerators have to do math in the field, or when respondents add extra zeroes by mistake.

• A farmer reports owning 3 acres of wheat, 2 acres of rice, and 1 acre of corn. But at the bottom of the form, the "Total Land Owned" is written as 60 acres. (3 + 2 + 1 = 6, not 60).
• A respondent writes their monthly salary as ₹5,000,000 instead of ₹50,000.

The Solution: The editor recalculates all totals, subtotals, and percentages on the form. If the parts do not add up to the total, the editor corrects the math. The editor also looks for "outliers"—numbers that are so wildly high or low that they must be mistakes. If a factory worker's reported income is higher than the CEO's, the editor flags it for accuracy.

4. Editing for Uniformity (Homogeneity)

Uniformity means that all the data is recorded in the exact same units and formats. You cannot compare apples to oranges, and you cannot run statistics on mixed units.

The Problem: Different people measure things differently.

• In a survey of agricultural land, Farmer A reports his land in Acres, Farmer B reports in Hectares, and Farmer C reports in Bighas (a local unit).
• In a survey of income, Person A writes down their monthly salary, while Person B writes down their annual salary.
• In a survey of weight, some write Kilograms and others write Pounds.

The Solution: The editor must convert all answers into one single, standard unit. If the standard for the survey is Hectares, the editor will use a conversion formula to change the Acres and Bighas into Hectares. If the standard is annual income, the editor will multiply the monthly salaries by 12. This ensures that when the data is finally grouped together, every number speaks the same language.

Editing Secondary Data

So far, we have mostly talked about editing primary data (data you collected yourself via questionnaires). But what if you are using secondary data—data that someone else already collected and published, like a government census report or a financial magazine's dataset?

Do you still need to edit it? Yes.

Even if data is published in a shiny book, you cannot blindly trust it for your specific purpose. When editing secondary data, the investigator must check it for three specific qualities:

1. 1Reliability: Who collected this data? Was it a neutral government body like the National Sample Survey Office (NSSO), or a private company with a hidden agenda? Did they use proper sampling methods? If the source is not reliable, the data must be rejected.
2. 2Suitability: Data might be perfectly accurate, but completely unsuitable for your inquiry. For example, if you are studying the "Cost of Living in Jammu," and you find a dataset titled "Cost of Living in India," that data is unsuitable because it averages out the whole country, which won't give you the specific local picture you need.
3. 3Adequacy: Is there enough data? If you are studying a trend over the last 10 years, but the secondary data only covers the last 2 years, it is inadequate for your needs.

The Concept of Coding Data

Once the raw data has been edited—meaning it is now complete, consistent, accurate, and uniform—it is clean. But it is still just a massive pile of words and numbers.

Before we can easily classify this data into groups (which is the next step in the statistical process), we often need to code it.

Why do we code data?

Imagine you have 10,000 questionnaires. One of the questions is: "What is your marital status?" The answers written by respondents might be:

• "I am married"
• "Single"
• "Divorced"
• "Widowed"
• "Unmarried"

If you try to count these by hand, or type these exact words into a computer, it will take forever. Furthermore, a computer might think "Single" and "Unmarried" are two totally different categories because the words are spelled differently.

To fix this, we create a Coding Frame (a rulebook for codes). We assign a simple number to each possible category:

• Single / Unmarried = 1
• Married = 2
• Divorced = 3
• Widowed = 4

Now, the editor goes through the 10,000 questionnaires. Wherever they see "Single" or "Unmarried", they just write a big "1" in the margin. Wherever they see "Married", they write a "2".

The Benefits of Coding

1. 1Speed: It is much faster to type the numbers 1, 2, 3, 4 into a spreadsheet than to type out long words.
2. 2Space: Codes take up very little space on a page or in a computer's memory.
3. 3Accuracy in Classification: It removes ambiguity. By forcing every written answer into a specific numbered code, it makes the next step—classification—automatic. All the "1"s go into one pile, all the "2"s go into another.

Pre-coding vs. Post-coding

• Pre-coding: This is when the codes are already printed on the questionnaire before it is even handed to the respondent. For example: "Are you a resident of J&K? (1) Yes (2) No". The respondent circles the number. This saves the editor a massive amount of time.
• Post-coding: This happens after the data is collected. It is necessary for "open-ended" questions where the respondent can write whatever they want. The editor has to read all the different written answers, group similar ones together, and invent codes for them after the fact.

A Complete Walkthrough: From Raw to Ready

Let’s put everything we just learned into a concrete example. Imagine you are the editor for a small health survey. You pick up a questionnaire filled out by a respondent. Here is what it looks like:

Raw Questionnaire #042

• Name: Amit Sharma
• Age: 12
• Gender: [Blank]
• Highest Education: Bachelor's Degree in Commerce
• Height: 5 feet 8 inches
• Weight: 70 kg
• Do you smoke? Yes, 10 cigarettes a day.

As an editor, you apply the four pillars and coding:

1. 1Completeness Check: The "Gender" field is blank.

Action: You look at the name "Amit" and infer the gender is Male. You fill it in.

1. 1Consistency Check: The Age is listed as 12, but the Highest Education is a Bachelor's Degree. A 12-year-old cannot have a B.Com degree. Furthermore, they smoke 10 cigarettes a day.

Action: You realize "12" is almost certainly a typo for "21". You correct the age to 21 to make the data logically consistent.

1. 1Accuracy Check: There are no complex math totals here, but you verify that 70 kg is a realistic weight for someone who is 5'8". It is accurate.
2. 2Uniformity Check: The survey standard requires height in centimeters, not feet and inches.

Action: You convert 5 feet 8 inches into 172.7 cm.

1. 1Coding: You apply your survey's coding frame to the cleaned answers.

• Male = Code 1
• B.Com = Code 4 (Undergraduate)
• Smoker = Code 1 (Yes)

The Final, Edited, and Coded Data Row: `[ID: 042] [Age: 21] [Gender: 1] [Edu: 4] [Height: 172.7] [Weight: 70] [Smoker: 1]`

This data is now perfectly clean. It is no longer messy raw data. It is ready to be handed over to the statistician for the next major step: Classification (which we will cover in the next chapter).

Quick Revision

Before moving on, make sure you can recall these high-yield points:

• Raw Data is unorganized, unedited data exactly as it was collected from the field. It cannot be used for analysis without cleaning.
• Data Editing is the process of inspecting raw data to detect and correct errors, omissions, and contradictions.
• The four pillars of editing primary data are Completeness (no blanks), Consistency (no logical contradictions), Accuracy (correct math and facts), and Uniformity (standardized units of measurement).
• If a questionnaire is missing too much critical information, the best statistical practice is to discard it entirely rather than guess wildly.
• When using Secondary Data, editing means checking the source for Reliability, Suitability, and Adequacy.
• Coding is the assignment of numbers, letters, or symbols to responses. It compresses data, removes ambiguity, and makes it easy to enter into computers.
• Editing and Coding are preparatory steps. They must be completed before data can be classified and tabulated.