UPI Study › Courses › Ethics in Technology › Free lessons › Lesson 5.2 Free with a free account
The Four Vs: Volume, Velocity, Variety, and Veracity as the Defining Characteristics of Big Data
Ethics in Technology · Chapter 5: Making Sense of Massive Datasets: Big Data, Mining, and the Privacy Trade-OffSection 5.2 · lesson 18 of 11614 min · free with a free account
Free preview
Create a free account to keep watching
It takes a minute and costs nothing. Your free account unlocks all of Chapter 1 and 3 more lessons of Ethics in Technology.
Lesson complete
Earn the 3 college credits for Ethics in Technology
Enrolling adds a quiz after each lesson, the graded work and the official transcript. Recommended for credit by NCCRS.
Next lesson: Techniques for Extracting Meaningful Business Intelligence from Enormous Datasets →← What Big Data Is and Why Traditional Processing Methods Fall ShortNext lesson: Techniques for Extracting Meaningful Business Intelligence from Enormous Datasets →
In this lesson
Volume, velocity, variety, and veracity describe different problems.
- Use decimal data units accurately
- Distinguish arrival speed from response time
- Compare 3 forms of data
- Explain quality, value, and variability
Free study notes
Get this lesson's notes by email
The key points and 4 practice questions with answers, in your inbox. One email, free.
One email. Unsubscribe in one click.
Transcript
Everything said in the video, section by section (1,871 words). Read along, or search it.
Chapter 5
The 4 Core Characteristics of Big Data
The four core characteristics of big data, describe different ways a processing task becomes difficult. Volume concerns quantity. Velocity concerns arrival rate and the time allowed for a useful answer. Variety concerns forms and structures. Veracity concerns how far the observations can be trusted. No fixed byte threshold captures those problems, for every organization or every workload. This lesson develops each characteristic, using units, timing, data forms, and quality examples. The decimal units increase by a thousandfold, making large differences easy to underestimate. Arrival speed and decision deadlines are separate, even when the same system must address both. Structured, semi-structured, and unstructured material require, different kinds of handling before analysis. Quality failures can produce convincing results, rather than an obvious processing error. We also distinguish later additions, including value and variability, from the four core characteristics. The purpose is diagnosis: identify the actual problem, then choose a response that addresses it.
Diagnose the Problem
Different Characteristics Require Different Remedies
A volume problem concerns how much information, must be held, scanned, or processed. Suitable storage and distributed processing may help, when capacity is the actual constraint. A velocity problem concerns incoming flow, and whether useful results arrive before the required deadline. It needs a design that meets the delay requirement, rather than simply a larger archive. A variety problem concerns different forms, or structures that the current tools cannot handle conveniently. It may require flexible handling, transcription, extraction, or another interpretation step. A veracity problem concerns trustworthiness, including duplicate entities, inaccurate fields, and biased collection. Adding machines does not correct those inputs, even if it calculates their misleading totals more quickly. Several characteristics can matter together, so the diagnostic is not a rigid labeling exercise. Ask which constraint is preventing the intended result, and what evidence establishes that constraint. Then choose a remedy that changes the problem, rather than treating every difficulty as insufficient hardware.
Volume: Units
Each Decimal Step Is 1,000 Times the Previous Step
The source uses decimal data units, with each step a thousand times the previous step. A kilobyte is one thousand bytes, under that stated measurement convention. A megabyte is one thousand kilobytes, and a gigabyte is one thousand megabytes. A terabyte is one thousand gigabytes, extending the same multiplication to a larger scale. A petabyte is one thousand terabytes, while an exabyte is one thousand petabytes. A zettabyte is one thousand exabytes, which is also one billion terabytes. These large jumps change what a practical workload requires, rather than merely adding a little waiting time. Large sorting or scanning jobs can require partitioning, when a typical machine cannot handle the collection conveniently. Binary units use different names and symbols, so the convention must be specified when comparing quantities. Separately, distinguish generated data from retained data, because an organization need not store every possible observation. Retention is a decision with costs and obligations, rather than an unavoidable consequence of data generation.
Velocity: 2 Questions
A Small Reading Can Produce a Large Daily Stream
Velocity has two halves: how quickly information arrives, and how quickly a response must follow. The source uses ten thousand sensors, each reporting once every second. A day contains eighty-six thousand four hundred seconds, which determines the total reporting opportunities. Multiplying the sensors by those seconds produces, eight hundred sixty-four million readings per day. That calculation assumes continuous reporting at the stated rate, without gaps or extra messages. Even a small individual reading can therefore create, a demanding stream of incoming observations. The second question is the time allowed, before the result loses its intended usefulness. A fraud check arriving four hours later, cannot serve the same role as a pre-completion decision. Storing the stream successfully does not establish, that the processing meets that deadline. Conversely, not every result needs an immediate answer, because some reporting tasks can wait. Specify both arrival rate and response requirement, before deciding what the system must support.
Batch and Streaming
Choose a Design That Meets the Required Delay
Batch processing gathers a set of observations, and processes the accumulated material together. An overnight report is the source's familiar example, where the answer can wait until the run completes. Streaming processes an ongoing flow, and updates results incrementally as new information becomes available. That design is useful when waiting would undermine, the purpose of the decision. Implementations can handle individual events, or use small batches within the ongoing stream. Continuous input does not guarantee instant output, because processing and delivery still have delays. Batch can remain appropriate, where its simpler schedule meets the required result and timing. Streaming adds challenges involving continuing state, errors, monitoring, and recovery over time. A faulty rule can affect many decisions quickly, because the system repeatedly applies it to incoming events. Speed amplifies mistakes as well as useful decisions, making validation part of the design requirement. Choose by the delay the task can tolerate, rather than assuming the fastest-looking label is always best.
Variety: 3 Forms
Structure Describes How Meaning Is Represented
Structured data follows an established schema, such as typed columns describing an order. That structure states what the fields mean, and allows appropriate validation and queries. Semi-structured data carries labels with its values, while the fields may differ between records. JSON and XML messages are common examples, though applications can still impose agreements about their structure. The tool must handle missing or additional fields, without silently changing the meaning of a value. Unstructured material does not arrange its analytical meaning, as a convenient set of predefined fields. A photograph may require recognizing an object, while a call may require identifying a complaint. That does not mean the file has no format or metadata, because technical descriptions can still exist. It means the meaning needed for this analysis, remains to be interpreted or extracted. These forms describe a processing requirement, rather than a strict boundary between every modern tool. Ask what structure already exists, and what uncertainty remains before the intended question can be answered.
Interpretation
Variety Can Become a Veracity Problem
Structured records support queries over known fields, and checks against expected types or requirements. Those checks help, but do not guarantee that every accepted value accurately describes the world. Semi-structured handling must account for varying fields, alongside consistent meanings and suitable query behavior. A labeled value is useful only when its label, format, and interpretation are understood. Unstructured handling adds recognition, transcription, or extraction, before selected information can enter many analytical tasks. A recorded complaint can become searchable text, through a transcription process. An image can become an object label, through a recognition process. Each interpretation may introduce mistakes, that later calculations treat as reliable observations unless reviewed. This is how variety can become veracity: the material is processable, but its extracted meaning is uncertain. Modern tools can combine several forms, without eliminating the need to validate that interpretation. Check the quality of the intermediate representation, instead of trusting it merely because the pipeline completed.
Veracity
Successful Processing Can Still Produce a Plausible Wrong Answer
Veracity concerns accuracy, completeness, reliability, and whether the collection suits the intended inference. Duplicates can inflate counts, making one person or event appear several times. Missing values can exclude records quietly, when a query filters on the absent field. Entry and format errors can produce plausible values, that conflict across dates, units, or text conventions. Stale records describe a world that has changed, making an otherwise careful decision inappropriate. Collection bias arises when the measured sample, does not represent the population being discussed. Deliberate falsity includes manipulated reviews, fake accounts, or inputs designed to distort a pattern. A system can process all of those records successfully, and still calculate a misleading answer. An error message is therefore not required, for a serious data-quality failure to occur. The source distinguishes cleaning existing records, from correcting how a collection was created. Both matter, especially when a result influences decisions about people who are unevenly represented.
Worked Source Figure
Is This One Customer, or Several?
The source's worked figure presents, the same fictional customer recorded in four systems. The billing system lists an abbreviated name, while the delivery system spells out Rohan Nair. Support includes a middle initial, and the loyalty record uses two initials instead of full names. The address appears as Mill Road, with one system shortening Road to its abbreviation. The date entries use different forms, including a missing date in the support record. Two entries also reverse the order, of the apparent month and day values. A naive count can therefore treat these records, as several customers rather than one. An overly aggressive merge can create the opposite problem, by combining different people with similar details. In a real collection, investigate identity and date conventions, rather than assuming the intended match. Use evidence to resolve duplication, missing fields, and incompatible formats before relying on the total. The example explains why a plausible customer count, can be wrong without any software failure.
Scale and Bias
More Records Do Not Remove a Systematic Error
A large collection does not automatically remove errors, simply because there are many observations. Under suitable assumptions, some random variation can diminish, in an aggregate estimate. That is different from a systematic error, repeated through the same flawed process. A shared date convention mismatch can misinterpret, more records as the collection grows. A drifting sensor can push readings, consistently away from the true condition. A collection route may repeatedly miss the same group, leaving its circumstances absent from the evidence. Tidying recorded rows cannot fully recover, people who were never represented in the collection. A larger biased sample can appear more authoritative, while preserving the same underlying distortion. Cleaning can address duplicate or malformed records, but coverage may require improved collection and analysis. Assess how observations were produced, and which groups or situations the process does not adequately capture. Scale strengthens the need for quality review, rather than replacing it with an assumption of correctness.
Later Additions
Value and Variability Ask Different Questions
Value asks what useful outcome, justifies the cost and exposure of holding a collection. Years of stored calls without a defined use, do not become valuable merely through accumulation. Storage and processing consume resources, while identifiable information also creates duties and breach exposure. Variability asks whether one source changes behavior, over time or under different conditions. The source's sale-day example increases flow, to forty times the typical rate. A system designed for the average may fail, even though the message formats remain unchanged. Variety instead concerns the different forms, such as orders, photographs, and written reviews. A platform can handle those forms well, while still being unable to manage the sale-day peak. Changing meanings and aging sensors provide other examples, of behavior that shifts over time. Additional characteristics are useful when they identify, a distinct problem with a distinct response. The purpose remains clear diagnosis, rather than extending a vocabulary list without improving the decision.
Transport Scenario
Classify the Workload and Inspect the Collection
The transport scenario asks you to examine, four collections with different processing requirements. Ticket validations are structured records, where volume and reliability deserve attention. Six years of recorded announcements involve audio, requiring interpretation and a reason for retention. Vehicle positions updating every three seconds emphasize velocity, with structured or labeled location records. Free-text complaints emphasize variety, and may not represent every rider's experience equally. Then examine the ticket records, before treating their total as a trustworthy measurement. Duplicate events can inflate counts, and incomplete or inconsistent fields can change filtered results. Collection gaps may omit some activity, requiring better coverage rather than only cleaning existing rows. Several characteristics can apply together, so explain which matters most for the intended task. Retained data is a choice, requiring purpose, quality, suitable response time, and an appropriate retention boundary. Use volume, velocity, variety, and veracity, to name the problem and select a meaningful response.
About Ethics in Technology at UPI Study
Ethics in Technology (Computer Science 195) is a self-paced online course of 116 video lessons in 15 chapters, from cybersecurity basics, privacy and surveillance to big data, hacking, malware, network defense, intellectual property, cybercrime, encryption, emerging technology and digital citizenship. It carries a NCCRS credit recommendation for 3 semester credits.
Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Enroll now → Course details Download the syllabus All Computer Science courses
Compare: Cybersecurity vs Ethics in Technology · Which ethics course?
Why take it here
Why students take Ethics in Technology with UPI Study
Where the credit goes
Universities that work with NCCRS
Ethics in Technology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Thomas Edison State UniversitySigned partner
University of the PeopleSigned partner
Charter Oak State CollegeSigned partner
Strayer UniversitySigned partner
Columbia Southern UniversitySigned partner
Indiana TechSigned partner
Penn StateWorks with NCCRS
Southern New Hampshire UniversityWorks with NCCRS
SUNYWorks with NCCRS
+many more cooperating universitiesFind my college →Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.
FAQ
Questions students ask us
About The Four Vs: Volume, Velocity, Variety, and Veracity as the Defining Characteristics of Big Data
What are the four Vs of big data?
What is the difference between structured, semi-structured, and unstructured data?
What is the difference between batch processing and stream processing?
Why doesn't having more data fix poor data quality?
About the Ethics in Technology course
How many college credits is Ethics in Technology?
How much does Ethics in Technology cost and what is included?
Which universities accept UPI Study credits?
How long does Ethics in Technology take?
How are the exams done?
How do my credits reach my college?
Ready for the full course? Earn 3 credits in Ethics in Technology.
116 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.
3 credits · NCCRSEnroll →