UPI Study › Courses › Ethics in Technology › Free lessons › Lesson 5.1 Free with a free account
What Big Data Is and Why Traditional Processing Methods Fall Short
Ethics in Technology · Chapter 5: Making Sense of Massive Datasets: Big Data, Mining, and the Privacy Trade-OffSection 5.1 · lesson 17 of 11615 min · free with a free account
Free preview
Create a free account to keep watching
It takes a minute and costs nothing. Your free account unlocks all of Chapter 1 and 3 more lessons of Ethics in Technology.
Lesson complete
Earn the 3 college credits for Ethics in Technology
Enrolling adds a quiz after each lesson, the graded work and the official transcript. Recommended for credit by NCCRS.
Next lesson: The Four Vs: Volume, Velocity, Variety, and Veracity as the Defining Characteristics of Big Data →← The Legal Framework Governing Electronic Eavesdropping, Interception, and SurveillanceNext lesson: The Four Vs: Volume, Velocity, Variety, and Veracity as the Defining Characteristics of Big Data →
In this lesson
Diagnose the workload before choosing how to process it.
- Use the 5 characteristics as a diagnostic
- Identify 6 major source categories
- Explain traditional processing constraints
- Compare scaling, Hadoop, and Spark
Free study notes
Get this lesson's notes by email
The key points and 4 practice questions with answers, in your inbox. One email, free.
One email. Unsubscribe in one click.
Transcript
Everything said in the video, section by section (1,854 words). Read along, or search it.
Chapter 5
What Makes Data “Big”?
Big data describes a mismatch, between a collection and the methods available to handle it. The difficulty may involve size, speed, format, reliability, or the usefulness of the intended result. That makes the definition relative, rather than a threshold shared by every organization. A hospital and a search company may have, very different tools and processing requirements. This lesson uses five characteristics, to diagnose where the existing approach stops meeting the task. We identify major sources of collection, including human activity, logs, sensors, operations, science, and purchased records. We then examine the strengths and limits, of the source's traditional single-server database model. Scaling up adds capability to a machine. Scaling out coordinates additional machines. Distributed storage and processing require, recovery mechanisms and attention to where the data resides. Hadoop and Spark illustrate different approaches, to large collections and repeated analysis. The final question is ethical as well as technical: what useful purpose justifies collection and retention?
5 Characteristics
Size Is Only One Reason Existing Methods Can Fail
Volume concerns the quantity of information, that must be stored, read, or analyzed. An imaging archive can create a volume problem, when existing capacity cannot meet the required task. Velocity concerns arrival speed and response time, such as a continuous stream of sensor readings. A modest total collection can still be demanding, when results are required almost immediately. Variety concerns different forms, including free text, images, recorded calls, and scanned documents. Those forms may require interpretation, before they can support the questions an organization wants answered. Veracity concerns reliability, including errors, gaps, duplicates, and inconsistent versions of the same event. A technically successful calculation can still be misleading, when its inputs are wrong. Value concerns the useful outcome, that justifies storage, processing, obligations, and exposure. Keeping more records does not automatically create, a benefit that outweighs those costs. Use the five characteristics diagnostically, identifying the specific pressure rather than merely calling the collection large.
Sources
Growth Comes From Many Streams of Activity
Human activity online produces messages, posts, images, videos, reviews, and search terms. Their different formats create variety, even when part of the collection has a regular structure. Machine and service logs record requests, errors, sign-ins, and timing measurements during ordinary operations. A busy service can produce extensive logs, making volume an important processing concern. Sensors and connected devices produce readings continuously, including location, motion, vital signs, and machine condition. Their steady arrival can stress velocity, especially where timely action is required. Transactions and operations produce orders, payments, shipments, stock movements, and appointments across several systems. Inconsistent versions of an event raise veracity concerns, even when every individual table looks well organized. Scientific instruments create large imaging, genome, particle, and satellite collections that can stress volume. Public or purchased records vary in format and quality, including filings, maps, registers, and broker material. Much collection accompanies activity, while cheaper storage, faster connections, and inexpensive sensors make retention easier.
Relational Strengths
Tables and Transactions Still Solve Important Problems
A relational database organizes records in linked tables, with rows, columns, and shared keys. SQL provides a common way to query, and work with those structured relationships. Schema on write defines fields and their types, before records enter the table. That structure can reject unsuitable input, and make established business records easier to validate. Transactions support related changes, that either complete together or are rolled back together. A bank transfer illustrates the importance, of coordinating a debit with its corresponding credit. Accounts, payroll, stock, and other structured operations still benefit, from those capabilities. The source contrasts a traditional single-server model, with distributed approaches for very large analytical workloads. That simplified contrast does not describe, every capability of every modern relational system. Modern databases can support additional data types, replication, and other designs beyond one isolated machine. Choose according to the actual workload, rather than assuming relational tools have become obsolete.
Traditional Constraints
A Single-Server Design Can Reach Several Different Limits
The traditional model can reach four constraints, when its design no longer matches the workload. First, a fixed structure may make mixed recordings, images, and changing formats awkward to represent. Forcing everything into convenient columns can discard detail, or produce many largely empty fields. Second, growth tied to one server encounters, hardware limits and increasingly difficult purchasing choices. A larger server can help, but it does not create unlimited capacity. Third, moving data may dominate processing time, even when the calculation itself is simple. Slow reads and transfers can extend a report, beyond its required completion window. Fourth, coordination for strong guarantees consumes resources, and may reduce throughput for some designs. That cost can be necessary, depending on the consequences of an incorrect intermediate result. These constraints concern particular designs and requirements, rather than proving every relational workload fails at scale. Diagnose the bottleneck before buying hardware, because processor speed alone does not fix every limitation.
Correctness and Timeliness
A Bank Transfer and a View Count Need Different Guarantees
A bank transfer requires paired updates, so one account's debit matches another account's credit. The transaction should commit both changes, or leave neither change as a completed transfer. An incomplete update creates an invalid state, making that coordination worth its cost. A casual video view display has, a different requirement in the source's comparison. The shown count may safely lag behind, the latest viewing activity by a short interval. Its display can catch up, without an equivalent incomplete financial transfer occurring in the meantime. That tolerance may permit less coordination, improving throughput where the task allows it. The distinction is about required behavior, rather than some data being inherently unimportant. View records used for billing or fraud analysis, may need stronger guarantees than a popularity display. Specify acceptable correctness and timeliness, before choosing what coordination can be relaxed. The appropriate system preserves the guarantees that matter, instead of paying for or discarding them indiscriminately.
Adding Capacity
Scaling Up and Scaling Out Change Different Parts of a System
Scaling up, also called vertical scaling, adds capability to a single machine. The organization might use more memory, faster storage, or a more powerful replacement server. This can be effective, while the workload remains within affordable hardware and design limits. Scaling out, or horizontal scaling, adds machines that work together on the collection. Ordinary units share storage and computation, rather than one exceptional server carrying the entire task. Capacity can be extended by adding suitable units, instead of replacing everything with a rarer machine. The approach also introduces coordination, including scheduling, communication, and distributed recovery. More machines do not guarantee proportional speed, because some tasks cannot divide their work efficiently. Network traffic or a shared bottleneck can also limit, the benefit of additional capacity. The source's key change is moving beyond, the assumption that growth must remain on one server. Evaluate the workload and its overhead, before deciding whether scaling up or scaling out helps.
Distributed Reliability
Plan for Failure and Keep Work Near the Data
Large distributed systems must expect failures, rather than treating every failed component as an unprecedented event. Replication stores suitable additional copies, so one failed node need not destroy the only data copy. The system detects unavailable components, and restores protection through the appropriate recovery mechanisms. Unfinished tasks may be retried or reassigned, according to the processing framework's design. Recovery still takes work and time, so replication does not mean a failure costs nothing. Shared failures or network partitions also require planning, beyond protection from one isolated disk problem. Data locality addresses another practical concern: where the computation occurs relative to stored information. Moving a huge collection across a network can be, more expensive than sending processing instructions nearby. Frameworks therefore try to place work, near the blocks or partitions it needs. Some data still moves, particularly when partial results must be regrouped across machines. Fault tolerance and locality make distribution practical, while the actual design determines its limits and costs.
Hadoop and MapReduce
Split, Process, Group, and Combine
Hadoop established an important pattern, combining distributed storage with distributed batch processing. Its distributed file system, commonly called HDFS, stores large files as blocks across machines. Replication and other protection mechanisms support reliability, when individual storage components fail. MapReduce expresses processing as local work, followed by grouping and combination of intermediate results. The source's example counts terms, across a large body of text divided into partitions. Each processing task works on its assigned portion, producing partial counts for the terms it finds. Those counts are then grouped, so contributions for the same term meet in the next stage. The final step combines each group, producing the term's total across the processed collection. Different workers can therefore handle parts of the text, without one reading the entire original collection. Grouping can require substantial network transfer, so distributed processing does not eliminate movement altogether. The useful pattern is split, process, group, and combine, with attention to both locality and coordination.
Spark
Reusing Working Data Can Reduce Repeated Disk Work
Repeated analysis can make intermediate storage work expensive, when the same information is used many times. The source contrasts disk-heavy MapReduce stages, with Spark's ability to reuse working data efficiently. Spark can cache or persist suitable data, including keeping it in memory where appropriate. That reuse can reduce repeated reading and writing, during iterative analysis or model training. Its persistence choices also include disk, so Spark does not keep every operation entirely in memory. Available memory becomes an important constraint, alongside storage, network movement, and the processing task. Spark supports distributed SQL, making structured queries part of its analytical capabilities. It also supports streaming workloads, enabling continuing results rather than only a single overnight batch. The improvement depends on the workload, rather than one speed multiplier applying universally. Describing Spark as simply faster Hadoop misses, differences in execution and supported kinds of analysis. Compare the actual job and resource requirements, instead of assuming one framework replaces every other option.
Compare Approaches
Choose by Shape, Scale, Guarantees, and Timing
The traditional single-server relational model emphasizes structured records, established queries, and transaction guarantees. Its suitability depends on capacity, workload design, and the guarantees required by the operation. Hadoop storage and MapReduce emphasize large distributed collections, often analyzed through periodic batch jobs. Their costs include storage, grouped transfer, recovery, and overhead between processing stages. Spark emphasizes repeated analysis, distributed SQL, and streaming, alongside other distributed analytical work. Its useful memory reuse still depends, on capacity and suitable persistence choices. These descriptions are starting points, rather than rigid boundaries separating all modern systems. Organizations can keep transactional records in one system, and analyze suitable collections through another. The analytical platform does not remove, the need for trustworthy operational updates. The transactional platform does not automatically answer, every large mixed-format analysis requirement. Choose using data shape, scale, correctness, timing, and practical cost, rather than a single technology label.
Source Scenario
A Supermarket Has Several Data Problems, Not One
The source asks you to examine, a regional supermarket's four different collections. Twelve years of till receipts raise volume questions, while retaining a structured transactional form. Four years of loyalty activity also raise volume, and connect behavior to customer accounts. Live shelf-sensor readings emphasize velocity, when stock information must support timely action. Recorded contact-center calls emphasize variety, because speech needs interpretation before many analytical tasks can proceed. These are useful diagnoses, not proof that each collection has only one relevant characteristic. Then ask about value: which records serve, a defined operational or analytical purpose? Cheap retention does not establish, that keeping every record indefinitely is justified. More capable processing can make combining collections easier, increasing both usefulness and privacy exposure. Retention, access, quality, and obligations therefore remain choices, that engineering capability cannot make on its own. Diagnose the workload, choose suitable tools, and define the purpose and boundaries of the collection.
About Ethics in Technology at UPI Study
Ethics in Technology (Computer Science 195) is a self-paced online course of 116 video lessons in 15 chapters, from cybersecurity basics, privacy and surveillance to big data, hacking, malware, network defense, intellectual property, cybercrime, encryption, emerging technology and digital citizenship. It carries a NCCRS credit recommendation for 3 semester credits.
Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Enroll now → Course details Download the syllabus All Computer Science courses
Compare: Cybersecurity vs Ethics in Technology · Which ethics course?
Why take it here
Why students take Ethics in Technology with UPI Study
Where the credit goes
Universities that work with NCCRS
Ethics in Technology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Thomas Edison State UniversitySigned partner
University of the PeopleSigned partner
Charter Oak State CollegeSigned partner
Strayer UniversitySigned partner
Columbia Southern UniversitySigned partner
Indiana TechSigned partner
Penn StateWorks with NCCRS
Southern New Hampshire UniversityWorks with NCCRS
SUNYWorks with NCCRS
+many more cooperating universitiesFind my college →Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.
FAQ
Questions students ask us
About What Big Data Is and Why Traditional Processing Methods Fall Short
What is big data?
What is the difference between scaling up and scaling out?
How does MapReduce work?
What is the difference between Hadoop and Spark?
About the Ethics in Technology course
How many college credits is Ethics in Technology?
How much does Ethics in Technology cost and what is included?
Which universities accept UPI Study credits?
How long does Ethics in Technology take?
How are the exams done?
How do my credits reach my college?
Ready for the full course? Earn 3 credits in Ethics in Technology.
116 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.
3 credits · NCCRSEnroll →