UPI Study › Courses › Ethics in Technology › Free lessons › Lesson 5.4 Free with a free account
How Algorithms Dig Through Raw Data to Discover Hidden Patterns and Relationships
Ethics in Technology · Chapter 5: Making Sense of Massive Datasets: Big Data, Mining, and the Privacy Trade-OffSection 5.4 · lesson 20 of 11614 min · free with a free account
Free preview
Create a free account to keep watching
It takes a minute and costs nothing. Your free account unlocks all of Chapter 1 and 3 more lessons of Ethics in Technology.
Lesson complete
Earn the 3 college credits for Ethics in Technology
Enrolling adds a quiz after each lesson, the graded work and the official transcript. Recommended for credit by NCCRS.
Next lesson: Applying Data Mining Techniques to Understand Behavior and Trends on Social Platforms →← Techniques for Extracting Meaningful Business Intelligence from Enormous DatasetsNext lesson: Applying Data Mining Techniques to Understand Behavior and Trends on Social Platforms →
In this lesson
The algorithm is one stage in a larger knowledge-discovery process.
- Explain the 5 KDD stages
- Match 5 mining tasks to questions
- Calculate support, confidence, and lift
- Evaluate meaning, uncertainty, bias, and use
Free study notes
Get this lesson's notes by email
The key points and 4 practice questions with answers, in your inbox. One email, free.
One email. Unsubscribe in one click.
Transcript
Everything said in the video, section by section (1,847 words). Read along, or search it.
Chapter 5
Discovering Patterns in Raw Data
Data mining seeks regularities in prepared records, including patterns that were not specified as a simple query. That capability can create useful discoveries, while extending how previously collected information is used. Knowledge discovery in databases describes the whole process, commonly abbreviated as KDD. Mining is one stage within it, surrounded by selection, preparation, transformation, and evaluation. This lesson explains those five stages, and why the work can return to earlier steps. We distinguish classification, clustering, association rules, regression, and anomaly detection, by their questions. Classification uses known categories, while clustering produces groups that need interpretation. Association rules describe co-occurrence, using support, confidence, and comparison with an overall base rate. Regression estimates a quantity. Anomaly detection identifies departures from an expected pattern. We work through two basket examples, preserving the denominator used by each measure. The final question is whether the pattern is valid, useful, and appropriate to act on.
Knowledge Discovery
Selection and Interpretation Surround the Mining Step
Selection chooses the records and fields, relevant to a defined and permitted question. It can fail by excluding an important factor, or including information that should not be used. Pre-processing handles errors, missing values, duplicates, and differences in format across the collection. Treating an absent value as zero can change meaning, rather than merely filling a blank. Transformation prepares usable representations, including derived fields and suitable handling of different scales. An apparently helpful field may leak the answer, creating a result that fails at decision time. Mining applies a method suited to the question, producing candidate patterns rather than finished knowledge. Interpretation and evaluation assess whether those patterns, are genuine, useful, and defensible. The surrounding stages can require substantial effort, even when the algorithm itself runs quickly. Findings can reveal earlier preparation problems, or suggest that the original question needs revision. Knowledge discovery is therefore iterative, rather than one inevitable forward march from storage to truth.
Preparation and Testing
A Good-Looking Result Can Depend on an Unusable Input
A feature is a measurable input, such as purchase total or number of visits. Choosing features influences what the method can learn, and what it may inadvertently encode. Training data builds a model, while separate evidence assesses performance on cases not used to fit it. That distinction matters because fitting the known records well, does not guarantee useful behavior on new cases. Leakage occurs when preparation supplies information, unavailable when the intended decision would actually be made. The result may look excellent during development, because an input reveals the outcome indirectly. It can then fail in use, when that information no longer exists at the required time. Duplicates, missing values, and inconsistent scales create, other reasons to revisit preparation before trusting a pattern. Later interpretation may also improve the question, changing which records should have been selected. The source emphasizes preparation effort, without giving one universal percentage for every project. Treat the surrounding process as part of the method, instead of concentrating only on the final algorithm.
Classification and Clustering
Known Categories and Discovered Groups Are Different Tasks
Classification assigns a case to a known category, using labeled examples in the source's ordinary learning setup. An email filter learns from messages, already marked as wanted or unwanted. It then applies the learned distinction, to incoming messages it has not previously seen. The categories existed before analysis, rather than being discovered as its main output. Clustering instead starts without supplied category labels, grouping records by similarity in their features. A retailer might examine buying patterns, and obtain groups it had not previously named. Those groups need human evaluation, because their apparent separation may depend on feature scaling or method settings. Some methods ask for a chosen number of clusters, such as four or six. Other methods determine group structure differently, so a fixed requested count is not universal. A cluster does not automatically establish, a meaningful type of person or a reason to treat them differently. Distinguish the input categories from the output groups, then evaluate what either result actually means.
Association Rules
What Appears Together in the Recorded Baskets?
Association rules examine items that appear together, using the shopping basket as the source's example. Consider one thousand recorded baskets, with bread appearing in four hundred of them. Butter appears in three hundred baskets, and both products appear together in two hundred forty. The rule asks about butter, among the baskets that contain bread. Its direction matters, because the bread-containing baskets provide the denominator for that conditional question. The overall butter rate is three hundred out of one thousand, or thirty percent. That base rate supplies a comparison, rather than assuming every high conditional rate is informative. The recorded relationship concerns co-occurrence, without establishing a reason the products were bought together. It does not show that bread caused the butter purchase, or that intervening would change either behavior. Support, confidence, and lift summarize different parts, of this same observed basket collection. Keep the population and rule direction clear, before calculating or interpreting those measures.
3 Measures
Each Denominator Answers a Different Question
Support asks how often the combination appears, across all recorded baskets. Two hundred forty divided by one thousand gives, zero point two four, or twenty-four percent. Confidence asks how often butter appears, among the bread-containing baskets. Two hundred forty divided by four hundred gives, zero point six zero, or sixty percent. Lift compares that conditional rate, with butter's overall thirty-percent base rate. Zero point six zero divided by zero point three zero, gives a lift of two. Butter therefore occurs twice as frequently among bread baskets, as it does among all baskets in this sample. The denominator changes across measures, because each calculation answers a different question. A lift above one indicates positive sample association, rather than automatically establishing population reliability. It also does not establish causation, even when the sample relationship is substantial. Evaluate uncertainty, stability, and the intended use, before treating the descriptive calculation as sufficient justification.
A Common-Item Trap
High Confidence Alone Can Be Uninformative
High confidence can be misleading, when the second item is common in nearly every basket. The source illustrates carrier bags, appearing in ninety percent of all baskets. A bread-and-bag rule may then look impressive, mainly because bags are widespread regardless of bread. Lift addresses that problem, by comparing the conditional rate with the overall rate. A value near one indicates little association, for that rule in the observed data. This does not mean all possible relationships disappear, because the measure describes this particular combination. The result still needs evaluation, including sample size and uncertainty around the observed rates. Repeatedly searching many combinations creates, additional opportunities to find apparently interesting patterns by chance. Check whether the relationship remains useful, in separate evidence or another relevant period. Also ask whether the rule supports, a decision the organization is justified in making. The arithmetic summarizes a candidate finding, while interpretation determines how much confidence the finding deserves.
Regression
Predict a Quantity and Explain Its Uncertainty
Regression estimates a numerical quantity, rather than assigning a case to a known category. Examples include price, temperature, delivery time, and expected value over a customer relationship. The source's property example uses area, age, rooms, and locality alongside prior sale prices. A fitted relationship estimates the price, of a property not yet sold. The source expresses its illustration in lakh, converted on the slide into equivalent rupees. Forty-eight lakh is four point eight million rupees, the same estimate in both comparisons. Two lakh is two hundred thousand rupees, while twenty lakh is two million rupees. Those very different typical errors change, how useful the same estimated price would be. The uncertainty matters alongside the estimate, rather than being an optional detail after the number. A typical error is not automatically, a formally calculated confidence interval for this particular property. Ask what evidence supports the error description, and whether the estimate suits the intended decision.
Anomaly Detection
Unusual Does Not Automatically Mean Wrong
Anomaly detection identifies records that depart, from an expected or learned pattern. The source's card example begins, with three years of small purchases near home. It then shows a large purchase abroad, at four in the morning. The transaction may deserve attention, because it differs from the account's recorded history. Legitimate travel can create the same departure, so unusual does not automatically mean fraudulent. Other uses include equipment problems, network intrusions, and manufacturing defects. A threshold that flags more departures, may catch more genuine problems and interrupt more legitimate cases. A threshold that flags fewer departures, may reduce false alarms while missing more problems. The balance depends on the costs, of both kinds of mistake in the actual setting. Approaches and training evidence vary, including unlabeled records or a reference collection of normal behavior. Treat the flag as an assessment needing appropriate handling, rather than a guaranteed verdict about wrongdoing.
Compare the Tasks
Match the Question Before Choosing a Method
Match the task to its question, before choosing an algorithm or interpreting its output. Classification asks which known category, a new case belongs to. Its ordinary supervised form learns, from examples with category labels. Clustering asks what groupings emerge, without requiring predefined category labels in the input. Association rules ask what occurs together, using occurrence records rather than a labeled outcome. Regression asks what number to expect, learning from recorded target quantities in the supervised setup. Anomaly detection asks what departs from the expected pattern, with evidence requirements varying by approach. The same algorithm family can support, more than one kind of task. That makes the question and evidence, more useful identifiers than a tool name alone. The source's comparison provides common patterns, rather than exhaustive rules for every possible learning system. For each output, state what it answers, what assumptions it uses, and what remains unproved.
Interpretation
A Strong-Looking Pattern Can Fail in 4 Different Ways
Interpretation matters because a strong-looking pattern, can fail in several distinct ways. First, many tested combinations create opportunities, for spurious correlations arising by chance. Second, a genuine correlation may not reveal, the direction or cause of the relationship. Ice-cream sales and drowning can rise together, under warmer-weather conditions. Their co-movement does not establish, that increasing one causes the other. Third, historical decisions or measurement can encode bias, which a model may preserve or amplify. That is a risk to assess, rather than proof that every method reproduces every historical preference unchanged. Fourth, an accurate pattern may still be, inappropriate or harmful to use. A sensitive inference can expose something, the person did not choose to disclose. A group-level pattern can also be misapplied, as a definitive judgment about one individual. Statistical evidence, causal interpretation, and ethical permission are different questions, requiring evaluation rather than one automatic conclusion.
Practice and Judgment
Calculate the Rule, Then Ask Whether It Should Be Used
The practice example contains two thousand baskets, with coffee in six hundred and sugar in five hundred. Three hundred baskets contain both, supplying the joint count for the coffee-to-sugar rule. Support is three hundred divided by two thousand, giving fifteen percent. Confidence is three hundred divided by six hundred, giving fifty percent among coffee baskets. Sugar's overall rate is five hundred divided by two thousand, giving twenty-five percent. Lift is fifty percent divided by twenty-five percent, giving two for this sample. That compares conditional occurrence with the base rate, without establishing causation or automatic permission to act. The source then asks about a hypothetical purchase pattern, accurately predicting early pregnancy. Even under that assumption, accuracy alone does not settle, whether the retailer should use the inference. Consider purpose, permission, disclosure, individual impact, and who is accountable for reviewing the decision. Knowledge discovery ends with judgment about the finding, rather than treating an algorithmic pattern as its own authorization.
About Ethics in Technology at UPI Study
Ethics in Technology (Computer Science 195) is a self-paced online course of 116 video lessons in 15 chapters, from cybersecurity basics, privacy and surveillance to big data, hacking, malware, network defense, intellectual property, cybercrime, encryption, emerging technology and digital citizenship. It carries a NCCRS credit recommendation for 3 semester credits.
Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Enroll now → Course details Download the syllabus All Computer Science courses
Compare: Cybersecurity vs Ethics in Technology · Which ethics course?
Why take it here
Why students take Ethics in Technology with UPI Study
Where the credit goes
Universities that work with NCCRS
Ethics in Technology carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Thomas Edison State UniversitySigned partner
University of the PeopleSigned partner
Charter Oak State CollegeSigned partner
Strayer UniversitySigned partner
Columbia Southern UniversitySigned partner
Indiana TechSigned partner
Penn StateWorks with NCCRS
Southern New Hampshire UniversityWorks with NCCRS
SUNYWorks with NCCRS
+many more cooperating universitiesFind my college →Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.
FAQ
Questions students ask us
About How Algorithms Dig Through Raw Data to Discover Hidden Patterns and Relationships
What is knowledge discovery in databases?
How do you calculate support, confidence, and lift in association rules?
What is the difference between classification and clustering?
Why can a pattern found by data mining be misleading?
About the Ethics in Technology course
How many college credits is Ethics in Technology?
How much does Ethics in Technology cost and what is included?
Which universities accept UPI Study credits?
How long does Ethics in Technology take?
How are the exams done?
How do my credits reach my college?
Ready for the full course? Earn 3 credits in Ethics in Technology.
116 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.
3 credits · NCCRSEnroll →