Learning outcomes - Describe the parts of fintech that matter for gathering and analyzing financial data. - Describe Big Data, artificial intelligence, and machine learning. - Describe applications of Big Data and data science to investment management.

Core Idea

1. This module is about what happens when finance collides with massive data, fast computing, and machines that can learn patterns humans would miss.
2. The source frames this through fintech. What is fintech: technology-driven innovation in the design and delivery of financial services and products.
3. In common usage, fintech can also refer to the companies building these tools and the business sector around them.
4. Early fintech automated routine tasks. Later systems executed decisions through explicit rules. Now some systems learn and make decisions using far more complex logic.
5. These tools matter not only for quant managers but also for fundamental managers using hybrid decision-making processes.

Fintech in Investment Analysis

1. The source highlights two areas that matter most for quantitative investment analysis: analysis of large datasets and analytical tools.
2. The first area is straightforward. There is far more traditional data and far more alternative data than older workflows were built to handle.
3. Traditional data include prices, volumes, corporate financial statements, economic indicators, annual reports, regulatory filings, earnings figures, and conference calls.
4. Alternative data come from non-traditional sources like social media, sensor networks, web traffic, emails, texts, satellites, and company exhaust.
5. What is company exhaust: information generated in the normal course of doing business.
6. The second area is the toolset itself. AI-based techniques may spot complex, non-linear relationships that older statistical methods can miss or process too slowly.
7. Analysts can use these tools to sort through gigantic filing sets, annual reports, and earnings calls to identify what matters most.

Big Data

1. Big Data refers to the vast amount of information generated by industry, governments, individuals, and electronic devices.
2. What is Big Data: extremely large and diverse datasets that often arrive quickly and require new ways to store, process, and analyze them.
3. The source says Big Data is characterized by three Vs: volume, velocity, and variety.
4. Volume means the dataset is huge, often millions or billions of data points.
5. Velocity means the data are recorded and transmitted much faster than before, often in real time or near real time.
6. Variety means the data come from many sources and in many formats rather than one clean tabular structure.
7. When Big Data is used for inference or prediction, the source adds a fourth V: veracity.
8. What is veracity: the credibility and reliability of the data source and the truthfulness of the data content.
9. This is the grown-up warning in the whole section. Big Data does not solve the old quality problem; it amplifies it.

Structured, Semistructured, and Unstructured Data

1. Big Data can be structured, semistructured, or unstructured.
2. Structured data fit naturally into tables and databases, where each field represents the same type of information.
3. Semistructured data have some organizational features but do not fit neatly into traditional rows-and-columns storage.
4. Unstructured data are disparate and unorganized and cannot be represented cleanly in tabular form.
5. Social media posts, emails, texts, voice recordings, pictures, blogs, scanners, and sensors often create unstructured data.
6. Why does this matter: unstructured data usually need specialized applications or customized code before analysts can use them meaningfully.

Sources of Big Data

1. The source names five broad origins of Big Data: financial markets, businesses, governments, individuals, and sensors.
2. Financial-market data include equity, fixed income, futures, options, and other derivatives data.
3. Business data include corporate financials, commercial transactions, credit card purchases, supply-chain data, and point-of-sale scanner data.
4. Government data include trade, economic, employment, and payroll data.
5. Individual-generated data include product reviews, search logs, personal data trails, and social media posts.
6. Sensor-generated data include satellite imagery, geolocation, shipping information, traffic patterns, and other machine-generated streams.
7. The Internet of Things sits inside this sensor story. What is the Internet of Things: a network of physical devices embedded with electronics, sensors, software, and connections so they can interact and share information.

Alternative Data

1. Alternative data are non-traditional data used to support investment decisions and models.
2. The source classifies the three main alternative-data sources as data generated by individuals, business processes, and sensors.
3. Data generated by individuals are often unstructured and can appear as text, video, photos, audio, website clicks, and browsing behavior.
4. Data generated by business processes are often structured and can act as leading or real-time indicators of performance.
5. What is a leading indicator here: information that may signal performance before traditional quarterly or annual reports arrive.
6. Sensor data often have the greatest scale, sometimes orders of magnitude larger than the other streams.
7. Investors use alternative data to search for factors affecting prices, improve asset selection, improve trade execution, and uncover trends.

Legal and Ethical Caution

1. The source explicitly warns that alternative data can create legal and ethical issues.
2. Web scraping may capture personal information protected by regulation or shared without explicit knowledge and consent.
3. Best practices are still evolving, and guidance can differ across jurisdictions.
4. Exam-wise, remember this as a compliance smell test: non-public-looking data are not automatically safe just because a machine collected them.

Big Data Challenges

1. Big Data creates practical problems before analysis even begins.
2. The source asks whether the dataset has selection bias, missing data, outliers, enough volume, and fitness for the intended analysis.
3. What is selection bias: a dataset distortion caused by how observations were chosen rather than by the true underlying phenomenon.
4. What are outliers: observations far away from the rest that can distort models or summaries.
5. Most datasets must be sourced, cleansed, and organized before analysis.
6. Alternative data make this harder because they are often unstructured and more qualitative than quantitative.

Artificial Intelligence

1. Artificial intelligence refers to computer systems capable of performing tasks that traditionally required human intelligence.
2. What is AI used for here: to perform or assist with complex analysis, pattern recognition, and decision-support tasks at human-comparable or superior levels.
3. An early AI example is the expert system, which tried to simulate human expertise using rule-based logic like if-then rules.
4. Since the 1980s, finance has used AI tools such as neural networks in fraud detection for abnormal charges and claims.
5. The broader story is that stronger processors and better networks allowed AI to move from toy problems into logistics, data mining, finance, and diagnosis.

Machine Learning

1. Machine learning is a subset of this evolution. What is machine learning: computer-based techniques that extract knowledge from large datasets by learning patterns from examples.
2. The source gives a very exam-friendly summary: find the pattern, apply the pattern.
3. ML does not require assumptions about the data's underlying probability distribution in the way many traditional methods do.
4. In ML, the algorithm is given inputs and may also be given outputs, depending on the learning setup.
5. Training occurs as the algorithm identifies relationships in the data and refines its learning process.
6. The source says ML usually splits data into training, validation, and test datasets.
7. Training data help the model learn historical relationships between inputs and outputs.
8. Validation data help tune the model and check whether the learned relationships hold up.
9. Test data evaluate whether the model predicts well on new data.
10. Human judgment still matters. Why is human judgment used: people must understand the data, clean the data, and choose appropriate analytical techniques.

Overfitting and Underfitting

1. Overfitting is one of the biggest exam traps in this reading.
2. What is overfitting: the model learns the training data too precisely and treats noise as if it were true signal.
3. An overfitted model can look brilliant on the training set and fail embarrassingly on a different dataset.
4. Underfitting is the opposite mistake. What is underfitting: the model is too simple and treats true structure as if it were noise.
5. Underfitted models fail to discover important patterns that actually exist in the data.
6. Some ML methods also look like black boxes. What is black box here: a model whose path from input to output is hard to explain clearly.

Types of Machine Learning

1. Supervised learning uses labeled training data, meaning the algorithm is given both inputs and known outputs.
2. Supervised learning is useful when you want prediction, such as forecasting returns or predicting whether a market will be up, down, or flat.
3. Unsupervised learning uses data without labels and tries to uncover structure, groupings, or relationships.
4. Grouping companies into peer clusters based on characteristics instead of sector labels is a clean unsupervised example from the source.
5. Deep learning uses neural networks, often with many hidden layers, to perform multistage, non-linear processing.
6. Deep learning can operate in supervised or unsupervised settings.
7. Why are hidden layers used: they let the model build from simple patterns toward more complex representations in stages.

Data Science

1. Data science is the interdisciplinary field that combines computer science, statistics, and related disciplines to extract information from data.
2. Data scientists are the people trying to convert raw data into usable insight for business and investment decisions.
3. The structure of the data matters because unstructured alternative data often need special treatment before analysis begins.

Data Processing Methods

1. The source highlights five key processing methods: capture, curation, storage, search, and transfer.
2. Capture refers to how data are collected and transformed into a format the analytical process can use.
3. Low-latency systems matter when automated trading depends on real-time prices and market events.
4. What is low latency: minimal delay in communication and processing.
5. Curation refers to cleaning the data and ensuring quality and accuracy.
6. Storage refers to how data are recorded, archived, accessed, and designed at the database level.
7. Search refers to how the system queries and retrieves the requested content from very large datasets.
8. Transfer refers to how data move from the source or storage location into the analytical tool.

Data Visualization

1. Visualization is not cosmetic. It is a thinking tool for understanding large and complex datasets.
2. Traditional structured data can be shown with tables, charts, and trends.
3. Unstructured or multidimensional data often need richer visual forms like interactive 3D graphics, heat maps, tree diagrams, network graphs, tag clouds, and mind maps.
4. What is a tag cloud: a visual display where words appear larger when they occur more frequently in the source text.
5. What is a mind map: a visualization showing how concepts relate to one another rather than just how often words appear.

Text Analytics and NLP

1. Text analytics uses computer programs to analyze and derive meaning from large text- or voice-based datasets.
2. This includes filings, reports, earnings calls, social media, emails, postings, and surveys.
3. More advanced text analytics can include lexical analysis, meaning frequency analysis of words and recognition of patterns in words and phrases.
4. Natural language processing is an important application inside this area.
5. What is NLP: a field combining computer science, AI, and linguistics to analyze and interpret human language.
6. NLP can handle translation, speech recognition, text mining, sentiment analysis, and topic analysis.
7. NLP can also support compliance monitoring, fraud detection, and confidentiality controls in employee communications.
8. In investing, NLP can analyze analyst commentary, annual reports, call transcripts, news articles, and policy-maker communications at a scale no human team can match.
9. It can tag sentiment changes before an analyst formally changes a buy, hold, or sell recommendation.
10. It can also detect subtle topic shifts in central-bank or policy communications around inflation, output, or rate policy.

Programming Languages and Databases

1. The source names several common programming languages used in data science.
2. Python is open-source and approachable and underlies many fintech applications.
3. R is open-source and historically strong in statistics, ML, optimization, econometrics, and financial analysis.
4. Java runs across different machines and operating systems and supports many internet applications.
5. C and C++ are specialized for speed and processing performance and are used in algorithmic and high-frequency trading.
6. Excel VBA helps automate workflows, update data, run macros, gather web data, and build customized reports.
7. On the database side, SQL works for structured data stored in rows and columns on servers.
8. SQLite is a structured-data database embedded into programs and is common in mobile applications.
9. NoSQL is used for unstructured data that cannot be summarized in traditional tables.

Final Exam Traps

1. Big Data is not defined by size alone. The full exam frame is volume, velocity, variety, and often veracity.
2. Alternative data are not the same as traditional corporate or market data.
3. ML still needs human judgment, clean data, and enough data.
4. Overfitting means learning noise as signal; underfitting means missing real structure.
5. Supervised learning uses labels; unsupervised learning does not.
6. NLP sits inside text analytics and is directly useful for sentiment, topic, and language interpretation in finance.