Reference
Data Science & Analytics
Data analysis languages and platforms, big data tooling, data interchange formats, and the AI / machine learning / deep learning hierarchy.
Data Analysis Languages
| Language | Type | Description |
|---|---|---|
| Python | General Purpose | Dominant language for data science and ML; pandas, NumPy, scikit-learn, PyTorch, TensorFlow ecosystem |
| R | Statistical | Purpose-built for statistical computing and visualization; strong in academia and biostatistics |
| Julia | Numerical | High-performance numerical and scientific computing with Python-like syntax and C-like speed |
| SQL | Query | Standard language for querying and aggregating structured, relational data |
| SAS | Statistical | Legacy enterprise statistical software language, common in pharma, banking, and insurance |
| Scala | Big Data | JVM language behind Apache Spark's core API; used for large-scale distributed data processing |
| MATLAB | Numerical | Matrix-oriented language for engineering simulation and numerical analysis |
Platforms, Notebooks & Software
| Tool | Type | Description |
|---|---|---|
| Jupyter Notebook | Interactive Notebook | Open-source browser-based environment mixing code, output, and text in cells (.ipynb files) |
| Google Colab | Cloud Notebook | Free hosted Jupyter environment with GPU/TPU access, backed by Google Drive storage |
| RStudio | IDE | Dedicated integrated development environment for the R language |
| SPSS | Statistical Software | IBM's GUI-driven statistical analysis package, widely used in social sciences and survey research |
| Databricks | Unified Analytics Platform | Cloud platform built around Apache Spark for large-scale data engineering and ML workflows |
| Anaconda | Distribution & Package Manager | Python/R distribution bundling data-science packages, environments managed via conda |
Big Data & Data Mining
| Tool / Technique | Category | Description |
|---|---|---|
| Hadoop | Distributed Storage & Compute | HDFS (distributed file system) + MapReduce processing framework for very large datasets across clusters |
| Apache Spark | Distributed Processing | In-memory distributed data processing engine, much faster than MapReduce for iterative workloads |
| Apache Hive | Query Layer | SQL-like querying interface on top of Hadoop-stored data |
| Apache Kafka | Event Streaming | Distributed platform for publishing, storing, and processing continuous streams of records |
| Clustering | Data Mining Technique | Groups similar data points together without predefined labels (e.g. k-means) |
| Association Rule Mining | Data Mining Technique | Finds relationships between variables in large datasets (e.g. market-basket analysis) |
| Anomaly Detection | Data Mining Technique | Identifies data points that deviate significantly from the norm |
Data Interchange & Serialization Formats
| Format | Encoding | Description |
|---|---|---|
| JSON | Text | Human-readable key-value format; the default for web APIs and config files |
| Protocol Buffers (protobuf) | Binary, schema-based | Google's compact binary serialization format; core to gRPC service communication |
| BSON | Binary | Binary-encoded JSON superset used natively by MongoDB, adds types like dates and binary data |
| Avro | Binary, schema-based | Apache's row-oriented binary format with embedded schema; common in Kafka and Hadoop pipelines |
| Parquet | Binary, columnar | Column-oriented storage format optimized for analytical queries over large datasets |
AI / ML / DL Hierarchy
| Term | Relationship | Description |
|---|---|---|
| Artificial Intelligence (AI) | Broadest field | Any technique enabling machines to perform tasks that normally require human-like intelligence |
| Machine Learning (ML) | Subset of AI | Algorithms that learn patterns directly from data rather than following explicit rules |
| Deep Learning (DL) | Subset of ML | Uses multi-layer (deep) neural networks to learn hierarchical representations of data |
| LLM (Large Language Model) | Subset of DL | Deep learning model trained on massive text corpora to understand and generate language |
| Regression | ML technique | Predicts a continuous numeric value (e.g. linear regression); logistic regression predicts a class probability |
| Classification | ML technique | Predicts a discrete category or label for each input |
| Clustering | ML technique | Groups data into clusters without labeled outcomes (unsupervised learning) |