Taxonomy of Learning Types in AI

A near-exhaustive, structured classification of AI and Machine Learning paradigms. This updated master edition combines full geographic breadth (covering classical statistical theory to quantum computing) with actionable depth (Mental Models, Usage Rules, Failure Modes, and Go-To Tools).

mindmap
  root((ML Paradigms))
    Supervision & Data
      Supervised
      Unsupervised
      Semi-Supervised
      Self-Supervised
      Weakly Supervised
      Multi-Instance
      Reinforcement
    Structure & Geometry
      Graph & GNNs
      Knowledge Graphs
      Manifold
      Equivariant / Group
      Graph Signal Processing
      Structured Prediction
    Uncertainty & Generative
      PGMs & Bayesian
      Generative Models
      Energy-Based Models
    Adaptation & Efficiency
      Transfer & Domain Adaptation
      Meta & Few-Shot
      PEFT / Adapters
      Continual Learning
      Transductive
    Representation & Metric
      Contrastive Learning
      Metric Learning
      Info-Theoretic
    Reasoning & Autonomy
      Agentic AI & Tool-Use
      Neuro-Symbolic
      Symbolic Regression
    Robustness & Safety
      Adversarial
      Causal Inference
      Federated & Privacy
      XAI & Mech. Interp
    Hardware & Frontiers
      Edge ML
      Neuromorphic
      Quantum ML
      Topological Data Analysis
      Physics-Informed ML

1. Core Supervision Paradigms

Methods defined by feedback mechanism and label availability.

Supervised Learning

Mental Model: Train models using paired inputs and ground-truth labels to predict discrete classes or continuous values.
When to Use: You have abundant, high-quality labeled data \((X, Y)\) and a clear target variable (e.g., spam detection, object detection).
When NOT to Use: Ground-truth labels are unavailable, too expensive, or you want to discover unknown patterns.
Primary Failure Mode: Overfitting to the training distribution; failure to generalize to out-of-distribution (OOD) data.
Go-To Tools: Scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow.

Unsupervised Learning

Mental Model: Discovers inherent structure, clusters, or underlying probability distributions in data without any target labels.
When to Use: Exploratory data analysis, customer segmentation, anomaly detection, or initial representation learning.
When NOT to Use: You have a specific prediction objective and ground-truth target variables.
Primary Failure Mode: Constructing uninterpretable or arbitrary clusters due to sensitivity to distance metrics and scaling.
Go-To Tools: Scikit-learn (K-Means, DBSCAN, PCA, GMM), HDBSCAN, UMAP.

Semi-Supervised Learning

Mental Model: Leverages a small set of labeled data alongside a large volume of unlabeled data to improve decision boundaries.
When to Use: Labeling requires expensive human expertise (e.g., medical pathology), but raw unlabeled data is plentiful.
When NOT to Use: You already possess enough labeled data to hit model performance ceilings.
Primary Failure Mode: Confirmation bias / Error propagation (pseudo-labeling wrong instances and reinforcing errors).
Go-To Tools: Scikit-learn (Label Propagation/Spreading), PyTorch (FixMatch, FlexMatch).

Self-Supervised Learning (SSL)

Mental Model: Generates its own supervisory signal from raw data by hiding parts of the input and training the model to predict the missing pieces (pretext tasks).
When to Use: Pretraining Foundation Models (LLMs, Vision Transformers) on massive internet-scale data before downstream fine-tuning.
When NOT to Use: Extreme compute/budget constraints, or simple tabular regression tasks where baseline ML models excel.
Primary Failure Mode: Representation collapse (the model outputs identical trivial embeddings for all inputs).
Go-To Tools: Hugging Face, OpenCLIP, PyTorch (SimCLR, MoCo, MAE, DINOv2).

Weakly Supervised Learning

Mental Model: Trains models using noisy, programmatic, or higher-level heuristic labels rather than hand-annotated ground truth.
When to Use: You can write programmatic labeling functions (heuristics, regex, noisy rules) to label millions of records quickly.
When NOT to Use: Safety-critical applications (e.g., autonomous driving, medical dosage) requiring high precision guarantees.
Primary Failure Mode: The model overfits to the inherent bias and false positives of the heuristic rules.
Go-To Tools: Snorkel, Cleanlab, Rubrix/Argilla.

Multi-Instance Learning (MIL)

Mental Model: Labels are assigned to “bags” (groups) of instances; a bag is positive if at least one instance inside it is positive.
When to Use: Computational pathology (gigapixel Whole-Slide Images), long video tag detection, or chemical binding prediction.
When NOT to Use: You already have fine-grained, instance-level annotations.
Primary Failure Mode: Witness isolation failure—the model over-indexes on a single noisy patch and ignores broader context.
Go-To Tools: PyTorch (Attention-based MIL / ABMIL, TransMIL, CLAM).

Reinforcement Learning (RL)

Mental Model: An agent learns optimal policy behavior by taking actions in an environment to maximize cumulative scalar rewards.
When to Use: Sequential decision-making problems with dynamic environments (robotics, gaming, LLM alignment via RLHF/RLAIF).
When NOT to Use: Static datasets where actions do not change future states, or where simulators do not exist.
Primary Failure Mode: Reward hacking (agent exploits simulator bugs for points without solving the task) and extreme sample inefficiency.
Go-To Tools: Ray RLlib, Stable-Baselines3, Gymnasium, TRL (Transformer Reinforcement Learning).

2. Geometric, Structural & Relational Learning

Methods for non-Euclidean spaces, graphs, manifolds, and structured output spaces.

Graph-Based Learning & GNNs

Mental Model: Models entities (nodes) and their explicit pairwise relationships (edges) via message passing along graph topologies.
When to Use: Social network analysis, molecular property prediction, citation networks, and recommendation graphs.
When NOT to Use: Data consists of independent, identically distributed (IID) tabular rows or rigid Euclidean grids (use CNNs/Transformers).
Primary Failure Mode: Over-smoothing (deep GNN layers cause node features to converge to a uniform average).
Go-To Tools: PyTorch Geometric (PyG), Deep Graph Library (DGL), Spektral.

Knowledge Graphs & Relational Learning

Mental Model: Encodes world facts as semantic triples \((Subject, Predicate, Object)\) to perform logical reasoning, link prediction, and GraphRAG.
When to Use: Grounding LLMs in verifiable enterprise facts (GraphRAG), fraud detection, and multi-hop reasoning.
When NOT to Use: Continuous signal processing tasks (e.g., raw audio wave processing).
Primary Failure Mode: High maintenance overhead, schema/ontology drift, and sparse incomplete graphs.
Go-To Tools: Neo4j, LlamaIndex (GraphRAG), PyKEEN, AmpliGraph (TransE, RotatE).

Manifold Learning

Mental Model: Assumes high-dimensional data lies on a lower-dimensional continuous manifold embedded within the high-dimensional space.
When to Use: Visualizing complex high-dimensional embeddings (e.g., single-cell RNA, word embeddings) in 2D/3D.
When NOT to Use: You require an exact, mathematically invertible mapping back to the original space.
Primary Failure Mode: Distorting global topology in favor of preserving local distances (or vice versa).
Go-To Tools: UMAP, t-SNE, Isomap, LLE (Scikit-learn).

Group & Equivariant Networks

Mental Model: Enforces mathematical symmetry constraints (e.g., rotational, translational, reflectional invariance) directly into network architectures.
When to Use: 3D molecular conformation, protein folding (AlphaFold), and 3D point-cloud processing where orientation shouldn’t change the physics.
When NOT to Use: Standard image classification where orientation matters (e.g., distinguishing a 6 from a 9).
Primary Failure Mode: High computational expense and complex tensor product implementations.
Go-To Tools: e3nn, Geometric3D, Escnn.

Graph Signal Processing (GSP)

Mental Model: Extends traditional harmonic analysis (Fourier transforms, filtering) from Euclidean domains to irregular graph domains.
When to Use: Analyzing sensor networks, brain connectomics, and power grid signals structured on graphs.
When NOT to Use: Standard time-series data with uniform sampling intervals.
Primary Failure Mode: Matrix decomposition bottlenecks on massive graphs (\(>1M\) nodes).
Go-To Tools: PyGSP, GraphSignal.

Structured Prediction

Mental Model: Predicts structured objects (trees, sequences, graphs) rather than scalar labels, accounting for inter-dependencies between output components.
When to Use: Sequence labeling (NER), syntactic parsing, scene-graph generation, and image segmentation.
When NOT to Use: Independent multi-class prediction problems.
Primary Failure Mode: Exponential output space exploration during inference (requiring approximate decoding like Beam Search).
Go-To Tools: Conditional Random Fields (CRFs), Structured SVMs, PyTorch.

3. Probabilistic and Bayesian Approaches

Methods for modeling uncertainty, prior beliefs, and exact/approximate probability distributions.

Probabilistic Graphical Models (PGMs)

Mental Model: Expresses conditional independence structure between random variables using directed (Bayesian Networks) or undirected (Markov Random Fields) graphs.
When to Use: Medical diagnosis, fault trees, and domains where human experts must inspect conditional probability tables.
When NOT to Use: High-dimensional perception tasks (raw image or video classification).
Primary Failure Mode: Exact inference is NP-hard, forcing reliance on approximate inference methods.
Go-To Tools: pgmpy, BayesNet, Netica.

Bayesian Learning & Neural Networks

Mental Model: Places probability distributions over model parameters (weights) to capture epistemic uncertainty alongside predictions.
When to Use: High-stakes decision making (drug discovery, autonomous driving, finance) where knowing when the model is uncertain is critical.
When NOT to Use: Low-latency real-time applications requiring sub-millisecond inference.
Primary Failure Mode: Compute overhead during inference (sampling across posterior distributions) and difficult prior selection.
Go-To Tools: PyMC, Pyro, TensorFlow Probability, GPyTorch.

Probabilistic Programming

Mental Model: Allows developers to specify complex statistical models as code and automates inference workflows (MCMC, Variational Inference).
When to Use: Custom hierarchical modeling, causal discovery, and Bayesian statistical modeling.
When NOT to Use: Standard black-box deep learning tasks.
Primary Failure Mode: MCMC convergence failure or slow sampler performance on high-dimensional parameter spaces.
Go-To Tools: Stan, Pyro, PyMC, NumPyro.

Latent Variable Models (LVMs)

Mental Model: Explains observed variables through unobserved (latent) variables, inferring hidden factors that drive data generation.
When to Use: Matrix factorization for recommender systems, topic modeling (LDA), and factor analysis.
When NOT to Use: Simple direct supervised mapping where internal hidden state analysis isn’t needed.
Primary Failure Mode: Latent variable non-identifiability (multiple latent structures producing identical observed distributions).
Go-To Tools: Scikit-learn (NMF, LDA), PyTorch.

4. Generative and Energy-Based Models

Methods for learning data distributions, sampling realistic synthetic instances, and scoring configurations.

Generative Models (Diffusion, VAEs, GANs, Autoregressive)

Mental Model: Learns the underlying data distribution \(P(X)\) or \(P(X|Y)\) to generate brand-new, realistic data instances.
When to Use: Image/Video generation, text synthesis (LLMs), molecular generation, audio synthesis.
When NOT to Use: Simple classification tasks where estimating density is an unnecessary intermediate step (Vapnik’s principle).
Primary Failure Mode: Mode collapse (GANs), posterior collapse (VAEs), or slow iterative sampling speed (Diffusion).
Go-To Tools: Hugging Face Diffusers, vLLM, PyTorch.

Energy-Based Models (EBMs)

Mental Model: Associates an unnormalized scalar energy value to every data configuration; plausible points get low energy, implausible points get high energy.
When to Use: Compositional generation, out-of-distribution detection, and flexible multimodal generation without normalized densities.
When NOT to Use: Settings requiring fast single-pass generation without iterative Langevin dynamics sampling.
Primary Failure Mode: Unstable training due to intractable partition functions requiring Markov Chain Monte Carlo (MCMC) approximations.
Go-To Tools: PyTorch (EBM frameworks).

5. Transfer, Adaptation, and Parameter-Efficient Learning

Methods for transferring pre-existing knowledge, adapting across tasks, and resource-friendly fine-tuning.

Transfer Learning & Domain Adaptation

Mental Model: Takes features learned on a source domain (e.g., ImageNet, C4) and adapts them to a target domain with different distributions.
When to Use: Target dataset is small, but massive pretrained base models exist in a related domain.
When NOT to Use: The source and target domains share zero underlying structure (e.g., pretrained natural language model applied to raw EEG signals).
Primary Failure Mode: Negative transfer (source knowledge actively degrades performance on the target task).
Go-To Tools: Hugging Face, PyTorch Image Models (timm).

Meta-Learning & Few-Shot Learning

Mental Model: “Learns how to learn”—optimizes model parameters or architectures across a distribution of tasks so it can adapt to new tasks in 1–5 steps.
When to Use: Edge cases where new classes/tasks arrive continuously with only 1–5 examples per class.
When NOT to Use: Single static tasks with large amounts of training data available.
Primary Failure Mode: Meta-overfitting (generalizes well to seen training tasks, fails on unseen task distributions).
Go-To Tools: Learn2Learn, Higher (PyTorch).

Parameter-Efficient Fine-Tuning (PEFT)

Mental Model: Keeps base model parameters frozen and inserts tiny trainable rank-decomposition matrices or adapter modules.
When to Use: Fine-tuning massive modern foundation models (LLMs/Vision models) on limited consumer hardware GPU memory.
When NOT to Use: Pretraining models from scratch or fine-tuning tiny models where memory isn’t a bottleneck.
Primary Failure Mode: Adapter capacity bottlenecks (r-rank too small to absorb domain-specific domain nuance).
Go-To Tools: Hugging Face PEFT (LoRA, QLoRA, Prefix Tuning), Unsloth, Axolotl.

Continual / Lifelong Learning

Mental Model: Sequentially updates a model on streaming tasks while preserving previous task capabilities without full retraining.
When to Use: Production systems receiving continuous streaming domain updates where historical raw data cannot be stored (privacy/bandwidth).
When NOT to Use: Systems where periodic batch retraining from scratch on accumulated historical data is feasible.
Primary Failure Mode: Catastrophic forgetting (new task gradients completely erase weight configurations for old tasks).
Go-To Tools: Avalanche (PyTorch), Continual AI.

Transductive Learning

Mental Model: Makes predictions specifically for a known, fixed set of unlabeled test instances given at training time, without inducing a general rule.
When to Use: Semi-supervised graph node classification where all graph nodes (train and test) are fixed and present upfront.
When NOT to Use: Inductive settings where new unseen samples will arrive dynamically in production.
Primary Failure Mode: Complete inability to handle out-of-sample inference without re-running optimization.
Go-To Tools: Scikit-learn (Label Propagation), Graph Neural Networks (transductive split).

7. Representation, Metric, and Contrastive Learning

Methods explicitly designed to learn structured feature spaces, distance metrics, and semantic embeddings.

Representation Learning

Mental Model: Transforms raw unstructured input data into compact, dense vector representations that make downstream tasks easier.
When to Use: Feature extraction for recommendation systems, semantic search, and clustering.
When NOT to Use: Interpretable decision tree tasks on low-dimensional tabular data.
Primary Failure Mode: Extracting task-irrelevant background noise into the representation vector.
Go-To Tools: Sentence-Transformers, Autoencoders, PyTorch.

Contrastive Learning

Mental Model: Trains embeddings by pulling positive (similar) pairs together in vector space while pushing negative (dissimilar) pairs apart.
When to Use: Unsupervised pretraining, image/text search indexing, and deduplication.
When NOT to Use: Small datasets with abundant task-specific supervision.
Primary Failure Mode: False negative toxicity (pushing apart two samples that are semantically identical due to random batch sampling).
Go-To Tools: SimCLR, MoCo, InfoNCE, Lightly.

Metric Learning

Mental Model: Directly learns a distance function \(d(x_1, x_2)\) over vector spaces such that semantic similarity matches distance metrics.
When to Use: Face recognition, signature verification, and few-shot classification.
When NOT to Use: Fixed, non-metric relational data.
Primary Failure Mode: Triplet mining bottlenecks (finding hard positives and hard negatives becomes computationally intensive).
Go-To Tools: PyTorch-Metric-Learning, Faiss.

Information-Theoretic Methods

Mental Model: Optimizes data representations by maximizing Mutual Information \(I(X; Z)\) between inputs and latent codes while bounding information capacity.
When to Use: Disentangled representation learning, Information Bottleneck framework applications.
When NOT to Use: Applications where exact density bounds are difficult to estimate.
Primary Failure Mode: Mutual information estimation in high dimensions is notoriously noisy and unstable.
Go-To Tools: Barlow Twins, VicReg, MI estimators.

8. Kernel and Classical Statistical Learning

Traditional statistical frameworks rooted in functional analysis, VC dimension, and regularized empirical risk.

Kernel Methods

Mental Model: Implicitly maps data into an infinite-dimensional inner-product Hilbert space using a kernel function \(K(x, x')\).
When to Use: Small-to-medium tabular datasets requiring strict non-linear decision boundaries with strong theoretical guarantees.
When NOT to Use: Large-scale datasets (\(>100k\) samples) due to \(O(N^2)\) to \(O(N^3)\) kernel matrix memory costs.
Primary Failure Mode: Scaling bottleneck; highly sensitive to kernel hyperparameter selection (e.g., RBF gamma).
Go-To Tools: Scikit-learn (SVM, Kernel Ridge, Kernel PCA), LIBSVM.

Statistical Learning Theory

Mental Model: Analyzes generalization bounds, Probably Approximately Correct (PAC) learning framework, and Empirical Risk Minimization (ERM).
When to Use: Theoretical safety verification, bounding generalization error, and establishing formal guarantees.
When NOT to Use: Applied empirical engineering where empirical validation suffices.
Primary Failure Mode: Vacuous bounds (theoretical VC bounds are often far too loose for modern overparameterized deep networks).
Go-To Tools: Theoretical proofs, CVXPy.

Regularization Frameworks

Mental Model: Penalizes model complexity (L1, L2, Dropout, Weight Decay) to constrain search spaces and prevent overfitting.
When to Use: Standard inclusion across almost all ML model training pipelines.
When NOT to Use: Underfitting models with insufficient capacity for the task.
Primary Failure Mode: Over-regularization (destroying model capacity and introducing high bias).
Go-To Tools: Built into PyTorch, Scikit-learn, XGBoost.

9. Optimization and Algorithmic Variants

Iterative search techniques, meta-heuristics, and non-gradient optimization strategies.

Gradient-Based Optimization

Mental Model: Uses first- or second-order derivatives of loss functions to iteratively update parameters along steepest descent pathways.
When to Use: Differentiable continuous neural networks and deep learning models.
When NOT to Use: Non-differentiable step functions, discrete combinatorial search, or black-box objective functions.
Primary Failure Mode: Vanishing/exploding gradients, getting trapped in bad local minima or saddle points.
Go-To Tools: PyTorch (AdamW, SGD, L-BFGS), JAX (Optax).

Convex vs Non-Convex Optimization

Mental Model: Convex problems guarantee that any local minimum is global; non-convex problems feature complex landscapes with many local minima.
When to Use: Convex: SVMs, Linear Regression, LASSO. Non-Convex: Deep Learning.
When NOT to Use: N/A (Dictated by problem formulation).
Primary Failure Mode: Assuming local convergence equals global optimality in non-convex settings.
Go-To Tools: CVXPY, SciPy Optimize.

Meta-Optimization & Hyperparameter Tuning

Mental Model: Automates the search for optimal training hyperparameters using Bayesian Optimization, Population-Based Training, or Bandits.
When to Use: Systematic model tuning before production deployment.
When NOT to Use: Early exploratory prototyping phases where default parameters suffice.
Primary Failure Mode: High compute waste evaluating unproductive search spaces.
Go-To Tools: Optuna, Ray Tune, Hyperopt.

Evolutionary / Neuroevolution

Mental Model: Evolves population candidates using mutation, crossover, and selection rules inspired by biological evolution.
When to Use: Non-differentiable environments, neural architecture search (NAS), and reinforcement learning without gradients.
When NOT to Use: Standard supervised problems where exact analytical gradients are easily computed.
Primary Failure Mode: Slow convergence rates compared to first-order gradient descent on high-dimensional parameter spaces.
Go-To Tools: DEAP, PyGaggle, Evotorch.

Swarm Intelligence

Mental Model: Coordinates decentralized agents (particles, ants) that interact locally to solve global optimization problems.
When to Use: Combinatorial routing problems (Traveling Salesperson), continuous black-box parameter optimization.
When NOT to Use: High-dimensional deep neural network weight optimization.
Primary Failure Mode: Premature convergence to suboptimal local clusters.
Go-To Tools: PySwarms, Scikit-opt.

10. Robustness, Safety, Privacy, and Fairness

Methods ensuring trustworthiness, ethical compliance, security against attacks, and interpretability.

Adversarial Training & Robustness

Mental Model: Trains models on adversarially perturbed inputs (\(x + \epsilon\)) to build immunity against deliberate security attacks.
When to Use: Security-critical systems (biometrics, autonomous vehicles, spam/malware filters).
When NOT to Use: Low-risk internal applications where compute overhead outweighs security risks.
Primary Failure Mode: Robustness-Accuracy trade-off (improving adversarial robustness often slightly reduces clean test accuracy).
Go-To Tools: Adversarial Robustness Toolbox (ART), CleverHans, Foolbox.

Causal Learning and Inference

Mental Model: Moves beyond statistical correlation (\(P(Y|X)\)) to model structural cause-and-effect relationships via interventions (\(P(Y|do(X))\)).
When to Use: Policy making, clinical trials, A/B testing evaluation, and root-cause failure analysis.
When NOT to Use: Pure prediction problems where environmental distributions are guaranteed to remain static.
Primary Failure Mode: Unobserved confounders violating structural assumptions and distorting causal estimates.
Go-To Tools: DoWhy, EconML, CausalML, CausalDiscovery.

Federated & Distributed Learning

Mental Model: Trains a shared global model across decentralized devices (hospitals, mobile phones) holding local data without aggregating raw data centrally.
When to Use: Strict data privacy regulations (GDPR, HIPAA) preventing centralized data collection.
When NOT to Use: All data is already legally centralized in a secure cloud data lake.
Primary Failure Mode: High communication latency overhead and performance degradation on non-IID client data distributions.
Go-To Tools: Flower (flwr.dev), TensorFlow Federated, PySyft.

Privacy-Preserving ML (Differential Privacy & SMPC)

Mental Model: Adds calibrated noise or encryption primitives to guarantee individual records cannot be reconstructed from trained models.
When to Use: Training models on sensitive medical, financial, or personal user data.
When NOT to Use: Public non-sensitive open-source datasets.
Primary Failure Mode: Privacy-Utility trade-off (adding privacy noise reduces overall downstream prediction accuracy).
Go-To Tools: Opacus (PyTorch DP), PySyft, Concrete-ML (Fully Homomorphic Encryption).

Fairness-Aware Learning

Mental Model: Injects statistical constraints (Demographic Parity, Equalized Odds) into training to mitigate demographic bias.
When to Use: High-stakes human scoring systems (credit scoring, hiring algorithms, judicial risk assessments).
When NOT to Use: Pure physical perception tasks (e.g., satellite weather analysis).
Primary Failure Mode: Mitigation trade-offs (optimizing for one definition of fairness can violate another mathematical definition of fairness).
Go-To Tools: Fairlearn, AIF360 (IBM).

Explainable AI (XAI) & Mechanistic Interpretability

Mental Model: Post-hoc explanation tools (SHAP/LIME) score feature importance; Mechanistic Interpretability reverse-engineers neural circuits directly.
When to Use: Audit requirements, regulatory compliance, and understanding internal model reasoning.
When NOT to Use: Non-critical applications where latency and raw predictive accuracy are the only requirements.
Primary Failure Mode: Post-hoc feature attributions can be tricked or give plausible-sounding explanations for flawed predictions.
Go-To Tools: SHAP, Captum, TransformerLens (Mechanistic Interpretability).

11. LLMs, Agentic Frameworks & Symbolic ML

Methods involving multi-step reasoning, external tool execution, and hybrid neural-symbolic systems.

Agentic AI & Tool-Use

Mental Model: Couples an LLM with external tools (APIs, web browsers, code interpreters) and an iterative reflection/planning loop.
When to Use: Complex multi-step workflows (autonomous coding, research synthesis, enterprise automation).
When NOT to Use: Simple single-turn text classification or deterministic tasks.
Primary Failure Mode: Infinite execution loops, tool parameter hallucination, and cascading errors in long chains.
Go-To Tools: LangChain, LlamaIndex, AutoGen, CrewAI, DSPy.

Neuro-Symbolic Learning

Mental Model: Combines neural network perceptual capabilities with formal symbolic logic rules to guarantee rule compliance.
When to Use: Safety-critical domains, automated theorem proving, and strict rule-based business compliance.
When NOT to Use: Standard unstructured perception tasks where formal logic rules cannot be written down.
Primary Failure Mode: The interface boundary between continuous gradient optimization and discrete symbolic logic is difficult to optimize.
Go-To Tools: DeepProbLog, Scallop, PyTorch.

Symbolic Regression

Mental Model: Searches the space of mathematical expressions to find an explicit analytical equation \(y = f(x)\) that fits data cleanly.
When to Use: Uncovering closed-form scientific formulas from experimental observational data.
When NOT to Use: High-dimensional image or text perception.
Primary Failure Mode: Exponential combinatorial search explosion over expression tree structures.
Go-To Tools: PySR, gplearn.

12. Sparse, Low-Rank, and Signal Processing

Methods exploiting sparsity, low-rank structure, and parameter compression.

Sparse Learning

Mental Model: Enforces zero-value parameter constraints (L1 regularization, pruning) to compress models and accelerate computation.
When to Use: Deploying models under strict memory/latency limits or identifying key sparse features.
When NOT to Use: Tasks where every input feature contributes a small, non-zero dense signal.
Primary Failure Mode: Pruning too heavily reduces model capacity and causes accuracy drop-offs.
Go-To Tools: Neural Magic (SparseML), PyTorch Pruning.

Dictionary Learning & Sparse Coding

Mental Model: Represents data vectors as linear combinations of a small number of atomic elements chosen from an overcomplete dictionary.
When to Use: Image denoising, signal reconstruction, compressed sensing.
When NOT to Use: Massive end-to-end deep learning visual representation tasks.
Primary Failure Mode: High computational cost of dictionary updates (e.g., K-SVD) on massive data volumes.
Go-To Tools: Scikit-learn (DictionaryLearning), SPAMS.

13. Hardware-Inspired and Edge Learning

Methods tailored specifically for constrained hardware, spiking brain-like chips, or quantum processing units.

Edge ML & Model Compression

Mental Model: Optimizes models specifically for low-power edge hardware via quantization (INT8/INT4), pruning, and knowledge distillation.
When to Use: Running local AI models on smartphones, microcontrollers (MCU), and IoT devices.
When NOT to Use: Cloud server deployments with unconstrained GPU resources.
Primary Failure Mode: Quantization degradation (converting FP16 to INT4 damages precision on complex reasoning tasks).
Go-To Tools: TensorRT, ONNX Runtime, TFLite, GGML/llama.cpp.

Neuromorphic Learning (Spiking Neural Networks - SNNs)

Mental Model: Mimics biological brain dynamics using event-driven, time-dependent “spikes” to process information with ultra-low power consumption.
When to Use: Processing continuous event-based camera sensors (DVS) with milliwatt energy budgets.
When NOT to Use: Standard offline batch image or tabular processing.
Primary Failure Mode: Non-differentiable spike activation functions complicate standard backpropagation training.
Go-To Tools: sNNorch, Lava, BindsNET.

Quantum Machine Learning (QML)

Mental Model: Maps data onto quantum states (qubits) and uses variational quantum circuits to execute calculations in Hilbert spaces.
When to Use: Quantum chemistry simulations, quantum materials discovery, and complex optimization.
When NOT to Use: General everyday data processing on classical compute infrastructure.
Primary Failure Mode: Barren plateaus (flat optimization gradients) and noisy intermediate-scale quantum (NISQ) hardware errors.
Go-To Tools: PennyLane, Qiskit, Cirq.

14. Topological, Physical, and Scientific ML

Methods extracting topological invariants or embedding fundamental physical differential equations.

Topological Data Analysis (TDA)

Mental Model: Measures topological shape features (loops, voids, connected components) that persist across multiple spatial scales.
When to Use: Analyzing complex biological shapes, structural genomic data, and geometric point-cloud structures.
When NOT to Use: Standard structured tabular data with no spatial/topological structure.
Primary Failure Mode: High computational complexity (\(O(N^3)\) or worse) for persistent homology calculations on large point clouds.
Go-To Tools: Giotto-tda, Ripser, Gudhi.

Physics-Informed ML (PINNs) & Neural Operators

Mental Model: Embeds partial differential equations (PDEs) directly into the loss function to ensure outputs obey conservation laws.
When to Use: Computational fluid dynamics, weather forecasting, heat transfer, and structural engineering.
When NOT to Use: Social science or financial data with no known differential physical laws.
Primary Failure Mode: Optimization difficulty balancing PDE residual losses with empirical data losses.
Go-To Tools: NVIDIA Modulus, DeepXDE, NeuralOperators.

15. Societal, Operational & Special Settings

Real-world execution modes, interactive loops, and alignment settings.

Value Alignment & Moral Learning (RLHF / DPO)

Mental Model: Aligns AI model behavior with human preferences, values, and safety boundaries using reward models or direct preference optimization.
When to Use: Post-training foundation models for public-facing chat, safety filtering, and helpful tone alignment.
When NOT to Use: Pure technical predictive tools where subjective preference plays no role.
Primary Failure Mode: Sycophancy (model tells users what they want to hear) and reward hacking.
Go-To Tools: TRL (Transformer Reinforcement Learning), Alignment-Handbook, Axolotl.

Human-in-the-Loop & Active Learning

Mental Model: Interactively queries a human expert to annotate only the most uncertain or high-information data points.
When to Use: Labeling budgets are strictly capped and raw data is vast (e.g., active learning for rare defect detection).
When NOT to Use: Fully automated real-time systems where human response latency is unacceptable.
Primary Failure Mode: Sampling bias (active learning over-samples noisy outlier points instead of true boundaries).
Go-To Tools: MODAL, Label Studio, cleanlab.

Online & Streaming Learning

Mental Model: Incrementally updates model weights row-by-row on real-time data streams without storing history.
When to Use: High-frequency trading, real-time ad click-through rate prediction, handling concept drift.
When NOT to Use: Static offline datasets where batch processing is easily achievable.
Primary Failure Mode: Catastrophic drift if catastrophic unexpected noise Enters the streaming pipeline.
Go-To Tools: River (Python), Scikit-multiflow.

Ensemble Methods

Mental Model: Combines predictions from multiple weak base models (bagging, boosting, stacking) to produce a stronger ensemble prediction.
When to Use: Standard benchmark competitions, tabular datasets, and maximizing raw predictive accuracy.
When NOT to Use: Low-latency edge applications where running \(100\) models simultaneously is impossible.
Primary Failure Mode: High compute latency, complex deployment maintenance, and loss of interpretability.
Go-To Tools: XGBoost, LightGBM, CatBoost, Scikit-learn (Voting/Stacking).

Cheat Sheet: Core Paradigm Decision Matrix

Paradigm	Primary Input Requirement	Core Objective	Primary Failure Mode	Example Use Case
Supervised	Labeled \((X, Y)\)	Predict \(Y\) from \(X\)	Overfitting training distribution	Medical imaging diagnosis
Unsupervised	Unlabeled \(X\)	Find latent clusters/structure	Arbitrary, uninterpretable clusters	Customer segmentation
Self-Supervised	Raw Unlabeled Data	Pretext task completion	Representation collapse	Pretraining Foundation LLMs
Weakly Supervised	Programmatic Heuristics	Learn despite noisy labels	Learning heuristic rule biases	Massive dataset auto-labeling
Multi-Instance	Bags of instances	Bag-level classification	Witness isolation failure	Gigapixel Pathology slides
Reinforcement	Environment + Reward	Maximize cumulative reward	Reward hacking / Sample inefficiency	Game AI, Robotics, RLHF
Graph-Based	Nodes + Edges	Topological message passing	Over-smoothing deep layers	Drug discovery, Social graphs
PEFT / LoRA	Pretrained Model + Small \(Y\)	Low-rank adapter updates	Adapter capacity bottleneck	Fine-tuning LLM on 1 GPU
PINNs	Empirical Data + Physical PDEs	Minimize PDE residuals	Loss balance optimization traps	Fluid dynamics, Weather AI
Federated	Decentralized Local Data	Global training without sharing	Network bottlenecks, non-IID data	Mobile phone predictive text

Aliases and Overlaps:

Geometric Deep Learning \(\approx\) Graph / Manifold / Equivariant Networks
Contrastive Learning \(\subset\) Representation & Metric Learning
Domain Adaptation \(\subset\) Transfer Learning
Continual Learning \(\approx\) Lifelong / Incremental Learning
Weak Supervision \(\supset\) Multi-Instance Learning & Noisy-Label Learning
PEFT \(\subset\) Adaptation & Parameter-Efficiency