Gradient descent
Gradient descent is a method for unconstrained mathematical optimization.
At a glance
Key signals
0% cross fields · reaches 6 more
- Computer Science
- Machine Learning
- Optimization
- Explanation
- Examples
- Misconception
- Sourced relations 2
- Attribution
Dependencies
What this concept builds on and what it makes possible — derived from the atlas’s dependency, causal and structural relations, not from every related edge.
Foundations · builds on
Gradient descentrequiresLoss functionEstablished
Gradient descent requires Loss function.
Mechanism: Gradient descent minimizes the loss function.
Gradient descentdepends onTraining dataEstablished
gradient descent depends on Training data.
Mechanism: Each step of gradient descent measures the model's error on training data and adjusts to do better next time.
- Wikipedia (English & German editions) verifiedmoderate evidence
Enables · leads to
Regressiondepends onGradient descentEstablished
Regression depends on Gradient descent.
Structural role & consequence
Interpreted from the current atlas graph — what the connections mean, not just how many there are.
Currently dark in the atlas: no key date stored · 5 of 5 of its relations lack claim-level evidence.
atlas representation · Describes the current Thinking OS representation, not the state of the world.
Builds on 2 foundations (requires / depends-on / derived-from / emerges-from).
structural · Structural graph analysis — not a claim of importance, causation or history.
Exercises 3 annotated mental models — a concept that connects several thinking patterns.
curated · Curated annotations, not a derived measure.
Structural neighbourhood: 5 → 30 → 89 concepts reachable within 3 hops.
structural · Structural reach — being reachable is not the same as being understood.
cross-field
5 within-field, 0 cross-field
Strengths & constraints
Constraints
- Evidence coverage currently thin in the atlas — few of its relationships carry claim-level evidence. atlas representation
- No dated history stored — the atlas records no key date for this concept. atlas representation
Conditions
- Its dependency reading rests on 2 foundation relations. structural
- Read structurally — most of its relationships carry no external evidence yet, so claims here are graph-derived. structural
Dependency radial
What this concept builds on (left) and what it makes possible (right) — derived from dependency and causal relations.
What builds on this
1 concept build on this directly, 1 in total, across 1 discipline.
Structural downstream reach along dependency edges — not a claim of historical necessity.
Seen through each discipline
How this concept sits in each of its fields — derived from its real connections in the graph, not asserted.
Through this lens it connects to Training data and Optimization landscape.
Through this lens it connects to Training data.
Through this lens it connects to Optimization.
Through this lens it connects to Loss function and Regression.
Formula
Source: Wikidata
Check yourself
A quick check against a common misconception. Nothing is scored — picking the tempting-but-wrong answer just flags an idea worth revisiting.
Which statement is correct?
Gradient descent always finds the best possible answer.
It follows the slope downhill and can settle in a local minimum — a dip that is not the deepest. Where it lands depends on where it starts and how it steps.
Look for: Learner assumes training always reaches the global optimum.
Related ideas to explore
Concepts that look related but are not yet connected here — candidates for a connection to reason about, not established links.
This idea also appears in…
The same structure shows up in other disciplines. These are real recurrences drawn from the graph — a starting point for asking “what carries over, and what changes?”
Feedback loop45 disciplines · 45 concepts
Optimization13 disciplines · 11 concepts
Variation & selection10 disciplines · 11 concepts
Concepts
shares a mental model · shares a mental model · shares a mental model
shares a mental model · shares a mental model
shares a mental model · shares a mental model
shares a mental model · shares a mental model
shares a mental model
shares a mental model
Gradient descent is how a learning system improves itself. Imagine walking downhill in fog by always stepping in the steepest downward direction: the system keeps adjusting its settings a little at a time to make its mistakes smaller.
Gradient descent is the optimisation method that trains most machine-learning models. It measures how the error changes as each parameter changes (the gradient) and nudges every parameter a small step in the direction that reduces the error, repeating until it settles.
Mental models at work here
Feedback loop
A loop where an effect feeds back to change its own cause. Reinforcing loops amplify change; balancing loops resist it.
Ask: does the result push the system further in the same direction, or pull it back?
Optimization
Searching a space of options for the best one under constraints — following a gradient of 'better' toward a maximum or minimum.
Ask: what is being maximised or minimised, and what limits the search? Beware settling in a 'local' best that isn't the global one.
Variation & selection
When many varying candidates are filtered by a criterion and the survivors are copied, the population adapts to the criterion over rounds — with no designer.
Look for three ingredients: variation, a selection pressure, and inheritance/copying. Together they design without a designer.
- Connects 5 other ideas across 4 disciplines.
- A cross-disciplinary bridge — its connections reach into 6 other fields.
- Most of its connections are of the “Dependency” kind.
- It exercises 3 reusable thinking patterns.
Derived from the graph’s real structure — observations, not a score.
Sources
- Wikipedia (English & German editions) verifiedmoderate evidence
- Wikidata verifiedmoderate evidence
- Accelerated Gradient Descent Using Instance Eliminating Back Propagation (2020) verified
- Analyzing the Characteristics of Gradient Descent and Non-Gradient Descent-Based Algorithms in Neural Network Learning (2024) verified
- Gradient Descent (2018) verified
- Large-Scale Machine Learning with Stochastic Gradient Descent Léon Bottou (2011) verified
- Stochastic Gradient Descent (2015) verified
- Stochastic Gradient Descent (2014) verified
- Gradient Descent (2021) verified
- Accelerated Gradient Descent (2021) verified