What is a neural network? A developer's explanation

A neural network is a function. It takes numbers in, does a long chain of multiplications and additions, and gives numbers out. What makes it special is that the function has millions of adjustable constants inside it, and those constants were set by a program rather than by a person.

That is the whole answer to "what is a neural network". Everything else is detail about how the constants get set and what the function is bad at. I've spent six years building web applications and only started reading about this properly when I found myself calling these things through an API and realising I couldn't explain what was on the other end. This is what I've worked out so far.

What is a neural network, as code

Start with the smallest version. One input, one output, two constants:

def tiny_network(x, w, b):
    return x * w + b

w is a weight, b is a bias. Pick w = 2 and b = 1 and you've hand-written a function that doubles and adds one. Nobody would call that learning.

Now make it bigger. Several inputs, several outputs, and a non-linear squashing step:

def layer(inputs, weights, biases):
    outputs = []
    for w_row, b in zip(weights, biases):
        total = sum(x * w for x, w in zip(inputs, w_row)) + b
        outputs.append(max(0, total))   # the "activation"
    return outputs

Each output is a weighted sum of the inputs, plus a bias, passed through max(0, total). That last step is the activation function (this one is called ReLU). Without it, stacked layers would collapse into one linear equation, and linear equations can't bend around interesting shapes.

A neural network is this layer called on its own output, repeatedly:

def network(inputs, all_weights, all_biases):
    x = inputs
    for weights, biases in zip(all_weights, all_biases):
        x = layer(x, weights, biases)
    return x

Real ones have cleverer wiring (the transformer layers behind chat models route information between positions in a sequence) and run on GPUs rather than Python loops, but the idea holds. "Deep" just means many layers.

The part that took me a while to accept is how dumb the pieces are. There's no if is_cat: anywhere. There's no symbol for "cat". There are only weights, and the pattern of which ones are large and which are near zero is where all the behaviour lives.

Training is a loop that nudges the weights

Where do the weights come from? From a loop any developer would recognise. You need a pile of examples with known right answers. You start with random weights. Then:

1. Run an example through the network.
2. Measure how wrong the output was. That number is the loss.
3. For every weight, work out whether nudging it up or down would make the loss slightly smaller.
4. Nudge every weight a tiny step in its improving direction.
5. Next example.

Step 3 is backpropagation: because the network is a chain of simple arithmetic, calculus can compute each weight's effect on the loss in one backward pass. Step 4 is gradient descent, walking downhill on the loss.

for example, correct in training_data:
    prediction = network(example, weights, biases)
    loss = error(prediction, correct)
    gradients = backprop(loss)           # one number per weight
    for w, g in zip(weights, gradients):
        w -= learning_rate * g           # tiny step downhill

Run that over billions of examples and the weights drift into a configuration that gives good answers on the examples it saw. Nothing in the loop understands anything. It's a numerical optimiser making an error number go down.

"It learned" means "the numbers were tuned"

When someone says a model learned to recognise handwriting, the literal truth is: a loss was minimised over labelled handwriting, and the resulting weights happen to produce the right label on new handwriting that resembles the training set.

That's impressive. It's also specific, and the gap between the literal truth and the word "learned" is where most of my wrong intuitions used to live.

The weights encode statistical regularities in training data. They don't contain a rule for what makes a sentence true, or any notion of the world beyond "inputs like this tended to go with outputs like that". When the model gives you a correct answer, it's because that answer was the likely continuation. When it gives you a wrong one, it's for exactly the same reason. There is no truth-checking module. Confidence in the output measures how strongly the weights favoured it, not whether it's right.

What it's bad at, and why

Once you see the network as a tuned function rather than a mind, its failure modes stop looking like bugs and start looking like properties.

It interpolates. The function is good where training examples were dense and fills the gaps by blending nearby behaviour. Ask it something far from anything it saw and it still produces fluent output, because producing output is all it does. It can't say "I don't know" unless that phrase was itself trained in, and even then it's just another tuned output.

It's sensitive to phrasing. Two prompts that mean the same thing to you are two different points in input space. Rewording a request and getting a different answer isn't flakiness in the service. It's the shape of the function.

It has no model of truth. A language model is trained to predict plausible text. Plausible and true overlap often enough to be useful and diverge often enough to be dangerous. The confident paragraph citing a function that doesn't exist in your framework is the model doing precisely what it was tuned to do.

It can't tell instructions from data. The network receives one sequence of tokens. Your system prompt, the user's message, and the web page you pasted in are all input to the same function. There is no privileged channel for "commands from the developer". Whatever looks most like an instruction, statistically, gets followed.

Why a developer should care now

I used to treat a model API like any other third-party service: send a request, parse a response, handle errors. A database that speaks English.

That's wrong in a way that creates security problems, and I say that as someone still learning the security side. A traditional API has a contract: bad input gets a 400. A neural network has no contract. Every input produces output, delivered in the same tone whether it's right, wrong, or steered by someone else's text in the context window.

That last case is prompt injection, and Kushagra wrote it up properly in What is prompt injection, really. It's hard to fix not because vendors are sloppy but because of the point above: one input channel, no concept of who said what. Once you see the model as a tuned function over a single token sequence, the attack stops looking exotic and starts looking inevitable. The lethal trifecta post takes it one step further, to what happens when that function also has tools and your private data.

So the mental model I try to hold now is "a very good autocomplete over everything in the context window, including the bits I didn't write". Validate the output like user input. Don't give the model a capability you wouldn't give the untrusted text it's reading. Test with rephrasings, not one golden prompt.

If you've already shipped something on top of a model, run it through the AI Audit. It works in the browser and asks the questions I wish I'd asked earlier.

Frequently asked questions

Is a neural network the same thing as AI?

No. AI is a loose umbrella term. A neural network is one technique: a function with many tuneable weights, trained by gradient descent. Most of what people currently call AI, including large language models, is built on neural networks, but rule-based systems and classic machine learning methods are also called AI and don't work this way.

Does the network understand what it's saying?

Not in the sense a colleague understands something. The weights encode statistical patterns, and the output is whatever those patterns make most likely. Whether that counts as understanding is a philosophical argument; for engineering purposes, treat the output as a plausible guess, not a verified claim.

Why does rewording my prompt change the answer so much?

Because a prompt is a point in a very high-dimensional input space, and two rewordings are two different points. Models are trained to be fairly stable across paraphrases, but stability is a tendency, not a rule, and it weakens the further you get from typical training data.

Can't the model just be trained not to follow injected instructions?

Training can reduce it, and vendors do, but it can't remove it. Training makes behaviours more or less likely; it doesn't add a hard boundary between instruction and data, because the architecture has none. Defences that hold are outside the model: limiting what it can do and treating its output as untrusted.

Where should a developer start?

Write the tiny network above in a language you know and train it on something trivial, like adding two numbers. Watching the loss fall and the weights move does more for intuition than any diagram. Then read about transformers, which is the wiring in the models you're actually calling.