The Smallest Brain You Can Build: A Perceptron in Python

I like the ability of this article to do the tiny training runs in browser. It makes the point of a bias clear. Too many tutorials get sucked into the proof of zero times anything is zero. Everyone knows that. What you should show is where that mstters in the problem at hand.

3blue1brown does one of the best depictions of why we need an activation function.

Karpathy's videos are a little tougher for a beginner to grasp, but excel at solving a complete problem. I knew all of the theory behind what it takes to make micrograd before I made my own by following the video, but what you get from doing it can't be understated.

It's hard to describe but it what you learn is more of a feel than pure knowledge. It gives you a better sense of knowing when the principles apply in other circumstances.

Perhaps it's the distinction of understanding how springs and gears work, then looking at a clock and understanding how the gears and springs move the hands. There's still more needed if you want to make a clock. And that stuff is what let's you also make a wind up toy.

Back up a bit please! Analog computing is a thing. And it isn't even new - not by a long shot.

There are good reasons why practically all computing today is the digital kind. But electronic 'equivalents' of neural nets is one area where analog might make sense. Adding inputs can be as simple as a bunch of resistors + a transistor. Even on modern silicon nodes, that might be a more efficient setup than digital inputs, N-bit adders/multipliers etc. Not saying that's the case, and AI hardware should be based on analog circuitry. But it could be, and perhaps found to be practical.

Artificial neurons are significantly more complex that single transistors, and even a minimal hardwired circuit to implement just one neuron requires quite a number of transistors.

It's the same thing as stimulus, response.

Unchanging in response to circumstances is static.

Changing in the absence of circumstances is randomness.

The conditional is all that remains. Changing in response to circumstances

(Arguably, unchanging in the absence of circumstances completes the truth table, but it's a whole lot of nothing)

There exist several equivalent sets of primitive operations. While the sets containing only NAND or only NOR, or both AND and NOT or both OR and NOT are more notorious, these logical operations are more abstract and they do not indicate precisely a hardware implementation, i.e. there are many distinct hardware methods to make such logical gates.

Other sets of primitive operations map directly to hardware devices, e.g. the sets of primitive operations composed of maximum and complement or of minimum and complement map directly to a hardware implementation using rectifier diodes and inverting amplifiers (which can be made with either semiconductor devices or with vacuum tubes, or also with pneumatic or hydraulic devices).

Other sets of primitive operations are obtained by replacing the maximum or minimum circuits with series or parallel connections of switches, like in the CMOS logic that is nowadays dominant.

The alternative IF expression corresponds in hardware to a 2-way multiplexer, which, together with the 2 constant functions "0" and "1" (a.k.a. "false" and "true" or "low voltage" and "high voltage"), is sufficient for a complete set of computational primitives.

Besides those mentioned above, the main remaining variant for a complete set of computational primitives consists of an analog (possibly weighted) adder and an analog comparator, which had been used in the so-called RTL circuits (resistor-transistor logic) and which also corresponds to perceptrons. RTL had been used in some early integrated circuits, before being replaced by DTL and TTL circuits (which are based on minimum and complement functions).

In hardware, e.g. in RTL circuits, a combined analog adder+comparator can be made with a single high-gain amplifying device, together with a set of weighting resistors and a bias resistor. RTL circuits can implement complex logic with fewer devices (e.g. they can implement a neural network in the analog domain), but they were replaced during the sixties of the past century with DTL, then TTL, because those were faster (in RTL, the resistors limit the charging currents for input capacitors and parasitic capacitors, which slows down the logical transitions) and the fact that they needed more devices was not important, due to the quick increase in circuit density.

A transistor (driven to saturation) is a much better model.

I included a remark about how time travellers would find Rosenblatt a better target than Miles Dyson.

I was never quite sure on how close, or over, the line that was on appropriateness. It was definitely thought provoking.

and its an a great learning resource - which is arguably more important :-)

(before numarray and numpy merged!)

EDIT: His son is co-author?

    Ix = x
    Kxy = x
    Sxyz = xz(yz)

[1] https://www.nand2tetris.org/