Quick Heuristics and Network Saturation

This presentation, “The Marvels and Flaws of Intuitive Thinking”, is part of a series from Edge.org which looks most interesting,

http://edge.org/conversation/the-marvels-and-flaws-of-intuitive-thinking

We ended up studying something that we call “heuristics and biases”. Those were shortcuts, and each shortcut was identified by the biases with which it came. The biases had two functions in that story. They were interesting in themselves, but they were also the primary evidence for the existence of the heuristics. If you want to characterize how something is done, then one of the most powerful ways of characterizing the way the mind does anything is by looking at the errors that the mind produces while it’s doing it because the errors tell you what it is doing. Correct performance tells you much less about the procedure than the errors do.

If it weren’t for Nature’s “cheap and dirty tricks” of the mind, we would not be alive today.  On the other side of the coin is the science of information saturation in complex adaptive systems, as told by Geoffrey West, also at Edge.org,

http://edge.org/conversation/geoffrey-west

The work I got involved in was to try to understand these scaling laws. And to make it a very short story, what was proposed apart from the thinking was, look, this is universal. It cuts across the design of organisms. Whether you are insects, fish, mammals or birds, you get the same scaling laws. It is independent of design. Therefore, it must be something that is about the structure of the way things are distributed.

You recognize what the problem is. You have ten¹⁴ cells. You have this problem. You’ve got to sustain them, roughly speaking, democratically and efficiently. And however natural selection solved it, it solved it by evolving hierarchical networks.

There is a very simple way of doing it. You take something macroscopic, you go through a hierarchy and you deliver them to very microscopic sites, like for example, your capillaries to your cells and so on. And so the idea was, this is true at all scales. It is true of an ecosystem; it is true within the cell. And what these scaling laws are manifesting are the generic, universal, mathematical, topological properties of networks.

Read the whole article, especially the part about network saturation along S-curves, and about singularity/collapse of those networks.  Also note his discovery about the growth curve of companies, which is a semi-vindication of Coasean economics.

George Bergeron Was a Genius

Kurt Vonnegut, Jr., in his short story, “Harrison Bergeron“, developed Harrison’s father, George, as a genius.  Harrion’s father however was “handicapped” with a headset he was forced to wear.  This headset was equipped with a wireless receiver.  Every few minutes, the office of the Handicapper General would broadcast an amazingly loud sound to wearers of this class of headset.  This amazingly loud sound would temporarily stun the wearers, forcing them to forget what they were thinking about.  This reduced the thinking level of geniuses to the lowest common denominator of thinking capabilities, beautifully illustrated in contrast by Harrison’s mother, Hazel.

http://en.wikipedia.org/wiki/Harrison_Bergeron

Recently, scientists have discovered that class of cells, “K” (keniocellular) cells, at the end of optic nerve in primates which produce a sleep-like pulsing rhythm to the gateway of the optical cortex,

http://medicalxpress.com/news/2011-08-scientists-window-brain.html

In previous postings, I have suggested that “pulsing” behavior in neural networks in the brain is one of nature’s “cheap and dirty tricks” to “stir the pot” as it were, or to prevent our neural networks from remaining saturated and otherwise unresponsive to continued adaptation to an animal’s ever-changing environment.  The rhythmic pulsing of the “K” optic layer might to induce a regular series of George Bergeron moments in the optical networks which would prevent easy saturation.

Neural networks probably rely on multiple methods for reducing saturation effects.  I am sure other network methods of de-saturation will be found along the visual pathway over time.

Parahippocampal Cortext (PHC) Saturation

An interesting study shows how we are less likely to learn something new if our memory systems are “busy”,

http://medicalxpress.com/news/2011-08-ready-brain-scans.html

In short, when the parahippocampal cortex (PHC) is saturated, it is less likely to allow new long term learning.  What these researchers ought to do next is track down the “cheap and dirty tricks” which de-saturate the PHC.  How does the cortical region become calm once again?  What are the mechanisms which reset the network so that it can be “open for business” once again?

That a homogeneous neural network saturates itself and becomes immune to change or is otherwise dysfunctional is not new.  Nature’s real magic lies in how de-saturation takes place, and, actually, that de-saturation happens at all!

Also related is an interesting theory about humor.  Humor, it seems, is also one of nature’s cheap and dirty tricks for not only de-saturating our neural networks, but also for describing what seems like a rewarding experience we receive when it occurs!

http://www.slate.com/content/slate/blogs/browbeat/2011/05/13/5_leading_theories_for_why_we_laugh_and_the_jokes_that_prove_them_wrong.html

Memory, Irreversibility and Transactions

A “system” is a finite set of memory components interrelated through causative event maps.

Phwew, that was a mouthful!  What does that mean?

Memory is the ability of matter to change state and maintain that state for a non-zero period of time.  At the smallest scales of existence, atoms have memory when, for instance, chemical changes influence the electron configuration of those atoms.  The ability of paper to hold graphite markings throughout its lifetime is also a form of memory.

An event is a directional transfer of energy from one memory component to another, from source to target, in a way that induces a state change in the target which lasts for a non-zero period of time.  An event is an event if it alters the memory configuration of its target.  An event map is a set of source/target associations.  Causality is the study of the effects of event maps upon their state-absorbing targets.

To study a system is to study a well-defined, finite set of memory components and the causative event maps which affect those components.  For every system under study, there exists that which is outside of that system which we call the system’s environment.  Causative events flow from system to environment, and from environment to system, composing a causative event map called a feedback loop.

Entropy is the degree to which a system has been affected by its causative event map.  Low entropy implies that a system has “room” to absorb new state changes in an unambiguous way.  A set of aligned, untoppled dominoes has low entropy.  High positive entropy implies that a system has attained a degree of ambiguity with regard to its ability to absorb specific kinds of changes.  A set of toppled dominoes has a high degree of entropy relative to “toppling” events.  One can attempt to topple already-toppled dominoes, but the result is ambiguous in that it is more difficult to leave evidence of a toppling event (a finger push) than it was prior to toppling.  Negative entropy is a condition in which a system is to some degree “reset” so that it can once again, unambiguously, absorb more events than it could before.  To induce negative entropy into a system of toppled dominoes is to set them back up again to be retoppled.

All physical systems tend to increase in measures of entropy over time.  They do so because they have memory and exhibit hysteresis.  To memorize a change is to freeze that change in time.  Changes induced by previous events interfere with the ability of new events to be absorbed.  A thermodynamically hot system imparts kinetic events to cold systems they are connected to, at the cost of the energy stored in its own memory.  Slowly, the cold systems absorb the kinetic energy of the hot until a point is reached which the cold memory systems reach capacity, or become saturated.  Such a point of memory capacity saturation is called “equilibrium”.  If the cold system had no memory, for instance if it were a vacuum, it would never have increased in temperature and the hot system would have eventually become absolutely cold since it would be connected to systems with infinite capacities to absorb events.

As noted by Erwin Schrödinger, life in general has a “habit” of reversing entropy and in fact could be defined by this single, dominant habit.  Lifeless physical systems tend towards maximum positive entropy and tend to remain that way.  Life, on the other hand, does its damnedest to reverse entropy.  For life, it is not merely enough to keep entropy from increasing.  Like all systems, life which is saturated to its limit of information capacity can fail to adapt to a changing environment.  Life is a process through which its subsystems are continually de-saturated in order to make room for new information.  Life depends on entropy reversal.

This is not to say that entropy reversal does not happen to lifeless systems; entropy may be reversed here and there and for short periods of time.  Random, isolated reversals of entropy in any system however are always—even in the case of life—compensated for by an increase of entropy in the outer environment.  Ultimately, the Great Environment we call the Universe is continually losing more and more of its ability to unambiguously absorb new events.  The arrow of time since the Big Bang is the story of how the memory components of the Universe are reaching capacity saturation.

The metaphor of the economic transaction is useful for describing the flow of events leading to entropy reversal.  Financial transactions follow the same entropy build-up and subsequent decrease.  Even in the simplest of cases, financial participants form a “memory system” which saturates before it collapses.  Work is done between participants before money is exchanged.  The exchange of money allows the information of the transaction to “compress”, and entropy to reverse in the well-defined, temporary system of the particular transaction.  This entropy reversal occurs, of course, at the expense of the outer environment.  Quantum transactions also follow the same build-up and tear-down in terms of the memory capacities of participating elements of matter.

For true de-saturation to occur within a system, a system’s memory must be irreversibly erased.  If memory erasure were reversible, then memory would not have been erased and the system would have remained saturated.  “Reversible” memory loss is not true memory loss, but an illusion, a shuffling, a card trick.  Irreversibility however, comes at a price for a system.  One can shuffle sand in a sandbox from one side to another, but to truly increase the capacity of a sandbox one must expend energy to remove sand from it and returning that sand to the outer environment.  “Irreversibility” however, is not some separate, measurable feature of entropy reversal, but is a necessary part of its definition.  If a transaction is reversible, then entropy was not reversed.  If entropy has not been reversed, either partially or completely, then the transaction metaphor does not apply.  Irreversibility is a necessary test to determine the appropriateness of the transaction metaphor.

Remembering and Forgetting, Saturation in Neural Networks

This study by Rosenzweig, Barnes, and McNaughton highlights the importance of forgetting in order to make the best use of the brain cells we have,

http://frank.itlab.us/forgetting/making_room.html

If we fail to forget, our neural networks will saturate and become useless.  Saturation in a neural network does not merely mean that a network cannot learn more, it can mean that a network could fail to respond to input in an appropriate manner.

Consider a very simple network consisting of two input neurons I₁ and I₂, and two output neurons O₁ and O₂.  A neural network learns by increasing the strength of connections between associated inputs and outputs.  For instance should an input signal be present at I₁ while an output signal is also present at O₂, then the connection I₁↔O₂ would be strengthened.  Consider Ivan Pavlov, his dog, a dog treat, and Pavlov’s bell.

A trained neural network acts by pro-actively triggering appropriate output neurons when specific input signals are present.  Should a signal trigger the first input neuron in our example, I₁, the second output neuron would be triggered pro-actively.

• Given the “learned” synaptic connection: I₁↔O₂
• Assuming: I₁
• Triggered: O₂ (I₁→O₂)

Consider Pavlov’s dogs salivating when the bells rang regardless if treats were provided.

If a second training exercise triggered input I₁ but instead the first output neuron was triggered in lieu of the second, the connection I₁↔O₁ would also be strengthened.  We now have,

• I₁↔O₁
• I₁↔O₂

After the second training session, should I₁ be triggered once again, which output neuron would trigger?  Without any further weighting functions to apply to our connections, a I₁ signal would trigger both outputs,

I₁→O₁∪O₂

Consider a situation where Pavlov’s dogs were sometimes offered treats when the bells rang, or sometimes were given electric shocks.  What would the dogs have expected the next time bells rang?  Would they have expected treats, electric shocks, or both?

Perhaps this state of affairs is desirable, perhaps it is not.  Now that this cross-association is saturated however, there exists no way to trigger only O₂ given I₁.  Even if all future training sessions reinforce the I₁↔O₂ connection, the system will remain ambiguous forever.

It is likely that nature’s first, simple neural networks exhibited this kind easy saturation.  Perhaps early critters could only adapt to very limited environmental conditions during their very short lives.  Perhaps these critters simply died from indecision if they encountered natural oddities they weren’t prepared for.  In the competitive evolutionary race however, those critters who occasionally reset their saturated networks would have an evolutionary advantage over those who did not.  To reset an easily saturated neural network would have been to allow the forgetting of anomalies.  These critters would have had a better chance of survival in the real, random natural world.  They would relearn their most common and important lessons and forget the oddities which simply did not pertain to most circumstances of their lives.

In the context of the article, 4-(3-phosphonopropyl) piperazine-2-carboxylic acid (CPP) provides an occasional “reset” function to spatial memory that allows de-saturation and re-learning.  CPP is one of nature’s “dirty tricks” that helps to alleviate the downsides of easily saturated neural networks.  Nature has converged upon many such dirty tricks over the eons, including:

• Chemical washes (CPP)
• Inhibition, “pulsing” and other mild periodic reset mechanisms
• Network segmentation (slows saturation)
• Physical growth and degeneration
• Specialty circuits (e.g., “instinct”)
• Preferential learning such as that which provides increased weight to electric shocks versus pleasurable food treats
• Consciousness (self-awareness)
• Concept formation and other information compression mechanisms
• Emotion, heuristic, magical thinking, social deference and economic behavior in humans

The basic lesson is that, short of ameliorating effects, all neural networks easily saturate.  For any cognitive function, researchers should ask two questions:

• How does the associated network saturate?  What are the effects?
• What solutions has evolution converged upon to de-saturate the network?

Memory, Adaptation and Entropy

I will write more in the coming weeks and months about the various types of memory a life form may leverage in order to adapt to its environment.  An interesting article from ScienceDaily illustrates how epigenetics, those chemical changes which alter the way DNA is processed (or not processed) in our cells, provide an organism with an adaptation subsystem that helps it better fit its environment,

http://www.sciencedaily.com/releases/2011/07/110724135553.htm

Adaptation cannot occur without memory.  Organisms, including plants, leverage many forms of memory.  Other than chemical and physical construction, perhaps the most important characteristic which differentiates kinds of memories is the informational entropy capacities of those memories.  Memory systems with higher entropy capacities can assimilate larger informational variety.  As the informational variety (entropy) capacity of a memory system increases, so will rise the organisms potential to adapt to a greater number of environmental conditions.  That is, the higher the entropy capacity, the higher the potential utility of the adaptive system.

From the article,

Epigenetic memory comes in various guises, but one important form involves histones — the proteins around which DNA is wrapped. Particular chemical modifications can be attached to histones and these modifications can then affect the expression of nearby genes, turning them on or off. These modifications can be inherited by daughter cells, when the cells divide, and if they occur in the cells that form gametes (e.g. sperm in mammals or pollen in plants) then they can also pass on to offspring.

I will also illustrate in the coming weeks and months that adaptive system utility is not merely a function of higher information entropy capacity.  Adaptive system utility can also be extended by the system’s ability to “clean house”, “collect the garbage” and reduce information variety when the system has become saturated.