Representation

From Vision, by David Marr:

"A representation is a formal system for making explicit certain entities or types of information, together with a specification of how the system does this. And I shall call the result of using a representation to describe a given entity a description of the entity in that representation (Marr and Nishihara, 1978).

"For example, the Arabic, Roman, and binary numeral systems are all formal systems for representing numbers. The Arabic representation consists of a string of symbols drawn from the set (0, 1, 2, 3, 4, 5, 6, 7, 8, 9), and the rule for constructing the description of a particular integer n is that one decomposes n into a sum of multiples of powers of 10 and unites these multiples into a string with the largest powers of 10 on the left and the smallest on the right. Thus, thirty-seven equals 3 x 10(1) + 7 x 10(0), which becomes 37, the Arabic numeral system's description of the number. What this description makes explicit is the number's decomposition into powers of 10. The binary numeral system's description of the number 37 is 100101, and this description makes explicit the number's decomposition into powers of 2. In the Roman numeral system, thirty-seven is represented as XXXVII..

"This definition of a representation is quite general. For example, a representation for shape would be a formal scheme for describing some aspects of shape, together with rules that specify how the scheme is applied to any particular shape. A musical score provides a way of representing a symphony; the alphabet allows the construction of a written representation of words; and so forth. The phrase "formal scheme" is critical to the definition, but the reader should not be frightened by it. ... To say that something is a formal scheme means only that it is a set of symbols with rules for putting them together--no more and no less.

"A representation, therefore, is not a foreign idea at all--we all use representations all the time. However, the notion that one can capture some aspect of reality by making a description of it using a symbol and that to do so can be useful seems to me a fascinating and powerful idea. But even the simple examples we have discussed introduce some rather general and important issues that arise whenever on chooses to use one particular representation. For example, if one chooses the Arabic numeral representation, it is easy to discover whether a number is a power of 10 but difficult to discover whether it is a power of 2. If one chooses the binary representation, the situation is reversed. Thus, there is a trade-off; any particular representation makes certain information explicit at the expense of information that is pushed into the background and may be quite hard to recover.

"This issue is important, because how information is represented can greatly affect how easily it is to do different things with it. This is evident even from our numbers example: It is easy to add, to subtract, and even to multiply if the Arabic or binary representations are used, but it is not at all easy to do these things--especially multiplication--with Roman numerals. This is a key reason why the Roman culture failed to develop mathematics in the way the earlier Arabic cultures had.

"... But even though one is not restricted to using just one representation system for a given type of information, the choice of which to use is important and cannot be taken lightly. It determines what information is made explicit and hence what is pushed further into the background, and it has a far-reaching effect on the ease and difficulty with which operations may subsequently be carried out on that information. [Bold and italics added]"

--David Marr (1982, pp. 20-22)

Kinds of Representation

From Whispering in the Wind, by Carmen Bostic St Clair and John Grinder:

"The implication is that there are essential characteristics of each of the modes of representation and communication (the representational systems) that distinguish them from one another in deep and fundamental ways. For example, kinesthetic representations have characteristics for which there are no corresponding counterparts in visual representations and vice versa. More specifically, for example, visual representations may contain contradictory representations (or better, representations of contradictory information) in a stable form (and without any spontaneous movement to integrate) while kinesthetic representations when containing contradictory representations will be unstable and the contradictory representations will (except under conditions of extreme disassociation such as long established multiple personalities or sequential incongruity) spontaneously integrate. This spontaneous movement to integrate simultaneously presented kinesthetic representations (different feeling states) is the basis of all of the integration patterning in all NLP’s anchoring formats.

"In other words, a well trained agent of change will choose to put the contradictory representations in the kinesthetic system through the judicious use of anchors just in case he or she wants their clients to spontaneously integrate the contradictory parts of themselves. That same agent of change will select a simultaneous display of contradictory parts in the visual representational system just in case he or she does NOT want the parts to spontaneously integrate. " --Bostic St. Clair & Grinder (2001, p. ?)

----------------

"A third issue that arises with respect to coding is the following, in any coding exercise, it rapidly becomes clear that there are an arbitrarily large number of different representations of complex behavior that could potentially serve as a possible description. A classic example of this is the set of three representations immediately below--a binary number, a decimal number and a phrase in English:

1011101011010010000
186,000
the velocity of light

"In fact these three representations are equivalent--they are simply three distinct codes for the same information [relationship]. Note that the example makes explicit that information is independent of the code selected for the expression."

--Bostic St. Clair & Grinder (2001, p. 272)

Process

From Vision, by David Marr:

"The term process is very broad. For example, addition is a process, and so is taking a Fourier transform. But so is making a cup of tea, or going shopping. For the purposes of this book, I want to restrict our attention to the meanings associated with machines that are carrying out information-processing tasks. So let us examine in depth the notions behind one simple such device, a cash register at the checkout counter of a supermarket.

There are several levels at which one needs to understand such a device, and it is perhaps most useful to thinks in terms of three of them. The most abstract is the level of what the device does and why. What it does is arithmetic, so our first task is to master the theory of addition. Addition is a mapping, usually denoted by +, from pairs of numbers into single numbers; for example, + maps the pair (3, 4) to 7, and I shall write this in the form (3 + 4) --> 7. Addition has a number of abstract properties, however. It is commutative: both (3 + 4) and (4 + 3) are equal to 7; and associative: the sum of 3 + (4 + 5) is the same as (3 + 4) + 5. Then there is the unique distinguished element, zero, the adding of which has no effect: (4 + 0) --> 4. Also, for every number there is a unique "inverse", written (-4) in the case of 4, which when added to the number zero gives: [4 + (-4)] --> 0.

Notice that these properties are part of the fundamental theory of addition. They are true no matter how the numbers are written--whether in binary, Arabic, or Roman representation--and no matter how the addition is executed. Thus part of this first level is something that might be characterized as what is being computed.

The other half of this level of explanation has to do with the question of why the cash register performs addition and not, for instance, multiplication when combining the prices of the purchased items to arrive at the final bill. The reason is that the rules we intuitively feel to be appropriate for combining the individual prices in fact define the mathematical operation of addition. These can be formulated as constraints in the following way:

1. If you buy nothing, it should cost you nothing; and buying nothing and something should cost the same as buying just the something. (The rules for zero.)
2. The order in which goods are presented to the cashier should not affect the total. (Commutativity.)
3. Arranging the goods into two piles and paying each pile separately should not affect the total amount you pay. (Associativity; the basic operation for combining prices.)
4. If you buy an item and then return it for a refund, your total expenditure should be zero. (Inverses.)

It is a mathematical theorem that these conditions define the operation of addition, which is therefore the appropriate computation to use.

This whole argument is what I call the computational theory of the cash register. Its important features are (1) that it contains separate arguments about what is computed and why and (2) that the resulting operation is defined uniquely by the constraints it has to satisfy. In the theory of visual processes, the underlying task is to reliably derive properties of the world from images of it; the business of isolating constraints that are both powerful enough to allow a purpose to be defined and generally true of the world is a central theme of our inquiry.

In order that a process shall actually run, however, one has to realize it in some way and therefore choose a representation for the entities that the process manipulates. The second level of the analysis of a process, therefore, involves choosing two things: (1) a representation for the input and for the output of the process and (2) and algorithm by which the transformation may actually be accomplished. For addition, of course, the input and output representations can both be the same , because they both consist of numbers. However this is not true in general. In the case of a Fourier transform, for example, the input representation may be the time domain, and the output, the frequency domain. If the first of our levels specifies what and why, this second level specifies how. For addition, we might choose Arabic numerals for the representations, and for the algorithm we could follow the usual rules about adding the least significant digits first and "carrying" if the sum exceeds 9. Cash registers, whether mechanical or electronic, usually use this type of representation and algorithm.

There are three important points here. First, there is usually a wide choice of representation. Second, the choice of algorithm often depends rather critically on the particular representation that is employed. And third, even for a given fixed representation, there are often several possible algorithms for carrying out the same process. Which one is chosen will usually depend on any particularly desirable or undesirable characteristics that the algorithms may have; for example, one algorithm may be much more efficient than another, or may be slightly less efficient but more robust (that is, less sensitive to slight inaccuracies in the data on which it must run). Or again, one algorithm may be parallel, and another, serial. The choice, then, may depend on the type of hardware or machinery in which the algorithm is to be embodied physically.

This brings us to the third level, that of the device in which the process is to be realized physically. The important point here is that, once again, the same algorithm may be implemented in quite different technologies. The child who methodically adds two numbers from left to right, carrying a digit when necessary, may be using the same algorithm that is implemented by wires and transistors of the cash register in the neighborhood supermarket, but the physical realization of the algorithm is quite different in these two cases. Another example: Many people have written computer programs to play tic-tac-toe, and there is more or less a standard algorithm that can not lose. This algorithm has in fact been implemented by W. D. Hillis and B. Silverman in quite different technology, in a computer made out of tinker toys, a children's wooden building set. The whole monstrously ungainly engine, which actually works, currently resides in a museum at the University of Missouri in St. Louis.

Some styles of algorithm will suit some physical substrates better than others. For example, in conventional digital computers, the number of connections is comparable to the number of gates, while in a brain, the number of connections is much larger ( x 10⁴ ) than the number of nerve cells. The underlying reason is that wires are rather cheap in biological architecture, because they can grow individually and in three dimensions. In conventional technology, wire laying is more or less restricted to two dimensions, which quite severely restricts the scope for using parallel techniques and algorithms; the same operations are often better carried out serially."

--David Marr (1982, p. 22)

Enacting a World

The following is the an excerpt from Varela, Thompson, and Rosch (1993), The Embodied Mind:

"Consider once again Minsky's discussion toward the end of Society of Mind. There he writes, "Whenever we speak about a mind, we're speaking of the processes that carry our brains from state to state ... concerns about minds are really concerns with relationships between states - and this has virtually nothing to do with the natures of the states themselves." How, then, are we to understand these relationships? What is it about them that makes them mind like?

The answer that is usually given to this question is, of course, that these relationships must be seen as embodying or supporting representation of the environment. Notice, however, that if we claim that the function of these processes is to represent an independent environment, then we are committed to construing these processes as belonging to the class of systems that are driven from the outside, that are defined in terms of external mechanisms of control (a heteronomous system). Thus we will consider information to be a prespecified quantity, one that exists independently in the world and can act as the input to a cognitive system. This input provides the initial premises upon which the system computes a behavior - the output. But how are we to specify inputs and outputs of highly cooperative, self-organizing systems such as brains? There is, of course, a back-and-forth flow of energy, but where does information end and behavior begin? Minsky puts his finger on the problem, and his remarks are worth quoting at length.

Why are processes so hard to classify? In earlier times, we could usually judge machines and processes by how they transformed raw materials into finished products. But it makes no sense to speak of brains as though they manufacture thoughts the way factories make cars. The difference is that brains use processes that change themselves - and this means we cannot separate such processes from the products they produce. In particular, brains make memories, which change the ways we'll subsequently think. The principal activities of brains are making changes in themselves. Because the whole idea of self-modifying processes is new to our experience, we cannot yet trust our commonsense judgement about such matters.

What is remarkable about this passage is the absence of any notion of representation. Minsky does not say that the principal activity of brains is to represent the external world; he says that it is to make continuous self-modifications. What has happened to the notion of representation?

In fact, an important and pervasive shift is beginning to take place in cognitive science under the very influence of its own research. This shift requires that we move away from the idea of the world as independent and extrinsic to the idea of a world as inseparable from the structure of these processes of self-modification. This change in stance does not express a mere philosophical preference; it reflects the necessity of understanding cognitive systems not on the basis of their input and output relationships but by their operational closure. A system that has operational closure is one in which the results of its processes are those processes themselves. The notion of operational closure is thus a way of specifying classes of processes that, in their very operation, turn back upon themselves to form autonomous networks. Such networks do not fall into the class of systems defined by external mechanisms of control (heteronomy) but rather into the class of systems defined by internal mechanisms of self-organization (autonomy). The key point is that such systems do not operate by representation. Instead of representing an independent world, they enact a world as a domain of distinctions that is inseparable from the structure embodied by the cognitive system.

We wish to evoke the point that when we begin to take such a conception of mind seriously, we must call into question that idea that information exists ready-made in the world and that it is extracted by a cognitive system, as the cognitivist notion of an informavore vividly implies." pp. 138 to 140.

--------------

"We propose as a name the term enactive to emphasize the growing conviction that cognition is not the representation of a pregiven world by a pregiven mind but is rather the enactment of a world and a mind on the basis of a history of the variety of actions that a being in the world performs. The enactive approach takes seriously, then, the philosophical critique of the idea that the mind is a mirror of nature but goes further by addressing this issue from within the heartland of science." p. 9.

Varela, Thompson, and Rosch (1993)

Nystagmus and the Retinal Image

The following is from Malloy, Jensen, and Song (2003).

Physiological Nystagmus. Visually static objects in everyday life require the constant movement of our eyes (physiological nystagmus) in order to be seen. Physiological nystagmus is produced by tremors in the eye muscles. These tremors prevent the light from stable objects from falling on the same retinal cells from moment to moment. They do that by shifting the eye less than one degree of visual angle (which is about the distance between two foveal cones) around 10 times a second. Experiments with stabilized images (see Palmer, 1999, p. 521ff) indicate that if an image moves synchronously with these tremors, the perception of that image disappears after a few seconds. Bateson (1979, pp. 90, 91) uses nystagmus as a fundamental example in arguing that difference is the basis of mental process. A static relation between a static object and a static eye would produce no differences and therefore no knowledge; this is in full concordance with the stabilized image results. The perception disappears under such circumstances. Nystagmus assures that the eye receives a continuous flow of differences from environmentally static objects.

Enacting a World, Enacting a Mind. We are proposing that the universe be conceived of as a web of dynamic relations some of which set up a field of dynamic relations on the neural receptors of the retina. This, however, is not a passive receiving, as the word "receptors" taken in its strongest sense might connote. A living being has eye muscles that actively twitch the eyes (nystagmus) so as to create motion relative to the context in which it is enmeshed. Add to this other actions (e.g., saccades and micro-saccades). All these actions, taken in relations to a dynamic universe, co-create the retinal image. That is, the image is not simply received from the world by the eye; the eye (i.e., the being) is an active partner in creating the retinal image. The model we prefer is of dynamically active eyes enacting a receptive field on the retina in relation to dynamic relations ongoing in the universe.

Varela, Thompson and Rosch (1993, p. 9) put it this way: "We propose the term enactive to emphasize the growing conviction that cognition is ... the enactment of a world and a mind on the basis of ... actions that a being in the world performs."