Parallel developments to isolate the operant behavior have been made in the sea slug Aplysia ([Nargeot et al., 1997], [Nargeot et al., 1999a], [Nargeot et al., 1999b], [Nargeot et al., 1999d], [Nargeot et al., 2007], [Nargeot et al., 2009], [Brembs et al., 2002], [Nargeot, 2002], [Lorenzetti et al., 2006], [Lorenzetti et al., 2008] and [Nargeot and Simmers, 2010]). There, freely moving animals generate feeding movements in the absence of eliciting stimuli. Even the isolated buccal ganglia, which control these feeding movements in the intact animal, generate spontaneous motor patterns in the dish (‘fictive feeding’). Implanted electrodes can then be used to provide the animal or the isolated ganglia with a virtual food reward as reinforcement for one class of feeding movements but not others. Analogous to the Pavlovian strategy of isolating the relevant stimuli and then tracing their pathway into the nervous system until the synapse of convergence between conditioned and unconditioned stimulus had been identified, we identified the neuron where spontaneous behavior and reinforcement converge: a neuron called B51 ([Plummer and Kirk, 1990], [Nargeot et al., 1999b], [Nargeot et al., 1999c] and [Brembs et al., 2002]). Importantly, we discovered that the mechanism of plasticity in B51, unlike the synaptic plasticity found in Pavlovian learning situations, increased the excitability of the entire neuron ([Brembs et al., 2002] and [Mozzachiodi and Byrne, 2010]). This finding makes intuitive sense, because neuron B51 is only recruited into the active feeding circuitry during the behaviors we rewarded and not during other feeding movements (Nargeot et al., 1999a). Later, other neurons in this same circuit have been identified which also change their biophysical membrane properties after operant conditioning, serving to increase the number of behaviors generated (Nargeot et al., 2009). It is tempting to generalize from this one occurrence that we may have stumbled across a dedicated mechanism which modifies behavioral circuits by altering the biophysical membrane properties of entire neurons in these circuits, biasing the whole network towards one or the other behavior by altering the probabilities with which certain network states can be attained. If this were the case, one should be able to find homologous mechanisms in other animals.

3. The neurogenetics of self-learning

Its genetic toolbox is the well-known strength of Drosophila as a neurobiological model system. However, without the isolation of the operant behavior from contingent external stimuli, no molecular tool would have been able to tease apart the operant/classical conundrum which has intrigued scholars for over 70 years. Making a non-directional infrared heat-beam contingent on spontaneous left or right turning maneuvers (Wolf and Heisenberg, 1991) provides instantaneous punishment without any other stimuli being contingent on the behavior (analogous to the virtual food reward being contingent on feeding motor programs in Aplysia). Within seconds, the fly learns (by trial and error) that its turning attempts control the unpleasant heat. After 8 min of training, it biases its spontaneous decisions towards the previously unpunished turning maneuvers, even if the heat is now permanently switched off (Fig. 2). Using mutant, wildtype and transgenic animals, we discovered that the canonical, cAMP-dependent synaptic plasticity pathway was not involved in this type of learning, but manipulating protein kinase C (PKC) signaling abolished learning in this paradigm completely (Brembs and Plendl, 2008) (Fig. 3a). In the most Skinnerian sense, the animal modifies its nonlinear decision-making circuitry such that the probability of initiating the unpunished behavior increases.

These results are mirror-symmetric compared to an experiment which is almost identical, albeit with one small difference. The smallchange in this experiment is that whenever the direction of turning maneuvers changes, the entire visual field of the fly instantaneously turns from one color (say, green) to another (e.g., blue). Because now the colors change both with the yaw torque and the heat, the fly has the option to learn that one of the colors signals heat, analogous to how rats in Skinner-boxes may learn that the depressed lever signals food (and the undepressed lever no food). This situation now requires the cAMP cascade of synaptic plasticity and is independent of any PKC signaling (Fig. 3b). To solve this situation, it is sufficient for the flies to learn that one of the colors is associated with the heat and then use whatever means necessary to avoid this color. In the most Pavlovian sense, the flies learn the color-heat contingency independently of the behavior with which it was acquired, they only learn about the world around them, without leaving an observable trace that the behavioral decision-making circuitry itself has been altered, even though, of course, the entire situation is still just as operantas without the stimuli ([Brembs and Heisenberg, 2000] and [Brembs, 2009b]).

These two experiments exemplify drastically why the operant/classical debate has been going on for so many decades without any solution. Here we have two obviously operant experiments, one in which the operant behavior has been experimentally isolated and one almost identical, except that in this second experiment, the behavior is not isolated any more. Instead, it is accompanied by an explicitly predictive stimulus. Both experiments are clearly operant conditioning experiments, but the biological processes cleanly separate the experiments into two different operant categories. Perhaps one factor hampering research in the previous decades was the lack of a proper terminology, which separates between procedures and mechanisms, allowing for the necessary experiments to be designed? Our discovery demonstrates that the procedural distinction between operant and classical conditioning is not helpful for understanding the neurobiological mechanisms engaged during the learning tasks. To distinguish the biological process which modifies the behavioral circuits from the one which modifies sensory pathways, we have suggested to call the firstprocess ‘self-learning’ (the animal learns about its own behavior) and the second process ‘world-learning’ (the animal learns about the properties of the world around it)(Colomb and Brembs, 2010). Evidence from Aplysia suggests that this distinction is highly conserved among bilaterian animals: also in the neuron B51, the type I adenylyl cyclase involved in synaptic plasticity is not required to show the change in excitability after self-learning, whereas PKC signaling is necessary (Lorenzetti et al., 2008).

Another piece of evidence, as yet not published in a peer-reviewed journal, ties these insights to analogous vertebrate learning mechanisms. Vocal learning, be it human language or birdsong, follows the same operant trial-and-error principle as both our paradigms in Drosophila and Aplysia: spontaneous, variable behavior is first generated and then the sensory feedback (in this case auditory) received from this behavior is evaluated. The consequence of this evaluation is a lasting modification of the circuit generating the behavior leading to a bias towards the rewarded and away from the unrewarded behaviors. The only difference between vocal learning and invertebrate self-learning is the primary reward/punishment in the invertebrate case, compared to the more implicit reward of speaking a word or matching the tutor song. One prominent gene which has been discovered studying the biological basis of vocal learning is the gene FOXP2 ([Lai et al., 2001] and [Fisher et al., 2009]). Every member of the now famous KE family who suffers from severe verbal dyspraxia carries a mutated FOXP2 allele (Lai et al., 2001). If FOXP2 is knocked down in the basal ganglia of zebrafinches, they fail to learn their song properly (Haesler et al., 2007). Drosophila and other invertebrate genomes also contain a FoxP gene (Santos et al., 2010). A mutation or RNAi-mediated knockdown of FoxP in Drosophila yields a phenocopy of our PKC manipulations, i.e., impaired self-learning and unaffected world-learning ([Brembs et al., 2010] and [Colomb et al., 2010]).

It is tempting to interpret these results as corroborating evidence for the hypothesis that there exists a highly conserved mechanism of plasticity which evolved specifically to modulate the excitability of neurons involved in behavioral choice. This mechanism appears to be based on PKC and FoxP signaling mediating changes to the biophysical membrane properties affecting the excitability of the entire neuron. These excitability changes, in turn, serve to bias the decision-making network towards rewarded and away from punished behaviors by making it less likely for the network to reach a state which would generate a punished behavior and making it more likely to reach a state which would generate a rewarded behavior.

4. Multiple memory systems interacting

One may wonder why synaptic plasticity was not sufficient for all learning situations. Was there a specific need for a second mechanism that could not have been served equally well by synaptic plasticity? These questions can be tackled experimentally by separate tests for the two learning systems after operant conditioning with both components present (i.e., ‘composite conditioning’). Specifically, we trained animals in the right/left turning paradigm with colors present and then tested them after varying amounts of training in different test situations (Fig. 3c). Corroborating our interpretation that the colors are learned independently from the behavior used to control them, flies do not show any preference for the unpunished turning direction once the colors are removed, even though the amount of training was exactly the same as without colors (Brembs, 2009b). Conversely, the flies can avoid the punished color with a novel, orthogonal behavior controlling the colors. Thus, world-learning inhibits self-learning which would take place without the colors. If the same tests are conducted after twice the regular amount of training (i.e., 16 instead of 8 min), the results are reversed: now the flies show a spontaneous preference towards the previously unpunished direction when the colors are removed and are not able to avoid the previously punished color any more, using the orthogonal behavior. Mimicking the formation of habits or skills in vertebrate animals, the extended training has overcome the inhibition of self-learning by world-learning such that world-learning could take place and modify the fly's decision-making circuits. The flexible, goal-directed actions controlling the heat have become stereotyped, habitual responses. The scientists studying mammals commonly refer to this process as the transformation of response–outcome (R–O) associations into stimulus–response (S–R) associations (Balleine and O’Doherty, 2010).

Thus, there is an interaction between the two learning systems which would seem difficult, if not impossible to implement if there was only one mechanism mediating all learning types. World-learning inhibits self-learning in a time-dependent manner which serves to delay modification of the behavioral circuitry until proper vetting, rehearsal or optimization has taken place. Using transgenic animals to silence certain populations of neurons in the fly brain, we discovered that this inhibition is mediated by a prominent neuropil in the insect brain, the mushroom-bodies (corpora pedunculata) (Brembs, 2009b).

There is yet another, reciprocal interaction between the two memory systems. This interaction facilitates world-learning when the to-be-learned stimuli are under operant control. Flies, just like virtually every animal ever tested ([Thorndike, 1898], Slamecka and Graf, 1978 N.J. Slamecka and P. Graf, Generation Effect - Delineation of a Phenomenon, J. Exp. Psych.: Hum. Learn. Mem. 4 (1978), pp. 592–604. Abstract | PDF (1129 K) | Full Text via CrossRef[Slamecka and Graf, 1978] and [Kornell et al., 2007]), learn a contingency in the world around them more efficiently if they can explore these contingencies themselves ([Brembs and Heisenberg, 2000] and [Brembs and Wiener, 2006]). This facility has also been termed learning by doing (Baden-Powell, 1908) or the generation effect (Slamecka and Graf, 1978).

Thus, learning in realistic situations comprising self- and world-learning components is mediated by two dedicated, evolutionary conserved learning systems which interact reciprocally. These interactions ensure efficient, yet flexible world-learning and allow for sufficient rehearsal/practice time before self-learning transforms goal-directed actions into stereotypic, habitual responses (Fig. 4).

5. Conflicting evidence

Clearly, I have painted a simplified and exaggerated hypothesis in an attempt to explain the concepts more clearly and facilitate comprehension of the biological results against the novel theoretical backdrop. Just as synaptic plasticity is rarely, if ever mediated exclusively by pre- or by postsynaptic mechanisms, exclusive recruitment of synaptic or intrinsic plasticity by any given experimental procedure will probably be equally rare. In this section, I will provide a brief summary of the currently available evidence which is difficult to reconcile with the hypothesis fleshed out so far.

Probably the most tentative aspect of the hypothesis is the conjecture that because there is a double dissociation between rutabaga (i.e., type I adenylyl cyclase-mediated cAMP cascades) and PKC manipulations on world- and self-learning, respectively, in Drosophila and in Aplysia, that the same double dissociation also holds for the proposed physiological mechanisms, namely synaptic and intrinsic plasticity, respectively. The main evidence for this conjecture stems from a single model system, Aplysia. In Drosophila, we do not know which form of plasticity mediates self-learning. We only know that whatever plasticity mediates self-learning, it appears to require the same molecular cascades as the intrinsic plasticity discovered in Aplysia. In addition to this deduction by analogy, there is a plausibility argument: if network states are defined by the neurons participating in any given network dynamics, intrinsic plasticity is a very plausible way in which such recruitment (and hence network states) can be biased. While behavior is known to be controlled by the state of the network controlling it, sensory input is often depicted as a ‘labeled line’ along which the sensory information is processed and transmitted. Finally, the dominant paradigm in the literature by which classical learning is explained is synaptic plasticity, while there is little neurophysiological data for operant conditioning.

Challenging to our tentative postulation of a double dissociation among physiological mechanisms are mainly reports that classical conditioning situations, where largely world-learning mechanisms would be assumed to be at play, also seem to involve intrinsic plasticity (reviewed, for instance, in (Mozzachiodi and Byrne, 2010). These examples include a type of photoreceptor in Hermissenda changing its biophysical membrane properties after classical conditioning ([Crow and Alkon, 1980], [Alkon, 1984] and [Farley and Alkon, 1985]), neuron B51 in Aplysia changing its biophysical properties in the opposite direction after classical conditioning compared to operant conditioning(Lorenzetti et al., 2006), changes in the input resistance of LE sensory neurons in in vitro classical conditioning of the Aplysia siphon withdrawal reflex(Antonov et al., 2001), nonsynaptic plasticity in modulatory neurons after classical conditioning in the freshwater snail Lymnaea ([Jones et al., 2003] and [Kemenes et al., 2006]) and trace eyelid conditioning producing changes in the excitability of pyramidal neurons in the CA1 area of the rabbit hippocampus ([Disterhoft et al., 1986] and [Moyer et al., 1996]).

The fact that all of these examples all dwarfed in number by the publications demonstrating synaptic plasticity in the sensory pathways during classical conditioning may be explained by researchers not being aware of intrinsic plasticity and therefore not explicitly looking for it. However, it may also be that these examples are exceptions to a more common rule. Some of these examples also fail to report if the observed phenomena are really crucial to the learning process, i.e., would preventing these processes from happening during conditioning really impair the performance of the animals in the test after training? Some of the effects are only observed long after the learning has taken place and may be a process helping in stabilize the memory but contributing little to the actual content of the memory trace. Along similar lines, most of these examples also fail to rule out motivational or contextual contributions of the observed modifications, rather than being specific components of a memory trace. All of the above arguments serve to make the point that there probably are many more than just two learning mechanisms engaged during the various forms of conditioning and these examples may eventually provide avenues for discovering them. Thus, while publications like the ones just cited do pose problems to our hypothesis, at this point they do not suffice to eliminate it conclusively.

A second, related class of observations also poses problems for our hypothesis. This is the rare case of observations of synaptic plasticity in self-learning situations, such as the enhanced strength of electrical synapses in the central pattern generator in the Aplysia buccal ganglia after operant conditioning (Nargeot et al., 2009). Indeed, this is exactly the kind of synaptic modification one would postulate to be sufficient for all kinds of learning (see above). However, the authors themselves conclude that the changes in biophysical membrane properties may be the source for the increase in electrical coupling: “it is likely that the measured increase in membrane input resistance accounts in large part for their increased coupling coefficients, with perhaps little or no additional contribution made by a direct alteration in junctional resistance” (Nargeot et al., 2009). There are also some reports on synaptic changes after operant conditioning in lever-pressing vertebrates ([Jaffard and Jeantet, 1981] and [Kokarovtseva et al., 2009]). However, as these experiments still include the lever, it is impossible to say if these changes are not triggered by learning about the lever, rather than modifying the decision-making circuitry.

Finally, objections may be raised with regard to the generality of PKC being engaged in self-learning and not in world-learning. Indeed, there are many publications showing clearly that PKC mediates synaptic plasticity in many different preparations ([Olds et al., 1989], [Olds and Alkon, 1991], [Byrne and Kandel, 1996], [Drier et al., 2002], [Zhao et al., 2006], [Sossin, 2007], [Shema et al., 2009] and [Villareal et al., 2009]). However, many, if not all, of these publications only show that PKC activity either takes place or is required after training, during the consolidation phase. Indeed, few of these experiments explicitly compare blocking of PKC during training as opposed to blocking PKC activity after training or during retrieval only. In fact, the Drosophila study where this was done showed that PKC was not required during training. The requirement for PKC was for memory maintenance not acquisition. Furthermore, there are a good dozen different PKC isoforms and it is currently unknown which isoform is responsible for the self-learning effect in Aplysia and Drosophila. Thus, even if one isoform of PKC would be discovered that is critically involved in world-learning, this may not be the same isoform that appears to be involved in self-learning.

In summary, while there are isolated publications in the literature which require ad hoc arguments, these reports alone do not seem sufficient to falsify our, admittedly still tentative and highly simplified hypothesis. Most likely, future research will elucidate which other biological processes are engaged during the various conditioning experiments and how they interact with self- and world-learning. At this point, the research on the biological processes of self-learning is still in its infancy and postulating a physiological mechanism to go with the genetic evidence may just trigger the necessary experiments to falsify the hypothesis.

6. Epilogue

Using modern neurobiological model systems with sophisticated behavioral testing has allowed us to make significant inroads into classic behavior-analytic problems which have been recognized and described already more than 70 years ago. Despite the evidence still being patchy and comprising only few genes in a small number of model systems, overall the data seem to fit an emerging picture that makes ecological sense. The new tools and methods have only just begun to open an avenue neither Pavlov nor Skinner could dream of. About a century after Pavlov started to study behavior using physiological and analytical approaches, we have now started to gather the first glimpses into the neurobiological mechanisms underlying the probably most fundamental brain function: how to balance internal processing with external demands in order to achieve adaptive behavioral choice. These glimpses could only be obtained using multiple model systems, each with their own, unique advantages. The time for invertebrates in neuroscience has not ended – it is just about to take off.