Systematicity is a pervasive property of cognitive behaviour (e.g., language and reasoning) whereby the ability to represent (systematicity of representation) and infer from (systematicity of inference) some instances of a structured object extends to other instances conforming to the same structure (Fodor & Pylyshyn, 1988; Fodor & McLaughlin, 1990). The problem of systematicity for a Connectionist approach to cognition is: (1) Can Connectionist models exhibit systematicity without implementing a Classical cognitive architecture (i.e., without resorting to symbolic representations and processes)?; and (2) Can Connectionist models exhibit the necessary acquisition of systematic behaviour (i.e., above chance level)?
In regard to the first question, after a review of the Classicist/Connectionist debate over alternative explanations for systematicity, I conclude that Connectionist models cannot exhibit systematicity without resorting to some form of Classical compositionality (i.e., tokening of component representations wherever complex representations are tokened). Essentially, Smolensky's (1991) weak (microfeatures) and strong (tensors) compositionality, and van Gelder's (1990) functional compositionality either: (1) cannot support systematicity because component representations are not (uniquely) accessible by other processes; or, (2) can support systematicity, but only by tokening component representations relative to the processes that access them (i.e., by implementing some form of Classical compositionality).
However, I also argue that Connectionism potentially offers an explanation for the acquisition of systematic behaviour, which is an issue not addressed in the Classical paradigm since systematicity is built into a Classical architecture. Consequently, the primary concern of this thesis is the second question, which I address by considering two criteria for the necessary acquisition of systematic behaviour, and then evaluating Connectionist models with respect to these criteria on several learning tasks. Thus, systematicity is treated as a problem of generalization rather than representation.
The main results of this thesis concern Hadley's (1993,1994) strong systematicity criterion (i.e., generalization to novel component positions) on inference tasks where a network must learn to infer, on request, the components of binary and ternary relations. Through an analysis of internal representations and network learning dynamics I show that, in general, three-layer first-order networks (including the feedforward network, Elman's, 1990 simple recurrent network and Pollack's, 1990 recursive auto-associative memory) cannot exhibit strong systematicity on the binary relations inference task. I attribute the lack of strong systematicity to an independence between weights that implement component mappings in each of their possible positions.
In response to the lack of strong systematicity with existing networks, I then developed the tensor-recurrent network, which has a dependency between such weights, by incorporating the representational capacity of Smolensky's (1987b) tensor network, with the learning capacity of the simple recurrent network. Constructing and manipulating tensor representations of complex objects, through the inner and outer product operators, assumes appropriate component, role and cue vectors (for representing and extracting component objects and their roles within a complex object). In the tensor-recurrent network, these vectors are learnt by backpropagating an error signal along weighted connections and units implementing the inner and outer product operators. I show that this architecture exhibits the necessary acquisition of systematic behaviour, as defined by Hadley's strong systematicity criterion, on a ternary relations inference task.
In summary, I conclude that:
The separation of internal representations into component and role vector spaces in the tensor-recurrent network is analogous to a type/token distinction characteristic of symbolic computation. Thus, in terms of systematic behaviour, Connectionism offers a ``neural-like" implementation of symbolic processing. However, where Connectionism goes beyond Classicism is in an explanation for the acquisition of systematic behaviour, which is an issue not addressed in the Classical paradigm since systematicity is built into a Classical architecture, not acquired.