Abstract
The encoder task [1] is a widely used benchmark for connectionist learning algorithms. Kruglyak [8] proved that weights exist to implement the most difficult encoder, the N-2-N encoder, for arbitrarily large N, but Lister [9] found that Backpropagation can only feasibly learn such encoders for small N (<= 19). We show that learning the encoder is as much a representation problem as a learning problem. If a suitable, block encoded representation is used on the output, standard Backpropagation is able to learn encoders up to N = 1000. A novel analysis of the error surface demonstrates that local encoding creates an error surface typical of exception tasks, which backpropagation is poor at traversing. The use of block encoding dramatically increases the size of the global minimum, and reshapes the error surface so that this minimum is easily found by gradient descent.
Citation
Bakker, P., Phillips, S. and Wiles, J. (1994). The 1000-2-1000 encoder: A matter of Representation. Neural Network World, 4(5), 527-534.