7. A Bottom-up Investigation of Internal Context in an Artificial Agent

7.6. Preliminary Results

Two runs were performed with the same learning agent but different environmental set-ups. The first run's environment consisted of a market with 5 traders, each of which had 20 models of initial depth 5, while the second had only 2 agents each with 10 models of initial depth 4. The first will be called the `harder' and the second the `easier' learning task. The prices series that results from these are illustrated in figure 6 and figure 7 respectively. The `harder' series change more abruptly and exhibit a more complex cycle than the `easier' series.

Figure 6. A price series output from the market (two stocks) - harder learning task

Figure 7. A price series output from the market (two stocks) - easier learning task

For each task the agent was given an identical network structure of: 23 input nodes, 20 intermediate nodes; and 11 output nodes. The input nodes were randomly connected to an average of four intermediate nodes each, which were connected with 3 output nodes and switching 3 other arcs using enabling arcs as described above. The agent then runs through the dates once, hence it is a one-shot incremental learning task. The network is deliberately limited in order to simulate the sort of learning a real agent might attempt in a similar situation.

Despite the considerable difficulties it was faced with the agent did manage to induce some effective (but, of course, fallible) rules about the prices series. For both tasks the agent specialised some of the nodes to act as indications of context, that is it learnt arc weights such that these nodes acted to `switch' other arcs and did not imply any of the output nodes directly themselves. In figure 8 I have plotted the number of these `context' nodes as the simulation progressed for both learning tasks. It can be seen that the agent induced more contexts for the `easier' task than the `harder' one.

Figure 8. Number of nodes acting as contexts - both learning tasks

In figure 9 I have summarised the contexts developed by the agent at the end of the `harder' learning task. These were constantly being selected in and out, due to the nature of the learning algorithm. The underlined headings indicate the conditions under which the context is recognised and the implications under them are the rules that are hold in that context (i.e. are enabled by them). Thus the consequent of one of these rules will only be inferred if the context is fired as well as the antecedent, but this does not mean that the context is functionally the same as the antecedent as they are reinforced by the learning algorithm in different ways. The enabling arcs' weights are changed at a slower rate than the node-node arcs (one quarter as slowly). Thus it will take many more occasions of negative feedback to change a context node's associations than one of the implicational arcs in that context. This allows learnt contexts to last longer. Likewise the arcs that lead to the contextual node (i.e. those representing the conditions which the context is recognised) can change more quickly than the `membership' of that context.

In this way I have allowed the emergence of context-like behaviour without imposing the sort of two-tier structures that have been employed in other machine learning algorithms (e.g. [16, 20]). If I had a network with more than one intermediate layer we could allow the emergence of contexts within contexts etc. but this would take longer runs in order for any results to emerge.

Figure 9. The `contexts' induced by the agent by the last 10 dates - harder learning example

Future work with this model will involve attempting more runs of the current model; runs with a more intricate network topology and runs on data generated by real-world processes. The idea is not to discover any `golden rules' for these domains but to start to tease out how and in what way different contextual structures can arise from different sorts of regularity (or lack of it) in the data.

Generated with CERN WebMaker