learning and recalling a sequence of frames

In this post, we tweak our sequence learning code to learn and recall a short sequence of random frames. Previously all our sequences have been over random 10 bit on SDR's. This post shows we can extend this to sequences of almost arbitrary SDR's. Perhaps the only limitation is that these SDR's need to have enough on bits. Thanks to the random-column[k] operator acting on our SDR's we have an upper bound of k^n distinct contexts, where n is the number of on bits in the given SDR. Though smaller than this in practice to allow for noise tolerance. Which means the 10 on bits, and k = 10 which we have been using is more than enough, even for the 74k sequences in the spelling dictionary example.

To learn and recall our frames, we need three new operators:

random-frame[w,h,k]
display-frame[w,h]
display-frame-sequence[w,h]

where w,h are the width and height of the frames, and k is the number of on bits. For example, here is a 15*15 frame with 10 on bits:

sa: display-frame[15,15] random-frame[15,15,10]
....#..........
.........#...#.
.......#.......
...............
...............
..............#
.......#.......
...............
...............
..........#....
...............
..#..........#.
...............
.........#.....
...............

And this is what the random-frame SDR's look like (we just store the co-ordinates of the on bits):

sa: random-frame[15,15,10]
|7: 7> + |6: 2> + |10: 8> + |4: 12> + |4: 5> + |13: 9> + |8: 8> + |12: 8> + |13: 0> + |12: 14>

And now displaying this exact frame:

sa: display-frame[15,15] (|7: 7> + |6: 2> + |10: 8> + |4: 12> + |4: 5> + |13: 9> + |8: 8> + |12: 8> + |13: 0> + |12: 14>)
.............#.
...............
......#........
...............
...............
....#..........
...............
.......#.......
........#.#.#..
.............#.
...............
...............
....#..........
...............
............#..

OK. On to the main event. Let's learn these two sequences defined using gm notation:

$ cat gm-examples/frame-sequences.gm
seq-1 = {1.2.3.4.5}
seq-2 = {2.4.1.5.3}

We convert that to sw using gm2sw-v2.py, and then manually edit it to look like this:

-- frames:
frame |1> => random-frame[10,10,10] |>
frame |2> => random-frame[10,10,10] |>
frame |3> => random-frame[10,10,10] |>
frame |4> => random-frame[10,10,10] |>
frame |5> => random-frame[10,10,10] |>
frame |end of sequence> => random-frame[10,10,10] |>

-- learn low level sequences:
-- empty sequence
pattern |node 0: 0> => random-column[10] frame |end of sequence>

-- 1 . 2 . 3 . 4 . 5
pattern |node 1: 0> => random-column[10] frame |1>
then |node 1: 0> => random-column[10] frame |2>

pattern |node 1: 1> => then |node 1: 0>
then |node 1: 1> => random-column[10] frame |3>

pattern |node 1: 2> => then |node 1: 1>
then |node 1: 2> => random-column[10] frame |4>

pattern |node 1: 3> => then |node 1: 2>
then |node 1: 3> => random-column[10] frame |5>

pattern |node 1: 4> => then |node 1: 3>
then |node 1: 4> => random-column[10] frame |end of sequence>

-- 2 . 4 . 1 . 5 . 3
pattern |node 2: 0> => random-column[10] frame |2>
then |node 2: 0> => random-column[10] frame |4>

pattern |node 2: 1> => then |node 2: 0>
then |node 2: 1> => random-column[10] frame |1>

pattern |node 2: 2> => then |node 2: 1>
then |node 2: 2> => random-column[10] frame |5>

pattern |node 2: 3> => then |node 2: 2>
then |node 2: 3> => random-column[10] frame |3>

pattern |node 2: 4> => then |node 2: 3>
then |node 2: 4> => random-column[10] frame |end of sequence>


-- define our classes:
-- seq-1 = {1.2.3.4.5}
start-node |seq 1> +=> |node 1: 0>

-- seq-2 = {2.4.1.5.3}
start-node |seq 2> +=> |node 2: 0>

After loading that into the console we have:

$ ./the_semantic_db_console.py
Welcome!

sa: load frame-sequences.sw
sa: dump

----------------------------------------
|context> => |context: sw console>

frame |1> => |0: 9> + |5: 1> + |5: 5> + |8: 1> + |8: 2> + |7: 7> + |9: 1> + |3: 0> + |6: 3> + |2: 8>

frame |2> => |0: 6> + |3: 8> + |1: 2> + |2: 5> + |2: 2> + |0: 9> + |4: 3> + |6: 6> + |9: 4> + |9: 0>

frame |3> => |5: 7> + |9: 1> + |9: 3> + |0: 8> + |0: 0> + |8: 6> + |7: 9> + |5: 9> + |2: 5> + |7: 0>

frame |4> => |9: 3> + |1: 7> + |6: 9> + |0: 0> + |4: 6> + |8: 1> + |7: 4> + |4: 8> + |9: 1> + |4: 1>

frame |5> => |3: 3> + |5: 3> + |5: 9> + |5: 8> + |7: 1> + |7: 6> + |3: 1> + |4: 1> + |2: 2> + |5: 2>

frame |end of sequence> => |3: 8> + |7: 7> + |1: 3> + |2: 0> + |9: 2> + |0: 7> + |5: 7> + |3: 1> + |4: 9> + |3: 0>

pattern |node 0: 0> => |3: 8: 7> + |7: 7: 4> + |1: 3: 0> + |2: 0: 0> + |9: 2: 3> + |0: 7: 8> + |5: 7: 2> + |3: 1: 0> + |4: 9: 1> + |3: 0: 6>

pattern |node 1: 0> => |0: 9: 0> + |5: 1: 9> + |5: 5: 1> + |8: 1: 6> + |8: 2: 1> + |7: 7: 7> + |9: 1: 6> + |3: 0: 6> + |6: 3: 4> + |2: 8: 2>
then |node 1: 0> => |0: 6: 0> + |3: 8: 4> + |1: 2: 4> + |2: 5: 7> + |2: 2: 9> + |0: 9: 5> + |4: 3: 3> + |6: 6: 2> + |9: 4: 9> + |9: 0: 4>

pattern |node 1: 1> => |0: 6: 0> + |3: 8: 4> + |1: 2: 4> + |2: 5: 7> + |2: 2: 9> + |0: 9: 5> + |4: 3: 3> + |6: 6: 2> + |9: 4: 9> + |9: 0: 4>
then |node 1: 1> => |5: 7: 8> + |9: 1: 1> + |9: 3: 1> + |0: 8: 2> + |0: 0: 6> + |8: 6: 8> + |7: 9: 3> + |5: 9: 6> + |2: 5: 7> + |7: 0: 0>

pattern |node 1: 2> => |5: 7: 8> + |9: 1: 1> + |9: 3: 1> + |0: 8: 2> + |0: 0: 6> + |8: 6: 8> + |7: 9: 3> + |5: 9: 6> + |2: 5: 7> + |7: 0: 0>
then |node 1: 2> => |9: 3: 9> + |1: 7: 0> + |6: 9: 9> + |0: 0: 9> + |4: 6: 5> + |8: 1: 3> + |7: 4: 5> + |4: 8: 0> + |9: 1: 8> + |4: 1: 7>

pattern |node 1: 3> => |9: 3: 9> + |1: 7: 0> + |6: 9: 9> + |0: 0: 9> + |4: 6: 5> + |8: 1: 3> + |7: 4: 5> + |4: 8: 0> + |9: 1: 8> + |4: 1: 7>
then |node 1: 3> => |3: 3: 7> + |5: 3: 6> + |5: 9: 4> + |5: 8: 4> + |7: 1: 7> + |7: 6: 9> + |3: 1: 8> + |4: 1: 3> + |2: 2: 5> + |5: 2: 3>

pattern |node 1: 4> => |3: 3: 7> + |5: 3: 6> + |5: 9: 4> + |5: 8: 4> + |7: 1: 7> + |7: 6: 9> + |3: 1: 8> + |4: 1: 3> + |2: 2: 5> + |5: 2: 3>
then |node 1: 4> => |3: 8: 6> + |7: 7: 0> + |1: 3: 7> + |2: 0: 9> + |9: 2: 0> + |0: 7: 1> + |5: 7: 3> + |3: 1: 9> + |4: 9: 2> + |3: 0: 9>

pattern |node 2: 0> => |0: 6: 7> + |3: 8: 5> + |1: 2: 9> + |2: 5: 4> + |2: 2: 3> + |0: 9: 2> + |4: 3: 4> + |6: 6: 1> + |9: 4: 8> + |9: 0: 6>
then |node 2: 0> => |9: 3: 8> + |1: 7: 3> + |6: 9: 0> + |0: 0: 2> + |4: 6: 8> + |8: 1: 1> + |7: 4: 0> + |4: 8: 4> + |9: 1: 0> + |4: 1: 1>

pattern |node 2: 1> => |9: 3: 8> + |1: 7: 3> + |6: 9: 0> + |0: 0: 2> + |4: 6: 8> + |8: 1: 1> + |7: 4: 0> + |4: 8: 4> + |9: 1: 0> + |4: 1: 1>
then |node 2: 1> => |0: 9: 9> + |5: 1: 6> + |5: 5: 0> + |8: 1: 0> + |8: 2: 8> + |7: 7: 2> + |9: 1: 1> + |3: 0: 2> + |6: 3: 7> + |2: 8: 3>

pattern |node 2: 2> => |0: 9: 9> + |5: 1: 6> + |5: 5: 0> + |8: 1: 0> + |8: 2: 8> + |7: 7: 2> + |9: 1: 1> + |3: 0: 2> + |6: 3: 7> + |2: 8: 3>
then |node 2: 2> => |3: 3: 9> + |5: 3: 8> + |5: 9: 6> + |5: 8: 7> + |7: 1: 8> + |7: 6: 5> + |3: 1: 5> + |4: 1: 9> + |2: 2: 8> + |5: 2: 0>

pattern |node 2: 3> => |3: 3: 9> + |5: 3: 8> + |5: 9: 6> + |5: 8: 7> + |7: 1: 8> + |7: 6: 5> + |3: 1: 5> + |4: 1: 9> + |2: 2: 8> + |5: 2: 0>
then |node 2: 3> => |5: 7: 0> + |9: 1: 3> + |9: 3: 8> + |0: 8: 5> + |0: 0: 8> + |8: 6: 6> + |7: 9: 9> + |5: 9: 5> + |2: 5: 9> + |7: 0: 8>

pattern |node 2: 4> => |5: 7: 0> + |9: 1: 3> + |9: 3: 8> + |0: 8: 5> + |0: 0: 8> + |8: 6: 6> + |7: 9: 9> + |5: 9: 5> + |2: 5: 9> + |7: 0: 8>
then |node 2: 4> => |3: 8: 8> + |7: 7: 1> + |1: 3: 7> + |2: 0: 1> + |9: 2: 3> + |0: 7: 1> + |5: 7: 4> + |3: 1: 6> + |4: 9: 1> + |3: 0: 8>

start-node |seq 1> => |node 1: 0>

start-node |seq 2> => |node 2: 0>
----------------------------------------

with one interpretation that each ket is the co-ordinate of a synapse. Eg, |5: 1> or |5: 9: 4>. Noting that frames are 2D, and random-column[k] maps frames to 3D. It is this extra dimension that allows SDR's to be used in more than 1 context. Indeed, an upper bound of k^n distinct contexts, where n is the number of on bits. Though in the current example we only have two distinct sequences. Now let's display a couple of frames:

sa: display-frame[10,10] frame |1>
...#......
.....#..##
........#.
......#...
..........
.....#....
..........
.......#..
..#.......
#.........

sa: display-frame[10,10] frame |2>
.........#
..........
.##.......
....#.....
.........#
..#.......
#.....#...
..........
...#......
#.........

Now a couple of frames in our first sequence, noting "extract-category" is the inverse of random-column[k] and hence converting the pattern SDR back to 2D:

sa: display-frame[10,10] extract-category pattern |node 1: 0>
...#......
.....#..##
........#.
......#...
..........
.....#....
..........
.......#..
..#.......
#.........

sa: display-frame[10,10] extract-category then |node 1: 0>
.........#
..........
.##.......
....#.....
.........#
..#.......
#.....#...
..........
...#......
#.........

And finally our sequences:

sa: display-frame-sequence[10,10] start-node |seq 1>
...#......
.....#..##
........#.
......#...
..........
.....#....
..........
.......#..
..#.......
#.........

.........#
..........
.##.......
....#.....
.........#
..#.......
#.....#...
..........
...#......
#.........

#......#..
.........#
..........
.........#
..........
..#.......
........#.
.....#....
#.........
.....#.#..

#.........
....#...##
..........
.........#
.......#..
..........
....#.....
.#........
....#.....
......#...

..........
...##..#..
..#..#....
...#.#....
..........
..........
.......#..
..........
.....#....
.....#....

|end of sequence>

sa: display-frame-sequence[10,10] start-node |seq 2>
.........#
..........
.##.......
....#.....
.........#
..#.......
#.....#...
..........
...#......
#.........

#.........
....#...##
..........
.........#
.......#..
..........
....#.....
.#........
....#.....
......#...

...#......
.....#..##
........#.
......#...
..........
.....#....
..........
.......#..
..#.......
#.........

..........
...##..#..
..#..#....
...#.#....
..........
..........
.......#..
..........
.....#....
.....#....

#......#..
.........#
..........
.........#
..........
..#.......
........#.
.....#....
#.........
.....#.#..

|end of sequence>

Anyway, a nice proof of concept I suppose.