episodicMemory [iCub modules]

The episodicMemory module maintains as associative memory of images: More...

The episodicMemory module maintains as associative memory of images:

• when an image is presented to the memory, it attempts to recall that image;
• if a previously-stored image matches the presented image sufficiently well, the stored image is recalled.
• if no previously-stored image matches sufficiently well, the presented image is stored;
• in both cases, the module outputs:
  1. the recalled or stored image;
  2. an image identification number and the match value for the presented image
  3. an image identification number and the match value for whatever image was previously presented
• Alternatively, when an image identification number is presented to the memory, the associated image is retrieved. If both image and image identification number are presented, the image identification number takes precedence.

Since images are streamed continuously, the module reads an image with a pre-set but definable frequency. However, the recall is triggered by the input of a gaze command, typically produced by the attentionSelection module. In addition, before reading the module checks to ensure that the iCub head is not in motion so that the eyes are fixating on something. A configuration option - offline - allows this check to be switched off.

The frequency at which images are read and the threshold defining whether or not an input image adequately matches a stored image are both provided as module parameters, set in the episodicMemory configuration file. The thresold can also be set interactively via the episodicMemory port.

The episodicMemory module has the following inputs:

• an input image
• an input image vector containing an image id. number for the required image
• an action vector input containing five values, the first two representing the gaze angles; The protocol corresponds to that of the attentionSelection module for controlling gaze. Note that the action tag value is not yet implemented

The episodicMemory has the following outputs:

• recalled/stored image
• recalled/stored image vector containing
  1. an image id. number for the matched image
  2. a match value r, 0 <= r <= 1, for the matched image
  3. the azimuth angle of the gaze at which the image was acquired
  4. the elevation angle of the gaze at which the image was acquired
  5. an image id. number for the previously matched image
  6. a match value r, 0 <= r <= 1, for the previously matched image
  7. the azimuth angle of the gaze at which the previously image was acquired
  8. the elevation angle of the gaze at which the previously image was acquired
• retrieved image (i.e. the image indexed by the image id. number)

Dependencies

YARP.

Parameters

Command Line Parameters

The following key-value pairs can be specified as command-line parameters by prefixing -- to the key (e.g. --from file.ini. The value part can be changed to suit your needs; the default values are shown below.

• from episodicMemory.ini
  specifies the configuration file

• context episodicMemory/conf
  specifies the sub-path from $ICUB_ROOT/icub/app to the configuration file

• name episodicMemory
  specifies the name of the module (used to form the stem of module port names)

• robot icub
  specifies the name of the robot (used to form the root of robot port names)

Configuration File Parameters

The following key-value pairs can be specified as parameters in the configuration file (they can also be specified as command-line parameters if you so wish). The value part can be changed to suit your needs; the default values are shown below.

• imageInPort /image:i
  specifies the port for input of an image

• imageIdInPort /imageId:i
  specifies the port for input of an image identification number

• actionInPort /action:i
  The input port name for the gaze angles; the action tag value is not yet implemented

• headPort /head:i
  specifies the input port name for the head encoder values used to check that the iCub head has stopped moving

• recalledImageOutPort /recalledImage:o
  specifies the image output port

• retrievedImageOutPort /retrievedImage:o
  specifies the image output port

• imageIdOutPort /imageId:o
  specifies the port for output of the image identification number corresponding to the output image

• database episodicDatabase
  specifies the directory name in which the database of images will be stored

• path ~/iCub/app/episodicMemory
  specifies the path the to directory in which the database of images will be stored

• threshold 0.75
  specifies the value defining whether or not an input image matches adequately matches a stored image

• frequency 2
  specifies the number of images to be streamed per second.

• offline 0
  The check on the robot gaze is switched off if this value is not zero.

The database parameter specifies the name of the directory in which the database of images is stored. This directory must be created before running the module.

The path parameter specifies the full path to the database directory. This is where the where the database of image files is stored.

The threshold parameter determines which, if any, of the stored images are recalled: images which match the input image with a value that equals or exceeds the threshold are recalled and if more than one image exceeds the threshold, the image with the highest match is recalled.

For example, if the configuration file episodicMemory.ini is located in C:/iCub/app/episodicMemory/conf and the database is C:/iCub/app/episodicMemory/episodicDatabase then

• episodicMemory module must be invoked with --context episodicMemory/conf
• episodicMemory.ini must contain "path C:/iCub/app/episodicMemory"
• the directory C:/iCub/app/episodicMemory/episodicDatabase must exist.

Ports Accessed

None

Ports Created

Input ports

• /episodicMemory This port is used to change the parameters of the module at run time or stop the module.
  The following commands are available

help
quit
set thr <n> ... set the threshold for image recall (where <n> is a real number in the range 0-1)

Note that the name of this port mirrors whatever is provided by the --name parameter value The port is attached to the terminal so that you can type in commands and receive replies. The port can be used by other modules but also interactively by a user through the yarp rpc directive, viz.: yarp rpc /episodicMemory This opens a connection from a terminal to the port and allows the user to then type in commands and receive replies.

• /episodicMemory/image:i
• /episodicMemory/imageId:i
• /episodicMemory/action:i
• /episodicMemory/head:i
  This port needs to be connected to /icub/head/state:o to get the gaze values.

Output ports

• /episodicMemory
• /episodicMemory/recalledImage:o
• /episodicMemory/retrievedImage:o
• /episodicMemory/imageId:o

The functional specification only names the ports to be used to communicate with the module but doesn't say anything about the data transmitted on the ports. This is defined by the following code.

I/O Port Types & Naming

• BufferedPort<ImageOf<PixelRgb> > imageInPort;
• BufferedPort<VectorOf<double> > imageIdInPort; // image_id
• BufferedPort<VectorOf<double> > actionInPort; // azimuth, elevation 'a' 0 0
• BufferedPort<ImageOf<PixelRgb> > recalledImageOutPort;
• BufferedPort<ImageOf<PixelRgb> > retrievedImageOutPort;
• BufferedPort<VectorOf<double> > imageIdOutPort; // image_id, match_value, azimuth, elevation, previous_image_id, match_value, azimuth, elevation
• BufferedPort<Vector> headPort;

Note that the format of the actionInPort is exactly that defined by the /pos port of the controlGaze2 module.

Input Data Files

episodicDatabase (see above)

Output Data Files

None

Configuration Files

episodicMemory.ini (see above)

Tested OS

Linux and Windows

Example Instantiation of the Module

episodicMemory --context episodicMemory/conf --from episodicMemory.ini

Author:

David Vernon (based on code written by Alberto Bietti, Logan Niehaus, and Gionvanni Saponaro for the autoAssociativeMemory module)

Copyright (C) 2009 RobotCub Consortium

CopyPolicy: Released under the terms of the GNU GPL v2.0.

This file can be edited at src/episodicMemory/include/iCub/episodicMemoryModule.h