Next: , Previous: , Up: Adaptive Probabilistic Mapping   [Contents][Index]


2.7 Generating an Optimal Action

To calculate probabilities of emitting output signals for an action choice state, which is stored in an actor window that holds current n-gram from the event history, the following function can be used.

Function: int qsmm_actor_calc_action_prob (qsmm_actor_t actor, int rez1, qsmm_sig_t sig_beg, qsmm_sig_t sig_end, enum qsmm_prob_e prob_type)

This function fills the internal array of actor with probabilities of type prob_type calculated for an action choice state, which is stored in an actor window that holds current n-gram from the event history. Argument rez1 is reserved for future use and must be equal to 0.

For a small actor, the contents of the internal array that holds probabilities calculated can be used to stochastically choose an optimal output signal by the function qsmm_get_actor_sig_action. For a large actor, the function qsmm_actor_calc_action_prob operates much more slowly and should be used only when values of probabilities have to be obtained. Large actors support calling the function qsmm_get_actor_sig_action, which returns control quickly, without prior calling the function qsmm_actor_calc_action_prob.

Arguments sig_beg and sig_end can be both zero or may specify identifiers of the first signal (inclusive) and of the last signal (exclusive) of a signal segment for which the function should calculate probabilities. In the internal array that holds probabilities calculated, the probability of the first signal of that signal segment has offset sig_beg. Array elements within the signal segment, which do not correspond to output signals, are set to 0. If sig_end is 0, then the total number of actor signals will be used for the value of the last signal of the signal segment. The function will execute faster when the signal segment is shorter, so it is recommended to specify the segment as precisely as possible.

The function returns a non-negative value on success or a negative error code on failure. Currently, the following error codes can be returned.

QSMM_ERR_INVAL

The value of sig_beg is greater than or equal to a value (incremented by 1) of the last signal of the signal segment, or the value of sig_end is greater than the total number of signals of the actor, or the value of prob_type is invalid.

QSMM_ERR_NGRAM

The contents of an actor window, which holds current n-gram from the event history, correspond to an invalid action choice state. To determine the validity of an action choice state, it is checked for accordance with allowed ranges of signal identifiers specified using the field range_sig_p of structure qsmm_actor_desc_s when creating the actor.

QSMM_ERR_STORAGE

Statistics storage failure. This could leave a large actor in indeterminate state. If the large actor is in indeterminate state, then after removing a reason of the error, the operation can be repeated, and if the function succeeds, then the actor’s state will become determinate.

QSMM_ERR_NOMEM

There was not enough memory to perform the operation. This could leave a large actor in indeterminate state. If the large actor is in indeterminate state, then after removing a reason of the error, the operation can be repeated, and if the function succeeds, then the actor’s state will become determinate.

A type of probabilities the function qsmm_actor_calc_action_prob must calculate is specified using an enumeration described below. The enumeration is also used in several other cases.

Enumeration: qsmm_prob_e

This enumeration specifies a type of probabilities. It contains the following elements.

QSMM_PROB_AGGR

Aggregate probabilities based on learned and profile probabilities.

For a small actor, to calculate probabilities of type QSMM_PROB_AGGR, probabilities of types QSMM_PROB_LEARNED and QSMM_PROB_PROFILE (see below) are calculated, each learned probability is multiplied by a corresponding profile probability, and the resulting array of values is normalized. For a large actor, the above operation is performed by the small actor for all nodes of the Huffman tree.

For a small actor, if you do not use profile probabilities, then it will be faster to calculate probabilities of type QSMM_PROB_LEARNED, which are equal to probabilities of type QSMM_PROB_AGGR in this case.

For a large actor, probabilities of type QSMM_PROB_AGGR are ordinarily equal to probabilities of type QSMM_PROB_LEARNED, because profile probabilities typically follow only from a positional relationship of nodes of the Huffman tree (probabilities assigned to edges, which come from tree nodes to their child nodes, are usually equal), and those profile probabilities are taken into account when calculating probabilities of type QSMM_PROB_LEARNED.

QSMM_PROB_LEARNED

Learned probabilities, according to which optimal actions may be produced.

QSMM_PROB_PROFILE

For a small actor, profile probabilities specified a priori in statistics storage (see Structures for Accessing Storage, for information on how to pass profile probabilities to storage access functions to write them to storage). For a large actor, profile probabilities, which follow from the structure of a Huffman tree that corresponds to an action choice state.

QSMM_PROB_FQ

Probabilities proportional to observed frequencies of emitting output signals. Those frequencies are automatically incremented for output signals with identifiers passed to the function qsmm_actor_reg_sig_action, but the increment is actually performed by the function qsmm_actor_reg_sig_in when the same action choice state occurs again in the event history.

QSMM_PROB_COUNT

The last element of the enumeration equal to the number of supported types of probabilities.

To obtain a pointer to the internal array that holds probabilities of emitting output signals calculated by the function qsmm_actor_calc_action_prob, the following function can be used.

Function: double * qsmm_get_actor_choice_sig_prob (qsmm_actor_t actor)

This function returns the pointer to an internal array of an actor specified by handle actor. The internal array holds output signal emitting probabilities, possibly relative ones. Using that pointer, the array can be examined or modified by an application program. This function never returns NULL. The number of elements in the array is equal to the total number of signals of the actor returned by the function qsmm_get_actor_nsig. The array can be filled with probabilities of desired type by the function qsmm_actor_calc_action_prob.

For a small actor, the content of the array is used by the function qsmm_get_actor_sig_action to stochastically generate an output signal. This means that the array may be modified by an application program to affect such stochastic generation. After such a modification, the array may contain values of positive infinity or relative probabilities that do not sum up to 1. Conversely, for a large actor, modifying the content of the array by an application program does not affect the behavior of the actor.

The function qsmm_get_actor_choice_sig_prob locks the internal array of probabilities: when the internal array is represented by a sparse vector, the sparse vector is converted to an ordinary vector; after the conversion, the actor always uses ordinary vectors until the internal array of probabilities is released by the function qsmm_actor_choice_sig_prob_release. Forgetting to call the function qsmm_actor_choice_sig_prob_release after a call to qsmm_get_actor_choice_sig_prob may dramatically decrease actor performance.

To release a locked internal array of probabilities, the following function can be used.

Function: void qsmm_actor_choice_sig_prob_release (qsmm_actor_t actor)

This function releases an internal array of actor that holds (possibly relative) probabilities of emitting output signals, which could have been locked by the function qsmm_get_actor_choice_sig_prob. If the array is not locked, then the function will have no effect. After the release, a pointer that might have been previously obtained by the function qsmm_get_actor_choice_sig_prob must not be used.

When the internal array is locked or if is called for a large actor, the function qsmm_actor_calc_action_prob uses an ordinary vector for storing probabilities of emitting output signals, so elements of that ordinary vector can be accessed via a pointer returned by the function qsmm_get_actor_choice_sig_prob. When the internal array is released, the function qsmm_actor_calc_action_prob called for a small actor can use an ordinary or sparse vector for storing probabilities of emitting output signals depending on properties of the probability profile and on the value of field sparse_fill_max of structure qsmm_actor_desc_s specified when creating the actor.

There is a way to obtain the handle of an ordinary or sparse vector that holds probabilities of emitting output signals. It is a cheap operation that does not perform locking or converting a sparse vector to an ordinary vector. The vector is returned as is, in read-only mode. See Ordinary and Sparse Vectors, for information on how to access the elements of the returned vector.

Function: qsmm_vec_t qsmm_get_actor_choice_sig_prob_vec (qsmm_actor_t actor)

This function returns the handle of an ordinary or sparse vector of actor, which holds (possibly relative) probabilities of emitting output signals. This function never returns NULL. The vector can be filled with probabilities of desired type using the function qsmm_actor_calc_action_prob. For a small actor, the contents of the vector are used by the function qsmm_get_actor_sig_action to stochastically generate an output signal.

To stochastically generate an output signal, the following function can be used.

Function: int qsmm_get_actor_sig_action (qsmm_actor_t actor, int rez1, qsmm_sig_t sig_beg, qsmm_sig_t sig_end, qsmm_sig_t *sig_action_p)

If an actor is a small one, this function stochastically generates an output signal according to (possibly relative) probabilities stored in the internal array of the actor. That array can be filled with probabilities of desired type using the function qsmm_actor_calc_action_prob. A pointer to the array is returned by the function qsmm_get_actor_choice_sig_prob. A read-only view of the array, which can speed up accessing its contents, is returned by the function qsmm_get_actor_choice_sig_prob_vec.

If an actor is a large one, this function stochastically generates an optimal output signal without using the aforementioned internal array. A Huffman tree is used to provide fast choice of an output signal (the number of those signals can be very large) by the actor according to probabilities of type QSMM_PROB_AGGR, which can be computed by the function qsmm_actor_calc_action_prob. However, for large actors, there is no need to call that function before calling the function qsmm_get_actor_sig_action; the number of calls to the function qsmm_actor_calc_action_prob should be made as little as possible, because for those actors that function operates very slowly.

A generated output signal is stored in *sig_action_p if sig_action_p is not NULL. Argument rez1 is reserved for future use and must be equal to 0.

Arguments sig_beg and sig_end can be both zero or may specify identifiers of the first signal (inclusive) and of the last signal (exclusive) of a signal segment, to which the generated output signal should belong. However, for a large actor, the signal segment must include all output signals of the actor; see Specifying Weights of Output Signals, for information on how to choose an optimal output signal from different subsets of output signals of a large actor. In the internal array of small actor, a probability of the first signal of that signal segment has offset sig_beg. If sig_end is 0, then the total number of actor signals will be used for the value of the last signal of the signal segment. The function will execute faster when the signal segment is shorter, so it is recommended to specify the segment as precisely as possible. For a small actor, the function does not check whether a signal it generates actually belongs to a set of output signals of the actor: if an element of the internal array within the signal segment has a positive value, then a corresponding signal can be potentially returned by the function. If at least one relative probability in the internal array within the signal segment has the value of positive infinity, then, for a small actor, the function will choose an output signal as a uniformly distributed random element from the set of signals to which such infinite relative probabilities correspond.

To stochastically generate an output signal, the actor uses either a random number generator supplied via the field rng of structure qsmm_actor_desc_s when creating the actor by the function qsmm_actor_create or an instance of default random number generator allocated automatically if 0 is specified in that field. The handle of a random number generator used by an actor can be obtained via the function qsmm_get_actor_rng, e.g. to seed the generator.

The function returns a non-negative value on success or a negative error code on failure. Currently, the following error codes can be returned.

QSMM_ERR_INVAL

One of the following conditions is met:

  • – the value of sig_beg is greater than or equal to a value (incremented by 1) of the last signal of the signal segment;
  • – the value of sig_end is greater than the total number of signals of the actor;
  • – the actor is the large one, and the value of sig_beg is greater than the lowest output signal identifier of the actor;
  • – the actor is the large one, and the value of sig_end is less than the highest output signal identifier of the actor plus 1;
  • – the actor is the small one and the internal array does not contain (within the signal segment) relative probabilities equal to positive infinity, but the sum of relative probabilities within the signal segment is not finite.
QSMM_ERR_WEIGHT

The actor is the small one, and a negative number or a non-finite number, which is not a positive infinity, was encountered in the internal array within the signal segment.

QSMM_ERR_NOCHOICE

The actor is the small one, and the internal array does not contain at least one positive element within the signal segment.

QSMM_ERR_NGRAM

The actor is the large one, and the contents of an actor window, which holds current n-gram from the event history, correspond to an invalid action choice state. To determine the validity of an action choice state, it is checked for accordance with allowed ranges of signal identifiers specified using the field range_sig_p of structure qsmm_actor_desc_s when creating the actor.

QSMM_ERR_STORAGE

Failure of statistics storage of a large actor. This could leave the large actor in indeterminate state.

QSMM_ERR_NOMEM

There was not enough memory to perform the operation by a large actor. This could leave the large actor in indeterminate state.

For a small actor, an optimal action is typically produced and registered using a sequence of function calls like this:

qsmm_sig_t sig_action=QSMM_SIG_INVALID;
if (qsmm_actor_calc_action_prob(actor, 0, NSIG_IN, NSIG_IN+NSIG_OUT,
                                QSMM_PROB_LEARNED)<0)
    goto Error;
if (qsmm_get_actor_sig_action(actor, 0, NSIG_IN, NSIG_IN+NSIG_OUT,
                              &sig_action)<0)
    goto Error;
if (qsmm_actor_reg_sig_action(actor,sig_action)<0) goto Error;

However, if, in a particular case, the application program chooses on its own which output signal to emit, then calls to qsmm_actor_calc_action_prob and qsmm_get_actor_sig_action can be omitted, but the small actor should be still notified of an output signal actually emitted using a call to qsmm_actor_reg_sig_action.

For a large actor, to produce and register an optimal action, a call to qsmm_actor_calc_action_prob is redundant and only decreases performance. The sequence of function calls looks like this:

qsmm_sig_t sig_action=QSMM_SIG_INVALID;
if (qsmm_get_actor_sig_action(actor, 0, NSIG_IN, NSIG_IN+NSIG_OUT,
                              &sig_action)<0)
    goto Error;
if (qsmm_actor_reg_sig_action(actor,sig_action)<0) goto Error;

Large actors do not support choosing an output signal by an application program. That is, for a large actor, it is incorrect to omit the call to qsmm_get_actor_sig_action before calling qsmm_actor_reg_sig_action, or to pass to the latter function a signal identifier other than that, which was returned by a preceding call to qsmm_get_actor_sig_action.

In QSMM version 1.16 there is introduced a function, which retrieves a probability of the last output signal generated. The function returns control quickly even when called for a large actor and can be used to calculate the time of emitting an output signal, e.g. according to the exponential probability distribution.

Function: double qsmm_get_actor_prob_action (qsmm_actor_t actor)

This function returns the probability of the last output signal generated by actor in the function qsmm_get_actor_sig_action. For a small actor, it is a probability that corresponds to the element of an internal array of the actor, which holds relative probabilities of emitting output signals. For a large actor, it is the probability of a path from the root of the Huffman tree to its leaf that corresponds to the output signal. The returned value is always in the range 0 to 1 (inclusive). If the function qsmm_get_actor_sig_action is not yet called, then 0 will be returned.


Next: , Previous: , Up: Adaptive Probabilistic Mapping   [Contents][Index]