Next: Specifying Weights of Output Signals, Previous: Generating an Optimal Action, Up: Adaptive Probabilistic Mapping [Contents][Index]

Efficient operation of a small or large actor, i.e. optimality of actions the actor produces, depends on the form of a function that returns the relative probability of output signal choice. It may well be true that, for different kinds of applications, different functions are more suitable. The QSMM package provides a few built-in functions the programmer may select for use by an actor. Unfortunately, the author of the package experiences problems with: (1) providing a sound theoretical basis for using those functions in the package (that basis might not exist for some functions); (2) devising functions that give better efficiency than the functions currently provided in the package. The author’s own experimenting had shown that, for some special cases, there do exist more effective functions, but he did fail to find general forms for those functions, making it possible to utilize them for wider ranges of values of actor parameters. To provide a way to solve the above problems by developers, in QSMM starting from version 1.15, a possibility to specify via the Actor API a custom relative probability function for use by an actor is supported.

Use the following functions to get or set the type of a relative probability function used by an actor.

- Function:
*enum qsmm_relprob_e***qsmm_get_actor_relprob_type***(qsmm_actor_t*`actor`) This function returns the type of a function used by an actor to calculate the relative probability of output signal choice. Argument

`actor`specifies the handle of the actor. If the actor is the large one, then the type of a function used by a small actor associated with the large actor will be returned.

- Function:
*void***qsmm_set_actor_relprob_type***(qsmm_actor_t*`actor`, enum qsmm_relprob_e`relprob_type`) This function sets the type of a function used by an actor to calculate the relative probability of output signal choice. Argument

`relprob_type`specifies that type. Argument`actor`specifies the handle of the actor. If the actor is the large one, then the type of a function used by a small actor associated with the large actor will be set. When the value of`relprob_type`does not match any valid element of enumeration`qsmm_relprob_e`

, behavior of the actor is undefined.

Below is a description of the enumeration that specifies supported relative probability function types.

- Enumeration:
**qsmm_relprob_e** -
This enumeration specifies supported types of a function used by an actor to calculate the relative probability of output signal choice.

The following notations are used in the description of elements of this enumeration.

*h*An action choice state.

*z*An output signal.

*n*_{e}The number of spur types.

*T*Actor temperature multiplied by some constant.

*F(h,z)*The relative probability of emitting output signal

*z*in action choice state*h*.*L*-
The mean discrete time period between an occurrence of action choice state

*h*, when output signal*z*was emitted, and the subsequent occurrence of*h*, multiplied by a value returned by the function`qsmm_get_actor_naction_per_evt`

. This discrete time period (with the foregoing multiplication applied) is called a*discrete cycle period*for short. *n*_{out}The mean number of output signals of the actor (for more information on computing this number, see Other Parameters of an Actor).

*W[i]*Spur weight for spur type

*i*.*C(h,z,i)*The ratio of spur increment velocities. It is computed depending on the way of spur perception (see further on in this section) for spur type

*i*.

The enumeration contains the following elements.

`QSMM_RELPROB_BUILTIN1`

The relative probability function is defined by the formula

where

*κ=2L*. The paper “An Approach to Optimal Action Generation for a System that Interacts with the Environment” available from the project homepage provides more details pertaining to the formula.`QSMM_RELPROB_BUILTIN2`

The relative probability function is defined by the formula

The author has empirically found this formula when developing version 1.15 of QSMM package and making an actor generate output signals with frequencies proportional to probabilities of those output signals. The probabilities had been specified a priori, and spur increments supplied to the actor just after emitting corresponding output signals were equal to logarithms of those probabilities.

`QSMM_RELPROB_BUILTIN3`

[New in QSMM 1.16] The relative probability function is defined by the formula

This formula is a simplification of the formula that corresponds to relative probability function type

`QSMM_RELPROB_BUILTIN2`

and is meant for developers who prefer beauty and/or simplicity.`QSMM_RELPROB_USER1`

The relative probability function is defined by the formula

where

*f(L)*is a function of type`qsmm_relprob_user1_func_t`

supplied by a programmer via a call to`qsmm_set_actor_relprob_helper`

(see further on in this section). If function*f(L)*is not supplied, then 1 will be used for it.`QSMM_RELPROB_USER2`

[New in QSMM 1.16] The relative probability function is defined by the formula

where

*g*is a function of type`qsmm_relprob_user2_func_t`

supplied by a programmer via a call to`qsmm_set_actor_relprob_helper`

(see further on in this section). Function*g*receives the value of*z*and statistics associated with*h*and*z*. These function arguments may be necessary to devise a sophisticated relative probability function.

When creating a small actor, the function `qsmm_actor_create`

initializes the type of the relative probability function of the small actor to `QSMM_RELPROB_BUILTIN1`

.
When creating a large actor, the function `qsmm_actor_create`

initializes the type of the relative probability function of a small actor associated with the large actor to `QSMM_RELPROB_BUILTIN2`

.

Below are descriptions of types for functions *f(L)* and *g*.

- Data type:
**qsmm_relprob_user1_func_t** This is a type for the pointer to a helper function that can be supplied by a programmer for relative probability function type

`QSMM_RELPROB_USER1`

. The following declaration corresponds to this type:typedef double (*qsmm_relprob_user1_func_t)( qsmm_actor_t

`actor`, double`cycle_period`, void *`paramp`);Argument

`actor`holds the handle of an actor that has called the helper function. Argument`cycle_period`holds the value of*L*. Argument`paramp`holds a user parameter specified when setting the helper function for the actor. The helper function should return the value of*f(L)*.

- Data type:
**qsmm_relprob_user2_func_t** This is a type for the pointer to a helper function that should be supplied by a programmer for relative probability function type

`QSMM_RELPROB_USER2`

. The following declaration corresponds to this type:typedef double (*qsmm_relprob_user2_func_t)( qsmm_actor_t

`actor`, qsmm_sig_t`sig_cycle`, const struct qsmm_state_s *`state_p`, const struct qsmm_cycle_s *`cycle_p`, const struct qsmm_sspur_s *`sspur_p`, const struct qsmm_cspur_s *`cspur_p`, void *`paramp`);Argument

`actor`holds the handle of an actor that has called the helper function. Argument`sig_cycle`holds the value of*z*. Arguments`state_p`,`cycle_p`,`sspur_p`, and`cspur_p`hold statistics associated with*h*and*z*. See Structures for Accessing Storage, for a description of the corresponding structures. Argument`paramp`holds a user parameter specified when setting the helper function for the actor. The helper function should return the value of*g*.

To obtain working parameters of an actor, in the body of a helper function such functions as `qsmm_get_actor_nsig_ctrl`

and `qsmm_get_actor_discrete_cycle_period_mean`

may be called.
See Other Parameters of an Actor, for a detailed description of these functions.
If the need exists to use application-specific information associated with *h*, then the n-gram of *h* can be retrieved by a call to `qsmm_get_actor_sig_ngram`

during execution of the helper function.

Use the following functions to get or set the helper function for an actor.

- Function:
*void***qsmm_get_actor_relprob_helper***(qsmm_actor_t*`actor`, void **`helper_func_pp`, void **`helper_func_param_pp`) This function retrieves information on a helper function used by an actor when calculating the relative probability of output signal choice. Argument

`actor`specifies the handle of the actor. If there is such helper function set for the actor, and`helper_func_pp`is not`NULL`

, then the function will set *`helper_func_pp`equal to the pointer to the helper function. If there is no such helper function set for the actor, and`helper_func_pp`is not`NULL`

, then the function will set *`helper_func_pp`equal to`NULL`

. If`helper_func_param_pp`is not`NULL`

, then the function will set *`helper_func_param_pp`equal to the user parameter of this helper function specified when setting it for the actor. If the actor is the large one, the function retrieves information on a helper function used by a small actor associated with the large actor. A particular interpretation of the pointer stored in *`helper_func_pp`depends on a relative probability function type set for the actor by a call to`qsmm_set_actor_relprob_type`

.

- Function:
*void***qsmm_set_actor_relprob_helper***(qsmm_actor_t*`actor`, void *`helper_func_p`, void *`helper_func_param_p`) This function sets the pointer to a helper function used by an actor when calculating the relative probability of output signal choice. Argument

`actor`specifies the handle of the actor. Argument`helper_func_p`specifies the pointer to the helper function. Additionally, the function sets the user parameter of that helper function to`helper_func_param_p`. If the actor is the large one, this function sets the helper function for a small actor associated with the large actor.A particular interpretation of pointer

`helper_func_p`depends on a relative probability function type set for the actor by a call to`qsmm_set_actor_relprob_type`

. Providing a pointer that does not conform with the relative probability function type will lead to undefined behavior of the library.The value of

`helper_func_p`can be`NULL`

. Such`NULL`

value means that there is no helper function set for the actor. Setting the`NULL`

value makes a certain sense when the relative probability function type is`QSMM_RELPROB_USER1`

.

To provide an example of using the functions `qsmm_set_actor_relprob_type`

and `qsmm_set_actor_relprob_helper`

, let us suppose a developer has proved that it is mathematically correct to use the function for calculating the relative probability of output signal choice like the function `QSMM_RELPROB_BUILTIN2`

but with the following one difference: the numerator of the exponent contains the expression *L* instead of expression *L+1*.
To make an actor calculate the relative probability of output signal choice using this function, the developer may define the helper function such as:

#include <math.h> static double get_relprob_exp_mlt( qsmm_actor_t actor, double cycle_period, void *paramp ) { return log(qsmm_get_actor_nsig_ctrl(actor))*cycle_period/2; }

To set this helper function for use by an actor, the developer may write these function calls:

qsmm_set_actor_relprob_type(actor,QSMM_RELPROB_USER1); qsmm_set_actor_relprob_helper(actor,&get_relprob_exp_mlt,0);

Alternatively, since QSMM version 1.16, to use that relative probability function, the developer may write the function calls shown below.
A description of function `qsmm_set_actor_ktemperature`

will be given later in this section.

qsmm_set_actor_relprob_type(actor,QSMM_RELPROB_BUILTIN3); qsmm_set_actor_ktemperature(actor,2);

Most of the relative probability function formulas given earlier in this section contain the function *C(h,z,i)* equal to the ratio of spur increment velocities.
The method of computing *C(h,z,i)* depends on the way of spur perception for spur type *i*.
The way of spur perception determines whether that ratio will be generally positive or negative for generally positive or negative spur increments supplied to the actor.
Two ways of spur perception are supported: normal and inverse.
The normal way of spur perception provides generally positive ratio for generally positive spur increments and generally negative ratio for generally negative spur increments.
The inverse way of spur perception provides generally negative ratio for generally positive spur increments and generally positive ratio for generally negative spur increments.

The following enumeration specifies the way of perception of the spur of a particular type.

- Enumeration:
**qsmm_spur_perception_e** This enumeration specifies the way of spur perception for a spur type.

Below are notations used in the description of elements of this enumeration. In the description of notations, a cycle of type

*<h,z>*is the event history segment between an occurrence of action choice state*h*, when output signal*z*was emitted, and the subsequent occurrence of*h*. When referring to a particular cycle type, all cycles of that type are meant.*t*Discrete or continuous time used to compute spur increment velocity.

*E[i]*The value of the spur of type

*i*the actor currently accumulated.*H*^{(h,z)}[i]The sum of increments of the spur of type

*i*over cycle type*<h,z>*since the beginning of actor operation.*ω*^{(h,z)}The sum of periods of discrete or continuous time over cycle type

*<h,z>*since the beginning of actor operation.

The enumeration contains the following elements.

`QSMM_SPUR_PERCEPTION_NORMAL`

The value of

*C(h,z,i)*is the ratio of the spur increment velocity of a cycle type to the absolute value of the mean spur increment velocity:When spur increments over the event history are non-negative, the value of

*C(h,z,i)*increases as spur increment velocity increases. When the spur increment velocity of a cycle type is positive and is equal to the mean spur increment velocity, the value of*C(h,z,i)*is equal to 1.`QSMM_SPUR_PERCEPTION_INVERSE`

The value of

*C(h,z,i)*is the negative ratio of the absolute value of the mean spur increment velocity to the spur increment velocity of a cycle type:When spur increments over the event history are negative, the value of

*C(h,z,i)*increases as spur increment velocity increases. When the spur increment velocity of a cycle type is negative and is equal to the mean spur increment velocity, the value of*C(h,z,i)*is equal to 1.The inverse way of spur perception can be used to countervail the negative spur with the other negative spur perceived as the positive one because of the inverse way of perception specified.

Use the following functions to get or set the way of spur perception for a spur type.

- Function:
*int***qsmm_get_actor_spur_perception***(qsmm_actor_t*`actor`, int`spur_type`, enum qsmm_spur_perception_e *`spur_perception_p`) - Function:
*int***qsmm_set_actor_spur_perception***(qsmm_actor_t*`actor`, int`spur_type`, enum qsmm_spur_perception_e`spur_perception`) If

`spur_perception_p`is not`NULL`

, the function`qsmm_get_actor_spur_perception`

sets *`spur_perception_p`equal to the way of spur perception by an actor. The function`qsmm_set_actor_spur_perception`

sets the way of spur perception by an actor to`spur_perception`. When`spur_perception`does not match any valid element of enumeration`qsmm_spur_perception_e`

, behavior of the actor is undefined.Argument

`actor`specifies the handle of the actor. Argument`spur_type`specifies the spur type and cannot be less than -1 or greater than or equal to the number of spur types returned by the function`qsmm_get_actor_nspur`

for the actor. If the actor is the small one, specifying spur type -1 is incorrect.If the actor is the large one, the functions get or set the way of spur perception for a corresponding spur type of a small actor associated with the large actor. Special spur type -1 of the large actor corresponds to the automatic spur of the small actor if this small actor uses the automatic spur.

On success, the functions return a non-negative value. On invalid

`spur_type`, the functions return negative error code`QSMM_ERR_INVAL`

. The value of`spur_type`is invalid in the following situations:- – the actor is the small one and
`spur_type`is negative; - – the actor is the large one, a small actor associated with it does not use the automatic spur, and
`spur_type`is negative; - – the actor is the large one, a small actor associated with it uses the automatic spur, and
`spur_type`is less than -1; - – the value of
`spur_type`is greater than or equal to the number of spur types returned by the function`qsmm_get_actor_nspur`

for the actor.

- – the actor is the small one and

The function `qsmm_actor_create`

initializes ways of spur perception for all spur types of a newly created actor to `QSMM_SPUR_PERCEPTION_NORMAL`

.

Use the following functions to get or set spur weight denoted by *W[i]* in the relative probability function formulas mentioned earlier in this section.

- Function:
*int***qsmm_get_actor_spur_weight***(qsmm_actor_t*`actor`, int`spur_type`, double *`weight_p`) - Function:
*int***qsmm_set_actor_spur_weight***(qsmm_actor_t*`actor`, int`spur_type`, double`weight`) If

`weight_p`is not`NULL`

, the function`qsmm_get_actor_spur_weight`

sets *`weight_p`equal to the weight of a spur type of an actor. The value of *`weight_p`obtained in this way will always be finite. The function`qsmm_set_actor_spur_weight`

sets the weight of a spur type of an actor to`weight`.Argument

`actor`specifies the handle of the actor. Argument`spur_type`specifies the spur type and cannot be less than -1 or greater than or equal to the number of spur types returned by the function`qsmm_get_actor_nspur`

for the actor. If the actor is the small one, specifying spur type -1 is incorrect.If the actor is the large one, the functions get or set the weight of a corresponding spur type of a small actor associated with the large actor. Special spur type -1 of the large actor corresponds to the automatic spur of the small actor if this small actor uses the automatic spur.

On success, the functions return a non-negative value. On failure, the functions return negative error code

`QSMM_ERR_INVAL`

. The following situations are failures:- – the value of
`weight`is not finite (is applicable only to the function`qsmm_set_actor_spur_weight`

); - – the actor is the small one and
`spur_type`is negative; - – the actor is the large one, a small actor associated with it does not use the automatic spur, and
`spur_type`is negative; - – the actor is the large one, a small actor associated with it uses the automatic spur, and
`spur_type`is less than -1; - – the value of
`spur_type`is greater than or equal to the number of spur types returned by the function`qsmm_get_actor_nspur`

for the actor.

- – the value of

The function `qsmm_actor_create`

initializes weights of all spur types of a newly created actor to 1.

In the relative probability function formulas mentioned earlier in this section, *T* denotes actor temperature multiplied by some constant.
Actor temperature is a concept that follows from the use of exponential forms for the relative probability function resembling formulas related to the Boltzmann distribution.
In most of the relative probability function formulas, actor temperature is a divisor of every spur weight.

The perfect relative probability function is the one that provides maximum operating optimality of an actor without the need to tune up its temperature by hand. That is, for such function, default temperature equal to 1 can always be used. If the function is imperfect, its application is not intended one, or when applying the simulated annealing approach for solving a problem, use the following API functions to adjust the temperature of an actor for increasing its operating optimality.

- Function:
*double***qsmm_get_actor_ktemperature***(qsmm_actor_t*`actor`) This function returns the temperature (multiplied by some constant) of an actor. Argument

`actor`specifies the handle of the actor. If the actor is the large one, this function returns the temperature of a small actor associated with the large actor. The returned value is always finite and positive.

- Function:
*int***qsmm_set_actor_ktemperature***(qsmm_actor_t*`actor`, double`val`) This function sets the temperature (multiplied by some constant) of an actor to

`val`. Argument`actor`specifies the handle of the actor. If the actor is the large one, this function sets the temperature of a small actor associated with the large actor.On success, the function returns a non-negative value. If

`val`is not a finite number or is not a positive number, then negative error code`QSMM_ERR_INVAL`

will be returned.

The function `qsmm_actor_create`

initializes the temperature (multiplied by some constant) of a newly created actor to 1.