Main Content

getMaxQValue

Obtain maximum estimated value over all possible actions from a Q-value function critic with discrete action space, given environment observations

所有的页面崩溃

Syntax

[maxQ,maxActionIndex] = getMaxQValue(qValueFcnObj,obs)
[maxQ,maxActionIndex,state] = getMaxQValue(___)

Description

example

[maxQ,maxActionIndex] = getMaxQValue(qValueFcnObj,obs)evaluates the discrete-action-space Q-value function criticqValueFcnObjand returns the maximum estimated value over all possible actionsmaxQ, with the corresponding action indexmaxActionIndex, given environment observationsobs.

[maxQ,maxActionIndex,state] = getMaxQValue(___)also returns the updated state ofqValueFcnObjwhen it contains a recurrent neural network.

Examples

collapse all

Open Live Script

Create an observation and action specification objects (or alternatively usegetObservationInfoandgetActionInfoto extract the specification objects from an environment. For this example, define the observation space as a continuous three-dimensional space, and the action space as a finite set consisting of three possible values (named -1, 0, and 1).

obsInfo = rlNumericSpec([3 1]); actInfo = rlFiniteSetSpec([-1 0 1]);

Create a custom basis function to approximate the Q-value function within the critic, and define an initial parameter vector.

myBasisFcn = @(myobs,myact) [...ones(4,1); myobs(:); myact; myobs(:).^2; myact.^2; sin(myobs(:)); sin(myact); cos(myobs(:)); cos(myact) ]; W0 = rand(20,1);

Create the critic.

critic = rlQValueFunction({myBasisFcn,W0},...obsInfo,actInfo);

UsegetMaxQValueto return the maximum value, among the possible actions, given a random observation. Also return the index corresponding to the action that maximizes the value.

[v,i] = getMaxQValue(critic,{rand(3,1)})
v = 9.0719
i = 3

Create a batch set of 64 random independent observations. The third dimension is the batch size, while the fourth is the sequence length for any recurrent neural network used by the critic (in this case not used).

batchobs = rand(3,1,64,1);

Obtain maximum values for all the observations.

bv = getMaxQValue(critic,{batchobs}); size(bv)
ans =1×21 64

选择相关系数最大值esponding to the 44th observation.

bv(44)
ans = 10.4138

Input Arguments

collapse all

Q-value function critic, specified as anrlQValueFunctionorrlVectorQValueFunctionobject.

Environment observations, specified as a cell array with as many elements as there are observation input channels. Each element ofobscontains an array of observations for a single observation input channel.

The dimensions of each element inobsareMO-by-LB-by-LS, where:

  • MOcorresponds to the dimensions of the associated observation input channel.

  • LBis the batch size. To specify a single observation, setLB= 1. To specify a batch of observations, specifyLB> 1. IfqValueFcnObjhas multiple observation input channels, thenLBmust be the same for all elements ofobs.

  • LSspecifies the sequence length for a recurrent neural network. IfqValueFcnObjdoes not use a recurrent neural network, thenLS= 1. IfqValueFcnObjhas multiple observation input channels, thenLSmust be the same for all elements ofobs.

LBandLSmust be the same for bothactandobs.

For more information on input and output formats for recurrent neural networks, see the Algorithms section oflstmLayer.

Output Arguments

collapse all

Maximum Q-value estimate across all possible discrete actions, returned as a 1-by-LB-by-LSarray, where:

  • LBis the batch size.

  • LSspecifies the sequence length for a recurrent neural network. IfqValueFcnObjdoes not use a recurrent neural network, thenLS= 1.

Action index corresponding to the maximum Q value, returned as a 1-by-LB-by-LSarray, where:

  • LBis the batch size.

  • LSspecifies the sequence length for a recurrent neural network. IfqValueFcnObjdoes not use a recurrent neural network, thenLS= 1.

Updated state ofqValueFcnObj, returned as a cell array. IfqValueFcnObjdoes not use a recurrent neural network, thenstateis an empty cell array.

You can set the state of the critic tostateusing thesetStatefunction. For example:

qValueFcnObj = setState(qValueFcnObj,state);

Version History

Introduced in R2020a

See Also

||

Topics