-1:
numpy必须为1.20.0，否则会报错，版本冲突0.rlvalue-based: 如q-learning（走迷宫），对当前状态下作出的动作进行价值计算，通过贪婪策略穷尽所有可能选择最佳state-action,但是对于连续的动作空间，动作的值是无穷的，把他们离散化，会维度爆炸，MSEpolicy-based: 比如移动机器人在一个室内环境中导航，策略网络以机器人当前的传感器信息（如激光雷达数据、摄像头图像）作为输入，输出动作的概率分布（对于连续动作）。通过策略梯度算法，根据机器人是否成功到达目标位置以及所花费的时间等奖励信号来更新策略网络的参数。训练过程可能会受到局部最优解的影响，并且策略梯度的估计可能存在较大方差，导致训练不稳定LOSS:负号是因为我们要使用梯度下降优化器，而策略梯度的目标是进行梯度上升。--PPO 是一种改进的策略梯度算法，旨在提高训练的稳定性和样本效率。它通过限制策略更新的步长，避免更新幅度过大导致性能下降。1.代码解析
1.0 on_policy_runner.py: class OnPolicyRunner.learntot_iter = self.current_learning_iteration + num_learning_iterationsfor it in range(self.current_learning_iteration, tot_iter):start = time.time()with torch.inference_mode():for i in range(self.num_steps_per_env):actions = self.alg.act(obs, critic_obs) # 2048，8，84(64+19)-->transformer 预测动作分布，再采样出2048*19个关节的动作, 84是观测值：观测值的计算综合了机器人的多种状态信息，如姿态、角速度、指令、关节位置和速度、动作等，并且可以根据配置添加感知输入和噪声。obs, privileged_obs, rewards, dones, infos = self.env.step(actions) #仿真环境中执行上面的动作,action--》compute_torques;compute observations,rewards, resetscritic_obs = privileged_obs if privileged_obs is not None else obsobs, critic_obs, rewards, dones = obs.to(self.device), critic_obs.to(self.device), rewards.to(self.device), dones.to(self.device)self.alg.process_env_step(rewards, dones, infos) # 对环境交互一个时间步结果的处理，包括奖励和终止信号的保存、超时情况的处理、转换信息的记录以及智能体的重置，为后续的学习和决策提供了基础。1.1 HST
--class H1(): legged_gym/env/h1/h1.py----init: self._super_init-->self.create_sim-->self._create_envs----step----post_physics_step----reset----compute_reward----compute_observations----create_sim----_compute_torques----_create_envs----render
--class H1RoughCfg( BaseConfig ): legged_gym/env/h1/h1_config.py----class human:----class env:----class terrain:----class commands:----class init_state:----class control:----class asset:----class domain_rand:----class rewards:----class noise:----class sim:
--class H1RoughCfgPPO(BaseConfig):----class policy:----class algorithm:----class runner:2.网络结构
1.ActorNET = Transformer + mlp
Actor MLP: Transformer((input_layer): Sequential((0): Linear(in_features=84, out_features=128, bias=True)(1): Dropout(p=0.1, inplace=False))(weight_pos_embed): Embedding(8, 128)(attention_blocks): Sequential((0): Transformer_Block((ln_1): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(attn): MultiheadAttention((out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True))(ln_2): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(mlp): Sequential((0): Linear(in_features=128, out_features=512, bias=True)(1): GELU()(2): Linear(in_features=512, out_features=128, bias=True)(3): Dropout(p=0.1, inplace=False)))(1): Transformer_Block((ln_1): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(attn): MultiheadAttention((out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True))(ln_2): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(mlp): Sequential((0): Linear(in_features=128, out_features=512, bias=True)(1): GELU()(2): Linear(in_features=512, out_features=128, bias=True)(3): Dropout(p=0.1, inplace=False)))(2): Transformer_Block((ln_1): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(attn): MultiheadAttention((out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True))(ln_2): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(mlp): Sequential((0): Linear(in_features=128, out_features=512, bias=True)(1): GELU()(2): Linear(in_features=512, out_features=128, bias=True)(3): Dropout(p=0.1, inplace=False)))(3): Transformer_Block((ln_1): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(attn): MultiheadAttention((out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True))(ln_2): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(mlp): Sequential((0): Linear(in_features=128, out_features=512, bias=True)(1): GELU()(2): Linear(in_features=512, out_features=128, bias=True)(3): Dropout(p=0.1, inplace=False))))(output_layer): Sequential((0): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(1): Linear(in_features=128, out_features=19, bias=True))
)==================================================================================================================
2.CriticNet = MLP + Transformer
Critic MLP: Transformer((input_layer): Sequential((0): Linear(in_features=84, out_features=128, bias=True)(1): Dropout(p=0.1, inplace=False))(weight_pos_embed): Embedding(8, 128)(attention_blocks): Sequential((0): Transformer_Block((ln_1): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(attn): MultiheadAttention((out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True))(ln_2): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(mlp): Sequential((0): Linear(in_features=128, out_features=512, bias=True)(1): GELU()(2): Linear(in_features=512, out_features=128, bias=True)(3): Dropout(p=0.1, inplace=False)))(1): Transformer_Block((ln_1): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(attn): MultiheadAttention((out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True))(ln_2): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(mlp): Sequential((0): Linear(in_features=128, out_features=512, bias=True)(1): GELU()(2): Linear(in_features=512, out_features=128, bias=True)(3): Dropout(p=0.1, inplace=False)))(2): Transformer_Block((ln_1): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(attn): MultiheadAttention((out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True))(ln_2): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(mlp): Sequential((0): Linear(in_features=128, out_features=512, bias=True)(1): GELU()(2): Linear(in_features=512, out_features=128, bias=True)(3): Dropout(p=0.1, inplace=False)))(3): Transformer_Block((ln_1): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(attn): MultiheadAttention((out_proj): NonDynamicallyQuantizableLinear(in_features=128, out_features=128, bias=True))(ln_2): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(mlp): Sequential((0): Linear(in_features=128, out_features=512, bias=True)(1): GELU()(2): Linear(in_features=512, out_features=128, bias=True)(3): Dropout(p=0.1, inplace=False))))(output_layer): Sequential((0): LayerNorm((128,), eps=1e-05, elementwise_affine=True)(1): Linear(in_features=128, out_features=1, bias=True))
)
