wordpress 名站,棋牌网站开发工程师,简单静态网页制作代码,系统页面模板上篇文章主要介绍了hook钩子函数的大致使用流程#xff0c;本篇文章主要介绍pytorch中的hook机制register_forward_hook#xff0c;手动在forward之前注册hook#xff0c;hook在forward执行以后被自动执行。
1、hook背景 Hook被成为钩子机制#xff0c;pytorch中包含forwa…上篇文章主要介绍了hook钩子函数的大致使用流程本篇文章主要介绍pytorch中的hook机制register_forward_hook手动在forward之前注册hookhook在forward执行以后被自动执行。
1、hook背景 Hook被成为钩子机制pytorch中包含forward和backward两个钩子注册函数用于获取forward和backward中输入和输出按照自己不全面的理解应该目的是“不改变网络的定义代码也不需要在forward函数中return某个感兴趣层的输出这样代码太冗杂了”。
2、源码阅读 register_forward_hook()函数必须在forward函数调用之前被使用因为该函数源码注释显示这个函数“ it will not have effect on forward since this is called after :func:forward is called”也就是这个函数在forward之后就没有作用了 作用获取forward过程中每层的输入和输出用于对比hook是不是正确记录。
def register_forward_hook(self, hook):rRegisters a forward hook on the module.The hook will be called every time after :func:forward has computed an output.It should have the following signature::hook(module, input, output) - None or modified outputThe hook can modify the output. It can modify the input inplace butit will not have effect on forward since this is called after:func:forward is called.Returns::class:torch.utils.hooks.RemovableHandle:a handle that can be used to remove the added hook by callinghandle.remove()handle hooks.RemovableHandle(self._forward_hooks)self._forward_hooks[handle.id] hookreturn handle3、定义一个用于测试hook的类 如果随机的初始化每个层那么就无法测试出自己获取的输入输出是不是forward中的输入输出了所以需要将每一层的权重和偏置设置为可识别的值比如全部初始化为1。网络包含两层Linear有需要求导的参数被称为一个层而ReLU没有需要求导的参数不被称作一层init()中调用initialize函数对所有层进行初始化。
**注意**在forward()函数返回各个层的输出但是ReLU6没有返回因为后续测试的时候不对这一层进行注册hook。
class TestForHook(nn.Module):def __init__(self):super().__init__()self.linear_1 nn.Linear(in_features2, out_features2)self.linear_2 nn.Linear(in_features2, out_features1)self.relu nn.ReLU()self.relu6 nn.ReLU6()self.initialize()def forward(self, x):linear_1 self.linear_1(x)linear_2 self.linear_2(linear_1)relu self.relu(linear_2)relu_6 self.relu6(relu)layers_in (x, linear_1, linear_2)layers_out (linear_1, linear_2, relu)return relu_6, layers_in, layers_outdef initialize(self): 定义特殊的初始化用于验证是不是获取了权重self.linear_1.weight torch.nn.Parameter(torch.FloatTensor([[1, 1], [1, 1]]))self.linear_1.bias torch.nn.Parameter(torch.FloatTensor([1, 1]))self.linear_2.weight torch.nn.Parameter(torch.FloatTensor([[1, 1]]))self.linear_2.bias torch.nn.Parameter(torch.FloatTensor([1]))return True4、定义hook函数 hook函数是register_forward_hook()函数必须提供的参数首先定义几个容器用于记录 定义用于获取网络各层输入输出tensor的容器
# 同时定义module_name用于记录相应的module名字
module_name []
features_in_hook []
features_out_hook []
hook函数需要三个参数这三个参数是系统传给hook函数的自己不能修改这三个参数hook函数负责将获取的输入输出添加到feature列表中并提供相应的module名字。
def hook(module, fea_in, fea_out):print(hooker working)module_name.append(module.__class__)features_in_hook.append(fea_in)features_out_hook.append(fea_out)return None5、对需要的层注册hook 注册钩子必须在forward函数被执行之前也就是定义网络进行计算之前就要注册下面的代码对网络除去ReLU6以外的层都进行了注册也可以选定某些层进行注册 注册钩子可以对某些层单独进行
net TestForHook()
net_chilren net.children()
for child in net_chilren:if not isinstance(child, nn.ReLU6):child.register_forward_hook(hookhook)6、测试forward返回的特征和hook记录的是否一致 6.1 测试forward提供的输入输出特征
由于前面的forward函数返回了需要记录的特征这里可以直接测试
out, features_in_forward, features_out_forward net(x)
print(**5forward return features**5)
print(features_in_forward)
print(features_out_forward)
print(**5forward return features**5)输出如下
*****forward return features*****
(tensor([[0.1000, 0.1000],[0.1000, 0.1000]]), tensor([[1.2000, 1.2000],[1.2000, 1.2000]], grad_fnAddmmBackward), tensor([[3.4000],[3.4000]], grad_fnAddmmBackward))
(tensor([[1.2000, 1.2000],[1.2000, 1.2000]], grad_fnAddmmBackward), tensor([[3.4000],[3.4000]], grad_fnAddmmBackward), tensor([[3.4000],[3.4000]], grad_fnThresholdBackward0))
*****forward return features*****6.2 hook记录的输入特征和输出特征 hook通过list结构进行记录所以可以直接print。
测试features_in是否存储了输入
print(**5hook record features**5)
print(features_in_hook)
print(features_out_hook)
print(module_name)
print(**5hook record features**5)得到和forward一样的结果
*****hook record features*****
[(tensor([[0.1000, 0.1000],[0.1000, 0.1000]]),), (tensor([[1.2000, 1.2000],[1.2000, 1.2000]], grad_fnAddmmBackward),), (tensor([[3.4000],[3.4000]], grad_fnAddmmBackward),)]
[tensor([[1.2000, 1.2000],[1.2000, 1.2000]], grad_fnAddmmBackward), tensor([[3.4000],[3.4000]], grad_fnAddmmBackward), tensor([[3.4000],[3.4000]], grad_fnThresholdBackward0)]
[class torch.nn.modules.linear.Linear,
class torch.nn.modules.linear.Linear,class torch.nn.modules.activation.ReLU]
*****hook record features*****6.3 把hook记录的和forward做减法 如果害怕会有小数点后面的数值不一致或者数据类型的不匹配可以对hook记录的特征和forward记录的特征做减法 测试forward返回的feautes_in是不是和hook记录的一致
print(sub result)
for forward_return, hook_record in zip(features_in_forward, features_in_hook):print(forward_return-hook_record[0])得到的全部都是0说明hook没问题
sub result
tensor([[0., 0.],[0., 0.]])
tensor([[0., 0.],[0., 0.]], grad_fnSubBackward0)
tensor([[0.],[0.]], grad_fnSubBackward0)7、完整代码
import torch
import torch.nn as nnclass TestForHook(nn.Module):def __init__(self):super().__init__()self.linear_1 nn.Linear(in_features2, out_features2)self.linear_2 nn.Linear(in_features2, out_features1)self.relu nn.ReLU()self.relu6 nn.ReLU6()self.initialize()def forward(self, x):linear_1 self.linear_1(x)linear_2 self.linear_2(linear_1)relu self.relu(linear_2)relu_6 self.relu6(relu)layers_in (x, linear_1, linear_2)layers_out (linear_1, linear_2, relu)return relu_6, layers_in, layers_outdef initialize(self): 定义特殊的初始化用于验证是不是获取了权重self.linear_1.weight torch.nn.Parameter(torch.FloatTensor([[1, 1], [1, 1]]))self.linear_1.bias torch.nn.Parameter(torch.FloatTensor([1, 1]))self.linear_2.weight torch.nn.Parameter(torch.FloatTensor([[1, 1]]))self.linear_2.bias torch.nn.Parameter(torch.FloatTensor([1]))return True# 定义用于获取网络各层输入输出tensor的容器并定义module_name用于记录相应的module名字
module_name []
features_in_hook []
features_out_hook []# hook函数负责将获取的输入输出添加到feature列表中并提供相应的module名字
def hook(module, fea_in, fea_out):print(hooker working)module_name.append(module.__class__)features_in_hook.append(fea_in)features_out_hook.append(fea_out)return None# 定义全部是1的输入
x torch.FloatTensor([[0.1, 0.1], [0.1, 0.1]])# 注册钩子可以对某些层单独进行
net TestForHook()
net_chilren net.children()
for child in net_chilren:if not isinstance(child, nn.ReLU6):child.register_forward_hook(hookhook)# 测试网络输出
out, features_in_forward, features_out_forward net(x)
print(**5forward return features**5)
print(features_in_forward)
print(features_out_forward)
print(**5forward return features**5)# 测试features_in是不是存储了输入
print(**5hook record features**5)
print(features_in_hook)
print(features_out_hook)
print(module_name)
print(**5hook record features**5)# 测试forward返回的feautes_in是不是和hook记录的一致
print(sub result)
for forward_return, hook_record in zip(features_in_forward, features_in_hook):print(forward_return-hook_record[0])
