dl::optimization::AdamW< T > Class Template Reference

AdamW optimizer with autograd support. More...

#include <optimizers.hpp>

Inheritance diagram for dl::optimization::AdamW< T >:

Collaboration diagram for dl::optimization::AdamW< T >:

Public Member Functions
	AdamW (std::vector< Variable< T > * > parameters, T lr=1e-3, T beta1=0.9, T beta2=0.999, T eps=1e-8, T weight_decay=1e-2)
	Constructor.
void	step () override
	Perform one AdamW step.
T	get_lr () const override
	Get learning rate.
void	set_lr (T lr) override
	Set learning rate.
Public Member Functions inherited from dl::optimization::AutogradOptimizer< T >
	AutogradOptimizer (std::vector< Variable< T > * > parameters)
	Constructor.
virtual	~AutogradOptimizer ()=default
virtual void	zero_grad ()
	Zero gradients of all parameters.

Additional Inherited Members
Protected Attributes inherited from dl::optimization::AutogradOptimizer< T >
std::vector< Variable< T > * >	parameters_

Detailed Description

template<typename T>
class dl::optimization::AdamW< T >

AdamW optimizer with autograd support.

Adam with decoupled weight decay regularization. Often performs better than Adam with L2 regularization.

Paper: "Decoupled Weight Decay Regularization" (Loshchilov & Hutter, 2017)

Definition at line 188 of file optimizers.hpp.

Constructor & Destructor Documentation

AdamW()

template<typename T >

dl::optimization::AdamW< T >::AdamW	(	std::vector< Variable< T > * >	parameters,
		T	lr = 1e-3,
		T	beta1 = 0.9,
		T	beta2 = 0.999,
		T	eps = 1e-8,
		T	weight_decay = 1e-2
	)

Constructor.

Parameters

parameters	Parameters to optimize
lr	Learning rate (default: 1e-3)
beta1	Coefficient for first moment estimate (default: 0.9)
beta2	Coefficient for second moment estimate (default: 0.999)
eps	Term for numerical stability (default: 1e-8)
weight_decay	Weight decay coefficient (default: 1e-2)

Definition at line 89 of file optimizers.cpp.

Member Function Documentation

get_lr()

template<typename T >

T dl::optimization::AdamW< T >::get_lr ( ) const

inlineoverridevirtual

Get learning rate.

Implements dl::optimization::AutogradOptimizer< T >.

Definition at line 214 of file optimizers.hpp.

set_lr()

template<typename T >

void dl::optimization::AdamW< T >::set_lr ( T  lr )

inlineoverridevirtual

Set learning rate.

Implements dl::optimization::AutogradOptimizer< T >.

Definition at line 219 of file optimizers.hpp.

step()

template<typename T >

void dl::optimization::AdamW< T >::step ( )

overridevirtual

Perform one AdamW step.

Implements dl::optimization::AutogradOptimizer< T >.

Definition at line 115 of file optimizers.cpp.

---

The documentation for this class was generated from the following files:

• include/optimization/optimizers.hpp
• src/optimization/optimizers.cpp