Please enable JavaScript.
Coggle requires JavaScript to display documents.
DL(19) - Autoencoders - Coggle Diagram
DL(19) - Autoencoders
aims to learn a compressed code
h
for the input x, learning the identity function
x
= g(f(
x
)), with constraint:
on the architecture of the net (
undercomplete autoencoder
)
adding a regularizing term to the loss (
overcomplete autoencoder
)
Undercomplete autoencoders
f(∙), g(∙) has low capacity (linear)
must discard some information in
h
h
has lower dimension than
x
Overcomplete autoencoders
h
has higher dimension than
x
must be regularized
Singolar Value Decomposition
compute the
SVD
of
Ξ
,
Ξ
=
V S Uᵀ
S
(n x m) : diagonal matrix (singolar values)
Uᵀ
(m x m) : orthonormal matrix (columns basis for columns of
Ξ
V
(n x n) : orthonormal matrix (columns basis for rows of
Ξ
reduced representation of
xᵢ
=
V S Uᵀ U
=
VS
collect data
xᵢ
as rows in a matrix
Ξ
Overcomplete Autoencoders
Denoising Autoencoder
the output has to be the original input without any noise/corruption
L
= - log p(
x
|
h
=
f(x')
)
there is a
C
(
x'
|
x
) corruption procer
we want to remove the noise, learn a way to project the corrupted version in its original position
Contractive Autoencoders
Ω
(
h
) = λ || ∂ f(
x
) / ∂
x
||²
practical issues
heavy computation of Jacobian for deep networks: build one layer at time
scale of decoder should be the same as encoder
Sparse Autoencoders
L(
x
, g(f(
x
) ) +
Ω
(
h
)
add a term to the cost function penalizing the entries
h
= f(
x
) for being larger
Autoencoders with Dropout on the hidden layer
Autoencoders for Sequences
Linear autoencoders for sequences
o
(t) =
C h
(t)
H
= [
h
(1)ᵀ, ...,
h
(n)ᵀ] = [
( Ax
(1)) ᵀ, (
Ax
(2) +
BAx
(1)ᵀ ), decomposed in :
Ξ
= [ [
x
(1)ᵀ, 0, ..., 0], [
x
(2)ᵀ,
x
(1)ᵀ, 0, ..., 0] , ..., [
x
(n)ᵀ, ...,
x
(1)ᵀ]
Ω
= [
A
ᵀ,
A
ᵀ
B
ᵀ ,
A
ᵀ
B
ᵀ², ...,
A
ᵀ
B
n-1ᵀ]
with
Ω
=
U
, the
space representation
is
VS
matrices
A
and
B
satisfying
Ω
=
U
do exist:
A
ᵀ =
U
₁ and
B
ᵀ =
Q
h
(t) =
A x
(t) +
B h
(t-1)