Please enable JavaScript.
Coggle requires JavaScript to display documents.
On a final note, the backpropagation variants described herein are by far…
On a final note, the backpropagation variants described herein are by far not the
only variants currently available.
The variants described here are only the ones which the author has implemented in the NetSim software
Another important class of variants are the second order methods, which use not only first order derivatives but also second order derivatives to guide the training process.
-
Adaptive Learning Rates
The method of adaptive learning rates has been implemented in the NetSim
neural network training software.
In addition, NetSim allows the user to mix the batching method, the momentum method, and the method of adaptive learning rates together to produce the most robust training combination for the particular problem being solved.
While the bookkeeping required to implement the combination of all three of these methods s—in
addition to pure backpropagation—is complex and will not be discussed here, the use of the software is straightforward
The user can specify the values of the four parameters which control the learning rate adaptation—(1) the initial learning rate, (2) the “recentness” factor, (3) the learning rate reduction factor, and (4) the learning
The final variant of backpropagation that will be discussed here is the method of
adaptive learning rates.
In the method of adaptive learning rates, the problems associated with correctly
choosing a network learning rate are eliminated.
It should be clear that because the learning rates are adapted during the training
process, the initial choice has little effect on the overall training process.
-
Now consider the opposite case in which changes of the same sign are
consistently made to a particular connection weight in the network
The use of increased learning rates can be critically important when the training
process must traverse an area of near-zero slope on the error surface.
When the method of adaptive learning rates is applied to this situation (see Figure 5.7), he learning rate is allowed to grow progressively larger as long as the slope is not identically zero.
In implementing the method of adaptive learning rates, an exponential averaging scheme is used to determine which direction each weight has been moving recently— where recently means the past few epochs.
-
Scaling Neural Network
Figure 6.10 ILLUSTRATES the basic layout of the scaling (magnitude) neural
networks USED in this research.
As the figure indicates, there ARE five input parameters
and a single output parameter for each of these networks.
Nine networks WERE CONSTRUCTED so that each combination of load type (Fz, Mx, My) and displacement type (Tz, Rx, Ry) WAS covered.
Therefore, the maximum magnitude displacements also HAD TO BE normalized
into the range [0,1] for neural network use.
This WAS ACCOMPLISHED by NORMALIZING all of the maximum displacement magnitudes with respect to the maximum values that OCURRED in the GSF=1.2 geometry case.
After normalization then, the maximum displacement magnitudes all FALL in the range [0,1].
Note that when using the networks—as opposed to when training them—the normalized maximum displacement magnitudes produced by the networks MUST BE SCALED from [0,1] to a true structural range.
The “overall” scaling factors (the maximum displacement magnitudes from the
GSF=1.2 case) used for purpose this are listed in Table 6.3 .
-
-
7.1 Introduction
One of the primary focal points of FEA research during the past few decades
has been the development of fast and efficient equation solvers.
This is because every finite element analysis requires that at least one set of simultaneous equations be solved as part of the analysis.
In cases such as nonlinear analysis or dynamic analysis, the equation solving step may have to be performed many times during the analysis.
Since the number of equations that must be solved will grow as the FEA model becomes more refined, the equation solving portion of an analysis can account for a significant portion of the total analysis time.
It is therefore desirable to create equation solvers which are as numerically efficient as possible.