An Introduction To Optimization (with Applications To Machine Learning)

Accessible introductory textbook on optimization theory and methods, with an emphasis on engineering design, featuring MATLAB^® exercises and worked examples

Fully updated to reflect modern developments in the field, the Fifth Edition of An Introduction to Optimization fills the need for an accessible, yet rigorous, introduction to optimization theory and methods, featuring innovative coverage and a straightforward approach. The book begins with a review of basic definitions and notations while also providing the related fundamental background of linear algebra, geometry, and calculus.

With this foundation, the authors explore the essential topics of unconstrained optimization problems, linear programming problems, and nonlinear constrained optimization. In addition, the book includes an introduction to artificial neural networks, convex optimization, multi-objective optimization, and applications of optimization in machine learning.

Numerous diagrams and figures found throughout the book complement the written presentation of key concepts, and each chapter is followed by MATLAB^® exercises and practice problems that reinforce the discussed theory and algorithms.

The Fifth Edition features a new chapter on Lagrangian (nonlinear) duality, expanded coverage on matrix games, projected gradient algorithms, machine learning, and numerous new exercises at the end of each chapter.

An Introduction to Optimization includes information on:

• The mathematical definitions, notations, and relations from linear algebra, geometry, and calculus used in optimization
• Optimization algorithms, covering one-dimensional search, randomized search, and gradient, Newton, conjugate direction, and quasi-Newton methods
• Linear programming methods, covering the simplex algorithm, interior point methods, and duality
• Nonlinear constrained optimization, covering theory and algorithms, convex optimization, and Lagrangian duality
• Applications of optimization in machine learning, including neural network training, classification, stochastic gradient descent, linear regression, logistic regression, support vector machines, and clustering.

An Introduction to Optimization is an ideal textbook for a one- or two-semester senior undergraduate or beginning graduate course in optimization theory and methods. The text is also of value for researchers and professionals in mathematics, operations research, electrical engineering, economics, statistics, and business.

Preface xv

About the Companion Website xviii

Part I Mathematical Review 1

1 Methods of Proof and Some Notation 3

1.1 Methods of Proof 3

1.2 Notation 5

Exercises 5

2 Vector Spaces and Matrices 7

2.1 Vector and Matrix 7

2.2 Rank of a Matrix 11

2.3 Linear Equations 16

2.4 Inner Products and Norms 18

Exercises 20

3 Transformations 23

3.1 Linear Transformations 23

3.2 Eigenvalues and Eigenvectors 24

3.3 Orthogonal Projections 26

3.4 Quadratic Forms 27

3.5 Matrix Norms 32

Exercises 35

4 Concepts from Geometry 39

4.1 Line Segments 39

4.2 Hyperplanes and Linear Varieties 39

4.3 Convex Sets 41

4.4 Neighborhoods 43

4.5 Polytopes and Polyhedra 44

Exercises 45

5 Elements of Calculus 47

5.1 Sequences and Limits 47

5.2 Differentiability 52

5.3 The Derivative Matrix 54

5.4 Differentiation Rules 57

5.5 Level Sets and Gradients 58

5.6 Taylor Series 61

Exercises 65

Part II Unconstrained Optimization 67