Vi­su­al­iz­ing Sad­dle Points and Min­i­mums

The two pre­vi­ous posts demon­strat­ed how to use the method of La­grange mul­ti­pli­ers to find the op­ti­mum so­lu­tion for a coin toss game with bi­ased coins of un­known weight. In one case, we found the min­i­mum of a cost func­tion based on the La­grangian func­tion. In the oth­er case, we found the sad­dle point of the La­grangian func­tion it­self. The pur­pose of this post is to pro­vide some vi­su­al rep­re­sen­ta­tions of these func­tion­s.

The Mod­el

The mod­el of the coin toss game we’ll use here is a sim­pli­fied ver­sion. It us­es on­ly two flips per round in­stead of three flips or four flips per round as used in pre­vi­ous ex­am­ples. Here is what the Markov mod­el looks like:

The first coin toss is al­ways a fair coin, by de­f­i­n­i­tion. Let’s as­sume the prob­a­bil­i­ty of toss­ing both a head and a tail, in any or­der, is twice that of get­ting ei­ther two heads in a row or two tails in a row. Let’s al­so as­sume a scor­ing func­tion that gives pref­er­ence to weights that are as close to that of a fair coin as pos­si­ble. The La­grangian func­tion for this mod­el looks like this:

Like we did pre­vi­ous­ly, we can de­rive a cost func­tion from the La­grangian func­tion by com­put­ing the mag­ni­tude of the gra­di­ent of the La­grangian func­tion. We’ll take the square of the mag­ni­tude to sim­pli­fy the cal­cu­la­tion. Here is the cost func­tion:

For such a sim­ple mod­el, it might be ob­vi­ous what the so­lu­tion is. The crit­i­cal points for both func­tions lie at the same lo­ca­tion. But for the La­grangian func­tion, the crit­i­cal point ex­ists at a sad­dle point, where­as the crit­i­cal point for the cost func­tion lies at the min­i­mum. The il­lus­tra­tions in the fol­low­ing sec­tions pro­vide the vi­su­al rep­re­sen­ta­tion­s.

La­grangian Func­tion

The op­ti­mum val­ue lies at the point at which the gra­di­ent is equal to the ze­ro vec­tor:

Here is a heatmap of the La­grangian func­tion:

Here is a sur­face plot of the func­tion at three slight­ly dif­fer­ent an­gles:

Here is the pro­file of a slice of the func­tion tak­en across a di­ag­o­nal line:

Here is the pro­file of an­oth­er slice tak­en across a dif­fer­ent di­ag­o­nal:

The sur­face plot re­sem­bles the shape of a cow­boy hat or a west­ern horse sad­dle. The left and right sides are tilt­ed up­ward­s, while the front and back sides are tilt­ed down­ward­s. In the pro­file chart­s, one is con­vex while the oth­er is con­cave. This type of cur­va­ture is char­ac­ter­is­tic of a func­tion with a sad­dle point.

Cost Func­tion

The op­ti­mum val­ue lies at the point at which the gra­di­ent is equal to the ze­ro vec­tor:

Here is a heatmap of the cost func­tion:

Here is a sur­face plot of the func­tion at three slight­ly dif­fer­ent an­gles:

Here is the pro­file of a slice of the func­tion tak­en across a di­ag­o­nal line:

Here is the pro­file of an­oth­er slice tak­en across a dif­fer­ent di­ag­o­nal:

The sur­face plot re­sem­bles the shape of an elon­gat­ed bowl or an en­closed val­ley. All sides are tilt­ed up­ward­s. In the pro­file chart­s, both pro­files show a con­vex curve. As a mat­ter of fac­t, all slices through the op­ti­mum point will be con­vex. This type of cur­va­ture is char­ac­ter­is­tic of a func­tion with a min­i­mum point.

Ac­com­pa­ny­ing source code is avail­able on GitHub.