Nor­mal and Laplace Dis­tri­b­u­tions

I’m in­ter­est­ed in study­ing the Laplace dis­tri­b­u­tion. I was once un­der the im­pres­sion that price fluc­tu­a­tions in the fi­nan­cial mar­kets were nor­mal­ly dis­trib­ut­ed. How­ev­er, as I plan to show in a lat­er post, stock prices seem to move up and down ac­cord­ing to a Laplace dis­tri­b­u­tion in­stead. Be­fore an­a­lyz­ing any his­tor­i­cal price data, I first want to lay some ground­work and com­pare the Laplace dis­tri­b­u­tion to the nor­mal dis­tri­b­u­tion.

The Nor­mal Dis­tri­b­u­tion

Sup­pose we have a con­tin­u­ous ran­dom vari­able whose pos­si­ble val­ues are dis­trib­uted ac­cord­ing to a nor­mal dis­tri­b­u­tion. The prob­a­bil­i­ty den­si­ty func­tion is:

If we have some sam­ples of a ran­dom vari­able that we ex­pect to have a nor­mal dis­tri­b­u­tion, we can es­ti­mate the pa­ra­me­ters of the den­si­ty func­tion us­ing the max­i­mum like­li­hood method de­scribed in some of my pre­vi­ous posts. Since it’s more con­ve­nient in this case, in­stead of max­i­miz­ing the like­li­hood func­tion, let’s max­i­mize the log­a­rithm of the like­li­hood func­tion:

We want to know what val­ues for the mean and stan­dard de­vi­a­tion pa­ra­me­ters have the high­est pos­si­ble like­li­hood. To do that, we can fig­ure out where the de­riv­a­tive of the log-like­li­hood func­tion with re­spect to each of the pa­ra­me­ters is equal to ze­ro. Here is the par­tial de­riv­a­tive of the log-like­li­hood func­tion with re­spect to the mean:

Set­ting the par­tial de­riv­a­tive to ze­ro and solv­ing for the mean, we ar­rive at the fol­low­ing es­ti­mat­ed val­ue:

Once we have the val­ue for the mean, we can fol­low the same steps to solve for the stan­dard de­vi­a­tion. Here is the par­tial de­riv­a­tive of the log-like­li­hood func­tion with re­spect to the stan­dard de­vi­a­tion:

Set­ting the par­tial de­riv­a­tive to ze­ro and solv­ing for the stan­dard de­vi­a­tion, we get this es­ti­mat­ed val­ue:

If you want to see a more de­tailed break­down of the steps above, you can ref­er­ence my post ti­tled Least Squares and Nor­mal Dis­tri­b­u­tions. As I men­tioned in that post, the max­i­mum like­li­hood es­ti­ma­tor for the stan­dard de­vi­a­tion can give an es­ti­mate that is too low for small sam­ple sizes. If us­ing a lim­it­ed sam­ple size, it might be a good idea to ap­ply Bessel’s cor­rec­tion to get a more ac­cu­rate es­ti­mate.

The Laplace Dis­tri­b­u­tion

Sup­pose we have a con­tin­u­ous ran­dom vari­able whose pos­si­ble val­ues are dis­trib­uted ac­cord­ing to a Laplace dis­tri­b­u­tion. The prob­a­bil­i­ty den­si­ty func­tion is:

If we have a set of sam­ples of a ran­dom vari­able that we know to have a Laplace dis­tri­b­u­tion, we can es­ti­mate the pa­ra­me­ters us­ing the same ap­proach we took for es­ti­mat­ing the pa­ra­me­ters of the nor­mal dis­tri­b­u­tion. We can use the max­i­mum like­li­hood method. Here is the log-like­li­hood func­tion we want to max­i­mize:

We want to know what val­ues of the lo­ca­tion and scale pa­ra­me­ters have the great­est like­li­hood. The an­a­lyt­i­cal ap­proach is to take the de­riv­a­tive, set it to ze­ro, and solve for the pa­ra­me­ter­s. But con­sid­er the ab­solute val­ue func­tion:

It’s a piece­wise func­tion. Tak­ing the de­riv­a­tive of the log-like­li­hood func­tion with re­spect to the lo­ca­tion pa­ra­me­ter can be a bit tricky be­cause the ab­solute val­ue func­tion, al­though con­tin­u­ous, is not dif­fer­en­tiable at all points:

To be more suc­cinc­t, we can rep­re­sent the de­riv­a­tive of the ab­solute val­ue func­tion us­ing the sign func­tion:

The sign func­tion sim­ply re­turns the sign of a val­ue:

We can ex­press the par­tial de­riv­a­tive of the log-like­li­hood func­tion with re­spect to the lo­ca­tion pa­ra­me­ter as:

This is re­al­ly just giv­ing us the num­ber of sam­ples with a val­ue greater than the lo­ca­tion pa­ra­me­ter mi­nus the num­ber of sam­ples with a val­ue less than the lo­ca­tion pa­ra­me­ter. Note al­so that the de­riv­a­tive is un­de­fined at points where the lo­ca­tion pa­ra­me­ter equals the val­ue of one of the sam­ples. While not ad­e­quate for an an­a­lyt­i­cal so­lu­tion, this does pro­vide a clue that the best es­ti­mate is at or near the me­di­an val­ue. Let’s rank our sam­ples in as­cend­ing or­der:

Let’s al­so choose a mid­dle val­ue that is about halfway be­tween the first and last sam­ple in the or­dered set. The ex­act val­ue de­pends on whether the to­tal num­ber of sam­ples is an even num­ber or an odd num­ber:

We can glean some in­sights by look­ing at a plot of the like­li­hood val­ue for pos­si­ble val­ues of the lo­ca­tion pa­ra­me­ter. When there is an even num­ber of sam­ples, the like­li­hood func­tion looks like this:

No­tice that there is a range of pos­si­ble val­ues where the like­li­hood is at a max­i­mum when there is an even num­ber of sam­ples. For an odd num­ber of sam­ples, the like­li­hood func­tion looks slight­ly dif­fer­en­t:

For an odd num­ber of sam­ples, there is a sin­gle point at which the like­li­hood is max­i­mized. By in­spec­tion, we can con­clude that the me­di­an val­ue of our sam­ples has the high­est like­li­hood for the lo­ca­tion pa­ra­me­ter:

If we have an even num­ber of sam­ples, we just take the mean of the two me­di­an val­ues. Once an es­ti­mate of the lo­ca­tion pa­ra­me­ter is known, solv­ing for the scale pa­ra­me­ter is a bit eas­i­er since there is an an­a­lyt­i­cal so­lu­tion. Here is the par­tial de­riv­a­tive of the log-like­li­hood func­tion with re­spect to the scale pa­ra­me­ter:

Set­ting the par­tial de­riv­a­tive to ze­ro and solv­ing for the scale pa­ra­me­ter, we get the fol­low­ing es­ti­mate:

I think it’s worth men­tion­ing here that this method of es­ti­mat­ing the pa­ra­me­ters of a Laplace dis­tri­b­u­tion does­n’t sit well with me. Choos­ing the me­di­an val­ue for the lo­ca­tion pa­ra­me­ter seems like a coarse ap­proach. In cas­es where there is a range of pos­si­ble val­ues for the lo­ca­tion, I won­der just how wide that range can be in prac­tice. There might be oth­er es­ti­ma­tion tech­niques worth look­ing in­to, but I want to see how well this one works with re­al da­ta be­fore ex­plor­ing al­ter­na­tives.

Com­par­i­son

The nor­mal dis­tri­b­u­tion and the Laplace dis­tri­b­u­tion are both sym­met­ri­cal. The den­si­ty func­tions of each have a sim­i­lar struc­ture. And with a small num­ber of sam­ples, it might be dif­fi­cult to de­ter­mine if a ran­dom vari­able has a nor­mal dis­tri­b­u­tion or a Laplace dis­tri­b­u­tion. How­ev­er, there are some im­por­tant dif­fer­ences that are best shown with an il­lus­tra­tion:

Both den­si­ty func­tions have the same ba­sic shape. The den­si­ty plot of the Laplace dis­tri­b­u­tion, how­ev­er, is taller and skin­nier in the mid­dle. It al­so has fat­ter tails than the nor­mal dis­tri­b­u­tion. I think those fat tails are worth tak­ing a clos­er look at. Here is the same chart with the den­si­ty plot­ted on a log­a­rith­mic scale:

No­tice the dif­fer­ence in mag­ni­tude for val­ues far from the mid­dle. The prob­a­bil­i­ty of ob­serv­ing a val­ue of a nor­mal­ly dis­trib­uted ran­dom vari­able far from the mean is quite smal­l. The prob­a­bil­i­ty of ob­serv­ing the same val­ue, while still smal­l, might be or­ders of mag­ni­tude greater if the ran­dom vari­able has a Laplace dis­tri­b­u­tion.

Ac­com­pa­ny­ing source code is avail­able on GitHub.