Sep­a­rat­ing Sig­nal from Noise

I want to ex­per­i­ment with mod­el­ing price changes over time as the com­bi­na­tion of a smooth trend com­po­nent over­laid with a ran­dom noise com­po­nen­t. My goal is to ex­am­ine the sta­tis­ti­cal prop­er­ties of each con­stituent com­po­nent and com­pare the re­sults to the sta­tis­ti­cal prop­er­ties of the un­de­com­posed mar­ket price.

In this post, I use a least squares mov­ing av­er­age to de­ter­mine the smooth com­po­nent of a price se­ries. This is our sig­nal, so to speak. The dif­fer­ence be­tween the mar­ket price and the smooth com­po­nent is the noise com­po­nen­t, which I have dubbed the “dither.” All cal­cu­la­tions are based on the log­a­rith­mic price val­ues.

S&P 500 ETF (Dai­ly)

The first set of da­ta I want to ex­am­ine is a se­ries of dai­ly clos­ing prices of an S&P 500 in­dex track­ing fund cov­er­ing a pe­ri­od of ap­prox­i­mate­ly 21 years. The first chart be­low shows the mar­ket price and the smooth com­po­nen­t. The sec­ond one shows the noise com­po­nen­t. Here are the chart­s:

The dither com­po­nent of the dai­ly price fluc­tu­a­tion is sep­a­rat­ed from the smooth com­po­nent that rep­re­sents the gen­er­al trend, cre­at­ing two dis­tinct da­ta sets that can be an­a­lyzed in­di­vid­u­al­ly. In my post ti­tled The Dis­tri­b­u­tion of Price Fluc­tu­a­tions, I plot­ted the dai­ly re­turns of the mar­ket price in a his­togram, along with a fit­ted nor­mal and Laplace den­si­ty func­tion. We can per­form the same analy­sis not on­ly on the mar­ket price da­ta but al­so on the sep­a­rat­ed smooth and dither com­po­nents. See charts be­low:

As might be ex­pect­ed, based on a pre­vi­ous study, the his­togram for the mar­ket price da­ta has the shape of a Laplace dis­tri­b­u­tion. For the smooth da­ta set, the shape of the his­togram al­so looks rough­ly like that of a Laplace dis­tri­b­u­tion, al­though it’s a bit dis­tort­ed. No­tice, how­ev­er, that the stan­dard de­vi­a­tion of the smooth da­ta is about an or­der of mag­ni­tude small­er than that of the mar­ket price da­ta. Al­so no­tice how the peak in the smooth price his­togram is shift­ed no­tice­ably to the right, in­di­cat­ing a gen­er­al up­trend in the data. The right­ward shift is present in the mar­ket price data as well, but it’s not as no­tice­able be­cause of the larg­er dis­per­sion of dai­ly price moves in the mar­ket price data.

Look­ing at the dither com­po­nen­t, the shape of the his­togram re­sem­bles that of a Laplace dis­tri­b­u­tion about as neat­ly as the shape of the his­togram for the mar­ket price does. The stan­dard de­vi­a­tion is about the same as that of the mar­ket price da­ta as well. To gain more in­sight­s, let’s look at some con­crete num­bers con­cern­ing the analy­sis of these three da­ta set­s:

These val­ues are the max­i­mum like­li­hood es­ti­mates for each da­ta set. The ta­bles above show the es­ti­mat­ed lo­ca­tion and scale pa­ra­me­ters for both the nor­mal den­si­ty func­tion and the Laplace den­si­ty func­tion. My post ti­tled Nor­mal and Laplace Dis­tri­b­u­tions pro­vides the de­tails on how these val­ues are cal­cu­lat­ed.

The lo­ca­tion pa­ra­me­ter for the nor­mal den­si­ty func­tion is rough­ly the same for both the mar­ket price and the smooth com­po­nen­t, while the es­ti­mat­ed pa­ra­me­ter val­ue for the dither com­po­nent is about an or­der of mag­ni­tude small­er. This sug­gests that the smooth com­po­nent em­bod­ies the gen­er­al di­rec­tion of the price trend, while the dither com­po­nent is rel­a­tive­ly neu­tral. We can see a sim­i­lar pat­tern in the lo­ca­tion pa­ra­me­ter val­ues for the Laplace den­si­ty func­tion.

The scale pa­ra­me­ter val­ues for both den­si­ty func­tions are rough­ly the same for both the mar­ket price and the dither com­po­nen­t, while the val­ues for the smooth com­po­nent are an or­der of mag­ni­tude small­er. This seems to im­ply that the dither com­po­nent em­bod­ies most of the noise that ob­scures the oth­er­wise smooth trend in the orig­i­nal mar­ket price.

S&P 500 ETF (In­tra­day)

The next set of da­ta I want to look at is a se­ries of in­tra­day prices of the same S&P 500 in­dex track­ing fund eval­u­at­ed pre­vi­ous­ly. This da­ta set con­tains one-minute in­tra­day da­ta cov­er­ing a sin­gle trad­ing day. The charts be­low show the mar­ket price along with the smooth trend com­po­nent and the dither noise com­po­nen­t:

The in­tra­day price se­ries con­tains a cou­ple of sud­den price moves that are not tracked very well by the least squares mov­ing av­er­age. This re­sults in large spikes on the noise chart. Let’s take a look at the his­togram for the mar­ket price da­ta se­ries and com­pare it to that of the sep­a­rate smooth and dither com­po­nents:

This his­togram for mar­ket price da­ta looks like it might ap­prox­i­mate the shape of the Laplace den­si­ty func­tion, but it has a set of shoul­ders not present in the mod­el func­tion. The his­togram for the smooth com­po­nent has a shape that is even less well de­fined. But look at the shape of the his­togram for the dither com­po­nen­t—it looks like an al­most ide­al ap­prox­i­ma­tion of the Laplace den­si­ty func­tion. Let’s take a look at the num­ber­s:

The lo­ca­tion and scale pa­ra­me­ters fit­ted to the nor­mal den­si­ty func­tion fol­low the same pat­tern we saw in the pre­vi­ous da­ta set. The smooth com­po­nent rep­re­sents the trend and the dither com­po­nent rep­re­sents the noise. For the Laplace den­si­ty func­tion, this pat­tern al­so holds for the scale pa­ra­me­ter but not for the lo­ca­tion pa­ra­me­ter. In­ter­est­ing­ly, the lo­ca­tion pa­ra­me­ter fit­ted to the Laplace den­si­ty func­tion im­plies a side­ways trend in the mar­ket price but in­di­cates an up­ward bias in both the smooth com­po­nent and the dither com­po­nen­t.

Japan­ese Yen (Dai­ly)

Now let’s take a look at the dai­ly ex­change rate be­tween the US dol­lar and the Japan­ese yen. This da­ta set cov­ers a range of about 18 years. Here are the chart­s:

The smooth com­po­nent seems to track the mar­ket price fair­ly well most of the time, but there does ap­pear to be some no­tice­able lag fol­low­ing re­ver­sal­s. The dither com­po­nent seems to os­cil­late up and down in cy­cles. Here are the his­togram­s:

The shape of the his­togram for both the mar­ket price da­ta and the dither com­po­nent close­ly re­sem­ble the shape of the Laplace den­si­ty func­tion. For the smooth com­po­nen­t, the his­togram has a gen­er­al bell shape, but it looks like it might be a bit too slop­py and asym­met­ri­cal to prop­er­ly char­ac­ter­ize it as hav­ing the shape of a nor­mal or a Laplace den­si­ty func­tion. Here are the num­ber­s:

The pat­tern here is very sim­i­lar to that of the pre­vi­ous da­ta set. These re­sults sug­gest that the smooth com­po­nent cap­tures the trend and the dither com­po­nent cap­tures the noise. As with the es­ti­mates com­put­ed for the pre­vi­ous sec­tion, the lo­ca­tion pa­ra­me­ters es­ti­mat­ed for the Laplace dis­tri­b­u­tion give con­fus­ing re­sult­s.

Chi­nese Yuan (In­tra­day)

The next da­ta set is a se­ries of in­tra­day ex­change rates be­tween the Chi­nese yuan and the US dol­lar cov­er­ing a pe­ri­od of ap­prox­i­mate­ly 24 hours. Each da­ta point is one minute apart. Here are the charts show­ing the in­tra­day mar­ket prices along with the sep­a­rat­ed smooth and dither com­po­nents:

In my post ti­tled The Very Strange Chi­nese Yuan, I ex­am­ined a dif­fer­ent set of in­tra­day ex­change rates be­tween the Chi­nese yuan and the US dol­lar. In that post, I demon­strat­ed a pe­cu­liar triple peak pat­tern in the dis­tri­b­u­tion of price move­ments. We can ob­serve the same phe­nom­enon in this da­ta set as well:

The shape of the his­togram for the mar­ket price da­ta shows the triple peak pat­tern that is char­ac­ter­is­tic of in­tra­day ex­change rates be­tween the yuan and dol­lar. The his­togram for the smooth com­po­nent ex­hibits a rough­ly bel­l-shaped dis­tri­b­u­tion with no in­di­ca­tion of the triple peak pat­tern at al­l. The fit­ted den­si­ty func­tions for the smooth com­po­nent are both shift­ed to the left, which can be at­tribut­ed to the down­ward trend vis­i­ble in the price chart. The his­togram for the dither com­po­nen­t, on the oth­er hand, clear­ly shows the triple peak pat­tern, in­di­cat­ing that this dis­tinc­tive noise pat­tern is al­most en­tire­ly re­moved from the price trend. Here are the pa­ra­me­ter es­ti­mates for the den­si­ty func­tion­s:

Again, we see a pat­tern here sim­i­lar to that of the pre­vi­ous da­ta set­s. For the pa­ra­me­ters fit­ted to the nor­mal den­si­ty func­tion, the mag­ni­tude of the val­ues give ev­i­dence that the smooth com­po­nent rep­re­sents the trend and the dither com­po­nent rep­re­sents the noise. We can see this mir­rored in the scale pa­ra­me­ter val­ues fit­ted to the Laplace den­si­ty func­tion, but the fit­ted lo­ca­tion pa­ra­me­ter val­ues are not as in­tu­itive. For the Laplace den­si­ty func­tion, the lo­ca­tion pa­ra­me­ter fit­ted to the mar­ket price da­ta is ze­ro even though there is an ob­vi­ous down­ward trend in the price chart.

Bit­coin (Dai­ly)

The fi­nal set of da­ta I want to ex­am­ine is the se­ries of dai­ly Bit­coin prices cov­er­ing a pe­ri­od of about five years. Here are the chart­s:

The price chart shows a fair­ly con­sis­tent mul­ti­-year trend fol­lowed by a dis­tinct re­ver­sal. The noise chart ex­hibits what ap­pears to be a cycli­cal pat­tern, al­though the pe­ri­ods don’t seem to be even­ly spaced. Here are the his­togram­s:

The his­tograms for both the mar­ket price and the dither com­po­nent have a shape that re­sem­bles the Laplace den­si­ty func­tion. The his­togram for the smooth com­po­nent has a slop­py and ir­reg­u­lar shape. Here are the num­ber­s:

Not sur­pris­ing­ly, these re­sults mir­ror what we’ve seen with the oth­er da­ta set­s. The pa­ra­me­ters fit­ted to the nor­mal den­si­ty func­tion and the scale pa­ra­me­ters fit­ted to the Laplace den­si­ty func­tion in­di­cate that the smooth and dither com­po­nents rep­re­sent the trend and noise re­spec­tive­ly. The lo­ca­tion pa­ra­me­ters es­ti­mat­ed for the Laplace den­si­ty func­tion re­main a bit more mys­te­ri­ous.

Fi­nal Thoughts

I think the most ob­vi­ous con­clu­sion to draw from this ex­per­i­ment is that a smooth price sig­nal can be sep­a­rat­ed from the un­re­lat­ed noise in price fluc­tu­a­tion­s. More specif­i­cal­ly, the par­tic­u­lar shape of the dis­tri­b­u­tion of price move­ments—a shape that typ­i­cal­ly re­sem­bles that of a Laplace dis­tri­b­u­tion—can be de­tached from the smooth price trend. The char­ac­ter­is­tics of the dis­tri­b­u­tion of mar­ket price fluc­tu­a­tions are large­ly a con­se­quence of the noise com­po­nent in­de­pen­dent of the trend com­po­nen­t. Even in the case of the in­tra­day Chi­nese yuan prices, the idio­syn­crat­ic triple peak dis­tri­b­u­tion can be iso­lat­ed to the noise com­po­nent on­ly.

An­oth­er in­ter­est­ing ob­ser­va­tion in the da­ta sets ex­am­ined here is that the noise is not en­tire­ly ran­dom. There is a struc­ture to it. There are un­de­ni­able up and down cy­cles. My ini­tial thought is to ap­ply Fouri­er analy­sis to ex­tract a cycli­cal com­po­nent from the resid­ual noise. I am cu­ri­ous what the dis­tri­b­u­tion char­ac­ter­is­tics of the resid­ual noise would look like if it could be iso­lat­ed from the cycli­cal com­po­nent as well as the trend com­po­nen­t. The up and down cy­cles are some­what ir­reg­u­lar, how­ev­er, which might make a Fouri­er analy­sis dif­fi­cult. I think this is some­thing worth fur­ther in­ves­ti­ga­tion.

The tech­niques used in this ar­ti­cle re­ly on a least squares mov­ing av­er­age to de­ter­mine the smooth trend com­po­nent of a price se­ries. While the least squares mov­ing av­er­age is great for track­ing sus­tained price trend­s, the dis­ad­van­tage is that it re­acts slow­ly to sharp re­ver­sals in the trend. The slow re­ac­tion to trend re­ver­sals can pro­duce ar­ti­fi­cial­ly large spikes in the noise com­po­nen­t. There might be bet­ter smooth­ing al­go­rithms worth ex­plor­ing.

Ac­com­pa­ny­ing source code is avail­able on GitHub.