Weighted correlation coefficient

Question

My problem is to classify input time series signal (128 points, equidistant in time, range 0.0..1.0) in limited amount (say 16) different classes. We have hand-classified 20000 samples and we are using 2000 as training set and other 18000 to evaluate our classification algorithm (test set).

In my analysis I found that classification using correlation coefficient works quite good. https://en.wikipedia.org/wiki/Pearson_correlation_coefficient

My classification "standard" class samples are calculated as mean values from training set. I correlate each sample from test set with standard class samples and the winning class is selected by the highest coefficient.

The problem is that I found out that different parts of signal is not equaly important. Some parts of signal fluctuate widely between samples, and same are very similar.

So I came upon idea to include some sort of weight for points in correlation coefficient. The weight would be inverse proportional to variance of selected point sample for the class in training set.

I can't believe this is a novel idea, but in my search up to now I have not found anything like this. If this is a known method, please let me know. If somebody had a similar problem and used similar methods, please report on results. If nowone has heard of this approach, please let me know what you (statistics-wise) think of it. Thank you!

Answer 1

Because you are only comparing the candidates to the mean sample of each class, you lose information about the distribution of each class. You are trying to compensate for this by assigning a different importance for different aspects of the signal, but a good model should include this uncertainty directly.

As a very simple improvement, you could compare a candidate to multiple samples of the class and then combine the correlations. The more samples you take the more precise the estimate of correlation will be but the longer will the calculation take. You could even use such a technique to determine the uncertainty of your correlation estimate through bootstrapping.

You could also use a different metric to indicate class membership, and that metric could make use of more information. Your classification problem can be interpreted as a model selection problem. Each class represents a model, and you want to select the model that best explains the candidate sample. With the Bayesian approach, you would select that model that has the best evidence.

Whereas you currently describe the model by the mean value of the distribution of its training samples, you could extend the model e.g. to also include the variance. Unimportant features will have a high variance within a class, and thus automatically consider those features less important. This can be done fairly easily if you can assume the features to have a normal/Gaussian distribution. If you manually assign a weight to features that can still be helpful to speed the algorithm up, but it's not necessary or fundamentally helpful.

Before you start building complicated models, you should consider whether a learning approach could work that doesn't need to abstract a model from the data. For example, a KNN algorithm might perform well: you select the k nearest training samples (with some distance metric in the feature space, which could be your correlation metric), and then assign the candidate to that class to which most of its neighbors belong.

However, this is more of a statistics and data science problem than a software engineering problem. From the software engineering standpoint I have found it useful to create some interface that all algorithms must implement, so that algorithms can easily be swapped out for testing and comparison. That also makes it fairly straightforward to create a boosted classifier that combines the output of multiple simpler classifiers.