How to find respective features from Principal Components

Hello Everyone!

I have a feature matrix of 4378*54. After doing PCA, I get a reduced feature matrix of 4378*29. I have used a threshold variance of 95%.

In summary, 29 principal components explain 95% of variance in my feature matrix.

How can I find the respective features in my feature matrix from the 29 principal components? Ofcourse, these 29 principal components correspond to 29 features in my feature matrix which have 54 features. How can I find those features?

I am using PCA function which gives me coeff, scores, latent and explained.

Secondly, do I have to use same PCA for both training and testing dataset? I have read in an article that I have to store the selected coeff (95% variance) from training dataset and multiply this with my testing dataset which will yield a reduced feature matrix. I have tried it and it works. But how it works, I dont understand.

Your comments will be highly appreciated.

Answer 1

[Ayush Aniket]

Hi Jamil,

Regarding your first question:

• Principal components are linear combinations of your original features. Each component is a vector in the direction of the dataset's maximum variance. The coefficients in these vectors tell you how much each original feature contributes to that principal component.
• Each principal component does not correspond directly to a single original feature. Instead, each principal component is a mixture of all the original features. Therefore, you cannot directly map the 29 principal components to 29 original features out of the 54. Instead, you can determine which original features contribute most to each principal component.
• The coeff matrix that you get is the matrix of coefficients for the principal components. Each column of coeff represents one principal component, and each row corresponds to the original features. The magnitude of the coefficients indicates the importance of each feature to the principal component.
• To find which original features are most important for each of the 29 principal components, you can look at the magnitude of the coefficients in coeff. For each principal component (column in coeff), the features (rows) with the highest absolute values are the ones that contribute most to that component.

For the second query:

• Yes, you should use the same PCA transformation for both training and testing datasets. You should use the coeff obtained from the training data to project the testing data onto the same principal component space. This is done by subtracting the mean of the training data from the testing data and then multiplying by coeff. The process ensures that the testing data is transformed in a way that is consistent with the training data, allowing for meaningful comparisons and predictions.

Refer to the documentation link to read more about PCA: https://www.mathworks.com/help/stats/principal-component-analysis-pca.html

Hope it helps.

[Jamil]

Thank you so much for the answer!

However, It would be nice if you can clear another confusion of mine.

First, (correct me if I am wrong) coeff matrix is arranged in descending order. So for example it is possible that feature number 10 has the highest contribution. How can I understand from coeff matrix that feature number 10 in my original feature matrix has the highest contribution?

Second, Before applying PCA to my training dataset, I standardized it using zscore. Should I also standardize my testing data before subtracting the mean of training data and then multiplying by the coeff? I tried with and without standardizing my test data and the scores changed significantly i.e., good scores when I standardized my test data and poor scores when I didnt.

[Ayush Aniket]

Sure.

Regarding the first query:

• The coeff matrix in PCA indeed shows the contribution of each original feature to each principal component, and the columns of the coeff matrix are indeed arranged in descending order of variance explained by the corresponding principal components. However, the arrangement of columns doesn't directly tell you which original feature contributes the most; instead, it tells you the direction in the original feature space that captures the most variance.
• To understand which original feature (e.g., feature number 10) has the highest contribution to a principal component, you look at the magnitude of the coefficients in a given column of the coeff matrix. A higher absolute value of the coefficient for a feature means a higher contribution of that feature to the principal component. Refer to the below code snippet for finding the the feature with the highest contribution to the first principal component:

    [coeff, ~, ~, ~] = pca(yourFeatureMatrix);
    % Extract the coefficients (loadings) for the first principal component
    firstPC = coeff(:,1);
    % Find the index of the feature with the highest absolute contribution
    [~, maxFeatureIndex] = max(abs(firstPC));

• maxFeatureIndex will give you the index of the original feature that has the highest contribution to the first principal component.

For the second one:

When you perform PCA, especially after standardizing (z-scoring) the training data, you must apply the same preprocessing steps to your testing data before transforming it with the PCA model obtained from the training data. This ensures that the testing data is on the same scale as the training data, which is crucial for the PCA transformation to be meaningful and for your model to perform as expected. This technique is common for any Machine Learning task.

[Jamil]

Thank you so much for your valuable insight.

Now, I fully understand the concept.

Thanks alot for the help.