Hi, I'm working on a project where we are validating a psychometric self-reported tool on personality traits and comparing to other personality measures (self-report and other). Myself and colleagues have identifies which items we expect to correlate on each tool by indicating that a correlation is expected and its direction.
I have run the correlations and now have data to compare what we expected to correlate vs. the actual outcomes.
We would like to run some significance testing to see whether the hits and misses are better than chance. We thought of using a chi-squared, but all the literature I have seen using this method have given actual values for their expected correlations (for example, we expect a correlation of .33 and the observed correlation was .56). We don't have the values for expected outcomes - is there a way we can still carry out a significance test?
I'd be grateful for any advice.