How to choose a sample of these input permutations?

Hello,

I have 38 inputs to my problem, but these 38 inputs are grouped into 12 groups. I allow 6 of these groups to change at once, which means in total there are 924 permutations of these groups that are being used as inputs to a simulation. For each of these permutations, I calculated the "decrease in Variance Accounted For (VAF)." This quantity isn't very relevant to my question, it's just a value I calculated for each set of inputs. It looks like this:

Now, I'm looking for a way to sample the set of 924 permutations, to obtain a set of inputs that are representative of the variability in the above bar plot. Ideally, each permutation could just be the inputs to one simulation, and I could run 924 simulations. But due to the nature of the work, this would take much too long. I need a way to pick a handful of these permutations and use those as inputs to only a handful of simulations, but I want to make sure the permutations I pick are representative. I've thought about random sampling until the mean/stdev (of the decrease in VAF) of the smaller set is within 10% of the mean/stdev of this entire 924 set. I've also been thinking of Design of Experiments, but I'm not sure that's exactly what I'm looking for. I already know the effect of each of the permutations (as seen in the bar plot). I'm also not trying to pick the "best" permutation to use, so I wouldn't say this is parameter optimization. I need a range of the VAF values from the input permutations to use.

Thanks for any advice!

Answer 1

[Jeff Miller]

Here's one possibility:

Start by making a histogram of the "Difference in VAF" (DVAF) scores.

Divide this histogram into k equal-probability regions, where k is the number of permutations you want to select. For example, with k=10, the first region has the smallest 92 (rounding) DVAF scores.

Use the permutation giving you the DVAF score in the middle of each of the k regions. Again with k=10, the first permutation is the one that gives you the 41'st smallest DVAF score, then second permutation is the one that gives you the 133'rd smallest, etc.

Each region of DVAF scores will be represented by the permutation giving the score in the middle of that region.

[Ashley Rice]

Oh, this is such a good idea! Thank you very much.