## Saturday, May 25, 2019

### Brief items

A number of interesting items:

Alex Small said...

The GRE paper is not very good. First, their measure of grad school performance is completion, i.e. a binary variable. They are suppressing the range of the dependent variable, which is guaranteed to weaken correlations.

Also, they include the ranking of the grad program as an independent variable, i.e. a variable that is strongly correlated with GRE scores. This collinearity means that estimated coefficients in their regression model will have huge uncertainties, making it harder to get any non- zero coefficients to be statistically significant.

This is all analyzed here:

https://arxiv.org/abs/1902.09442

Furthermore, I wrote something in response to another study, looking at how selection effects should reduce correlations in the data. Indeed, a perfectly efficient sitting process would remove predictive validity from all variables, even if we don't truncate the dependent variables:

https://arxiv.org/abs/1709.02895

In short, we should leave this sort of data analysis to statisticians and social scientists, who are much better at this than physicists.

Anonymous said...

Douglas Natelson said...

Alex, thanks for the links. Precise statements about statistical significance aside, in a study of our own program we saw a very similar situation- grad gpa and degree completion seemed uncorrelated with subject gre. Of course, there are big selection effects at work here, as we were only looking at the students who had actually matriculated into our program. Similarly,the published study is only looking at students who did get admitted to some doctoral program.

Anon, my impression was that materials parts of the DOE cosponsored the study. I’m not sure about the costs - presumably there are travel costs associated with bringing the panel together for some small number of face to face meetings, for “townhalls” at APS and MRS, and some staff support at the NAS for the actual editing and document preparation. Still, all of that seems like small numbers. Perhaps a reader with more knowledge could comment here.

Anonymous said...

Doug, the total cost of the decadal study must be around $1M. The NSF portion was$458,000. See here: https://nsf.gov/awardsearch/showAward?AWD_ID=1647113

Maybe not surprising NSF complained when they did not get what they paid for.

Douglas Natelson said...

Anon@3:13pm, thanks for the numbers; I should've thought of doing that. As for whether the NSF got "what they paid for", I need to read the whole study to have a more informed opinion. Generically, a charge is given to the study group and they go to work, often with the sponsors checking in occasionally to make sure that any ambiguities are resolved. The reason the comments are controversial is that it can come across like slagging the people who volunteered a bunch of time and effort to do the study.

Peter Armitage said...

And Ted Geballle turns 100 next year! I'm told he's still in everyday working for a few hours and just applied for beam time to do some work on a new class of cuprates.

Anonymous said...