Fritz - Curtin Old vs New test - why I don't trust it

[not telling]

This year's competition would be the perfect opportunity to find musicians and others who believe they possess better-than-random abilities of discernment. Let those people have an hour a day for a week, blindfolded. Compile an experienced audience who wants to hear great instruments played (not difficult, I imagine) and see what they think as the week progresses. Bring in more instruments. Maybe 2-3 that have fetched a high price in recent years among the olds...make sure no one has any interest in any of the instruments faring better than another...

I had one stats course in college, so maybe this is too many variables. But has audience perception been compared with musicians' impressions? Someone do this, please.

[legenyes]

Dan_S, I'm sorry, I don't seem to be able to find a way to explain this to you.  If you have a group of 10 people flipping coins, it is quite likely that someone will get 5 heads in 6 flips, just by chance.

 

Maybe Don Noon's post #89 above will make more sense to you.  He is saying the same thing I am.

[Janito]

 

Raw data of the second  Part 2  the experiment (Evaluation by specific criteria) is not available, moreover protocol has in my opinion majors flaws.I therefore discarded it as I saw no point to discuss it.

 

Thanks for your analysis!

 

You point out something critical - look at all the data to determine if there are inconsistencies.

[dan_s]

Dan_S, I'm sorry, I don't seem to be able to find a way to explain this to you.  If you have a group of 10 people flipping coins, it is quite likely that someone will get 5 heads in 6 flips, just by chance.

 

Maybe Don Noon's post #89 above will make more sense to you.  He is saying the same thing as I am.

 

No problem. These things take a bit to "digest".  The situation here is not "A GROUP OF PEOPLE FLIPPING COINS" .  Not even Don thinks that anymore.

[Michael Darnton]

The 2014 Paper aimed to confirm and further investigate conclusion made by the authors in 2010. It claim to asses some of the short comings of the initial experiment.  etc. etc. etc

THANK YOU, THANK YOU THANK YOU!

 

When my wife did her Master's, she came home from statistics class one day in wonder at how many people in the class were unable to design a survey that was logical and extracted the information they wanted, and then follow through and get something meaningful. From the responses and persistent disagreements in this thread, I'd say that problem is more universal than we would like it to be.

[Antoine Nedelec]

If I forgot 5% every year for 46 years didn't I forget 230% of what I used to know?

Marty,

you are in luck.

If you lost 5% every year for 1000 years (that doesn't sound so bad, does it?), you would get pretty close to knowing nothing but you would never reach total ignorance. ;-)

[Antoine Nedelec]

No problem. These things take a bit to "digest".  The situation here is not "A GROUP OF PEOPLE FLIPPING COINS" .  Not even Don thinks that anymore.

No, it is not a group of people flipping coins. But they did no better than people flipping coins.

Let's take the player who did "well' and see if he does it again.  Except I doubt Claudia would like to put the effort. And seeing how much venim has been spewed her way, I would not blame her.

We are all interrested to know if there is such a thing as a old sound and if one can hear it.

In fact, let's stop bickering and let's get to it. I can get a couple nice moderns and a few Strads. Anyone wants to participate?

[dan_s]

I find it very disturbing that Claudia Fritz is not joining in the discussion in order to offer some clarification.

She also, ignored my request for a link/copy of her CV. 

[Carl Stross]

I find it very disturbing that Claudia Fritz is not joining in the discussion in order to offer some clarification.

She also, ignored my request for a link/copy of her CV. 

 

She's under no obligation to do that.  

[legenyes]

Initial impression: the first thing that stroke me is that the numbers did not add up in Figure 1. Session 1 gives us 99 Choices but 12 violins 10 players means 120 choice. the explanation is no where to be find in the paper,

Only when I toke a look  at the raw data in Table S1 could I figure that there was some "intermediate violin" . Again no explanation of what that are or how they are taken into account or not .

You have to guess that  there were some instruments that were not discarded yet not in top 4 and that the author chose not to discuss those data....

When you read a paper like this, you have to be very careful to understand what was intended.

 

The caption clearly explains that Figure 1 shows only top choices and rejected violins.  This is also explained in the first paragraph of the Results section.  Just to be sure, I compared Figure 1 with Table S1.  All violins are accounted for.  The article does go on to discuss all the violins, although not individually.

 

lFred wrote:

"Lets take an other look at table 1:

"As far as the preferred instrument (the one the player would keep to play) is concerned

 

"For 18th century Italians there is a 400% increase in choice between session 1 and Session 2

"                                      there is a 200% increase in number of instruments chosen between session 1 and Session 2..."

 

 

 

Table 1 does not show "the preferred instrument (the one the player would like to keep)".  It shows point totals, based on being chosen first, second, etc., or rejected.  I note that some old instruments rose in the rankings between sessions, but some also fell.  The same is true for new instruments.

 

 

lFred wrote:

"What do we make of those numbers? 

 

"If I only took a look at the table 1 data (the one analysed be the authors) I would have concluded that there are evidence that you can not properly test a violin in an hotel room an  that large halls were in deed more beneficial to 18th Italian instruments than modern instruments. In other word the exact opposite of what the authors concluded."

 

 

Table 1 shows that the rankings changed somewhat, which is hardly surprising.  But here is what the authors actually wrote, which I do not find fault with:

     "By the end of their time in the hall, eight of ten the players chose an instrument that was one or their top-four in the rehearsal room. While no one would propose that real-life instrument searches should rely entirely on impressions formed in a small room, these results do suggest that meaningful testing about general preferences can be conducted outside a concert hall. There remains the important question of how well player-preferences correlate with those of listeners, particularly with regards to projection in a hall – the topic of a forthcoming study."

 

 

    I think everyone who has questions about this paper would do well to read it carefully and see what it actually does say and what it does not.  Don't just skim it and form an opinion until you have read it carefully.  The paper does have something to say beyond the numbers about new vs. old sound, by the way.

[not telling]

The problems in designing this sort of study are mind-boggling! Do they need to all have the same setup? Strings/tension? Chin rests?

It's nice that these studies are happening, but one assumption among many in this recent inquiry was that all setups are equal. Among five thousand other variables, this one exists. And although everyone knows that setup matters a lot, how would you even test that simple question??

I have a deep, abiding passion for qualitative research. Once you start plugging in z values and formulaes into SAS and that sort of thing, my eyes glaze over. It took me awhile, too, to understand the objective math behind an experiment in which your chance of being correct is 1/2. I actually had to make a chart once to figure out why the numbers get weird when you're flipping a coin consecutively and wanting to know the odds of having a correct response, say...4/4 times. It's obvious once you think about it.

Choosing a Strad isn't flipping a coin, but the same odds of giving a correct response are in play. So it may as well be the same exercise. I find it all interesting but it's easy to think of ways the study isn't answering any questions.

[Don Noon]

We are all interrested to know if there is such a thing as a old sound and if one can hear it.

In fact, let's stop bickering and let's get to it. I can get a couple nice moderns and a few Strads. Anyone wants to participate?

 

I would love to, althought there might be some practical issues.  I'm also interested in seeing if there are consistent MEASURABLE differences between modern and old, which I believe can be found with more than random chance.

[JohnCockburn]

No, it is not a group of people flipping coins. But they did no better than people flipping coins.

Let's take the player who did "well' and see if he does it again.  

Yes. Let's.

If the person who gets 5/6 heads happens to have ginger hair, it would be ridiculous to conclude that having ginger hair gives you an improved chance of tossing a head.

But if it is possible that the outcome of a trial is influenced by some factor other than random chance (e.g. knowledge, skill, expertise etc) then any individual who shows significantly better performance than average must be investigated further before safe conclusions can be reached about whether such expertise is real or illusory. 

[dan_s]

 

 

 

    I think everyone who has questions about this paper would do well to read it carefully and see what it actually does say and what it does not.  Don't just skim it and form an opinion until you have read it carefully.  The paper does have something to say beyond the numbers about new vs. old sound, by the way.

 

You are right. Tell us please : do you have any connection with the paper or the authors ? You seem to have come back here specifically to "protect" this paper.

[James M. Jones]

 

In fact, let's stop bickering and let's get to it. I can get a couple nice moderns and a few Strads. Anyone wants to participate?

I would LOVE to.....in September?

[Janito]

 

Let's take the player who did "well' and see if he does it again.

 

It's all about understanding 'regression to the mean' - you need to see how the 2 extremes perform.  Do they continue to separate or do they regression to some common value ( the mean) - in other words, the differences become less and less each time the experiment is performed.

 

I had a paper rejected recently by a severe statistical reviewer who asserted that the sample sizes was to small.  What the reviewer missed completely was the fact that the data were internally consistent across the 3 independent studies reported in the paper.  In other words, the biochemical differences between treatment groups did not 'regress to the mean'. 

 

Incidentally, once you have the first set of data you can use Bayesian methods to predict outcomes for the next study, and so forth. You can determine the potential for false positive and false negative results given the characteristics of the new study..   

 

Enough from me, I know nothing.

[Don Noon]

One last shot at trying to clarify the statistics and probability argument:

 

If you gave Player1 the 6 instruments, and he guessed 5 correctly, then he beat the odds (which are less than 10% of that happening by chance, or slightly over 10% if you include the result of getting all 6 correct).

 

If you take a group of 10 players, give each of them the 6 instruments, then look at the results, find the player who got 5 correct, and THEN call him Player1... that is a TOTALLY different statistical situation, and you could very easily get the same results by having the players just flip coins for their answers.  This is what is most like the actual test.

 

 

If I forgot 5% every year for 46 years didn't I forget 230% of what I used to know?

 

With that math, it is only possible if you absorb knowledge from others around you, and then forget it.  Good thing for me I'm on the opposite side of the country.  Some other posters recently must be too near, I fear.

[Carl Stross]

No, it is not a group of people flipping coins. But they did no better than people flipping coins.

 

 

"They" is of no concern here. Imagine that amongst "they" is that unique individual that can tell Strads. Perfectly, anytime, anywhere.  In the context of this study, his ability will be reflected but the researchers through the inept approach of looking at the players AS A GROUP, will fail to find him and will conclude :

 

"Soloists failed to distinguish new from old at better than chance levels"

 

Sentient humans understand this as : NO soloist from this group, distinguished New from Old better than chance level. 

 

And that's not true.

[Marty Kasprzyk]

The problems in designing this sort of study are mind-boggling! Do they need to all have the same setup? Strings/tension? Chin rests?

It's nice that these studies are happening, but one assumption among many in this recent inquiry was that all setups are equal. Among five thousand other variables, this one exists. And although everyone knows that setup matters a lot, how would you even test that simple question??

I have a deep, abiding passion for qualitative research. Once you start plugging in z values and formulaes into SAS and that sort of thing, my eyes glaze over. It took me awhile, too, to understand the objective math behind an experiment in which your chance of being correct is 1/2. I actually had to make a chart once to figure out why the numbers get weird when you're flipping a coin consecutively and wanting to know the odds of having a correct response, say...4/4 times. It's obvious once you think about it.

Choosing a Strad isn't flipping a coin, but the same odds of giving a correct response are in play. So it may as well be the same exercise. I find it all interesting but it's easy to think of ways the study isn't answering any questions.

Well for fun I did the coin experiment rather than rely on mathematical theories.  I took 6 pennies and dropped them on a table and counted the number of heads and tails.  I did this ten times.  Two times I found a combination of 5 heads and 1 tail,  and one time I found a combination of 1 head and 5 tails.

 

 

Trial heads tails 1 2 4 2 1 5 3 4 2 4 5 1 5 3 3 6 4 2 7 5 1 8 3 3 9 4 2 10 3 3   34 26

 

So I believe Claudia's comment that a player could correctly pick 5 out of 6 just due to chance is probably (pun) correct.

 

I got similar results with violas.

[Carl Stross]

Well for fun I did the coin experiment rather than rely on mathematical theories.  I took 6 pennies and dropped them on a table and counted the number of heads and tails.  I did this ten times.  Two times I found a combination of 5 heads and 1 tail,  and one time I found a combination of 1 head and 5 tails.

 

 

Trial heads tails 1 2 4 2 1 5 3 4 2 4 5 1 5 3 3 6 4 2 7 5 1 8 3 3 9 4 2 10 3 3   34 26

 

So I believe Claudia's comment that a player could correctly pick 5 out of 6 just due to chance is probably (pun) correct.

 

I got similar results with violas.

 

Are you sure ?

[Janito]

 

 

I had a paper rejected recently 

 

To avoid any confusion, this was not submitted to the Journal of Biological Effects of Easy Coastal Living (JBEECL).

[Wm. Johnston]

I find it very disturbing that Claudia Fritz is not joining in the discussion in order to offer some clarification.

She also, ignored my request for a link/copy of her CV. 

How dare she not spend every waking moment on Maestronet answering comments by anonymous posters? The nerve!

 

ps. I just realized that the name "Maestronet" is not in Maestronet's spell checker. The fact that I found that interesting shows that I spent too much time reading this thread.

[Marty Kasprzyk]

Well for fun I did the coin experiment rather than rely on mathematical theories.  I took 6 pennies and dropped them on a table and counted the number of heads and tails.  I did this ten times.  Two times I found a combination of 5 heads and 1 tail,  and one time I found a combination of 1 head and 5 tails.

 

 

Trial heads tails 1 2 4 2 1 5 3 4 2 4 5 1 5 3 3 6 4 2 7 5 1 8 3 3 9 4 2 10 3 3   34 26

 

So I believe Claudia's comment that a player could correctly pick 5 out of 6 just due to chance is probably (pun) correct.

 

I got similar results with violas.

The table didn't copy well.  Attached is a pdf file.

coin flips.pdf

[Michael Darnton]

Let's take the player who did "well' and see if he does it again.  Except I doubt Claudia would like to put the effort. 

I suspect that waiting for the authors of a paper to do the research that would specifically disassemble their previous research and assumptions is not going to happen in the real world, even though that is certainly the issue that should have been examined in the first place, and the current paper relegated to the gathering-background category. Nevertheless, it does appear that data within their paper contradicts their conclusion, and someone who's less invested in the current results should definitely look at that in more depth.

[Wm. Johnston]

I suspect that waiting for the authors of a paper to do the research that would specifically disassemble their previous research and assumptions is not going to happen in the real world, ....

This does actually happen.