Fritz - Curtin Old vs New test - why I don't trust it

[Carl Stross]

 

Let's take the player who did "well' and see if he does it again.  Except I doubt Claudia would like to put the effort. And seeing how much venim has been spewed her way, I would not blame her.

 

 

What a pity she did not take 5 ( FIVE ) minutes there and then and repeat the test with just that player.

[legenyes]

Sadly, no control and no statistics other than descriptive statistics.  ...It is really a shame that with the considerable time, effort, and money (presumably) put into this project that the authors did not design the methods in a manner that would allow for statistical analyses.  With descriptive statistics you really can't separate causal results from variance and random chance.

But several people have done a partial statistical analysis in this thread.  And one even flipped a coin to discover that we are correct.

 

There are really two questions that I think you want to analyze.  The first is how well players can distinguish between different violins.  Although the paper doesn't specifically test this, my impression is that given their similarity in rankings between first and second rounds, players do fairly well at this for certain violins.  If you want to test player 1 against other players, I suspect you would need several times more data.

 

As for significant differences between old and new violins as a group, I suspect that you would need a lot more violins to do this.  I think you would be surprised at the amount of data and number of violins you would need for this.

[legenyes]

You are right. Tell us please : do you have any connection with the paper or the authors ? You seem to have come back here specifically to "protect" this paper.

I don't know the authors.  I'm not even familiar with their work except for these two papers.  I read these discussions with interest, however, and joined this discussion because I was appalled and offended by some of the abusive language in the previous discussion.

 

I am also in academia, and I thought it appropriate to comment because I know how difficult it is to understand and use simple statistical concepts.

[dan_s]

But several people have done a partial statistical analysis in this thread.  And one even flipped a coin to discover that we are correct.

 

There are really two questions that I think you want to analyze.  The first is how well players can distinguish between different violins.  Although the paper doesn't specifically test this, my impression is that given their similarity in rankings between first and second rounds, players do fairly well at this for certain violins.  If you want to test player 1 against other players, I suspect you would need several times more data.

 

As for significant differences between old and new violins as a group, I suspect that you would need a lot more violins to do this.  I think you would be surprised at the amount of data and number of violins you would need for this.

 

Nobody has done a partial statistical analysis in this thread. Flipping coins does not "discover we are correct"

Testing player1 against other players is completely useless. The other players might be tone deaf. That will not

influence player1 abilities. There is no conection between player1 and the others.

[legenyes]

Dan_S, I have tried to explain to you many times with ordinary language, and I cannot continue to do so.  Sorry you don't agree, but I have to leave it there.  You really would use your time better by reading the paper carefully, and don't worry about any statistical arguments that actually were not made in the paper.

 

Some people are stuck on the idea that we assume that people are "flipping coins".  This is not true.  Technically, you must disprove the null hypothesis to demonstrate that there is a statistically significant effect.  At this level, that is extraordinarily simple mathematics.  I will NOT discuss that further, because it is unlikely to sway the opinion of anyone who does not already understand probability and statistics.

[PASEWICZ]

I suspect that waiting for the authors of a paper to do the research that would specifically disassemble their previous research and assumptions is not going to happen in the real world, even though that is certainly the issue that should have been examined in the first place, and the current paper relegated to the gathering-background category. Nevertheless, it does appear that data within their paper contradicts their conclusion, and someone who's less invested in the current results should definitely look at that in more depth.

Mr. Darnton,

You seem to have a lot of criticisms for a lot of people about a lot of things. How about you test your theory and subject your research and conclusions to peer review?

[Carl Stross]

I don't know the authors.  I'm not even familiar with their work except for these two papers.  I read these discussions with interest, however, and joined this discussion because I was appalled and offended by some of the abusive language in the previous discussion.

 

I am also in academia, and I thought it appropriate to comment because many people misuse and misunderstand simple statistical concepts, in my view.

 

I apologize for MY shortcomings in the previous thread - I can relate to you being offended.  It is common on MN but you are right, that does not mean it is appropriate. 

Next time please, pull my sleeve the moment you feel I'm out of order.  I REALLY don't mind. 

[legenyes]

I apologize for MY shortcomings in the previous thread - I can relate to you being offended.  It is common on MN but you are right, that does not mean it is appropriate. 

Next time please, pull my sleeve the moment you feel I'm out of order.  I REALLY don't mind. 

Wow!  Good man, Carl.  By the way, I'm trying not to dominate this discussion. 

[actonern]

"...Next time please, pull my sleeve the moment you feel I'm out of order. I REALLY don't mind..."

 

If he forgets, I'll pull your finger...

[Carl Stross]

Wow!  Good man, Carl.  By the way, I'm trying not to dominate this discussion. 

 

I don't find anything wrong with somebody dominating the discussion.

[Don Noon]

Nevertheless, it does appear that data within their paper contradicts their conclusion, and someone who's less invested in the current results should definitely look at that in more depth.

 

I have no investment in the current results; in fact, I still believe there is a recognizable difference, but not perhaps as slam-dunk obvious as some believe.  I did look at this study in sufficent depth to conclude that their statistical analysis is correct, and that there is no evidence of being able to tell the difference between new and old.  Say for example that Player1 COULD and DID get his results based on ability.  Then you have the problem that all the others did WORSE than flipping a coin, and you have to explain that one.

 

Trying to read deeper into the test will run into a fundamental problem:

[James M. Jones]

My test ...analog...4,4,1,1,3,2,3,2,5,4..... uncanny....maybe I should try old vs New penny.... 

[reedman]

What a lively discussion about math. My experience is that players don't care about math, unless it relates directly to their bank account. And when professionals are trying out instruments (and that is who this study is addressing, not a HS senior who wants to upgrade from their shop fiddle), they make a pretty fast assessment about whether they like an instrument or not, and KNOW what is needed to get the instrument going, regardless of the strings, bridge height, chin rest, etc. Sure, that figures in later, but not initially. They are already constantly reacting to what they hear while they play (I know I do, and I've been playing the same instrument for almost 40 years), changing, adjusting for acoustic differences (different venue, audience/no audience, etc), temperature, humidity differences, their own physical body responses today, assessing the sound/response today (which might be different than just last night), to get the optimum sound quality desired for the moment, as well as the instruments response to all of these variables (and it's potential for response)--playing is not a static art, and players are not a mindless lot (contrary to some of the opinions voiced here that I have seen over the years). Any time spent longer than a couple of minutes is only used to confirm their initial assessment. After they narrow the choice down, that is when the adjustment of set-up starts taking place, and playing the instrument in different environments, be it a dead acoustic environment to hear the flaws, or the big hall where they can open it up and push it). All this study confirms to me is that players need to be checking out new instruments AND old instruments, because there are some great instruments being made right now. It's often easier to just go to a shop and try what they have on hand, which often runs to the older side. It is MORE work to try moderns, but there are some informal groups of players (LA and San Fran for instance, and I am sure there are many more) that do this routinely. 

 

Most individuals on audition committees make an assessment of a new players ability within the first 3-5 notes that they play. They know what they like, and they recognize what is needed to perform successfully. They also know what they like in an instrument, but it also might be influenced by what they had for breakfast (or as one respected maker replied once, players like new instruments that are similar to what they currently perform on). I don't know why THOSE variables for the players in this study has not been raised. What instrument do they currently use for performance, and how does it relate to their choice instrument in this study? And everyone should have eaten the same thing beforehand! Oh, but what if the granola supplied for their breakfast was liked by only 23% of the test subjects, and 72%  were already mad before they played because there wasn't any bacon...not a good frame of mind to be in just before a test. 

 

Jim Fellows

[legenyes]

I don't think that the inability to show the effects that many of you want to see is due to any shortcoming of the test.  I'm not entirely sure of this, but I suspect that a LOT more data would be required to demonstrate any effect that's there.

 

While you are pondering that, have a look at what the paper actually says about the differences between new and old violins.  Good luck designing the next test, folks.  I'll be there to listen if you want.

[Michael Darnton]

Mr. Darnton,

You seem to have a lot of criticisms for a lot of people about a lot of things. How about you test your theory and subject your research and conclusions to peer review?

Maybe because as someone with a position in each camp, and an interest in serving the needs of musicians I don't actually have an agenda beyond them getting what they want to have, whatever that is. Because they don't give a rat's ass about my opinion.

[Michael Darnton]

(I know I do, and I've been playing the same instrument for almost 40 years)

With due respect to the good parts of your post, I think this is a problem that you aren't aware of: that you know how to approach one instrument well doesn't extend to all instruments. This is not coming from me, but from many, many people who have bought new instruments that have different capabilities from their own.

[PASEWICZ]

Maybe because as someone with a position in each camp, and an interest in serving the needs of musicians I don't actually have an agenda beyond them getting what they want to have, whatever that is.

Now this is certainly up for peer review.

[Carl Stross]

I don't think that the inability to show the effects that many of you want to see is due to any shortcoming of the test.  

 

 

I have no investment in the current results; in fact, I still believe there is a recognizable difference, but not perhaps as slam-dunk obvious as some believe.  I did look at this study in sufficent depth to conclude that their statistical analysis is correct, and that there is no evidence of being able to tell the difference between new and old.  Say for example that Player1 COULD and DID get his results based on ability.  Then you have the problem that all the others did WORSE than flipping a coin, and you have to explain that one.

 

 

 

It is slum-dunk.

 

If I come to you with the claim that I can guess which way a coin lands and I do that 5 times and miss once, no amount of misses from other people on the street will influence or invalidate  MY result. And while I already beat the odds by a long shot, if you want more certainty, the ONLY thing you can do to get that is to test ME.   I don't have to "explain that one" because the ability of guessing coins and Strads has not been shown to be

uniformly distributed amongst humans. ( but actually, I just explained it )

[Carl Stross]

Or, you can not conclude that I do not posses perfect pitch because between myself and ten others, we AS A GROUP, don't do much better than average. I can organize tests like that all day long and come up with anything I please.

And it happens ALL THE TIME.

[Michael Darnton]

Now this is certainly up for peer review.

Frankly, I'm a lot more interested in customers' reviews. As regards opinions of the "peers", don't much give a r....... well, you get the point. . .

[PASEWICZ]

Frankly, I'm a lot more interested in customers' reviews. As regards opinions of the "peers", don't much give a r....... well, you get the point. . .

And yet you continue to criticize these things and people on the making forum...all about the customer?

[Don Noon]

Carl,

No amount of ranting or armwaving will change the facts of testing and probilities.  The paper is correct in that regard; peer review would have caught an error in a flash.  I have had to deal with testing, probabilities, and confidence levels quite often in the aerospace biz, so I have complete confidence that you and others who argue similarly are reading stuff into the test that IS ABSOLUTELY NOT SUPPORTED BY THE DATA.

 

If you want to hold onto your belief that there is a definite audible difference between old and new (as do I), fine... the test results do not prove otherwise.  If you want to believe that Player1 can really tell the difference, fine... the test and the math do not prove otherwise.  But you can not use the test data to support an assertion that Player1 really, truely can tell the difference.

[lFred]

 Dear Legenyes

 

I stand by my understanding of the table :

Instruction given to the player states  "Imagine you are looking for a violin to replace your own for recitals and concerto performances in an upcoming tour. We will present you with a variety of instruments for evaluation. Please use your regular bow, and a shoulder-rest if you use one. And please feel free to comment out loud about the selection process"

=> when an instrument get a White rectangle (4 points) each time is was the preferred instruments of one of the violinist in the session, therefore in session 1,  old instruments were preferred only once, and in session 2 , four time , that 400% increase. etc 

 

 

As far as the conclusion of the authors concerned.

As stated their previous study of the authors  was criticised and judged improper because of the poor venue, it was also said that 18th Italian benefit more that new violin from large hall. This time they wanted to challenge those critics.

 

The data of the 2014 study shows that venue does modifies the preferences, and shows that large hall are more beneficial (as far as the preferred instrument is concerned) to 18 century Italian. 

The representation chosen for the data tend to minimise those variations but the data are there for everyone to see decode.

The authors suggest in there conclusion (to me it means conclude) that "that meaningful testing about general preferences is possible outside a concert hall" because "eight of ten players chose their Session 2 favourites from their Session 1 top-four-lists, suggesting" . 

This is not only false but leading if not misleading.

 

 1) False:   Players 10,9,7,5,4,2,1 Chosed their S2 favourite from their S1 top'4.  that's 7 out of 10  and not 8 out of 10.

 2) leading: the different choices of preferences by the musician do changes from S1 to S2. But instead of quantifying each of those changes look at consistency within parameters (selected instrument). If you take 4 instruments that a player like, rank them in order of his preference and that rank always change between the 2 venues, which by the way is the case here! (for 10 out of 10 players) You just do not conclude that "that meaningful testing about general preferences is possible outside a concert hall" can be made.. Now  if an instrument is  rejected in the small room and almost always stayed in a large  you can conclude  that players can tell if they do not like an instrument regardless of the venue Nothing more.

 

"There is certainly no evidence here to support the belief that Old Italian violins come into their own in concert halls, while new ones fall behind." 

 

For 18th century Italians there is a 400% increase in choice between session 1 and Session 2

                                      there is a 200% increase in number of instruments chosen between session 1 and Session 2 

 

For new instruments     there is a 33% decrease in choice between session 1 and Session 2

                                     there is a 40% decrease of number of chose instruments between session 1 and Session 2

 

If "These results confirm and extend those of the earlier study, and present a striking challenge to near-canonical beliefs about Old Italian violins."  Old Italian violins,might not just be the only one tainted by beliefs.

[lFred]

Don Noon,

peer review would have caught an error in a flash

 

 

page 8:eight of ten players chose their Session 2 favourites from their Session 1 top-four-lists   etc..

 

page 17:(raw data)

player 3 ( S1_top4= N7, O6, N2, N11 |  S2_top1 = O1)

player 6(S1_top4= N10, O6, N7, N9 | S2_top1= N5)

player 8(S1_top4= N11, N10, N7, N2  |  S2_top1=O1)

 

 

In a flash 

[Carl Stross]

Carl,

No amount of ranting or armwaving will change the facts of testing and probilities.  The paper is correct in that regard; peer review would have caught an error in a flash.  I have had to deal with testing, probabilities, and confidence levels quite often in the aerospace biz, so I have complete confidence that you and others who argue similarly are reading stuff into the test that IS ABSOLUTELY NOT SUPPORTED BY THE DATA.

 

If you want to hold onto your belief that there is a definite audible difference between old and new (as do I), fine... the test results do not prove otherwise.  If you want to believe that Player1 can really tell the difference, fine... the test and the math do not prove otherwise.  But you can not use the test data to support an assertion that Player1 really, truely can tell the difference.

 

Don,

It's not me doing the ranting and arm waving.  Please read my posts and if you can, prove me wrong. I wish you good luck.

 

But : "The paper is correct in this regard:peer review would have caught the error in a flash" is not an argument.

We should be better than that.

 

If you want me to explain again, all you have to do is ask.

 

I just had an idea : how would you want the data to look in order for player1 to be able to "tell the difference" ?