Fritz - Curtin Old vs New test - why I don't trust it

[Carl Stross]

Reserved for final version.

[Carl Stross]

Things I don't like about the latest Fritz / Curtin test

 

1. The selection of instruments.

 

 While the Moderns were probably as good as

one can get ( and further selected ! ), we know NOTHING about the Strads 

and it seems we'll never know because.... it's SECRET. We need to take the researchers

at their word that they were Strads and they were the good ones - we all know

perfectly well that there are really crappy Strads out there and that the Great Ones

tend to be either "at work" or resting in Museums and collections. Imagine a medical study

which assesses the efficacy of some treatment, gets stellar results, but

nobody is allowed to know if the patients were actually sick to start with. No names.

no charts. 

 

For a study which "attempts to close a perennially fruitless one ( investigation ) – 

the search for the “secrets of Stradivari.” (sic!) , this is a serious problem. 

If  Claudia Fritz" attempts to CLOSE" then she MUST compare with the best and

NOTHING else will do. No amount of comparing with the so-so will

do. 

 

2, The selection of players.

 

I'd very much like to believe that all the players

were "renowned" and "great". I am dead sure that they are all exceptionally

competent violin players. What I don't know is if they're good at selecting violins.

The researchers did not take any steps to verify that. 

That's because their "likes" are the data Claudia Fritz is analyzing, exactly like reading 

a resistance with a Wheatstone bridge. Problem is she did not calibrate

the equipment. Or read the " Operating Instructions" booklet. 

 

Let's look at the last one first : in her paper Claudia Fritz tells us that the players have 

indicated at her request, the time they believe they need to test 12 instruments

in order to select one to be used in a tour. (This used to be an activity which could take

a few weeks for the older generations but these ones are quick. ) The  slowest one

believes he needs some one and a half hours.

( NB See questionnaire after Seession1 where wiser now, some players indicate

"a few months" when asked "About how long would you need to play a violin in order

to get a good impression of its overall quality"?

 

Without any testing to see if these

times are reasonable or if this rush might not negatively affect their powers of 

discrimination, the researchers take them at their word  and allocate them

a time shorter than their ( players' ) max estimation, based on the "authors' experience".

Problem is the authors have never done that, they simply watched others doing

something else. Common sense here would dictate the researchers organize

a realistic simulation and allocate the longest needed time because if a single player

in this study misjudges due to lack of time the results may be completely 

compromised - there isn't enough of them to take any loses.

 

Now, let's look at the first one : are players good at selecting violins. Again, here

Claudia Fritz relies on her intuition - she believes they should be because they're...good

at playing violin. But lots of people are bad at selecting tools while brilliant at using

them. Sometimes one causes the other !  And here the real mess starts :

 

She believes that by asking players to select quickly a violin "they like" for a hypotetical 

tour, the "like" replaces the time/experience/competency issues. But it does not

and the "data" shows it. Take a look at fig.2, pg 9 and observe that Strads GAIN in ALL

categories in the second session. Makes one wonder what would've been the result

after a third session.  A third session was warranted to get a grip on the trend and stabilize the results.

It is clear that there is a powerful indication that as the players become more

familiar with the violins, the Strads gain i.e. time / experience is an issue. No more but no less.

But anyway, they seem to be quickly learning what's good for a career.

 

3. Old or New ?  

 

Here, the researchers ask the players to place the violins into the Old or New

categories based again, on their untested beliefs. They ask them to guess what each

violin is within 30 seconds. We don't know why 30 seconds was chosen or how this

might better player's odds. ( It absolutely must not lower them ! ).

After collecting the "data", Claudia Fritz draws a graph and concludes :

 

"Considering all guesses about all instruments, 33 were wrong, 31 right, and 5 indeterminate. 

These guesses were rather evenly divided between old and new violins (36 and 33 respectively - 

see Table 2), so the data rather clearly demonstrate the inability of the players to reliably 

guess an instrument‟s age, whether the instrument is in fact new or old. "

 

BUT IT DOESN'T ! 

 

Take a look at fig 4, pg 13.   The first player gets almost all of them, he KNOWS New from Old.

The last player also knows his stuff but has them confused. None the less, he can SEPARATE them.

Players 4,6 and 7 are clueless - they flip coins, while the reminder is not doing too bad.. You see,

even if they miss, they still win because Claudia Fritz did not establish a "rule" for New vs Old. 

She supposes they should be able to tell. They are, in their own ways.

 

And now we know that players CAN identify New or Old, only not ALL players.

But we never expected that all of them can.

( Superfluous, I know, but observe that 4,6,7 flipping coins will not affect 1's competency , whichever

that one is)

The question here is not if the players as a group can discriminate Old from New, is to find if there are

players who can. ONE is good enough. But, due to a fundamental modeling error, the study can not show

that. Accordingly, the only conclusion the researchers should've drawn is that... no conclusion can be drawn.

The "first rate soloists" could not show they can distinguish Old Italian violins from new ones at

"better than chance levels", simply because they were not given the chance.

 

 

 

 

 

 

("Great efforts have been made to explain why instruments by Stradivari and other Old Italian makers sound better than high-quality new violins, but without providing scientific evidence that this is in fact the case. Doing so requires that experienced violinists demonstrate (under double-blind conditions) both a general preference for Old Italian violins and the ability to reliably distinguish them from new ones. The current study, the second of its kind, again shows that first-rate soloists tend to prefer new instruments and are unable to distinguish old from new at better than chance levels." )

Edited April 16, 2014 by carl stross

[legenyes]

If you really think you have a point, I encourage you to publish your views as a response in PNAS, where they will get carefully considered scrutiny.  You will need to show that the results by the first and last player are statistically significant.

 

(Part of this message was removed later.) 

[Carl Stross]

... 

[legenyes]

...

[Torbjörn Zethelius]

Elmar Oliveira is one of the most experienced players with both new and old instruments. I was curious how he did in the test, but the data doesn't show who's who.

[Carl Stross]

Elmar Oliveira is one of the most experienced players with both new and old instruments. 

 

I'd agree. I've heard a superb recording with him on an Alf violin if I remember right. Fantastic instrument.  I also really liked Elmar's nicely balanced comments towards the end of the documentary. Great guy !

[Torbjörn Zethelius]

The test questioned established "truths" about old vs new instruments in a rather convincing way. I don't think they were out to prove that new instruments are better than old ones. That would be rather foolish.

[Carl Stross]

The test questioned established "truths" about old vs new instruments in a rather convincing way. I don't think they were out to prove that new instruments are better than old ones. That would be rather foolish.

 

I respectfully disagree on both counts.

[actonern]

If Carl's right... if some players in the study could tell the old vs. the new with high accuracy, well, aren't we again back to the question of discernment?  Also, is there a scientific method related reason that the instruments used in the study should not be made known after the fact?  After a medical test is completed, are the patients told who used the placebo?  (the answer is yes they are, or at least they're told if they want to know.)

[bmccarthy]

You're not very bright.

 

Not a very nice thing to say to anybody. Reminds me of one of my school teachers when I was a kid who was a bully.

[Carl Stross]

Not a very nice thing to say to anybody. Reminds me of one of my school teachers when I was a kid who was a bully.

 

True and regret it. This set me off :

 

"But try not to make it obvious that you dislike anything that you didn't do, and try not to filibuster.  Try not to be a troll."

If you check, he came out of a LONG retirement just for that.

[Michael Darnton]

The test questioned established "truths" about old vs new instruments in a rather convincing way. I don't think they were out to prove that new instruments are better than old ones. That would be rather foolish.

I think it you read the study critically you discover that they actually established truths about violinists, not violins. People have forgotten that this isn't a hard science where you put two chemicals together and they always act the same--that kind of data can't be achieved through opinons of humans. Nor can you test a physical fact through humans. For instance, today, using the method of this study I could go out on the street, pick 100 people at random, and prove that man cannot run a four-minute mile. In the average, the approach taken by this study, I think you could prove that humans can't do anything except eat, breath, and make waste.

 

The most interesting bit in the whole study was the one player who picked all of his selected violins and they were the same type ((new or old--we weren't told). There, apparently, is the person who gives lie to the supposed fact that people can't tell new from old. . . . and to disprove the lie only takes one instance to the contrary, doesn't it? At this point, I would concentrate entirely on his abilities: the lie that new from old can't be discriminated has already been tentatively demonstrated false.Working further with him would possibly give more information about how new and old differ. This was THE most important thing in the whole study because it appears to disprove the premise that there's no difference.

 

The second most interesting thing in the study was the horizontal bar graph showing how players became familiar with the instruments, their impression of the old ones improved. Given that in 35 years in the violin business I know of only one person who went into a shop and bought a serious violin in an hour, I have no idea why the player's word that they could all do this was taken, except that the players perceived that this wasn't a task where they would have to put out their own money and be stuck with the results of their decision. In the real world players need days or weeks for this task. While the first test "proved" that given 15 minutes in a hotel room average players would do a certain thing, this test only extended that idea out a bit farther. It still doesn't say a thing about the choices players make in the real world. One wonders how that bar graph would have looked with a third section representing two weeks' trial. Of couse we all know that, don't we? We already have a couple of hundred years of data on that. But the truth is not the "truth" that the testers wanted.

 

The last sentence in their abstract states "These results . . .  present a striking challenge to near-canonical beliefs about Old Italian violins." That's their true target, and I'd say they missed, considerably.

[not telling]

Dammit, Carl. David Burgess asked me to make sure you behave while he's gone. I will almost certainly be blamed for this.

[Carl Stross]

Dammit, Carl. David Burgess asked me to make sure you behave while he's gone. I will almost certainly be blamed for this.

 

Tell him not to worry, I'll behave. And send him my love.

[not telling]

Tell him not to worry, I'll behave. And send him my love.

Haha! You have his e-mail address, I'm sure. You can express your love to him yourself via e-mail, if you can't wait. . ; p

Now be nice! Heh

BTW I think you are right about how strangely short the allotment of time for instrument selection seems. You know more about musicians' needs than many on here, clearly. Whoever you are. Carl Stross isn't your real name, is it?

[Torbjörn Zethelius]

The most interesting bit in the whole study was the one player who picked all of his selected violins and they were the same type ((new or old--we weren't told). There, apparently, is the person who gives lie to the supposed fact that people can't tell new from old. . . . and to disprove the lie only takes one instance to the contrary, doesn't it? At this point, I would concentrate entirely on his abilities: the lie that new from old can't be discriminated has already been tentatively demonstrated false.Working further with him would possibly give more information about how new and old differ. This was THE most important thing in the whole study because it appears to disprove the premise that there's no difference.

 

It takes a lot of experience to develop this ability. It can be done (in most cases), just as telling a Strad from a Guarneri can also be done, despite what Mr. Hargrave believes. Mr. Elmar Oliveira could probably tell you more about that. 

 

There were no facts presented; only questions asked, as I understand it. Mr. Curtin have nothing to prove, does he? I mean, he's already established himself as one of the top makers. He doesn't need a longer waiting list. Just my guess...

[Michael Darnton]

 Mr. Curtin have nothing to prove, does he?

I wonder how many drug tests allow the manufacturer's representative total contact and interaction with the entire process, including administration of the drugs, full interaction with the subjects, and data crunching. Part of doing legitimate testing is preventing even the *appearance* of impropriety.

[not telling]

It only benefits Curtin to be among the researchers listed. Speculation and buzz as to which moderns were high-scoring is like indirect free advertising. He is great at marketing, for sure. Or his people are.

He may not need a longer waiting list but further justification of that list won't hurt.

[not telling]

I wonder how many drug tests allow the manufacturer's representative total contact and interaction with the entire process, including administration of the drugs, full interaction with the subjects, and data crunching. Part of doing legitimate testing is preventing even the *appearance* of impropriety.

Bad analogy, depending on your perspective.

I read a lot on this topic. Big pharma does what they want, including run their own tests, run their own statistical analysis, false comparisons of safety between two similarly unsafe drugs to prove safety of the test drug, lobbying gov't then doctors, etc. Essentially compelling everyone to accept their product as it is shuttled though the regulatory process by their own, and all the way down to individual prescribing doctors. Money will buy practically everything.

I find it fascinating.

Sorry for the non sequitur, but I believe the answer is "quite a lot"...

ETA: it is safe for everyone else to post now. I'll stay out of this like I meant to. Also, I think all involved have great intentions. I'm definitely not as cynical as I'm coming across.

[Torbjörn Zethelius]

My guess as to Mr. Curtin's involvement is that he's interested in what makes a good violin and to what extent it is psychological or not. Even the chance to participate in the test would probably make anyone excited. Could it not be that simple?

[Michael Darnton]

Bad analogy, depending on your perspective.

I read a lot on this topic. Big pharma does what they want, including run their own tests, run their own statistical analysis, false comparisons of safety between two similarly unsafe drugs to prove safety of the test drug, lobbying gov't then doctors, etc. Essentially compelling everyone to accept their product as it is shuttled though the regulatory process by their own, and all the way down to individual prescribing doctors. Money will buy practically everything.

You are right about my analogy, but as you say, this type of interference is one of the hallmarks of bad research, not good research. It's the type of research that proved tobacco, sugar, pesticides that aren't permitted anywhere except in the US, etc, were/are good for us, and why this particular project is seriously polluted. But really, there are so many problems with it that are real rather than theoretical that we don't really need to go there. There's a lot more to say, but I'm not going to say it--I think I've made my point adequately.

[Michael Darnton]

My guess as to Mr. Curtin's involvement is that he's interested in what makes a good violin and to what extent it is psychological or not. Even the chance to participate in the test would probably make anyone excited. Could it not be that simple?

A couple of hundred years of important soloists have chosen their instruments from one specific small group because those instruments are better, and everyone around them agrees. Could it not be that simple?

[Janito]

Bad analogy, depending on your perspective.

I read a lot on this topic. Big pharma does what they want, including run their own tests, run their own statistical analysis, false comparisons of safety between two similarly unsafe drugs to prove safety of the test drug, lobbying gov't then doctors, etc. Essentially compelling everyone to accept their product as it is shuttled though the regulatory process by their own, and all the way down to individual prescribing doctors. Money will buy practically everything.

 

 

Oh dear - you read what about what to arrive at what?

 

Stick to what you know first hand.

[Michael Darnton]

Janito, since you stuck your head up here, I wonder what your take is on this site: http://www.badscience.net/

Maybe that should be a side topic, but hey, you started it. :-)

 

Maybe mentioning the Monsanto Protection Act isn't out of place in the discussion, as well.

http://rt.com/usa/monsanto-bill-blunt-agriculture-006/