Brand reliability--Perception vs reality

Post by Elmo P. Shagnasty
http://consumerist.com/5127337/the-10-most-reliable-car-brands-vs-your-pr
econceived-notions
Brand Perception Top 10
1. Toyota
2. Honda
3. Ford
4. Cadillac
5. Mercedes-Benz
6. GMC
7. Lexus
8. BMW
9. Chevrolet
10. Volvo
Brand Reliability Top 10
1. Scion
2. Acura
3. Honda
4. Toyota
5. Lexus
6. Infinity
7. Subaru
8. Hyundai
9. Mitsubishi
10. Kia
yeah. That's right. The numbers show the reality.

infinity but no nissan?

e.meyer

2009-01-12 15:42:08 UTC

Post by jim beam

http://consumerist.com/5127337/the-10-most-reliable-car-brands-vs-you...
econceived-notions
Brand Perception ‹ Top 10
1. Toyota
2. Honda
3. Ford
4. Cadillac
5. Mercedes-Benz
6. GMC
7. Lexus
8. BMW
9. Chevrolet
10. Volvo
Brand Reliability ‹ Top 10
1. Scion
2. Acura
3. Honda
4. Toyota
5. Lexus
6. Infinity
7. Subaru
8. Hyundai
9. Mitsubishi
10. Kia
yeah. That's right. The numbers show the reality.

infinity but no nissan?

Nissan is still plagued by the Mississippi built Pathfinders &
Armadas. They bring down the whole name.

Mike Hunter

2009-01-12 20:45:05 UTC

At least among it 320,000 subscribers most of whom are reported to be import
buyers, not the average of the 16,000,000 buyers that bought NEW cars in
each of the past five years.

I wonder what the results would be if CR sampled the extra one to three
million buyers annually, that had GM being that far ahead of any other
manufactures in sales annually?

h***@hairdresser.net

2009-01-13 02:23:15 UTC

Post by Mike Hunter
At least among it 320,000 subscribers most of whom are reported to be import
buyers, not the average of the 16,000,000 buyers that bought NEW cars in
each of the past five years.
I wonder what the results would be if CR sampled the extra one to three
million buyers annually, that had GM being that far ahead of any other
manufactures in sales annually?

Aren't you familiar with statistical sampling theory?

C. E. White

2009-01-13 12:59:50 UTC

Post by h***@hairdresser.net

Aren't you familiar with statistical sampling theory?

I am not an expert, but I don't think you should consider a self
selected subgroup of a self selected group to be a valid sample
(people who choose to subscribe to CR who choose to respond to the CR
survey).

Ed

Michael Pardee

2009-01-14 03:35:20 UTC

I am not an expert, but I don't think you should consider a self selected
subgroup of a self selected group to be a valid sample (people who choose
to subscribe to CR who choose to respond to the CR survey).
Ed

That's been a complaint I have had about CR's "polling" for a very long
time. Their surveys are useless.

Mike

Mike Hunter

2009-01-14 22:32:34 UTC

I wonder how many of the 16,000,000 buyers the bought vehicles in 2006, are
among CR's reported 320,000 subscribers and the percentage of those 320,000
subscribers respond to their surveys?

I am not an expert, but I don't think you should consider a self selected
subgroup of a self selected group to be a valid sample (people who choose
to subscribe to CR who choose to respond to the CR survey).
Ed

That's been a complaint I have had about CR's "polling" for a very long
time. Their surveys are useless.
Mike

SMS

2009-01-14 23:48:13 UTC

I am not an expert, but I don't think you should consider a self selected
subgroup of a self selected group to be a valid sample (people who choose
to subscribe to CR who choose to respond to the CR survey).

For the perception part of it you're correct. But for the reliability
side, the owners are simply filling out the survey for the vehicles they
own. Unless you believe a Toyota or Honda owner is less likely to put
down actual problems in the survey than a Ford or GM owner, the surveys
are statistically sound. Who knows, maybe a Toyota or Honda owner has
much higher expectations and would complain more about defects than a
Ford or GM owner that expects more problems.

Michael Pardee

2009-01-15 03:14:40 UTC

Post by C. E. White
I am not an expert, but I don't think you should consider a self
selected subgroup of a self selected group to be a valid sample (people
who choose to subscribe to CR who choose to respond to the CR survey).

For the perception part of it you're correct. But for the reliability
side, the owners are simply filling out the survey for the vehicles they
own. Unless you believe a Toyota or Honda owner is less likely to put down
actual problems in the survey than a Ford or GM owner, the surveys are
statistically sound. Who knows, maybe a Toyota or Honda owner has much
higher expectations and would complain more about defects than a Ford or
GM owner that expects more problems.

You see the problem - that the group is self-selected and therefore
statistically unsound. We can't arbitrarily say there are no differences
that would cause a Toyota or Honda owner to have different motivations than
a Ford or GM owner, especially since we hear from Japanese or American brand
partisans so frequently. In the same way we can't say just what the effect
of those passions and prejudices are. The results would also vary depending
on when the owner bought the car. Was it brand new, in its second year or in
its fourth year? The owner experience and the gist of what he reports will
depend on that - particularly whether he owned it during its first year of
life - to a great extent but it is not included in CR's methodology.

The surveys are also susceptible to "gaming": if a 2004 Malibu (for example)
owner wanted to improve the market value of the car he is planning to sell,
he can send in very many responses claiming to be another perfectly
satisfied owner even if his car was actually a lemon. Similarly, if he
wanted to buy a 2008 Camry he could flood CR with reports that he had
nothing but trouble with his (fictional) 2008 Camry in hopes of driving the
market price down.

No matter how you slice it, CR surveys are a textbook example of sample
selection problems. Any resemblance to typical owner experience is
coincidental.

Mike

SMS

2009-01-15 09:55:09 UTC

For the perception part of it you're correct. But for the reliability
side, the owners are simply filling out the survey for the vehicles they
own. Unless you believe a Toyota or Honda owner is less likely to put down
actual problems in the survey than a Ford or GM owner, the surveys are
statistically sound. Who knows, maybe a Toyota or Honda owner has much
higher expectations and would complain more about defects than a Ford or
GM owner that expects more problems.

It all balances out with such a huge sample. It's not a double-blind
study, but you'd never get that. Even if you sampled randomly you'd
still have the different motivations of the different owners.

It's not perfect, but it's a very large sample with a very small margin
of error.

Post by Michael Pardee
In the same way we can't say just what the effect
of those passions and prejudices are. The results would also vary depending
on when the owner bought the car. Was it brand new, in its second year or in
its fourth year?

The CR subscriber base tends to be higher income and more highly
educated. It's unlikely that they'd be buying used cars.

Michael Pardee

2009-01-15 13:00:03 UTC

Post by Michael Pardee
You see the problem - that the group is self-selected and therefore
statistically unsound.We can't arbitrarily say there are no differences
that would cause a Toyota or Honda owner to have different motivations
than a Ford or GM owner, especially since we hear from Japanese or
American brand partisans so frequently.

It all balances out with such a huge sample. It's not a double-blind
study, but you'd never get that. Even if you sampled randomly you'd still
have the different motivations of the different owners.

No - the problem increases with larger samples. In fact, the poor sample
selection makes the size of the sample totally irrelevant. For example, if
you are doing a survey of luxury homes built by premiere architects and are
collecting your data by asking people who pass you at the exit of WalMart,
more answers only mean more useless responses. There is no cure for sample
selection errors.

Post by SMS
It's not perfect, but it's a very large sample with a very small margin of
error.

It's a sample with an indefinite margin of error. When samples are properly
selected the margin of error is a simple calculation, but when the samples
are contaminated the margin of error can't be calculated. With a properly
selected sample set the margin of error drops like a rock with more samples
and it is rare that more than a hundred samples are necessary, but when the
sample selection process is flawed the margin of error is unpredictable
after the first sample is collected and never improves.

Post by Michael Pardee
In the same way we can't say just what the effect of those passions and
prejudices are. The results would also vary depending on when the owner
bought the car. Was it brand new, in its second year or in its fourth
year?

The CR subscriber base tends to be higher income and more highly educated.
It's unlikely that they'd be buying used cars.

You have data on those assertions, perhaps? I gave up on CR when I was about
20 because they often incorporated their stupid ideas into their reviews; as
an avid cyclist their shocking recommendation of "dual action brake levers"
on bicycles (now a thing of the past - the current high end devices of the
same name are completely different from the dangerous levers of the 70s CR
was touting) tore it for me. I have six figures on my W2 again this year and
have worked in high tech jobs since I was 17, when I put my first class FCC
radiotelephone license to use. I have only recently bought my third new car
ever and have bought 11 used cars that I can recall. New cars are a blind
wager and the depreciation is not the sort of thing most intelligent people
welcome... unless they are so rich they don't care how much trouble a car is
anyway. Of course the CR readers (maybe not the ones who respond to the
surveys, but who knows?) could still be highly educated; intelligence and
education are distinct concepts.

Mike

Mike Hunter

2009-01-15 17:00:49 UTC

Where did you obtain that statistic? Are you referring to the type of
buyers that chose their vehicles like they choose their refrigerators and
toaster etc. or car guys that read Road & Track, Car & Driver, Auto Week an
Motor Trend to help them to determine what to buy?

Post by Michael Pardee
In the same way we can't say just what the effect of those passions and
prejudices are. The results would also vary depending on when the owner
bought the car. Was it brand new, in its second year or in its fourth
year?

The CR subscriber base tends to be higher income and more highly educated.
It's unlikely that they'd be buying used cars.

SMS

2009-01-15 19:21:04 UTC

Post by Mike Hunter
Where did you obtain that statistic?

Consumer Reports has done surveys of their subscribers and describe the
typical subscriber as "relatively affluent and well-educated."

Are you referring to the type of

Post by Mike Hunter
buyers that chose their vehicles like they choose their refrigerators and
toaster etc. or car guys that read Road & Track, Car & Driver, Auto Week an
Motor Trend to help them to determine what to buy?

Consumer Reports subscribers value reliability and safety. They are less
concerned about 0-60 in four seconds versus seven seconds.

Dave Kelsen

2009-01-15 12:21:34 UTC

For the perception part of it you're correct. But for the reliability
side, the owners are simply filling out the survey for the vehicles they
own. Unless you believe a Toyota or Honda owner is less likely to put down
actual problems in the survey than a Ford or GM owner, the surveys are
statistically sound. Who knows, maybe a Toyota or Honda owner has much
higher expectations and would complain more about defects than a Ford or
GM owner that expects more problems.

You see the problem - that the group is self-selected and therefore
statistically unsound. We can't arbitrarily say there are no differences
that would cause a Toyota or Honda owner to have different motivations than
a Ford or GM owner, especially since we hear from Japanese or American brand
partisans so frequently. In the same way we can't say just what the effect
of those passions and prejudices are. The results would also vary depending
on when the owner bought the car. Was it brand new, in its second year or in
its fourth year? The owner experience and the gist of what he reports will
depend on that - particularly whether he owned it during its first year of
life - to a great extent but it is not included in CR's methodology.
The surveys are also susceptible to "gaming": if a 2004 Malibu (for example)
owner wanted to improve the market value of the car he is planning to sell,
he can send in very many responses claiming to be another perfectly
satisfied owner even if his car was actually a lemon. Similarly, if he
wanted to buy a 2008 Camry he could flood CR with reports that he had
nothing but trouble with his (fictional) 2008 Camry in hopes of driving the
market price down.

In addition to the responses SMS gave you, Michael, your last paragraph
is incorrect. As I recall, each subscriber only gets one survey.

The factors which can (not will, mind you, but can) skew results are
reasonably well understood, and generally apply across the board to
responders. For example, CR subscribers, on average, take better care
of their vehicles than non-subscribers. This is attributable to the
higher than average level of education, as is their higher level of
income. But since these factors apply to the general population of
subscribers (rather than, say, only the Ford owners), no inherent skew
is apprehended.

There are other factors, and some of them may present variations in the
distribution curve; for there to be a great deal of variation, though,
we would have to impute that the results were invalid. Toyotas and
Hondas are not generally more reliable than other vehicles, for example.

The observable world doesn't seem to support such a conclusion.

RFT!!!
Dave Kelsen

--
Nihilism means nothing to me.

Michael Pardee

2009-01-15 13:05:28 UTC

Post by Michael Pardee
The surveys are also susceptible to "gaming": if a 2004 Malibu (for
example) owner wanted to improve the market value of the car he is
planning to sell, he can send in very many responses claiming to be
another perfectly satisfied owner even if his car was actually a lemon.
Similarly, if he wanted to buy a 2008 Camry he could flood CR with
reports that he had nothing but trouble with his (fictional) 2008 Camry
in hopes of driving the market price down.

In addition to the responses SMS gave you, Michael, your last paragraph is
incorrect. As I recall, each subscriber only gets one survey.

Thanks for the correction, Dave. I must be behind the times - there used to
be an on-line survey that was anonymous, but I could also be mistaken that
it was CR. I withdraw that objection.

Mike

Dillon Pyron

2009-01-17 04:46:15 UTC

I am not an expert, but I don't think you should consider a self selected
subgroup of a self selected group to be a valid sample (people who choose
to subscribe to CR who choose to respond to the CR survey).
Ed

That's been a complaint I have had about CR's "polling" for a very long
time. Their surveys are useless.
Mike

That's the problem with almost all "volunteer" polls. You get two
types of people, the lovers and the haters. For two examples, go to
www.cruisecritic.com or www.tripadvisor.com

--
- dillon I am not invalid

When you wish upon a falling star, your dreams come true.
Unless it's really a meteorite hurtling to the Earth which
will destroy all life. Then you're pretty much hosed no
matter what you wish for. Unless it's death by meteor.

ACAR

2009-01-17 23:14:01 UTC

That's the problem with almost all "volunteer" polls. You get two
types of people, the lovers and the haters. For two examples, go towww.cruisecritic.comorwww.tripadvisor.com
--

which should balance out?

after all these years and all the CR data it should be pretty easy to
refute CR's frequency of repair reports with real world data.

can anyone *prove* the data are not correct?

what, no one has access to frequency of repair data collected by other
organizations?

the CR data is as good as your gonna get.
I've yet to find an independent mechanic who claimed the CR data are
not representative.
CR data may not be perfect but it's MUCH BETTER THAN NOTHING;
which is what you've got without the CR data.

Bob

2009-01-12 21:48:41 UTC

Whoah!!.... wait a sec...

Ford is perceived as reliable??

On Mon, 12 Jan 2009 08:58:29 -0500, "Elmo P. Shagnasty"

J Golden

2009-01-13 01:32:02 UTC

If you are going to include it, please spell INFINITI correctly!

Post by Bob
Whoah!!.... wait a sec...
Ford is perceived as reliable??
On Mon, 12 Jan 2009 08:58:29 -0500, "Elmo P. Shagnasty"

Hachiroku ハチロク

2009-01-13 04:11:50 UTC

But...but...a Scion is a Camry with a coupe body and Celica suspension and
brakes...

Tegger

2009-01-13 04:32:19 UTC

Post by Elmo P. Shagnasty
http://consumerist.com/5127337/the-10-most-reliable-car-brands-vs-your
-pr econceived-notions
Brand Perception Â Top 10
1. Toyota
2. Honda
3. Ford
4. Cadillac
5. Mercedes-Benz
6. GMC
7. Lexus
8. BMW
9. Chevrolet
10. Volvo
Brand Reliability Â Top 10
1. Scion
2. Acura
3. Honda
4. Toyota
5. Lexus
6. Infinity
7. Subaru
8. Hyundai
9. Mitsubishi
10. Kia
yeah. That's right. The numbers show the reality.

But...but...a Scion is a Camry with a coupe body and Celica suspension
and brakes...

Depends what Scion. Some are Yaris/Echo-based.

Still, it's passing strange that Scion is several points above Toyota's
other two brands when they all share the exact same parts. Sort of makes
one question the quality of the surveys' methods, no?

--
Tegger

St. John Smythe

2009-01-13 12:18:02 UTC

Post by Tegger
Still, it's passing strange that Scion is several points above Toyota's
other two brands when they all share the exact same parts. Sort of makes
one question the quality of the surveys' methods, no?

On the surface, yes, but consider the case of two cars with the "exact
same parts" under the skin, one marketed to people that drive like
maniacs and the other marketed to people that drive like Grandma. In
such a case, a difference in reliability wouldn't be surprising.

--
sjs

Elmo P. Shagnasty

2009-01-13 12:57:33 UTC

Ergo, the fact that Buick keep coming in tops in the JD Power and
similar ratings.

When you drive the car 6000 miles in 7 years, what do you expect?

B. Peg

2009-01-13 13:40:30 UTC

Post by Elmo P. Shagnasty
Ergo, the fact that Buick keep coming in tops in the JD Power and
similar ratings.
When you drive the car 6000 miles in 7 years, what do you expect?

LOL!

Ain't that the truth! Need to show it to my 90 year old mom who still
drives a Buick, maybe 5 miles a week. Incredible gas mileage, according to
her.

B~

Tegger

2009-01-13 13:29:55 UTC

Post by Tegger
Still, it's passing strange that Scion is several points above
Toyota's other two brands when they all share the exact same parts.
Sort of makes one question the quality of the surveys' methods, no?

That's exactly my point.

The survey results are not as valid as people think they are because the
survey makers have not controlled for owner behavior.

In other words, the survey reflects the owner as well as the car, which is
useless to me unless I'm buying a used car.

Consumer Reports reliability surveys have the same problem.

--
Tegger

The Unofficial Honda/Acura FAQ
www.tegger.com/hondafaq/

Tegger

2009-01-13 13:31:49 UTC

Post by Tegger
Consumer Reports reliability surveys have the same problem.

Sorry, I failed to realize this *IS* CR's own survey.

--
Tegger

The Unofficial Honda/Acura FAQ
www.tegger.com/hondafaq/

CharlesTheCurmudgeon

2009-01-13 15:26:00 UTC

Post by Tegger
Consumer Reports reliability surveys have the same problem.

Sorry, I failed to realize this *IS* CR's own survey.
--
Tegger
The Unofficial Honda/Acura FAQ
www.tegger.com/hondafaq/

I take CR and JDPowers with a grain of salt and usually 10 grains of asprin.
JDPower's Initial Quality survey means very little. Most cars can be made
to look good on the showroom floor. What's the car look like and act like
at 5 years or 10? Is it still responding well in traffic? How's the body
holding up after a few Chicago winters with all the metal-eating salt?
What's the engine look like after sloging through some traffic jams with the
AC on in some of our 95-degree summers? How's it doing for repairs? Mostly
scheduled maintenance, or a shop queen? Any handling quirks?

When I buy a car, I expect about 5 years out of it before I have to start
thinking about replacing it. It's not like a computer. I just need to get
from A to B, reliably.

Sir Charles the Curmudgeon

C. E. White

2009-01-13 15:49:33 UTC

Post by CharlesTheCurmudgeon
When I buy a car, I expect about 5 years out of it before I have to
start thinking about replacing it. It's not like a computer. I
just need to get from A to B, reliably.

ONLY 5 years????

I trade cars often, but not because they have problems. I just like
new cars or the type of car I want changes (I like convertibles, but
usually after three years of owning one, I decide to go back to a
sedan). However, at times I have had to hang on a car for a long time.
I won't even consider buying a car that I don't think will last 12 to
15 years and 200k miles. I have only kept four vehicles anywhere near
that long, and they were all Fords (all relatively problem free), but
I don't intend to start out buying a car with the idea it is going to
be reliable for only 5 years. I've never sold a car of my own that
wasn't in good condition at the time of the sale (but I have traded-in
a few that were horrid). For instance, when I sold my 1997 Expedition
with 150k miles, I was sure it was capable of doing another 50k miles
with no significant problems. Or when I sold by 1986 Sable to a
co-worker, I had no doubt he could drive it to at least 200k miles (he
actually exceeded that before totaling it in an accident). I have
purchased cars that after I bought them I decided were not going to be
reliable (notably a Toyota Cressida, Plymouth Reliant, Audi Coupe,
Saturn Vue, and multiple British Sports Cars) and I usually got rid of
them as soon as practical (well except for the British Sports Cars -
they were like a drug to me).

Ed

Leftie

2009-01-14 06:10:41 UTC

Post by Tegger
Still, it's passing strange that Scion is several points above
Toyota's other two brands when they all share the exact same parts.
Sort of makes one question the quality of the surveys' methods, no?

That's exactly my point.
The survey results are not as valid as people think they are because the
survey makers have not controlled for owner behavior.
In other words, the survey reflects the owner as well as the car, which is
useless to me unless I'm buying a used car.
Consumer Reports reliability surveys have the same problem.

It seems to me that you are overlooking the real reason. Scion only
makes about 4 vehicles, and they are based on solid, simple platforms.
Toyota makes many more vehicles, and there is no Scion V-6 Camry
automatic, so Toyota takes the hit for the few models it makes that are
less reliable. Pretty obvious, to me at least...

Mike Hunter

2009-01-14 22:28:19 UTC

That's true once they start selling a half million or more of a model more
of the 2%, that they all make that are not up to snuff, begin to appear.

Post by Leftie

Post by Tegger
Still, it's passing strange that Scion is several points above
Toyota's other two brands when they all share the exact same parts.
Sort of makes one question the quality of the surveys' methods, no?

That's exactly my point.
The survey results are not as valid as people think they are because the
survey makers have not controlled for owner behavior.
In other words, the survey reflects the owner as well as the car, which
is useless to me unless I'm buying a used car.
Consumer Reports reliability surveys have the same problem.

Hachiroku ハチロク

2009-01-13 21:02:18 UTC

Then why does Scion rate higher?

I don't really pound on my tC, but I don't baby it either. It has 165 HP,
and I use it to a fair advantage.

Did I ever mention this car is *FAST* ?

Mike Hunter

2009-01-13 21:21:07 UTC

Perhaps, any car can go fast but it certainly is anything but quick if one
needs to get our of the way.

Post by St. John Smythe
On the surface, yes, but consider the case of two cars with the "exact
same parts" under the skin, one marketed to people that drive like
maniacs and the other marketed to people that drive like Grandma. In
such a case, a difference in reliability wouldn't be surprising.

Then why does Scion rate higher?
I don't really pound on my tC, but I don't baby it either. It has 165 HP,
and I use it to a fair advantage.
Did I ever mention this car is *FAST* ?

C. E. White

2009-01-13 13:01:01 UTC

Post by Elmo P. Shagnasty
http://consumerist.com/5127337/the-10-most-reliable-car-brands-vs-your
-pr econceived-notions
Brand Perception Â< Top 10
1. Toyota
2. Honda
3. Ford
4. Cadillac
5. Mercedes-Benz
6. GMC
7. Lexus
8. BMW
9. Chevrolet
10. Volvo
Brand Reliability Â< Top 10
1. Scion
2. Acura
3. Honda
4. Toyota
5. Lexus
6. Infinity
7. Subaru
8. Hyundai
9. Mitsubishi
10. Kia
yeah. That's right. The numbers show the reality.

But...but...a Scion is a Camry with a coupe body and Celica
suspension
and brakes...

Depends what Scion. Some are Yaris/Echo-based.
Still, it's passing strange that Scion is several points above
Toyota's
other two brands when they all share the exact same parts. Sort of makes
one question the quality of the surveys' methods, no?

Why would a Toyota owner ever question CR opinions?

Ed

Hachiroku ハチロク

2009-01-13 21:00:27 UTC

But...but...a Scion is a Camry with a coupe body and Celica suspension
and brakes...

Depends what Scion. Some are Yaris/Echo-based.

I was referring to my tC...

One thing I was thinking about after reading this post last night: Scions
aren't pumped out in the same mass quantities as the Camry or other Toyota
units, and there might be a little more attention to detail when the cars
are rolling down the line. The one disturbing thing is that a *LOT* of tC
owners complain about rattles and squeaks, especially from the hatch.
Luckily, this is something mine does not suffer from, although there is a
nasty squeak from behind the dash when the temperature gets below 40
degrees F.

Interesting that Scion has shuch high numbers, but the Celica which was
made in limited quantities from 1999-2006 had a LOT of complaints...

Dave D

2009-01-13 08:22:57 UTC

Post by Elmo P. Shagnasty
http://consumerist.com/5127337/the-10-most-reliable-car-brands-vs-your-pr
econceived-notions
Brand Perception < Top 10
1. Toyota
2. Honda
3. Ford
4. Cadillac
5. Mercedes-Benz
6. GMC
7. Lexus
8. BMW
9. Chevrolet
10. Volvo
Brand Reliability < Top 10
1. Scion
2. Acura
3. Honda
4. Toyota
5. Lexus
6. Infinity
7. Subaru
8. Hyundai
9. Mitsubishi
10. Kia
yeah. That's right. The numbers show the reality.

To quote, "There are three kinds of lies. Lies; Damn Lies and Statistics."
Mark Twain....He was right on once more....

DaveD

C. E. White

2009-01-13 12:57:03 UTC

They are basing the "real" numbers on the bogus Consumer Reports
survey. A non-random, non-scientific, beauty poll of CR readers who
feel like responding. The results are interesting, but not
statistically significant.

Ed

tww1491

2009-01-14 21:01:25 UTC

Howing plowed through all of the posts -- or at least most of them -- I
guess the conclusion is that there are no reliable statistical data vis
"reliability." So, you go with what has worked for you over the years, I
suppose.

Michael Pardee

2009-01-15 03:56:00 UTC

Post by tww1491
Howing plowed through all of the posts -- or at least most of them -- I
guess the conclusion is that there are no reliable statistical data vis
"reliability." So, you go with what has worked for you over the years, I
suppose.

That's about all you can do for new cars. I prefer used cars, with at least
70K miles on them. 100K miles is the sweet spot; at 50K it is hard to tell
how a car has been treated and how it is holding up, but at 100K it is hard
to hide. I have developed a method that works for me... at considerable cost
over the years. There are a handful of mechanical checks I do for any used
car, taking about ten minutes, a flashlight and a pair of coveralls (plus an
ODBII reader for 1996 and later cars). But even more important is some
internet searching for complaints. Usenet, reviled as it is, is a gold mine.
I have ruled out Subarus in the year range I am interested in because of the
prevalence of head gasket and "torque bind" posts at alt.autos.subaru, and
the regulars responded very frankly when I asked what the deal was with
those problems. Good people on alt.autos.ford gave me their best skinny on
models to seek and models to avoid. These are not statistically valid, of
course, but I am not planning to write a doctoral dissertation. I want to
know what to look for. Instead of some insipid chart showing poorer than
average reliability for the engine (whatever that may mean) I know that
failure to "burp" a 2.5L Subaru engine after changing the coolant will
quickly lead to head gasket failure.

New cars are a riskier proposition. Changes from one year to another can
plunge buyers into nightmares, and if you are going to keep a car for, say,
five years the model isn't safe to buy until it has been relatively
unchanged for five years. I have owned only three new cars in my life, and
the first two (1970 Mercury Capri and 1984 Dodge 600ES) were two of the most
troublesome cars I've ever owned.

Mike

SMS

2009-01-14 23:42:55 UTC

Post by Elmo P. Shagnasty
http://consumerist.com/5127337/the-10-most-reliable-car-brands-vs-your-pr
econceived-notions
Brand Perception ‹ Top 10
1. Toyota
2. Honda
3. Ford
4. Cadillac
5. Mercedes-Benz
6. GMC
7. Lexus
8. BMW
9. Chevrolet
10. Volvo
Brand Reliability ‹ Top 10
1. Scion
2. Acura
3. Honda
4. Toyota
5. Lexus
6. Infinity
7. Subaru
8. Hyundai
9. Mitsubishi
10. Kia
yeah. That's right. The numbers show the reality.

It's interesting to see the perception, since the reliability surveys
from CR are simply the results of owners of each brand filling out a
survey about problems they've had with the vehicles. Since it's a huge
statistical sample, the CR reliability ratings are very accurate.

I think it was a good idea for CR to publish the perception list,
because I've often seen posts from people that don't understand both
statistical sampling and the survey methodology. Some people believe
that the CR reliability ratings are actually just the perceptions of the
people filling out the survey, and they don't realize that what's
being surveyed is actual owner's experiences with the vehicles _not_
what the survey takers believe is the most reliable vehicle. Similarly,
many people have no concept of statistical sampling and margins of
error, believing that if you don't survey every single owner you don't
have enough information to make reliability predictions. Separating
perception from reality should solve the first misconception, though
explaining sampling theory is more difficult.

Michael Pardee

2009-01-15 03:30:42 UTC

It's interesting to see the perception, since the reliability surveys from
CR are simply the results of owners of each brand filling out a survey
about problems they've had with the vehicles. Since it's a huge
statistical sample, the CR reliability ratings are very accurate.
I think it was a good idea for CR to publish the perception list, because
I've often seen posts from people that don't understand both statistical
sampling and the survey methodology. Some people believe that the CR
reliability ratings are actually just the perceptions of the people
filling out the survey, and they don't realize that what's being surveyed
is actual owner's experiences with the vehicles _not_ what the survey
takers believe is the most reliable vehicle. Similarly, many people have
no concept of statistical sampling and margins of error, believing that if
you don't survey every single owner you don't have enough information to
make reliability predictions. Separating perception from reality should
solve the first misconception, though explaining sampling theory is more
difficult.

The problem is that perennial bugaboo of statistics, sample selection. As
you say, a small *truly random* sample is plenty. Beginning stat textbooks
are fond of pointing out that even teh largest nation's election could be
precisely predicted by fewer than a hundred properly selected samples.
That's just way the math side of it works.

Getting a random sample is invariably the biggest challenge to any survey,
and CR doesn't even pretend to try. Self-selection - even when it is more
subtle than CR's open invitation format - will turn any survey to trash.
Many a survey has been invalidated by the mere fact that in most cases the
potential respondents can't be compelled to respond, and can't be compelled
to respond honestly. No quantity of self-selected respondents can produce
usable results. And that is the case with CR's surveys.

Mike

SMS

2009-01-15 10:09:07 UTC

Post by Michael Pardee
Getting a random sample is invariably the biggest challenge to any survey,
and CR doesn't even pretend to try. Self-selection - even when it is more
subtle than CR's open invitation format - will turn any survey to trash.
Many a survey has been invalidated by the mere fact that in most cases the
potential respondents can't be compelled to respond, and can't be compelled
to respond honestly. No quantity of self-selected respondents can produce
usable results. And that is the case with CR's surveys.

You're confusing "usable results" with the "perfect results" of a
double-blind random survey.

Unless you believe that a large number of the subscribers that respond
to the survey are intentionally lying only about certain vehicles, while
telling the truth about others, the reliability survey is in fact very
usable. You have to take it for what it is, a survey of owner's
experiences of the reliability of their vehicles, with results only
reported if a sufficient number of respondents own the vehicle in
question. If 30% of Camry owners report problems with the transmission,
and 5% of Accord or Taurus owners report similar problems, you have some
usable information. Maybe a double-blind random survey would have
slightly different percentages, but the information in the CR survey is
still valid.

It doesn't matter what the survey is, or what the source is, you always
have people that don't like the results trying to attack it if it's not
a double blind random survey. You see the same thing with the CR survey
on cellular carriers, the largest survey of its kind. Again, you have a
few subscribers of the carriers that do extremely poorly year after year
whining that the perception of Verizon of a carrier with superior
coverage is causing the Verizon subscribers to rate it highly, while the
AT&T subscribers somehow are out to bash AT&T. This is despite the fact
that every other survey from non-advertiser based organizations reports
the same results.

Then you have the people that are confused about quality versus
quantity, claiming that since McDonald's sell the most hamburgers of any
restaurant, that proves that McDonald's has the best hamburgers, and
that since GM sells more vehicles than Honda or Toyota in the U.S., that
proves that GM produces the best vehicles.

Michael Pardee

2009-01-15 14:06:08 UTC

Post by Michael Pardee
Getting a random sample is invariably the biggest challenge to any
survey, and CR doesn't even pretend to try. Self-selection - even when it
is more subtle than CR's open invitation format - will turn any survey to
trash. Many a survey has been invalidated by the mere fact that in most
cases the potential respondents can't be compelled to respond, and can't
be compelled to respond honestly. No quantity of self-selected
respondents can produce usable results. And that is the case with CR's
surveys.

You're confusing "usable results" with the "perfect results" of a
double-blind random survey.
Unless you believe that a large number of the subscribers that respond to
the survey are intentionally lying only about certain vehicles, while
telling the truth about others, the reliability survey is in fact very
usable. You have to take it for what it is, a survey of owner's
experiences of the reliability of their vehicles, with results only
reported if a sufficient number of respondents own the vehicle in
question. If 30% of Camry owners report problems with the transmission,
and 5% of Accord or Taurus owners report similar problems, you have some
usable information. Maybe a double-blind random survey would have slightly
different percentages, but the information in the CR survey is still
valid.
It doesn't matter what the survey is, or what the source is, you always
have people that don't like the results trying to attack it if it's not a
double blind random survey. You see the same thing with the CR survey on
cellular carriers, the largest survey of its kind. Again, you have a few
subscribers of the carriers that do extremely poorly year after year
whining that the perception of Verizon of a carrier with superior coverage
is causing the Verizon subscribers to rate it highly, while the AT&T
subscribers somehow are out to bash AT&T. This is despite the fact that
every other survey from non-advertiser based organizations reports the
same results.
Then you have the people that are confused about quality versus quantity,
claiming that since McDonald's sell the most hamburgers of any restaurant,
that proves that McDonald's has the best hamburgers, and that since GM
sells more vehicles than Honda or Toyota in the U.S., that proves that GM
produces the best vehicles.

You are demonstrating the worst effect of selection errors - the apparent
usability of useless results. Double blind techniques are used to prevent
reporting and collection bias when the sample has already been selected, and
that doesn't apply here. Randomness is important here, and that is where
CR's surveys are most seriously deficient.

In your example of transmission problems, we can see how that works. If ten
percent of Camry owners are interested in responding to the survey, and of
those 30% are interested because they had transmission problems while the
rest are interested because they loved their car so much or they had some
other complaint, the result would be to magnify the transmission problem
from as little as 3% in the real world to 30% in the survey results because
30% of respondents (not owners) complained about their transmissions.
Conversely, if 90% of Taurus owners felt compelled to respond because there
were daggers embedded in the driver's seat (seating comfort = poor) but only
5% of those had transmission problems, the survey results would be as you
describe... although the reality would be that the Camry transmission
failure rate was as little as half the Taurus rate rather than six times as
high. We can't even say what the actual rate was for Camrys even though we
know - as CR does not - only 10% of Camry owners responded, because we don't
know what the other 90% experienced. Did they have no complaints at all, or
did a lot of them have transmission trouble and were tired of dealing with
it? http://en.wikipedia.org/wiki/Berkson%27s_paradox

Check out http://en.wikipedia.org/wiki/Selection_bias and
http://en.wikipedia.org/wiki/Self-selection for more. In the first link, the
heading "Overcoming selection bias" warns, "In the general case, selection
biases cannot be overcome with statistical analysis of existing data
alone..."

Anyway, to each his own. When I have occasionally looked at CR's predictions
for cars I have owned I have been struck by how far off base they usually
were (although they got pretty close to my experience once!) And to give
them their due, when a problem is as pervasive as the AC and tranny failures
were for the 1993 and 1994 Volvo 850s they picked up on that. If you want to
credit them you can. Personally, I regard the CR surveys as not worth my
attention. I have access to a dartboard that is as reliable.

Mike

Michael Pardee

2009-01-15 14:26:50 UTC

Dumb Outlook express - this was a draft. Check the other post.

Michael Pardee

2009-01-15 14:25:38 UTC

Post by Michael Pardee
Getting a random sample is invariably the biggest challenge to any
survey, and CR doesn't even pretend to try. Self-selection - even when it
is more subtle than CR's open invitation format - will turn any survey to
trash. Many a survey has been invalidated by the mere fact that in most
cases the potential respondents can't be compelled to respond, and can't
be compelled to respond honestly. No quantity of self-selected
respondents can produce usable results. And that is the case with CR's
surveys.

You're confusing "usable results" with the "perfect results" of a
double-blind random survey.
Unless you believe that a large number of the subscribers that respond to
the survey are intentionally lying only about certain vehicles, while
telling the truth about others, the reliability survey is in fact very
usable. You have to take it for what it is, a survey of owner's
experiences of the reliability of their vehicles, with results only
reported if a sufficient number of respondents own the vehicle in
question. If 30% of Camry owners report problems with the transmission,
and 5% of Accord or Taurus owners report similar problems, you have some
usable information. Maybe a double-blind random survey would have slightly
different percentages, but the information in the CR survey is still
valid.
It doesn't matter what the survey is, or what the source is, you always
have people that don't like the results trying to attack it if it's not a
double blind random survey. You see the same thing with the CR survey on
cellular carriers, the largest survey of its kind. Again, you have a few
subscribers of the carriers that do extremely poorly year after year
whining that the perception of Verizon of a carrier with superior coverage
is causing the Verizon subscribers to rate it highly, while the AT&T
subscribers somehow are out to bash AT&T. This is despite the fact that
every other survey from non-advertiser based organizations reports the
same results.
Then you have the people that are confused about quality versus quantity,
claiming that since McDonald's sell the most hamburgers of any restaurant,
that proves that McDonald's has the best hamburgers, and that since GM
sells more vehicles than Honda or Toyota in the U.S., that proves that GM
produces the best vehicles.

Double blind techniques are used to prevent reporting and collection bias
when
the sample has already been selected, and that doesn't apply here.
Randomness
is important here, and that is where CR's surveys are most seriously
deficient.
Nor does it matter who complains about what survey; the real problems remain
with CR's methodology.

In your example of transmission problems, we can see how that works. If ten
percent of Camry owners are interested in responding to the survey, and of
those 30% are interested because they had transmission problems while the
rest are interested because they loved their car so much or they had some
other complaint, the result would be to magnify the transmission problem
from as little as 3% in the real world to 30% in the survey results because
30% of respondents (not owners) complained about their transmissions.
Conversely, if 90% of Taurus owners felt compelled to respond because there
were daggers embedded in the driver's seat (seating comfort = poor) but only
5% of those had transmission problems, the survey results would be as you
describe... although the reality would be that the Camry transmission
failure rate was as little as half the Taurus rate rather than six times as
high. This is an example of ascertainment bias
http://en.wikipedia.org/wiki/Ascertainment_bias
We can't even say what the actual rate was for Camrys even though we
know - as CR does not - only 10% of Camry owners responded, because we don't
know what the other 90% experienced. Did they have no complaints at all, or
did a lot of them have transmission trouble and were tired of dealing with
it? As long as we don't know why the responses were sent and the rate
of response for the various models there is no valid data to be had.

Check out http://en.wikipedia.org/wiki/Selection_bias and
http://en.wikipedia.org/wiki/Self-selection for more. In the first link, the
heading "Overcoming selection bias" warns, "In the general case, selection
biases cannot be overcome with statistical analysis of existing data
alone..."

Anyway, to each his own. When I have occasionally looked at CR's predictions
for cars I have owned I have been struck by how far off base they usually
were (although they got pretty close to my experience once!) And to give
them their due, when a problem is as pervasive as the AC and tranny failures
were for the 1993 and 1994 Volvo 850s they picked up on that. If you want to
credit them you can. Personally, I regard the CR surveys as not worth my
attention. I have access to a dartboard that is as reliable.

Mike

C. E. White

2009-01-15 14:39:02 UTC

Post by Michael Pardee
Getting a random sample is invariably the biggest challenge to any
survey, and CR doesn't even pretend to try. Self-selection - even
when it is more subtle than CR's open invitation format - will turn
any survey to trash. Many a survey has been invalidated by the mere
fact that in most cases the potential respondents can't be
compelled to respond, and can't be compelled to respond honestly.
No quantity of self-selected respondents can produce usable
results. And that is the case with CR's surveys.

You're confusing "usable results" with the "perfect results" of a
double-blind random survey.
Unless you believe that a large number of the subscribers that
respond to the survey are intentionally lying only about certain
vehicles, while telling the truth about others, the reliability
survey is in fact very usable. You have to take it for what it is, a
survey of owner's experiences of the reliability of their vehicles,
with results only reported if a sufficient number of respondents own
the vehicle in question. If 30% of Camry owners report problems with
the transmission, and 5% of Accord or Taurus owners report similar
problems, you have some usable information. Maybe a double-blind
random survey would have slightly different percentages, but the
information in the CR survey is still valid.
It doesn't matter what the survey is, or what the source is, you
always have people that don't like the results trying to attack it
if it's not a double blind random survey. You see the same thing
with the CR survey on cellular carriers, the largest survey of its
kind. Again, you have a few subscribers of the carriers that do
extremely poorly year after year whining that the perception of
Verizon of a carrier with superior coverage is causing the Verizon
subscribers to rate it highly, while the AT&T subscribers somehow
are out to bash AT&T. This is despite the fact that every other
survey from non-advertiser based organizations reports the same
results.
Then you have the people that are confused about quality versus
quantity, claiming that since McDonald's sell the most hamburgers of
any restaurant, that proves that McDonald's has the best hamburgers,
and that since GM sells more vehicles than Honda or Toyota in the
U.S., that proves that GM produces the best vehicles.

Here are the reason's I don't trust the CR auto survey:

1) CR uses a bad sampling method. Like it or not, you are sampling a
group of people who purchased CR. These people tend to agree with the
CR's editorial style / opinions / whatever. To me it is like only
asking Democrats what they think of a particular political figure. You
may get many responses, but you aren't getting the whole picture.

2) CR is not unbiased. They may not be influenced by advertising
dollars, but they clearly are not unbiased. CR readers are more likely
than non-readers to share the CR biases. I believe this influences how
people respond to the survey. I've responded to the surveys for years,
and it is my opinion that the surveys leave a lot of room for
interpretation and don't collect nearly enough information to make the
sort of dramatic claims the CR survey results portray.

3) CR's method of reporting the results deliberately hides the
information that would help me decide if it was significant. Why don't
they publish the actual number of respondents for each vehicles and
more details on the results for each vehicles. Distilling it down to a
few little circles makes it difficult to put a value on the
information.

4) I don't actually trust CR. I read it all the time, but I treat it
just like any other publication. The opinions of CR's editorial staff
often don't match my experiences. Sometimes the differences are just
honest disagreements, but sometimes I have to wonder. And it is pretty
clear that CR has deliberately monkey'd with tests in the past to make
them match their preconceived notions (just ask Suzuki or Isuzu what
they think of CR's methods). If they are willing to "adjust" tests to
make sure they get the results they desire, is it really so hard to
imagine they might adjust, massage, interpret, whatever, the
responses to their annual survey to get the results they believe to be
true?

Ed

J Golden

2009-01-16 02:53:01 UTC

I've been subscribing to CR since 1969 and have purchased and owned
several vehicles of various brands and national origins as have my
friends and family. In my opinion, CR's "frequency of repair records'
are probably the most accurate and objective items published by the
magazine, particularly in recent years. In recent years, there has
been a great deal of emphasis on "subjective feelings" regarding autos
(and other products) rather than objective statement of fact. With
today's auto lasting longer, I'd like to see the repair records
increase to 10 years +.

Jose