Ex-Xbox Boss Reflects on "RRoD", How it Almost Killed Xbox

Ex-Xbox Boss Reflects on "RRoD", How it Almost Killed Xbox

Red Ring of Death

Ex-Xbox Boss Peter Moore says that the Xbox 360's "Red Ring of Death" cost the company over a billion dollars.

The "Red Ring of Death" is somewhat of a legend among the gaming community, and refers to the unreasonably high failure rates of Xbox 360 consoles that no-one could figure out. Back in the day, it was not a question of would your console fail, but when. Ex-Xbox boss Peter Moore recently reflected on this issue to IGN, explaining that it cost the company over a billion dollars and almost killed the Xbox brand entirely.

"If we don't [address this problem], this brand is dead," Moore told Steve Balmer, the then-head of Microsoft. "It was sickening. I was doing a lot of interviews. It's not like today with social media, which would have been horrific. We couldn't figure it out." Moore says Microsoft initially thought the problem was due to a heating issue, and changed the solders in newer consoles, but was horrified to learn that it still wasn't doing the trick.

"I remember going to Robbie Bach, my boss, and saying, I think we could have a billion dollar problem here," Moore recalls. "As we started to do the analysis of what was going on, we were getting the defectors in, it was a challenging problem for our engineers, and we couldn't quite figure out what it was. We knew it was heat related. There were all kinds of fixes. I remember people putting wet towels around the box."

Moore talks of his nail-biting meeting with Balmer, where he and Bach gave him the cost estimation of their solution - which involved taking back every defective console, fixing them, and FedEx-ing them back with overnight priority mail.

"He said, 'what's it going to cost?' I remember taking a deep breath, looking at Robbie, and saying, 'we think it's $1.15bn, Steve.' He said, 'do it.' There was no hesitation."

Moore said that if Balmer had balked at the price, the Xbox brand would be in a very different place today, and the Xbox One would probably never have been made.

Source: IGN

Permalink

Smart move by Balmer, if Microsoft had of not gone out of their way to deal with RROD then the Xbox and even Microsoft brand would have been suffered far more perminant damage and the fallout would have been worse than it was.

It's not too often that I hear of positive things associated with Balmer.

As someone who suffered through two RRoD on my Xbox 360, I'm glad MS took this issue seriously. If they didn't, then their place in the console race would be over, and done with.

It's still hard to believe such a problem existed in the first place, I mean yes hardware problems are common, but the RRoD was all over the place, and costed this company a billion dollars.

Let hope a similar problem won't ever happen again to any console in the future.

Go big or go home. He went big. It worked.

RicoADF:
Smart move by Balmer, if Microsoft had of not gone out of their way to deal with RROD then the Xbox and even Microsoft brand would have been suffered far more perminant damage and the fallout would have been worse than it was.

Think about were the Game Industry would be if that happened. With Xbox in the toilet a lot of people would have had to deal with the PS3's cell processing bullshit, PC Gaming's dead market or trying to get things to work on the Wii. A lot of people would be out of the industry.

And their competitors forever hated their far-sighted spending to fix the problem.

Did they ever sort out exactly what was causing all the failures? I mean, Microsoft isn't a neophyte in the hardware sector, but to see so many units have problems speaks of a serious design flaw.

The Rogue Wolf:
Did they ever sort out exactly what was causing all the failures? I mean, Microsoft isn't a neophyte in the hardware sector, but to see so many units have problems speaks of a serious design flaw.

If i remember right, it had to do with the mounting solution they used to mount the cooler to the motherboard.
When the heatsink got really hot, it caused it to start bending. And because of the way the cooler was mounted, that also caused the motherboard to bend. Which in turn broke solder joints.

Their solution was a new spring mounting system which allowed for more flexing.

"He said, 'what's it going to cost?' I remember taking a deep breath, looking at Robbie, and saying, 'we think it's $1.15bn, Steve.' He said, 'do it.' There was no hesitation."

It's rare that I read something and stop to think "Holy shit!" in a positive light. This was one of those moments, especially considering who it came from. More importantly, I am completely overcome by the fact that someone acknowledged a mistake they made and wanted to fix it despite the costs.

Of course this all happened after a bunch of denial like everything else when it comes to this sort of thing, but in the end this was still a great move that paid off.

ParasiteX:

The Rogue Wolf:
Did they ever sort out exactly what was causing all the failures? I mean, Microsoft isn't a neophyte in the hardware sector, but to see so many units have problems speaks of a serious design flaw.

If i remember right, it had to do with the mounting solution they used to mount the cooler to the motherboard.
When the heatsink got really hot, it caused it to start bending. And because of the way the cooler was mounted, that also caused the motherboard to bend. Which in turn broke solder joints.

Their solution was a new spring mounting system which allowed for more flexing.

Apparently they used a lead-free solder which caused hairline cracks. The multiple instances of heating and cooling only made it more brittle and ultimately caused many units to fail. As they said in the article they tried changing solders and it still failed, but ultimately it came down to poor heat dissipation and the wrong type of solder.

The brand is dead in my world. In the end, I had to replace three 360s to continue playing my game collection for it. Never again, Microsoft, never again.

Knowingly releasing a faulty system SHOULD cost them money. They almost got beaten though in "Biggest fuck you" to consumers by a company later on but I think it's probably even.

EndlessSporadic:

ParasiteX:

The Rogue Wolf:
Did they ever sort out exactly what was causing all the failures? I mean, Microsoft isn't a neophyte in the hardware sector, but to see so many units have problems speaks of a serious design flaw.

If i remember right, it had to do with the mounting solution they used to mount the cooler to the motherboard.
When the heatsink got really hot, it caused it to start bending. And because of the way the cooler was mounted, that also caused the motherboard to bend. Which in turn broke solder joints.

Their solution was a new spring mounting system which allowed for more flexing.

Apparently they used a lead-free solder which caused hairline cracks. The multiple instances of heating and cooling only made it more brittle and ultimately caused many units to fail. As they said in the article they tried changing solders and it still failed, but ultimately it came down to poor heat dissipation and the wrong type of solder.

It was a bit of both. The original design called for a more flexible x-clamp but the company that manufactured the public consumption 360's went with a more rigid clamp. That paired with the heat dissipation issues and the solder choice made for a 3-stage failure. It may have been cost-cutting by Microsoft directly or the manufacturer just pocketing more cash for less expensive materials, the situation is murky at best but having Ballmer directly take action after being advised is a cool thing. Also MS did repair most Xbox's that failed for free, including sending a postage paid box. My wife and I owned an original xbox that failed for other reasons and Microsoft still sent us a free repair, so they gained a huge plus in my book.
In my view, companies can screw up but its how they handle said screw ups is what makes or breaks them. Microsoft screwed up, whether it was a cost-cutting gaff or the manufacturer's fault I don't have the information from reliable sources to say, but they fixed it for free and didn't require people to buy a new system. They also changed the design of newer releases to have better heat dissipation and flexible clamps. We had an Elite (the black box) and it never failed in the 4+ years we had it, and that thing got major usage.

Well it's not like it was a first really. I still remember some the original xbox there were some editions who had reported fires breaking out in the box, when it was constantly connected to power.
Then they just send a better powercable that could be turned off and on so the box would only get power when it needed it.
That also worked, my old xbox is still running without problems. Though i still haven't figured out how they knew i had an xbox. They didn't even adress the package with the new powercable to my mother who bought it, but directly to me.

As annoying as it was to have my launch bought Xbox 360 die within 6 months the way Microsoft handled did take the sting out of out, it happened at the weekend and I called them. Monday a courier bought a box, came back on the Tuesday to collect the stripped down (HHD taken out etc) and packaged XBox away and then on Thursday dropped another new/refurbished Xbox 360 off.

A fairly fast and painless process as these things go, so I will give them that at least.

Just shows you how bad it was. No console I've ever owned has failed, and some of them are close to 20 years old.
Some have a tiny bit become glitchy, (usually more so individual games though), but none have ever failed.

A PC I had did fail. It came from a period where a very large batch of capacitors being used for many motherboards were faulty.

Also had a few bits of cheap chinese 'off brand' devices fail on me... But that's different in general.

This must have happened after I dropped the Xbox brand. I had 3 consoles RRoD on me in the span of 6 months and needless to say I converted back the PS brand and didn't look back.

Given the 360's longevity and the One's success, the move payed off even if they didn't make the money back. I still remember my friend's 360 RRoD'ing which I think was very shortly after I got mine, which was a scary sight to say the least.

CrystalShadow:
Just shows you how bad it was. No console I've ever owned has failed, and some of them are close to 20 years old.
Some have a tiny bit become glitchy, (usually more so individual games though), but none have ever failed.

A PC I had did fail. It came from a period where a very large batch of capacitors being used for many motherboards were faulty.

Also had a few bits of cheap chinese 'off brand' devices fail on me... But that's different in general.

Just curious about the PC with the faulty capacitors. Was it a Dell?

Frezzato:

CrystalShadow:
Just shows you how bad it was. No console I've ever owned has failed, and some of them are close to 20 years old.
Some have a tiny bit become glitchy, (usually more so individual games though), but none have ever failed.

A PC I had did fail. It came from a period where a very large batch of capacitors being used for many motherboards were faulty.

Also had a few bits of cheap chinese 'off brand' devices fail on me... But that's different in general.

Just curious about the PC with the faulty capacitors. Was it a Dell?

Nah. Custom build. Lost 2 different mainboards that day within about 2 years. Can't remember what manufacturer it was.
I know the first system was an AMD duron system, while the second was a Celeron from the era when slot 1 pentium 3 systems were a thing...

Fairly sure the Mainboards were different brands too.
This was a rather widespread issue. Lots of bad power capacitors.

Adverising these days pointing out the use of 'high quality solid state power capacitors' (or something similar) can probably be traced to the widespread problems back then.

The Rogue Wolf:
Did they ever sort out exactly what was causing all the failures? I mean, Microsoft isn't a neophyte in the hardware sector, but to see so many units have problems speaks of a serious design flaw.

The Xbox 360 Slim solved the issue, by totally redesigning the main board, GPU and processor.

Microsoft never admitted specifics, but the consensus is that Microsoft designed the 360's GPU chip in house instead of going to Nvidia or ATI. Either through inexperience or cheapness Microsoft did not properly test the chip's ability to cope with thermal stress and they also didn't get their manufacturing up to the kind of standard the big two employ for their chips, it's been alleged that in early production as much as two thirds of chipsets made were defective.

The end result of that was that the GPU of the 360 ran extremely hot, so hot that it would bend under the stress and pull itself out of it's soldered joints to the main board over time, leading to the RRoD. The various minor redesigns of the 360 never addressed the GPU design for whatever reason until the 360-S came out, the 360-S was essentially an all new console.

I don't remember any of that "they took it back and repaired it free of charge" part, I remember peoples early-adopted 360 breaking just out of warranty and them just getting a new one or people binge-buying 360s as they were going through each.
People were actually putting up with that.

$1 billion, huh? Yeah, a 30% failure rate will do that to ya.

My launch-day 360 is still working fine, and I took horrible care of it.
I sometimes wonder if this RROD is a joke I'm not privy to along the lines of the drop bears.

I've left this bad boy on for weeks at a time, tripped on it, sat it on carpet for the majority of its life and still no issue. Same as the original Xbox. But those things were probably ready to be used as armor plating by the military.

loa:
I don't remember any of that "they took it back and repaired it free of charge" part, I remember peoples early-adopted 360 breaking just out of warranty and them just getting a new one or people binge-buying 360s as they were going through each.
People were actually putting up with that.

http://www.gamespot.com/articles/microsoft-extending-360-warranty-to-three-years/1100-6173633/

My flatmate went through seven, yes SEVEN! machines... and yet he still forgave them and now is on an XBONE they got it right somewhere as there fanboys are damned loyal beasts.

Seen more people jump ship this genreation over the microsoft BS than over the ring of death.(I've worked in game retail for more than 10 years). I think it is probally the social media age we now live in as was said above, the ring would've destroyed them if it had happened this genration.
It was silly when I worked the launch week, we had a wall covered in machines out back but within a week we had a wall of faulties and were running short of machines to swap for faulties let alone have any to sell, it was hell.

Alweays felt sorry for Sony they made a better machine in every way, just a shame it cost way too much to make and sell at the time. But i'm a pc gamer that only bacause I work in game retail own a very unloved ps3 and ps4 oh and a 3DS(gets the most use).

Consoles are odd, never sure how i feel about them really maybe they will just turn into true hybrid pc in the next generation or stagger on as a simple easy to use gaming platform they should always have been for those who can't cope with pc gaming's ever complex, improving, advancing technolgy and cost BS. I Love pc gaming but the rig i'm going to have to build for Star Citizen is going to bankrupt me.

LordMonty:
My flatmate went through seven, yes SEVEN! machines... and yet he still forgave them and now is on an XBONE they got it right somewhere as there fanboys are damned loyal beasts.

Seen more people jump ship this genreation over the microsoft BS than over the ring of death.(I've worked in game retail for more than 10 years). I think it is probally the social media age we now live in as was said above, the ring would've destroyed them if it had happened this genration.
It was silly when I worked the launch week, we had a wall covered in machines out back but within a week we had a wall of faulties and were running short of machines to swap for faulties let alone have any to sell, it was hell.

Alweays felt sorry for Sony they made a better machine in every way, just a shame it cost way too much to make and sell at the time. But i'm a pc gamer that only bacause I work in game retail own a very unloved ps3 and ps4 oh and a 3DS(gets the most use).

Consoles are odd, never sure how i feel about them really maybe they will just turn into true hybrid pc in the next generation or stagger on as a simple easy to use gaming platform they should always have been for those who can't cope with pc gaming's ever complex, improving, advancing technolgy and cost BS. I Love pc gaming but the rig i'm going to have to build for Star Citizen is going to bankrupt me.

Well, to be quite fair, this time, the Xbox One is now built like a ****ing tank. I saw an Xbox One getting burnt with Gasoline 2-3 times and it STILL ran. And THEN,

In case you don't want to watch the video, they dropped the Xbox One from a massive height and it STILL ran. Guy was up and playing games on it in seconds just after the drop.

And also, the PS3 was so NOT a better machine in every way. For one, dat Cell processor was effing hard to design games for. All that power was locked behind its arcane code. And the UI was not as good at all as the Xbox 360's. And on top of that, although it may seem like a small thing, you can play music when playing games on the 360. With the PS3, it's either one or the other.

no one knew why it was happening?! people used WET towels?! bullshit, ill tell you exactly why it happened. if you crack open your 360 and look at the backside of the motherboard you'll see these things that look like pink foam squares. they're supposedly heat syncs, and what happened is they melted because all the things generating heat had copper syncs running to the tiny little fan in the back. this was horribly insufficient in terms of heat management. so those little pink squares melted all over the board and jacked up your 360.

the solution? peel the pink squares off and replace them with two pennies sealed in electrical tape. then towel trick your 360 into thinking the problem has been solved (because it has).

a couple of pennies solved the problem. just throwing that out there

"...the unreasonably high failure rates of Xbox 360 consoles that no-one could figure out".

Nobody except the xbox insiders who came out and confirmed that the 360 was knowingly rushed to market with a high failure rate because Microsoft attributed the success of the PS2 over the original xbox with being first to market.

I went through seven of these things. They will never get my business again. Ever.

What a bunch of self-serving, lying crap. My 360 was the last MS product I'll ever purchase.

I got a 360 a month or 2 after they came out, still working to this day. I consider myself very lucky :)

 

Reply to Thread

Posting on this forum is disabled.