At GoogleGOOG+0.53%, hundreds of fans funnel hot air from the computer servers into a cooling unit to be recirculated at a Google data center in Mayes County, Okla.
A few months ago, James Weatherell updated his online resume on LinkedIn with his recent experience working with new cloud-computing software known as Open Stack. Since then, the phone of the 27-year-old computer engineer who works at the Broad Institute of Harvard and the Massachusetts Institute of Technology has been ringing off the hook.
“I get three or four calls a day from headhunters,” said Weatherell.
Weatherell’s mojo reflects the growing popularity of Open Stack, a new type of open software that more companies are using to manage their private cloud-computing systems. In December, Weatherell helped the research institute to create a proof of concept using Open Stack software and computer servers from Cisco SystemsCSCO+1.41%.
The new system, said Weatherell, is cheaper and more reliable than public cloud computing services from Amazon.comAMZN-0.05%, and also enables the institute to more easily protect its data and quickly add capacity to support the large computing needs of its researchers.
“It worked really well,” he said. “It made everyone feel this could work.”
The rise of so-called hyper-scale or high-performance computing in which companies increasingly manage thousands to hundreds of thousands of computer servers was pioneered by Google and FacebookFB-1.57%, but now most large companies and organizations are following their direction. Faced with exploding data growth and demand for computing gear to support it, Google and Facebook now build much of their own hardware, including servers that cost $1,500 apiece instead of the standard $5,000. Last year, Microsoft saMSFT+2.39%id the company had more than 1 million servers in its data centers.
It’s a shift that carries profound implications for the way technology is built and bought, the way software is developed and the future of the technology job market.
“Google figured out how to solve big problems using cheap computers,” said Mark Shuttleworth, founder of cloud computing company Canonical. “It sucked the importance out of paying for physical hardware or software that failed.”
Computer engineers like Shuttleworth and Weatherell often turn to an analogy to describe this shift. In the old style of computing, engineers say companies treated their computers like pets. They spent a lot of money buying those servers, from thousands of dollars for low-end machines, to tens of thousands of dollars for proprietary UNIX servers or hundreds of thousands for mainframes.
Every server was given a name. And given their high price, companies spent large sums of money caring and feeding these precious machines. Armies of system administrators were hired to work day and night to make sure they served up Web pages for their users. If a server broke down, companies dispatched the administrators to fix the problem and try and bring the server back to life.
“In the old world you would go in, take the top off, and test the expensive memory,” said Weatherell, a former IBMIBM+0.33% systems administrator. “You needed to be an A+ information technology guy.”
But in the new world of hyper-scale computing, engineers treat servers like cattle. It doesn’t matter if a cow dies as long as the herd survives. Servers can be treated like throwaway items because they are much cheaper than they used to be, and because companies now have software that can connect these machines and enable engineers to shift tasks automatically from one box to another without a hiccup.
“Now you shoot servers in the head and leave them in the field,” said Joshua McKenty, a former technical architect of NASA who co-founded a cloud software company based on his work there called Piston Cloud Computing. “Eventually enough die and then you swap out the whole rack.”
Mr. Weatherell said his team “does not even bother supporting our hardware now.” His Open Stack system started out with about 80 cores, or computer servers, and is growing to 1,000 to 2,000 servers in the near term. Thanks to the new software, one administrator can now manage several thousand servers, instead of just 100 in the old model, say engineers.
The rise of the new massive computer networks is extracting the value out of computing hardware and making the software and expertise that runs these machines more valuable.
Tens of thousands of computer administrators and software developers will have to retrain themselves in the new arts of cloud software or risk obsolescence. But it’s also an opportunity for engineers that leap to the new world.
“Companies that jumped into cloud early are the kinds of companies” that headhunters are targeting, said Weatherell.
TAIPEI — MediaTek not only provides manufacturers with chips for their
smartphones, but also tells how to build and run them, cutting costs as
they compete with the iPhone.
PAINFUL TEST FOR TAIWAN'S CHIPMAKERS
By Robin Kwong in Taipei , Robin Harding in Tokyo 2009-07-24
Changes in the technology sector have typically been good news for Taiwan's chipmakers.
Since the government set up Taiwan Semiconductor Manufacturing Company and United Microelectronics Company in the 1980s, the Taiwan semiconductor industry has grown to become the world's chip factory.
Navin Shenoy, general manager for Intel in Asia Pacific, says this was largely achieved through “winning in transition”.
Taiwan semiconductor groups and their computer makers “were very smart to get on board early” in the shift from desktop to notebook computing.
They held on to their lead by continuing to invest even in troubled times, such as the Asia financial crisis in the late 1990s and the bursting of the tech bubble.
But Taiwan has suffered its worst-ever falls in exports and economic performance in the past half year, and the economic crisis is testing the island's chipmakers' resilience.
Taiwan semiconductor vendors saw their revenues fall 8.7 per cent last year to $14.89bn, a steeper drop than the global average of 5.4 per cent, according to Gartner.
Among the worst hit were D-Ram memory chip groups, which had already been dealing with industry overcapacity.
Taiwan's D-Ram companies are desperately raising capital, while the government has established Taiwan Memory Company to reform the industry.
Taiwan Memory, which has chosen Japan's Elpida as partner, has said it hopes to develop more customised, higher-margin memory chips but has given few details on how it plans to do so.
Taiwan said yesterday it would extend state funds to local makers of D-Ram chips provided they could demonstrate plans to develop technology with foreign groups and propose restructuring options, such as mergers or acquisitions that would benefit the wider industry.
Beyond the memory chip industry, Taiwan's top chipmakers are scrambling to adapt to industry changes in the hope of emerging stronger from the crisis.
TSMC, which managed to remain profitable but saw its biggest-ever revenue drop in the first quarter of this year, is considering diversifying away from chips for the first time in its 22-year history.
Rick Tsai, the former chief executive, was named head of a unit to explore options in the green energy and LED lighting sectors.
TSMC, the world's biggest contract chipmaker, has appealed to clients to co-operate on research and development efforts.
The company, which will make available some of its research and development to clients, hopes co-operation will help contain the escalating costs of designing and producing ever more complicated chips.
TSMC was one of the first big chipmakers to restore investment plans that had been curtailed by the downturn, and is expanding its R&D team.
Bhavtosh Vajpayee, analyst at CLSA, said TSMC would also probably benefit from increased outsourcing by integrated design manufacturers.
Such companies, which include businesses such as STMicroelectronics and Intel, outsource less than 10 per cent of their production but are expected to do so increasingly given the rising cost and complexity of making advanced chips.
“TSMC's massive lead in technology and rising competitive leadership within the industry has helped it grab at least 56 per cent of the IDM [Integrated Device Manufacturers] outsourcing shift since 2001, and this is set to grow further,” Mr Vajpayee said in a recent note.
TSMC has signed an outsourcing deal with Japan's Fujitsu to produce future Fujitsu chips, allowing the Japanese company gradually to run down its domestic plants and quit parts of the semiconductor business where it cannot make money.
Mainland tempts makers to set up shop and offers vital market
The mainland is a growing force in chip manufacturing.
Intel's first semiconductor plant in China is expected to begin production next year.
Taiwan's United Microelectronics Corp, the world's second-biggest contract chipmaker, plans to acquire the rest of Hejian Technology, a Chinese foundry in which it holds a 15 per cent stake.
“We believe a production base in China is essential,” said Richard Yu of UMC.
“China's market was relatively strong even during [the global economic crisis], attracting many customers that preferred the option of local production.”
But China's domestic foundries are years away from mounting a serious challenge to their Taiwan counterparts, partly because of restrictions on technology transfer by the US and Taiwan governments.
Nearly all China's domestic foundries are struggling to stay profitable.
An advantage of being in Taiwan, according to Mr Yu, is that “the critical mass of semiconductor companies in Taiwan has really helped us optimise our operating efficiency.
“Within Taiwan's dedicated science parks in Hsinchu and Tainan, there are companies that represent the full spectrum of the semiconductor supply chain to fulfil industry needs”.
But China is becoming a vital market for Taiwan's semiconductor groups.
Donald Lu at Goldman Sachs estimates that China accounts for 13 per cent of demand for the contract chipmakers, meaning “demand from China and other emerging markets has become as important as demand from developed countries”.