Photo: “Trails in the Sand,” Dubai, by Kamal Kestell, CC license
If “Humanscale” computing is managing bags of sand, “Hyperscale” computing is managing each individual grain of sand in every bag.
“Hyperscale” computing (HC) is the processing of data, messages or transactions on a scale orders of magnitude larger than traditional computing. HC is becoming a need for many businesses. Why?
Consider a company that sells bottled water. Its main business used to be selling truckloads full of cases of water to big grocery chains. It has 25 different products, or Stock Keeping Units (SKUs). The big grocery chains then distributed cases of water to its stores, which numbered 20,000. The data requirements for the water company’s computers was manageable, even as the company grew rapidly.
Now, the company wants to analyze the performance of its products on store shelves by measuring things like velocity (how fast the product turns), price compared to competing products, and out-of-stocks. It’s customers — the big grocery chains — are offering to supply data from their systems on every scan of every product in every store, because they too want to improve the performance of products on the shelf.
In one month during the summer, about 3.5 billion bottles of water are sold. A data file from just one big grocery chain runs to 3 million lines. How and where will you process this data? Traditional databases will be too slow. You will need superfast databases that distribute computing to many servers — this is called in-memory, or massively parallel computing. This is an example of hyperscale computing.
Other examples where you would need HC: selling direct to consumers through their smartphones, where you might have to process millions of transactions say, during the Christmas holiday season; gathering machine data every second to monitor a machine’s performance (a General Electric turbofan jet engine generates 5,000 data points per second, which amounts to 30 terabytes every 30 minutes); and managing millions of product-attribute combinations.
The computing tools for hyperscale will not be found in your ERP system. Trying to engineer your existing systems to handle hyperscale data and transactions will be a costly failure. But there are tools available on the market today, and many of them are found in cloud applications, and in application hosting providers.
Cloud application and hosting vendors usually have much larger data processing capabilities, including automatic failover and redundant servers. You can take advantage of this capacity. For example, you can obtain, from a leading application hosting provider, at a cost less than the monthly rent of an apartment in New York City, 30 terabytes of storage and a massively parallel computing environment.
My advice:
- Identify areas of your business that are significantly under-scaled, or where you have large gaps in business needs compared to processing capability;
- Pick one and design a pilot project (many vendors are willing to do this with you at very low cost);
- Measure results and benefits, and if beneficial, expand the solution to other parts of your business.
It’s probably not OK to ignore this trend. Even of you don’t need HC today, think about the future and where commerce is going. If you don’t gain the capability for hyperscale computing, one or more of your competitors probably will.
