For as long as electronic computation has been around, scientists and engineers have had to warehouse their most challenging problems, waiting for a system to come along that would be capable of handling them. That hasn’t changed; right now, leading laboratories, universities and R&D-based firms would like to refine models, run simulations, and analyze data streams, but are unable because the computational power required exceeds what is available to them by a factor of ten, or a hundred, or a thousand. Think about this: the fastest supercomputer on the most recently published list, an IBM system at LANL which uses both standard x86 and specialty Cell processors in a hybrid configuration, produces as many computations per second as the bottom 180 systems on the list. Specialty systems may make up a small fraction of the total systems delivered into HPC, but they are an outsized force when it comes to solving the world’s collective scientific problems.

Supercomputer vendors that seek to serve customers with the most challenging problems must make a long-term investment, and have a vivid imagination of what the requirements will be years in advance of delivery. IBM’s Blue Gene, the fastest, most energy efficient supercomputer when it was delivered in 2005 was initiated as a full-fledged project in 1999. It contained numerous innovations that could be called “specialty” components, yet it adhered closely to programming, administrative and IT lab standards so that customers’ investments in software and skills were protected.

The reason start-ups seeking to develop and market “specialty” supercomputers fail has less to do with the market turning its back on such offerings in favor of commodity clusters, and more to do with the enormous investment needed to get from specs to final product and a payback that takes years to recover, if ever.

Standardized components are absolutely crucial to the HPC market, for they have helped break down price barriers; and for most of the HPC market, “off the shelf” products are fine. But there will always be a segment of the market that will have problems that lean out ahead of wherever the state-of-the-art may be, and those customers will continue to depend on the vendors that can produce something special for them.

1. SiCortex system software, application libraries, and programming model were all industry standard: MPI, OpenMP, C, C++, Fortran95. They were also “open” as in “open source.” So the software wasn’t proprietary.

2. If “proprietary” means “not x86″ then I think we’ve got a new definition of “proprietary.” How many new x86 licensees are there? None. If “proprietary” means “not ethernet” then you ignore most of the better infiniband implementations (that are specially designed for cluster use — e.g. infinipath) and the IBM federation switch, and the Cray interconnect.

(Note that at the time of SGI’s sale, they were manufacturing x86 / infiniband clusters and had been for a long time. Is that “proprietary” hardware? The irony is that if they’d fielded their shared memory x86 large SMP (a distinctly non-commodity solution) the fate of the company might have been much different. SGI failed because they had no product differentiation that could produce real profits.)

SiCortex failed because they ran out of money. Raising money is hard in the best of times. The founders (I was one of them) took on raising money in 2002 (in the second worst capital market in recent memory) as a full time job. Raising money required full time dedication, knocking on lots and lots of doors — at prospective investors as well as at customer prospects — a thorough understanding of the competitive situation, and a good bit of luck. That didn’t come together this time.

Don’t let the SiCortex collapse serve as a bogus argument against innovation. The big companies don’t have all the answers, and the answers they do have are often so overconstrained by inertia, internal internescine games, and quarterly results that innovation is reduced to a slogan.

Think big. Work hard. Build stuff.