The infrastructure of the Surgical Planning Lab is designed to give doctors, computer scientists, and other researchers the freedom to treat, understand, and innovate without computational limitations. In effect, we have designed a supercomputer center to serve the needs of medicine, where the hospital's network acts as a huge, lifesaving peripheral device. The SPL's computer facilities provide not just brute computational horsepower, but also abundant and reliable data storage, a fast, high-capacity network, and a large number of well-equipped graphics workstations for use by the SPL and its collaborators.
The SPL's computational facility rivals the power and capacity of many university computer science sites throughout the world. What really sets the SPL apart, though, is its integration into the clinical environment. Located inside Brigham and Women's Hospital, the main SPL laboratory space is immediately adjacent to forty active operating rooms. The Radiology Department's main MRI (Magnetic Resonance Imaging) and CT (Computed Tomography) scanners are located just down the hallway; data from these scanners is available directly from the SPL's network. With such proximity, close-knit clinical and research collaborations can be developed on a day-to-day basis.
The SPL Network
The backbone of the SPL's computing facilities is its own dedicated high-speed network. The center of the network is an Alcatel 7020/Omnicore 5052 enterprize-class routing switch, capable of managing the traffic from hundreds of 10 or 100 Mb/Sec Fast Ethernet and scores of 1Gb/Sec Gigabit Ethernet connections. This central switch services direct connections from the machines in the SPL's central computer facility, including approximately 220 workstations and laptops. The switch also connects directly to four other sites: offices in the the Thorn Research Building and at 1249 Boylston Street, the Open Magnet MRI system in the MRT operating room, and the Center for Neurologic Imaging located about 1 Mile away. Gigabit Ethernet speeds are or will be possible to each of these sites.
The SPL depends on the network of Partners Healthcare to connect to other
parts of the hospital (research groups, imaging scanners, clinical display
stations, and patient information systems), other Partners institutions
such as Massachusetts General Hospital (MGH), and the outside Internet.
UNIX Workstations are the most common computational tool in the SPL. Graphics workstations from Sun Microsystems provide individual researchers, computer scientists, and physicians the ability to process data, develop software, or visualize results. Through the generous support and cooperation of Sun, SPL members have access to single- and multiprocessor UltraSPARC based workstations with accelerated graphics, fast networking, and abundant memory for processing large medical data sets. The most common computer in the SPL is the Sun Ultra 10 with Creator 3D or Elite 3D graphics cards. For more demanding tasks, users can access several Ultra80 workstations, each with four processors and about 4 GB of RAM.
In addition to these machines, the lab has about 10 other Ultra-class computers for general use. Peripheral devices such as DLT tape drives and CD-RW writers are usually connected to these computers. Sun workstations also form the core of the SPL's administrative infrastructure: dedicated Ultra5 computers provide the lab's mail, web, logging, gateway, and other administrative services.
Central Computing and HPC
The Surgical Planning Lab offers a powerful computation center to serve a wide range of medical applications. Some of these applications may deal with huge data sets, such as non-rigid matching of brain structures of a group of patients. Other uses of massive computation require near-real time results: for instance, processing image information about a patient in surgery using the Open MRI scanner in MRT and correlating it with the patient's conventional MRI or CT data taken before surgery. Finally, large-scale processing engines allow computer scientists to more rapidly develop new prototype segmentation, visualization, registration, and interaction tools without constraints before optimizing the algorithms to run on more common workstations and PCs.
The SPL's high performance computing system is made up of a variety of enterprise-class computers from Sun Microsystems. Currently, these machines include two ES5000 machines, each with 8 processors and 2GB of RAM, and one ES6000 with 18 processors and 5GB of RAM. These machines, which are optimized for fast internal data transport, are connected to our network using Gigabit Ethernet. In addition to these machines, we also have a four processor Sun ES450 application and file server designed to take the more commonplace computational load off of the larger machines.
In addition to these "big iron" machines, the computational infrastructure of the SPL is augmented by a large collection of Sun workstations that provide both power to individual users and the capability of clustering together large numbers of processors into a distributed supercomputer for large tasks.
The SPL's is engaged in a variety of research and clinical activities that involve processing and storage of large amounts of data:
For these applications, data needs to be available quickly to the PCs, workstations, and supercomputers that perform processing and analysis, in the lab or in the operating room. In addition, that same data needs to be well-organized and safe from disk failure or accidental modification; the results of important clinical studies may depend on it.
To meet these demanding storage needs, the SPL has moved from a system of workstations and file servers with individual disk drives to a high reliability network attached storage (NAS) device from Procom Technology. The SPL's Procom NetForce 3100 holds 2.5 Terabytes (2,500,000 Megabytes) of data using redundant FibreChannel disk arrays and a specialized disk computer called a filer head. Redundant power supplies, cooling fans, disks, and other components mean that all the data on the Procom will survive any single failure. Two Gigabit Ethernet links directly to our network switch permit individual client machines to write data at up to 30 Megabytes per second.
In addition, the Procom's file system allows us to conveniently manage and track disk usage by our core team and our collaborators, eliminating the need to think about individual disk drives, controllers, and file servers. Instead, we can focus on getting storage capacity quickly to the projects and groups that need it. As our storage needs grow, we can add more disk arrays to the NetForce and allocate more storage space to users and groups transparently.
Valuable data must be recoverable in the event of alteration or deletion, be it because of catastrophe or accident. The SPL uses three techniques for backup: file system checkpointing, tape backup, and data archiving. The Procom NAS provides a checkpointing service that allows individual users to recover their own data for themselves as it existed yesterday, a week ago, or any other time we choose to provide a checkpoint. This feature is extremely timesaving and useful for "oops" deletions or changes.
To deal with the unlikely occurrence of a major storage failure, the SPL backs up all data on the NAS onto a Sun DLT tape library with 4 drives and slots for a large number of extra DLT tapes. Finally, the SPL provides individual DLT tape drives so that groups can archive data onto reliable backup storage for an extra level of security or to allow original data to be removed from the file system.
The SPL ties together the talents of many different researchers and clinicians from different backgrounds in medicine and technology; the work of this team often dictates special computational needs. Working with collaborators at MIT's Artificial Intelligence Laboratory, for example, the SPL created a specialized cart with a Sun workstation, tracking hardware, and specialized software to provide a computer-augmented view of the patient's anatomy for surgeons in the operating room. Another research project is building a MRI-compatible robot and the associated software for use in Magnetic Resonance Therapy. These projects and many others depend on the SPL's flexible computational infrastructure and related resources.
Surgical Planning Laboratory, Department of Radiology, Brigham and Women's Hospital