Michael Gschwind

Michael Karl Gschwind is an American computer scientist at Nvidia in Santa Clara, California. He is recognized for his seminal contributions to the design and exploitation of general-purpose programmable accelerators, as an early advocate of sustainability in computer design and as a prolific inventor.

Accelerators

Gschwind led hardware and software architecture for the first general-purpose programmable accelerator Accelerators and is widely recognized for his contributionsHeterogeneous computing as architect of the Cell Broadband Engine processor used in the Sony PlayStation 3, and RoadRunner, the first supercomputer to reach sustained Petaflop operation. As Chief Architect for IBM System Architecture, he led the integration of Nvidia GPUs and IBM CPUs to create the Summit and Sierra supercomputers.

Gschwind was an early advocate for accelerator virtualization and as IBM System Chief Architect led I/O and accelerator virtualization.

Gschwind has had a critical influence on the development of accelerator programming models with the development of APIs and best practices for accelerator programming, application studies for a diverse range of HPC and non-HPC applications. and as co-editor of books and journals on practice and experience of programming accelerator-based systems.

AI acceleration

Gschwind was an early advocate of AI Hardware Acceleration with GPUs and programmable accelerators. As IBM's Chief Engineer for AI, he led the development of IBM's first AI products and initiated the PowerAI project which brought to market AI-optimized hardware (codenamed "Minsky"), and the first prebuilt hardware-optimized AI frameworks. These frameworks were delivered as the firstfreely installable, binary package-managed AI software stacks paving the path for adoption.

At Facebook, Gschwind demonstrated accelerated Large Language Models (LLMs) for Facebook's First Generation ASIC accelerators and for GPUs, leading the first LLLM production deployments at scale for embedding serving for content analysis and platform safety, and for numerous user surfaces such as Facebook Assistant, and FB Marketplace starting in 2020. Gschwind led the development of and is one of the architects of Multiray, an accelerator-based platform for serving foundation models and the first production system to serve Large Language Models at scale in the industry, serving over 800 billion queries per day in 2022.

Gschwind led the company-wide adoption of ASIC and Facebook's subsequent "strategic pivot" to GPU Inference, deploying GPU Inference at scale, a move highlighted by FB CEO Mark Zuckerburg in his earnings call. Among the first recommendation models deployed with GPU Inference was a Reels video recommendation model which delivered a 30% user surge within 2 weeks of deployment, as reported by FB CEO Mark Zuckerburg in his Q1 2022 earnings call, and a subsequent $3B to $10B growth for REeels year-over-year.

Gschwind also led AI Accelerator Enablement for PyTorch with a particular focus on LLM acceleration, leading the development of Accelerated Transformers (formerly "Better Transformer") and partnered with companies such as HuggingFace to drive industry-wide LLM Acceleration to establish PyTorch 2.0 as the standard ecosystem for Large Language Models and Generative AI.

Gschwind subsequently led expanding LLM acceleration to on-device AI models with ExecuTorch, the PyTorch ecosystem solution for on-device AI, making on-device generative AI feasible for the first time. ExecuTorch LLM acceleration (across multiple surfaces including NPUs, MPS, and Qualcomm accelerators) delivered significant speedups making it practical to deploy Llama3 unmodified on servers and on-device (demonstrated on iOS, Android, and Raspberry Pi 5) at launch with developers reporting up to 5x-10x speedups over prior on-device AI solutions.

Gschwind's multiple contributions to AI software stacks and frameworks, AI accelerators, mobile/embedded on-device AI and low-precision numeric representations in torchchat, representing a seminal milestone as the industry's first integrated softwarestack for servers and on-device AI with support for a broad set of server and embedded/mobile accelerators.

Gschwind is a pioneer and advocate of Sustainable AI.

Supercomputer design

Gschwind was a chief architect for hardware design and software architecture for several supercomputers, including three top-ranked supercomputer systems Roadrunner (June 2008 – November 2009), Sequoia (June 2012 – November 2012), and Summit (June 2018 – June 2020).

Roadrunner was a supercomputer built by IBM for the Los Alamos National Laboratory in New Mexico, USA. The US$100-million Roadrunner was designed for a peak performance of 1.7 petaflops. It achieved 1.026 petaflops on May 25, 2008, to become the world's first TOP500 LINPACK sustained 1.0 petaflops system. It was also the fourth-most energy-efficient supercomputer in the world on the Supermicro Green500 list, with an operational rate of 444.94 megaflops per watt of power used.

Sequoia was a petascale Blue Gene/Q supercomputer constructed by IBM for the National Nuclear Security Administration as part of the Advanced Simulation and Computing Program (ASC). It was delivered to the Lawrence Livermore National Laboratory (LLNL) in 2011 and was fully deployed in June 2012. Sequoia was dismantled in 2020, its last position on the top500.org list was #22 in the November 2019 list.

Summit is a supercomputer developed by IBM for use at Oak Ridge Leadership Computing Facility (OLCF), a facility at the Oak Ridge National Laboratory. It held the number 1 position from November 2018 to June 2020. Its current LINPACK benchmark is clocked at 148.6 petaFLOPS.

Many-core processor design

Gschwind was an early advocate of many-core processor design to overcome the power and performance limitations of single-processor designs. Gschwind co-authored an analysis of the limitations of frequency scaling which arguably led to an industry-wide transition to many-core designs. Gschwind was a lead architect for several many-core designs, including the first commercial many-core processor Cell with 9 cores, BlueGene/Q with 18 cores, and several enterprise and mainframe processors ( POWER7/POWER8/ POWER9 with up to 24 cores; z10-z15 with up to 12 cores).

As chip chief architect and chief microarchitect, Gschwind was critical to the reboot of the POWER architecture after the POWER6 high-frequency high-power dead-end, leading revival of the POWER5-style out-of-order design with POWER7, serving as unit lead and chief microarchitect for the instruction fetch, decode and branch prediction unit (also including logical instruction execution), and as acting lead for most other units at one point during the design. In subsequent generations of the POWER architecture, integration of the VMX SIMD design and FPU into VSX, little-endian support in POWER8 laying the foundation for little-endian PowerLinux (used in the Google POWER prototype and for GPU integration for the Minsky PowerAI system), and integration of NVLink for optimized GPU/CPU integration; and native support for Linux-style hardware-managed radix page-tables in POWER9, used in the world-leading Summit and Sierra Power+Nvidia supercomputers; and the introduction of PC-relative addressing and prefix instructions to transcend the limitations of the 32-bit instruction encodings of RISC architectures in POWER 10.

As architecture lead/manager and cross-platform chief architect, Gschwind also led the reboot of system z mainframe, with introduction of compiled code efficiency (with a particular view to C, C++ and Java) in IBM z10, out-of-order execution, PCIe-based I/O in z196 and z114, support for transactional memory in IBM zEC12, introduction of hardware multithreading and z/Vector SIMD architecture (including shared software infrastructure with Power's VSX) in IBM z13; and the sunsetting of ESA390 for operating systems substantially reducing verification and design complexity and improving time-to-market in IBM z14.

System reliability

Gschwind coined the term "reliability wall" for obstacles to sustained operation of large-scale systems. He has made major contributions to system-level reliability modeling and improvements, with a particular view to enabling sustained supercomputing system operation. As chief architect of BlueGene/Q, he led system-level reliability and processor design in addition to being the chief ISA architect and QPU vector floating point unit design lead.

Gschwind led the first processor and chip-level architectural vulnerability modeling and selective hardening to achieve target MTBF, first implemented in BlueGene/Q using stacked DICE latches for critical state-holding latches. To increase system reliability while avoiding the performance and power cost associated with ECC-based designs, Gschwind proposed and led the design of register files and minor buses protected with parity with state recovery. In accordance with this approach, error detection is implemented in datapaths which may occur in parallel with initiating compute operations, with a recovery operation when a soft error is detected in parallel with the operation. Recovery then proceeds from good-state maintained in alternate copies of the register file commonly used to scale the number of register file read portsa and reduce wiring delay from register file reads to execution units.

Compiler technologies

Gschwind has made seminal contributions to compiler technology, with a particular emphasis on pioneering contributions to just-in-time compilation, dynamic optimization, binary translation and compilers in supercomputing.

Just-in-time-compilation

Gschwid was an early proponent of just-in-time compilation and has been a driving force in the field. He has proposed critical improvements for the implementation of JIT compilation based systems, with a particular view to dynamic optimization, binary translation and virtual machine implementation. Gschwind's contributions includes implementation of precise exceptions with deferred state materialization, high-performance computing optimization such as software pipelining at JIT translation time, hardware/software co-design for binary emulation and dynamic optimization. Gschwind's seminal contributions to Virtual Machine design and implementation are reflected by being the most-cited author in the `Virtual Machines' textbook by Smith and Nair.

Compilation for accelerators and accelerator-based supercomputers

Gschwind is credited with seminal contributions for compilers and programming environments for general-purpose programmable accelerators and GPUs, building the first general purpose GPU frameworks, APIs and optimizations for Cell. His contributions include performance monitoring, code partitioning, code optimization, code partitioning and APIs for accelerators.

His innovations include compiler/hardware co-design for integrated register files to resolve phase ordering issues in auto-vectorization between unit assignment and vectorization decisions to simplify the cost model, an innovation adopted by general-purpose programmable accelerators, including the Cell SPU and GPUs, and general-purpose CPU designs, starting with Gschwind's pioneering work for SIMD CPU accelerators.

More recently, his contributions to HPC compilation have included pioneering work in enabling high-performance execution of AI workloads.

System and compiler APIs

Gschwind led the development of the ELFv2 Power execution environment, which has been broadly adopted for Power execution environments. Advantageously, the new environment updates the APIs and ABIs for object-oriented environments. Departing from traditional Power architecture big-endian data conventions, the ELFv2 ABI and APIs were first launched to support a new little-endian version of Linux on Power. This has since been adopted for all Linux versions on Power servers and to support GPU acceleration with Nvidia GPUs, e.g., in the Minsky AI-optimized servers and the Summit and Sierra supercomputers.

SIMD Parallel vector architecture

Gschwind is a pioneer of SIMD parallel vector architecture to increase the number of operations which can be performed per cycle. To enable efficient compilation, Gschwind proposed the implementation of merged scalar and vector execution units, eliminating the cost of copies between scalar and vectorized code, and simplifying compiler architecture by resolving phase ordering problems in compilers.

The Cell's accelerator cores (Synergistic Processor Unit SPU) contain a single 128 element register file with 128 bit per register. Registers may hold either scalar or a vector of multiple values. The simplified cost model leads to significantly improved vectorization success, improving overall program performance and efficiency.

The vector-scalar approach was also adopted by the IBM Power VSX (Vector Scalar Extension) SIMD instructions, BlueGene/Q vector instructions and System/z mainframe vector instruction set,SIMD Processing on IBM z14, z13 and z13s, https://www.ibm.com/downloads/cas/WVPALM0N the design of all three IBM vector-scalar architectures having been led by Gschwind as Chief Architect for IBM System Architecture.

Service, education, diversity, inclusion and digital inclusion

Gschwind is a strong believer in the power of education and its power to help overcome the effects of all types of discrimination and colonialism. He has served as faculty member at rincetonand U Wiento advance education. To overcome the effects of colonialism and bridge the digital divide, Gschwind has volunteered in Senegal to contribute to the expansion and improvement of Senegal's education and research network, snRER.

Background

Gschwind was born in Vienna and obtained his doctorate degree in Computer Engineering at the Technische Universität Wien in 1996. He joined the IBM Thomas J. Watson Research Center in Yorktown Heights, NY and also held positions IBM Systems product group and at its corporate headquarters in Armonk, NY. At Huawei, Gschwind served Vice President of Artificial Intelligence and Accelerated Systems at Huawei. Gschwind is currently a software engineer at Meta Platforms where he has been responsible for AI Acceleration and AI infrastructure.

References

{{DEFAULTSORT:Gschwind, Michael
Living people
Cell BE architecture
Computer architects
Computer designers
American computer scientists
Austrian computer scientists
Scientists from Vienna
Year of birth missing (living people)
Nvidia people