As AI models grow in size, the demand for resources requires innovative solutions for sustainability. A comprehensive survey has investigated algorithmic and systemic methods to enhance resource efficiency in large foundation models — spanning LLMs, Vision Transformers (ViTs), and multimodal models. The paper addresses the lifecycle of these AI giants from training to deployment, offering insights into current architectures, algorithms, and practical system designs.

Highlights of the Survey:

• Analysis of Model Architectures: Understanding the latest advances in scalable AI.
• Training and Serving Strategies: Investigating methods to reduce resource consumption.
• System Design Solutions: Exploring implementations that can aid resource-efficiency.
• Inspiring Future Research: Paving the way for revolutionary ideas and approaches.

This survey is significant for not only outlining the current state of the AI landscape but also for highlighting a critical aspect of future technology — sustainability. It emphasizes the necessity for ongoing innovation to maintain the momentum of AI advancements while mitigating environmental and economic impacts. With implications for policy makers, researchers, and practitioners, this work anchors a pivotal conversation on responsible AI development.