alignmentforum.org·2026/10/15·Vanessa Kosoy
为什么重要 偏理论与对齐议程阐述，无新工具或实验结果，对工程实践影响有限。
“artificial superintelligence [1]”（ASI）这一理念近来已在主流认知中从科幻领域，进入 AI 领域多数领先商业组织公开追求的近期技术目标范畴。不幸的是，ASI 对人类这一物种构成严重的生存风险。Eliezer Yudkowsky 自 2001 年以来一直提出这一论点 [2]，许多其他人也提出过类似论点（例如 Omohundro 2009、Bostrom 2014、Armstrong 2014、Russell 2019、Bengio et al. 2024（25 位作者）、Bengio et al. 2025（13 位作者），以及 Yudkowsky and Soares 2025）。
面对这一风险，问题便成为我们如何让 ASI 变得安全 [3]。这个问题在多个层面都具有挑战性，因为要解决它，我们需要弥补理论计算机科学以及与知识和决策相关哲学领域中的巨大理解空白。
The idea of “artificial superintelligence 1 ” (ASI) has recently crossed from the realm of science fiction (in the mainstream perception) to the realm of near-term technological goals openly pursued by most leading commercial organizations in the field of AI.
Unfortunately, ASI poses a grave existential risk to the human species.
This was argued by Eliezer Yudkowsky since 2001 2 , and by many others (see e.g., Omohundro 2009 , Bostrom 2014 , Armstrong 2014 , Russell 2019 , Bengio et al. 2024 (25 authors), Bengio et al. 2025 (13 authors) and Yudkowsky and Soares 2025 ).
In light of this risk comes the question of how can we make ASI safe 3 .
This question is challenging on many levels, because to solve it we need to complete vast gaps in our understanding both in theoretical computer …