Finding the next architecture is critical. We've seen limits on what transformers can do, inefficient at best. Dense models carry the cumulative knowledge of all humans but cannot do simple reasoning tasks that a child could. Adding to this, humans consume a very small amount of energy in calling upon information, making for an ultra efficient system. It is clear that transformers alone will not get us to super intelligence.
New breakthroughs will need to take place from now to then, and we believe they will come. We suspect that a 600 quadrillion parameter will be the threshold for ASI. This is at least a decade away, at best. By then, computing will have advanced to the point where 10 angstrom chips and trillions of transistors become commonplace, and cloud computing exceeds 10 exaflops--barely enough to power something like ASI.
