Exclusive: Emmett Shear Is Back With a New Company and A Lot of Alignment

Insert coup pun here

A new AI alignment player has entered the arena.

Emmett Shear, Adam Goldstein and David Bloomin have set up shop in San Francisco with a 10-person start-up called Softmax. The company is part research lab and part aspiring money maker and aimed at figuring out how to fuse the goals of humans and AIs in a novel way through what the founders describe as “organic alignment.” It’s a heady, philosophical approach to alignment that seeks to take cues from nature and the fundamental traits of intelligent creatures and systems, and we’ll do our best to capture it here.

“We think there are these general principles that govern the alignment of any group of intelligent learning agents or beings, whether it's an ant colony or humans on a team or cells in a body,” Shear said, during his first interview to discuss the new company. “And organic alignment is the kind of alignment where a bunch of peers come together and find their role in a greater whole together where they maintain their individual identity.

“Organic alignment centers on this shared whole idea, which is opposed to the kind of alignment that you see from most foundational model companies that is very much about steering and control and direction. We think of that as hierarchical alignment.”

There’s a lot to get into there, but first some context.

Shear last made headlines in 2023 during what the wider world calls The Coup, and people inside of OpenAI call The Blip. You guys remember. It was the totally insane five-day period in which OpenAI’s board fired Sam Altman in a bid to protect the future of humanity from Altman’s “long-standing patterns of behavior” and then didn’t provide any details on said “patterns of behavior” and then the board went all quiet while chaos ensued and the rest of us pulled large buckets of popcorn up to our X viewing window to watch adults flail around as they tried to hold onto a massive AI empire and all their hopes and dreams and Shear came in for a couple of days in a bid to right the ship but that gambit didn’t even almost work and we ended up right back where we started except that some members of the board were replaced by new members of the board and Harvard Business School began work on its “How Not To Coup” seminar.

Well, Shear had a couple of major takeaways from living through this calamity.

While the experience was far from ideal, it triggered a desire to pursue something important in the AI field. In his jobs running Twitch and working as a part-time partner at Y Combinator, Shear had watched AI developments and thought about them pretty deeply but had put off starting anything. The Coup, however, refocused him on the importance of the technology and the myriad unsolved questions surrounding it.

“The OpenAI experience was very intense, and it was a wakeup call for me,” Shear said. “Before it happened, I’d left my job to spend time with my family and my new son and was really enjoying that. But I had this realization that I wasn’t ready to be semi-retired and that I wanted to be part of AI because what’s going on is really important. And I wanted to be part of it from a basic research perspective.”

The second takeaway stemmed from being reminded of the fallibility of humans. As Shear saw it, the existing alignment strategies from large AI players were too human nature-focused in their approaches. They sought to manage AI like an employee that needed guidance and goading and came from a hierarchical place where humans were the top beings dishing out the rules. He concluded that the best laid plans were bad and needed fixing. As Shear put it,

“Upon reflection, my biggest lesson from the experience was something along the lines of, ‘Oh shit, we need a real source of alignment. No one has a secret plan to deal with this technology.’”

“If you look at what happened at OpenAI, there was an alignment failure. The board and Sam were not aligned, and they fell so far out of alignment that it nearly destroyed the company. I think that the form of alignment needed there can't be framed in terms of making the board members or Sam follow some rule book, some chain of command.

It can't be framed in terms of aligning people to some hierarchical person in charge who will just decide everything. Somehow people who all have something at stake have to come together and be something more than just a collection of individuals.

I think it just highlights how important this kind of cohesion is not just for AI systems but in life in general. I guarantee you some of the stuff we're trying to tell the AIs to do is wrong because we're not perfect. So, it's important that the AIs be able to push back when it's not right.”

Some of Softmax’s alignment ideas have their origins in Michael Levin’s lab at Tufts University. Levin describes his work as analyzing, “the processes by which intelligence, in myriad conventional and unconventional embodiments, operates in the physical world.” His experiments stretch across developmental biophysics, computer science, and behavioral science in a bid to study cognition in everything from single cells on up to groups of living organisms. Levin also has the most awesome laboratory website art that I’ve ever seen.

Goldstein, who founded Hipmunk back in the day, spent some time working in Levin’s lab where he started a biology-inspired research group. A lot of his research focused on Active Inference Agents. He and Shear often discussed this work and, for a spell, it looked like Shear might team up with Levin at the lab.

The AI researcher Bloomin, meanwhile, has been working on reinforcement learning through the MettaGrid project that tries to find emergent social behaviors between AIs in closed virtual worlds.

The more the three men talked, the more they tried to blend their areas of interest and expertise. This led to the formation of Softmax and its attempt to create a theory of alignment modeled on something like a multi-cellular organism or even an ant colony where there are individuals doing their jobs as part of a collective whole.

“In order for cells to align together as a multicellular organism, they have to learn how to differentiate into roles,” Softmax writes on its website. “Muscle cells, nerve cells, skin cells, liver cells, and so on are all roles that individual cells can play. Organic alignment happens not in the simple space of spatial direction or temporal frequency, but in the more complex space of healthy development. Healthy development and differentiation for the cell is defined relative to a purpose: the healthy development of the greater whole.”

Again, some of this borders on philosophy and how one defines a concept like “intelligence,” which can make this exercise feel very fuzzy. The fundamental notion, though, is that the most basic parts of complex systems can exhibit behaviors associated with thinking and goals. These behaviors often have elements tied to self-preservation and individual objectives and to the aspirations of a group as well. Softmax seems to want to capture this essence of intelligence and this interplay between the individual and the group in its version of alignment. More power to them.

To try and express these ideas in software, Softmax has begun running reinforcement learning experiments with agents in virtual worlds. The agents – a term of art in Reinforcement Learning Land and not like a coding or assistant agent – go about miniaturized video games and try to learn and thrive as they interact with each other. The models are relatively small – in the hundreds of thousands to millions of parameters range as opposed to billions of parameters with mainstream foundational models – and very much in the research and development realm for the moment.

Shear does not yet know what form the technology Softmax hopes to develop will take in terms of an actual product. He likens the company’s position to that of the early days of OpenAI when the then pure research lab was chasing big ideas regardless of their product potential. “I have a bunch of guesses for things that I think we're going to sell, and I refuse to say them because we probably have not realized what the actual answer is yet,” Shear said. “If we make the breakthroughs we hope for, it'll become obvious.”

Softmax has raised an undisclosed amount from Andreessen Horowitz, Lionheart Ventures and Plural.

Shear does not yet think existing AI models are likely peers to humans. They remain “less than” in the spirit of bacteria or fish. That said, he thinks the line could be crossed soon, and he’s an unabashed advocate for setting the groundwork for AI rights now. “We’re sort of at this point where we seem quite happy enslaving the wish-granting God machine,” he said. “In fact, that’s the current plan of record. That’s a bad plan, and it’s not going to work.”

As for the whole OpenAI saga, Shear said, “I think it's really unfortunate that it happened because ultimately it was a lot of thrash without that much of a real change. And I don't know what course would've been better. I wish that I had never needed to be involved. Still, I'm glad I was involved and, in some ways, I'm glad it did happen because I don’t know if I would be here doing this without it.”