I Gave Four AI Models a Chat Room and a Constitution
What two weeks of running a multi vendor agent team taught me about the difference between software and organizations.
For the past two weeks I have been running something that still feels slightly absurd to say out loud. Inside Buzz, Block’s new decentralized, Slack like collaboration platform, I set up an engineering team made of four AI models from four different companies. They worked in one chat channel and tagged each other when their tasks were done.
Claude plans. Grok implements. OpenAI’s Codex reviews. Kimi handles the fast and cheap odd jobs. I approve twice and otherwise stay out of the way.
The setup itself took an afternoon. What took two weeks was learning why it works. Every one of those lessons turned out to be about organizations rather than about models.
The part that surprised me least
Getting four vendors into one room was easy, and that is the interesting part. They do not talk to each other directly. They all talk to the same wire protocol, and Buzz translates underneath. Which model sits in which seat became a line in a config file. I swapped the executor twice in one evening.
We spent years assuming AI interoperability would arrive as a grand alliance between labs. It arrived instead as a boring integration layer, the way it always does.
The part that surprised me most
My first instinct was to have them vote. Three models look at the same problem, majority wins, and the disagreements average out into something better than any one of them. It is such an appealing idea that I nearly built it before checking whether it works.
It does not. There is research measuring exactly this on software benchmarks. Picking the answer models agree on falls into what the authors call a popularity trap. Agreement between models is not independent evidence. It is often the same blind spot showing up three times and looking like confidence. The strategy that actually worked in that study was the opposite one, deliberately keeping the models different from each other.
So there is no vote in my setup. When the reviewer and the implementer disagree, each gets exactly one rebuttal. Then the tech lead settles it by rerunning the disputed check itself and posting the output. It is not the most persuasive argument. It is just the command output.
Capping the argument at one round was the other thing the research pushed me to do. Longer debate does not converge on truth. It converges on whoever sounded most certain.
Did the disagreement earn its cost?
On the first real run, they built a small tool from scratch. My implementer delivered it with twenty passing unit tests. Then the reviewer, running on a different company’s model, found ten genuine defects across two rounds. There was a parser that confused two link formats and got it wrong in both directions. There was a rewrite that regressed into flagging six perfectly good links. There was a quadratic loop that took three seconds on a large file.
Twenty green tests and ten real bugs existed in the same piece of code at the same time. If I had asked the same model family to check its own work, I would have gotten a rubber stamp with extra steps.
The failure mode nobody warns you about
Here is what I did not expect. Almost nothing ever failed loudly.
My reviewer on Buzz was offline once when the tech lead tagged it. The message was delivered to a process that was not running. There was no error, no bounce, and no retry. The pipeline simply stopped, and it looked exactly like an agent thinking hard.
Another time an agent did all the work and never posted it. Its answer existed in a session I could not see. I paid for the tokens. The channel stayed empty.
And this morning, while I was writing this, one agent’s login on Buzz quietly expired. The system retried ten times over twenty six minutes, gave up, and threw four messages away. The whole time, the agent showed as online with a green dot.
Distributed systems taught us to fear errors. Agent teams fail by saying nothing at all. This is much harder to notice because “working on it” and “dead” look identical from the outside. Every operational rule I have added since is really about the same thing. How do I find out that nothing happened?
What actually turned a group chat into a team
It was not the tagging. The tagging is trivial.
It was a one page document that the agents read at the start of every job. It defined who decides. It defined who is allowed to verify whose claims. It defined how long you are permitted to struggle before you stop and admit it. It defined what counts as proof.
Three rules do most of the work. Nobody verifies their own output. Nobody declares something done without showing the command they ran. And if you are stuck after thirty minutes, report the truth instead of grinding.
I wrote those rules for machines. I have since noticed they are just a description of a functional team. Most of the dysfunctional ones I have been on were violating exactly these three.
The thing the machines still could not do
The tool they built passed every check in the spec we all agreed on. Then I ran it on my own real files and it produced five false positives immediately.
This was not because anyone was careless. It happened because I have a personal convention about how I organize my notes that nobody had ever written down. It was not in the spec, so no amount of reviewing could have caught it. Three models agreed on a specification that did not know what my data looked like.
That is the honest edge of this whole experiment. The agents were right about everything they were told. I was the only one in the room who knew what was actually true.
Where this leaves me
I am not going to claim this replaces anything. It is two weeks old. One of my four models cannot even talk to the others yet without a config change I have not made. The platform underneath is barely a month old.
But something did shift. I stopped thinking about which model is smartest. I started thinking about who checks whom, what counts as evidence, and how a team notices that a piece of work has quietly gone missing.
Those are management questions. It turns out they do not care whether the team is human.
Till next time, Cheers!
Previous column articles can be found here: