Comment on this story
comment
SAN FRANICSCO — Elon Musk’s Twitter is a house of cards.
On two recent occasions, almost exactly a month apart, minor changes to Twitter’s code appeared to break the site.
The latest outage came on Monday when thousands of users found they couldn’t access links, photos or other important aspects of the site.
“One small API change had a massive impact,” said Twitter CEO Elon Musk wrote in a tweet on Monday, referring to the tool used by third-party developers who run programs that pull Twitter data and post on its website. “The code stack is extremely brittle for no good reason. Will eventually have to be completely rewritten.”
It was the second time he had addressed that statement on Monday, both times vocation the side “brittle”.
Since Twitter’s acquisition, CEO Elon Musk has laid off more than two-thirds of the company’s workforce, embarked on aggressive cost-cutting and partially laid off employees by forcing them to commit to “extremely tough” jobs or leave the company. The massive layoffs led to widespread concerns about Twitter’s ability to maintain core functions, as critical engineering teams were reduced to one or zero employees.
In the months since the acquisition — and subsequent layoffs — Twitter has faced multiple outages that impacted key functions: loading tweets and notifications, sending tweets and direct messages, accessing links and photos. Everyone was told — by current and former employees, or Musk himself — to come when the company made changes to its code.
“Any mistake in code and operation is now deadly,” a former engineer told the Washington Post in November, explaining that those who remained were “overwhelmed, overworked and more likely to make mistakes because of that.” The former engineer spoke on condition of anonymity for fear of retribution.
Prior to Musk’s acquisition, the company had a risk assessment team that reviewed product changes for anticipated issues. Twitter’s risk assessment process was designed to identify potential problems before they occur. But the team was fired after Musk’s takeover, The Washington Post reported, resulting in product launches littered with bugs.
Musk and Twitter did not immediately respond to requests for comment.
Since acquiring Twitter, Musk has implemented a plan to cut 75 percent of the company’s workforce, aggressively cutting costs and finding new revenue streams, such as sales. But his tenure was also marked by embarrassing mishaps, like the botched rollout of the check mark feature, which led to a swarm of impersonators and prompted Twitter to temporarily pause the subscription service on multiple occasions.
Musk went after Twitter by promising to restore “free speech” on the platform, and fired the company’s previous list of executives he blamed for a content moderation approach built on protecting against the harms of hate speech and misinformation was based. Musk also promised transparency about the old regime’s decisions — such as efforts to limit the distribution of a New York Post story about the contents of Hunter Biden’s laptop — but has done so collapsed on leaks of company information under his own leadership.
Even before Musk’s takeover, Twitter employees had warned of weaknesses in the site in the event of a failure. Twitter whistleblower Peiter Zatko, for example, warned of a possible event of overlapping failures at Twitter’s external data centers in a complaint received by congressional committees.
That kind of outage, he said, could fail to recover critical data — and cause Twitter to be down for months. Despite concerns about Twitter’s vulnerable infrastructure, Musk ordered Twitter’s largest data center in Sacramento to shut down in December, the Washington Post reported at the time.
On Monday, Twitter users were greeted with glitches almost as soon as they opened the page.
When users clicked a link on Twitter, they received the message, “Your current API plan does not include access to this endpoint,” it said, redirecting them to a page for developers.
Complaints piled up on the Down Detector website, which tracks online outages: “User reports point to problems on Twitter,” it said.
“We made an internal change that had some unintended consequences,” according to the Twitter support account called in a tweet.
By late morning some of the functions appeared to be restored.
“Things should be working as usual now,” Twitter’s support account said in a tweet. “Thank you for staying with us!”
Monday’s pattern reflected an outage from early February, which occurred when Twitter made similar changes to its API, the data feed intended for developers.
Twitter faced a widespread outage on February 8, making it impossible for users to tweet and direct message, follow other accounts, and upload content to their timelines.
“Starting February 9th, we will no longer support free access to the Twitter API, both v2 and v1.1. A paid basic tier will be available instead,” Twitter said wrote this month.
Musk said Twitter tried to crack down because Twitter’s open data “abused‘ of bots that peddle scams, but later said the company would make a free version available amid criticism of its aggressive efforts to monetize aspects of the site that were previously free.
This series of problems followed a widespread global outage that Twitter faced in December.
In group chats between current and former engineers at the time, some speculated that the December outage had come after a software update gone awry.
Monday wasn’t the first time Musk suggested a complete rewrite of Twitter’s code. He’s maintained that stance for months since taking over the site last year. Speaking on Twitter Spaces, the site’s live audio feature, in December, he said the company’s codebase needed an overhaul.
Urged by a participant to explain what that meant, Musk became irritated.
“Amazing, wow,” he said after hesitations and pauses. “You are a donkey. … What an idiot.”