An update on the near.org / RPC outage on July 11, 2024

From Thursday, July 11 to Saturday, July 13, many visitors to near.org and its subdomains (like dev.near.org and docs.near.org) were unable to reach those websites. Also, NEAR applications that rely on RPC services hosted on near.org were affected. This was due to a security incident followed by an outage from one of our vendors, Squarespace. During this period, the NEAR protocol & blockchain was unaffected by this incident, as it does not rely on any centralized services.

Our Use of Squarespace DNS

The near.org domain is operated by Pagoda, an engineering arm of the NEAR Foundation. An important part of operating a domain like near.org is choosing a DNS provider; to learn more about DNS and how it operates, please see this introduction. For years, we happily used Google’s domain name registration & DNS service to manage near.org, as part of our broad usage of Google Cloud services. In June 2023, Google announced it had sold its Domains business to Squarespace, and that our account would be transitioned to Squarespace “in the next few months.” We expected that the new service from Squarespace would closely match the feature set, reliability, and security we had enjoyed while using Google Domains, and if there were any problems then we could easily switch from Squarespace to another DNS provider after the transition.

Our first domain, near.wiki, was automatically moved over to Squarespace in May 2024. As we explored the new offering, our Security & IT teams quickly had concerns about the feature set, security controls, and level of enterprise support we could get from Squarespace. After some meetings with Squarespace sales and support failed to assuage our concerns, we decided to explore other DNS providers and to start migrating our domains off of Squarespace, starting with the ‘simpler’ ones and ending with near.org, our most complex and important domain name. As of last week, when this incident began, we had not yet begun migrating near.org to our new provider, Amazon Web Services, so it was still using Squarespace as its DNS provider.

The Squarespace Security Incident

Unbeknownst to us, Squarespace had a major security vulnerability. You can read about the problem in this in-depth blog post from Security Alliance. In short, when each domain was migrated from Google to Squarespace, all existing users on the Google Domains account were sent an email inviting them to create a new Squarespace account. But not everybody clicked on that email right away – some people, e.g. managers or our accounting team, only needed to log into the DNS service rarely or in emergencies. From what we can tell, all the attacker needed to do was identify one of those email addresses and sign up for a new Squarespace account using that email address, and they would be instantly granted full access to change or delete DNS entries for near.org. From our investigation, Squarespace did not require the attacker to verify ownership of the email address before letting them have full control on our account. In fact, we see no evidence that Squarespace even sent a “welcome” email to that address upon account creation, which might have raised warning flags.

On July 11, an attacker gained access to our Squarespace account. They deleted the DNS entries for near.org and its subdomains, causing the outage described above. But unlike some other domains affected by this attack, the impact on near.org seems limited to an outage; we have seen no evidence that the attacker put in place an ‘imposter’ site to try and phish or scam users.

Through a combination of our actions and Squarespace’s actions behind the scenes, we were able to quickly regain control of the account. However, due to other issues with Squarespace, it took another two days for near.org and its subdomains and services to get fully back online.

The Squarespace Outage

Even with full and exclusive access to our Squarespace, we were unable to restore service on near.org. The attacker had deleted our DNS entries, and when we re-added them, Squarespace failed to propagate those new entries to other DNS providers around the world. We attempted to quickly execute a switch to Amazon Web Service’s DNS provider, but the feature to switch to a different DNS service was also broken on the Squarespace site.

The next morning, on Friday July 12, on a hunch we deleted and re-added all of our DNS entries once again. This time, it did take effect and DNS providers around the world quickly saw the new information. That resolved the outage for many users, but not for everyone. near.org had been set up with an additional security feature, DNSSEC, in which Squarespace was supposed to digitally sign our DNS entries to prove that the entries had not been forged by another DNS provider. But Squarespace wasn’t properly updating and signing new DNSSEC entries, so any other DNS provider that validates DNSSEC was detecting near.org’s DNS entries as a forgery and refusing to allow access to near.org. This affected approximately 34% of users, especially in Europe. There is a button on the Squarespace site to turn off DNSSEC, but unsurprisingly, that button also just showed an error message.

Finally, on Saturday July 13, we were able to make contact with a specialist on the Squarespace team, and later that day they were able to fix the DNSSEC issue. Once that change propagated to other DNS providers around the world over the next few hours, near.org and the RPC service was restored for everyone.

Reflections on this Incident

You may have noticed that we made a few assumptions in this blog post, saying “from what we can tell” and “we see no evidence of…” a few times. Ideally, almost a week after the incident began, we would have more hard truths and solid information about what happened.

Unfortunately, we’ve seen little to no communication from Squarespace throughout this period. We tried multiple support tickets, chat, personal contacts, LinkedIn messages, and going through our Google support team, and still didn’t hear anything back from Squarespace until late on Friday July 12, about 36 hours after the incident began. Other affected companies also reported total silence from the Squarespace team. As far as we know, Squarespace has still not acknowledged this incident publicly, let alone shared details of the issue and how they remediated it, and their status page showed no outages or issues during this time.

I personally find this lack of support, communication, and forthrightness to be unacceptable for any service provider. I’m also somewhat disappointed in Google for transitioning a security-critical service to a new provider without proper vetting. We are accelerating our plans to move near.org and other domains out of Squarespace to Amazon Web Services’s domain registrar and DNS provider, which has a great track record of reliability and security.

DNS is a core part of internet infrastructure, but it’s far from a perfect system. Every domain name owner must rely on one centralized DNS provider to maintain their DNS entries, and every user and application must rely on one or more centralized DNS providers to look up entries as they navigate the internet. Various projects in the blockchain industry have created non-custodial on-chain alternatives to DNS, such as Unstoppable Domains and 3DNS (FYI: neither have sponsored or were made aware of this post), but the existing DNS system is so entrenched that adoption has been a challenge. Hopefully as an industry we can make headway on a more decentralized, trustless open web before further incidents like this happen.

I’m deeply sorry to anyone affected by this outage, especially people exploring NEAR for the first time during the EthCC conference and EthGlobal hackathon. We will be more vigilant about using high-quality vendor services going forward – and, where possible, moving to decentralized on-chain solutions.

-Eric Winer
CTO & Managing Director, Pagoda