Note di Matteo

Wow, un datacenter AWS negli Emirati Arabi Uniti è stato colpito da un "oggetto" (missile/drone) e ha preso fuoco.

Mar 01 9:41 AM PST We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event. EC2 Instance launches will continue to be impaired in the impacted AZ. We recommend that customers continue to retry any failed API requests. If immediate recovery of an affected resource (EC2 Instance, EBS Volume, RDS DB Instance, etc.) is required, we recommend restoring from your most recent backup, by launching replacement resources in one of the unaffected zones, or an alternate AWS Region. We will provide an update by 12:30 PM PST, or sooner if we have additional information to share.

Il cloud europeo di AWS è una partizione di AWS e quindi completamente indipendente dall'infrastruttura USA, anche se l'entità legale è di proprietà di Amazon. Qua qualche altro dettaglio tecnico e quali servizi importanti sono mancanti (ad esempio CloudFront, in arrivo nel Q4 2026).

Da oggi è disponibile AWS European Sovereign Cloud (AWS.eu). Non ci sono tutti i servizi ma è comunque interessante. Account separato e (credo) region separata, per ora in Germania, con staff 100% europeo e nessuna dipendenza "tecnica" sugli USA.

AWS Bedrock (managed AI inference) perde clienti grossi per carenza di capacità hw e latenza peggiore:

Customers using Anthropic’s Claude models through Bedrock opted to switch to Anthropic’s own platform or Google Cloud because of “ongoing capacity, latency, and feature parity issues,” according to the July AWS document. Companies such as Figma, Intercom, and Wealthsimple were among those migrating their workloads “due to one or several of these challenges.

Thomson Reuters also chose Google Cloud over Bedrock for its CoCounsel AI product after finding AWS’s service was 15% to 30% slower and lacked key government compliance certifications, the document showed.

CloudFront ora supporta mTLS (mutual TLS authentication con certificato client) senza costi aggiuntivi. Molto interessante.

S3 server access logs at scale. Yelp spiega come usa e gestisce grandi quantità di log di accesso a S3, su S3.

Interessante il postmortem di Incident.io per l'outage AWS di ottobre in us-east-1:

We’re hosted in multiple regions of Google Cloud and so the majority of our product was unaffected by the outage. However, we do depend on third parties for some aspects of our platform, who themselves are hosted in AWS, or have their own dependencies that are.

Il paradosso è che Incident.io serve proprio nelle occasioni in cui ci sono outage, e molte feature avevano problemi, incluso il sistema di autenticazione, il bot di trascrizione dei meeting, le notifiche via SMS e le chiamate, ma soprattutto l'impossibilità di fare deployment di codice perché Docker Hub era offline.

We use Google Container Registry to host our built docker images, which wasn’t impacted by the AWS outage at all, so we were surprised to see failures. We quickly realised however, that the issue actually lay with our base image (golang-1.24.9-alpine). Crucially, this resolved to a Docker registry image, and Docker runs their registry on AWS.

I tentativi di aggirare il problema non sono andati a buon fine e non c'erano nell'immediato alternative che non avessero una dipendenza su AWS.

Differentemente il servizio Ably (API realtime) aveva un sistema multi-region che è funzionato correttamente:

During this week’s AWS us-east-1 outage, Ably maintained full service continuity with no customer impact. This was our multi-region architecture working exactly as designed; error rates were negligibly low and unchanged throughout.

Rivoluzione dei prezzi di AWS CloudFront, 50 TB per 15$ (in su), prima sarebbe costato oltre 4.000 $:

AWS Lambda networking over IPv6 is here!

Charting the life of an Amazon CloudFront request. Un po' di dettagli su come funziona la CDN AWS e l'ordine delle varie fasi di elaborazione di una richiesta.

Today, us-east-1 is the single largest AWS region. It generates an estimated 2.5GW capacity spread across 6 availability zones (AZs) and 158 data centers. In comparison, the second-largest region is us-west-1 in Oregon, with an estimated 1.7GW capacity spread across 4 availability zones (AZs) and 48 data centers. Not only is us-west-1 about smaller than us-east-1, but it also charges higher per-instance prices than us-east-1.

(da The Pragmatic Engineer)

TIL AWS chiama le istanze EC2 "droplet" internamente, come DigitalOcean:

The first subsystem is DropletWorkflow Manager (DWFM), which is responsible for the management of all the underlying physical servers that are used by EC2 for the hosting of EC2 instances – we call these servers “droplets”.

Giornata nerissima per AWS: siamo a 14 ore di problemi diffusi in us-east-1, a varie fasi e con diversi servizi coinvolti.