Northflank tells customers to assume all environment variables were decrypted as AI API keys are abused
TLDR
- Northflank told customers to assume all environment variables were decrypted and to rotate all secrets after a Wiz alert.12
- Northflank says users reported unauthorised use of AI API keys supplied to the platform.12
- Wiz lists at least nine AI developer platform incidents in October and says reports describe hunting for stored environment variables.3
Cloud hosting platform Northflank disclosed a security incident on 30 October, BleepingComputer reports, after receiving a threat alert from Wiz.2 Northflank's status page shows its first public update at 13:40 UTC telling customers to "Rotate environment variables immediately", followed at 13:58 UTC by "Assume all environment variables have been decrypted. Disable/rotate all keys."1 The company's published timeline puts the Wiz alert at 11:43 UTC, an engineer review at 11:44 UTC, an initial network policy patch at 12:02 UTC, further patching at 12:50 UTC and 13:05 UTC, and a request for Wiz's incident response team at 13:00 UTC.1
At 15:12 UTC Northflank enabled secret rotation for Postgres and MongoDB add-ons, said all users should assume their environment variables were decrypted, and listed secrets on workloads, secret groups and global secrets as affected.1 At 19:28 UTC it said its investigation was ongoing with no further confirmed findings, and that its guidance was to assume all environment variables had been compromised.1 At 23:38 UTC it said users had reported unauthorised use of AI API keys supplied to the platform, and told customers to revoke the credentials with their issuing providers, generate replacements, review AI provider usage and billing logs, and report suspicious usage to Northflank support and the provider.1 Northflank added that the reports "may be consistent with the broader pattern of attacks targeting AI credentials" and that similar IP addresses appear in Wiz's "Black October" report.1 It later enabled audit logs for all customers, including free users, and rolled out secret rotation for MySQL and Redis, then Minio and RabbitMQ add-ons, by 31 October.1
BleepingComputer reports that Northflank has not disclosed how attackers gained access, how many customers were affected, or whether credentials other than the reported AI API keys were stolen or abused, and that Northflank had not replied to its questions.2 BleepingComputer's account says customers were told to assume the secrets inside the variables were stolen, while Northflank's status page uses the words decrypted and, later, compromised.12 BleepingComputer describes Northflank as a platform on which developers deploy and manage applications, containers, databases and AI applications.2
Wiz's post, dated 30 October, lists at least nine AI developer platforms that disclosed incidents in the month: Fireworks AI (6 October), Andromeda (9 October), Zeabur (14 October), Lightning AI (15 October), OpenCode (16 and 17 October), Railway (20 October), Warp (23 October), CommandCode (28 October) and Northflank (30 October).3 Wiz says most are still being analysed and no single vulnerability or dependency links them, but that all public reports highlight targeted exploitation explicitly hunting stored environment variables, with a focus on those granting AI access.3 BleepingComputer adds that Railway's incident also involved potential exposure of decrypted customer environment variables, OpenCode disclosed exposed AI API keys, and Warp reported a breach exposing AI agent logs and user telemetry data, and that Wiz has not confirmed that one set of attackers is behind all nine.2
Wiz also describes two cases in which it observed abuse of enterprise credentials used for Bedrock calls, which it attributes to one of these vendor breaches, without naming the vendor beyond "a compromised vendor".3 It reports about 800,000 Bedrock calls over 30 hours in one case and about 1.7 million over 13 hours in the other, and says that at list price the two cases alone likely ran up over $100,000 in usage.3 BleepingComputer summarises this as roughly 2.5 million Bedrock calls and about $100,000.2 Wiz says it found infrastructure tied to LLM resale services that tested the stolen keys, but could not link those operations to the initial compromises or the large-scale abuse.3
Why it matters: customers of a hosting platform were told to treat every environment variable as exposed, Northflank has yet to say how attackers got in or how many customers are affected, and Wiz describes a wider run of AI platform incidents aimed at stored environment variables.123
Sources
- Security incident (Northflank)
- Northflank security breach leads to abuse of customers' AI API keys (BleepingComputer)
- "Black October" for AI Developer Platforms (Wiz)