A 40-minute window in March was enough to expose terabytes of credentials belonging to more than 2,500 organizations. Attackers compromised LiteLLM, an open source tool used to streamline AI-driven software development, and published tainted versions to the package's official location in the Python Package Index. Anyone who ran pip install litellm during that window pulled the malicious build straight from the trusted source. This analysis draws on reporting from Arstechnica.

Security firms CloudSEK and Hudson Rock disclosed the findings this week. Hudson Rock reached its conclusions after analyzing a 195TB file it obtained, and neither firm has identified where that data came from.

The stolen material covers most of what a build environment holds:

• Cloud provider keys and environment variables
• Repository tokens, including GitLab and GitHub personal access tokens
• SSH keys and Kubernetes service account secrets
• Package publishing credentials
• AI provider API keys

Microsoft, Amazon, Cisco, Samsung, and Salesforce are among the organizations whose secrets appear in the data. If your CI/CD pipeline injects any of these as environment variables at build time, the malicious package could read them the same way your own code does.

"A window of roughly 40 minutes in which the LiteLLM dependency was hacked led to over 430,000 instances in which millions of secrets were harvested," wrote Alon Gal, co-founder and CTO of Hudson Rock.

The second-order damage shows why package publishing credentials matter. CloudSEK reported that developers of Trivy, the widely used vulnerability scanner, rotated an automation token but failed to fully revoke it. That left a 20-day window in which attackers could force-push malicious code into third-party builds that consumed Trivy.

Nothing here is a flaw in LiteLLM's code or Trivy's scanning logic. This is a compromise of distribution channels and the credentials that govern them, which means patching the tool does not address the exposure. The credentials taken in March remain valid until someone revokes them.

Business Impact: Downstream Risk Across AI and DevOps Infrastructure

Hudson Rock's Alon Gal put the scale of the harvest at over 430,000 separate instances of secret collection, producing millions of individual credentials. If your development teams pulled the affected AI orchestration package during the exposure window, the question is not whether one key leaked. It is how many of your systems shared an environment with that process.

The credential types matter more than the volume. Package publishing credentials give an attacker the ability to push code under your organization's name, which means your customers download tainted builds from a source they have every reason to trust. Cloud keys and Kubernetes service account tokens give access to the infrastructure those builds run on. Repository tokens give access to the source code itself.

The Trivy case shows what that looks like in practice. Developers of the vulnerability scanner rotated an automation token but did not fully revoke it, leaving roughly a three-week window in which attackers could force-push malicious code into third-party builds that relied on the tool. If your build pipeline consumed one of those builds, your release artifacts inherited the problem without any compromise of your own network.

That is the part your executives need to understand. Your organization can be breached through a dependency of a dependency, and the first evidence may arrive as a customer support ticket about unexpected behavior in software you shipped.

Rotation on paper does not equal revocation in practice. Security researcher Kevin Beaumont reported that one affected US technology company told him the credentials had all been rotated and the incident was a non-event. Working within that company's own responsible disclosure policy, he tested the credentials anyway. Almost every one still worked.

Apply that to your own environment. If your engineering team closed the ticket in March, you still have an open question about whether the old keys were deactivated on the provider side or simply replaced with new ones alongside them. Access that survives a rotation gives an intruder the same permissions as your legitimate staff, which makes the activity look normal in your logs.

The compliance exposure follows the credentials wherever they lead:

• SOC 2 auditors will ask how you established scope, and "we rotated everything" without evidence of revocation and access review is a control failure you will be explaining at your next assessment.
• PCI-DSS obligations apply if any exposed cloud key or container registry credential touched systems in your cardholder data environment.
• HIPAA breach notification analysis is triggered by unauthorized access to systems holding protected health information, not only by proven exfiltration.
• Contractual notification clauses with enterprise customers often carry shorter deadlines than regulators do, and those clocks are measured from discovery.

Microsoft, Amazon, Cisco, Samsung, and Salesforce were among the organizations whose access secrets appeared in the data. Their presence tells you something useful about your own exposure: mature security programs with dedicated supply chain teams were caught by the same dependency. Your smaller footprint does not make you a harder target here, because the collection was indiscriminate.

The financial consequences are less about ransom and more about labor. Credential revocation across cloud accounts, container registries, CI/CD systems, and AI provider APIs consumes engineering hours that were budgeted for delivery, and rebuilding artifacts produced during the affected period extends that work further.

Attack Chain: How Credentials Were Harvested and Distributed

The compromise happened at the distribution point, not in a fork or a lookalike package. Tainted builds were published under the genuine project name at its official Python Package Index listing, which means every integrity assumption downstream held true: the name matched, the registry was the expected one, and the install completed normally. Nothing in a standard dependency pull would have flagged it.

That placement is what makes the technique effective. Automated dependency resolution in CI/CD pipelines, container image builds, and developer workstation setups all fetch from the same upstream. No one had to be phished, click anything, or make a configuration mistake for the malicious build to land in a production environment.

Credential collection worked against the runtime environment of the process itself. Anything readable by the installed package was in scope, which in a typical AI orchestration deployment means a wide surface:

• Environment variables injected by the CI runner or container orchestrator, where API keys and connection strings routinely live in plaintext
• Mounted service account tokens available inside a pod's filesystem
• SSH private keys and repository access tokens present on build agents that clone source during the job
• Provider keys for model endpoints held in local configuration files

The practical consequence for your build infrastructure is that the blast radius equals the secret scope of the runner, not the scope of the application. A shared pipeline executor holding organization-wide tokens hands over organization-wide tokens.

The second stage is where this stopped being a single-vendor problem. CloudSEK documented the Trivy case as the clearest example: the vulnerability scanner's developers rotated an automation token after the disclosure but did not fully revoke the old one. That left roughly a 20-day window in which the original token still authenticated. Attackers used it to force-push malicious code into third-party builds that consumed Trivy, turning a security tool into a delivery mechanism for further compromise.

Force-pushing matters technically because it rewrites commit history rather than adding to it. Downstream consumers pulling the affected branch received altered code under commit references that looked established. A rotation that issues a new credential without invalidating the old one leaves the attacker's access fully intact, which is the specific failure this case illustrates.

Credential validity persisted long past the March window. Kevin Beaumont reported that one large US technology company told him the exposure was a non-issue because everything had been rotated. Working within that organization's own responsible disclosure policy, which permits credential testing, he tried the leaked secrets and found almost all of them still worked. Stolen keys that remain live are indistinguishable from legitimate authentication in most logs, which is why this class of access often goes unnoticed for extended periods.

On indicators: neither CloudSEK nor Hudson Rock published affected version strings, artifact hashes, or attacker infrastructure addresses in the disclosures, and neither firm identified where the harvested data originated. The evidence trail for most organizations sits in their own records, specifically the lockfiles, resolved dependency manifests, and build logs covering March installs of the affected package.

Mapped to MITRE ATT&CK, the sequence runs through T1195.002 (Compromise Software Supply Chain) for initial delivery, T1552 (Unsecured Credentials) for the harvest, and T1078 (Valid Accounts) for everything the collected secrets subsequently unlocked in cloud and source control platforms.

Immediate Actions: Credential Rotation and Exposure Assessment

The most important action is revocation, not rotation alone. CloudSEK told affected organizations to perform aggressive credential revocation and to assume that any secret reachable from the LiteLLM environment is compromised. The distinction matters because rotation issues a new credential while the old one may stay valid until it is explicitly killed.

Trivy's maintainers show what that gap costs. They rotated an automation token but failed to fully revoke it, and over a 20-day window attackers used the still-live token to force-push malicious code into third-party builds that relied on the vulnerability scanner. Nearly three weeks of downstream builds were exposed because one old credential was never invalidated.

In your first 24 hours, work through the following:

• Inventory every place the affected package could have landed. Check lockfiles and package manager manifests across build agents, developer workstations, and container registries, including base images that were rebuilt during the exposure period.
• Invalidate and rotate all cloud keys, Kubernetes service account tokens, and GitLab and GitHub personal access tokens that were reachable from those hosts. Rotate AWS access keys for the IAM users and roles attached to the build runners that installed the package, not just the ones a developer remembers using.
• Verify the old credentials actually fail. Attempt authentication with each retired key and confirm the request is rejected rather than trusting the console's status field.
• Pull access logs for every rotated credential and look at what happened after March. New IAM users, new access keys created by an existing key, package publish events you cannot tie to a release, and API calls from unfamiliar regions all deserve a written explanation.

That verification step is not theoretical. Kevin Beaumont reported that one of the largest US technology companies told him it had rotated everything and the disclosure was a nothingburger, so he tested the credentials under the company's own responsible disclosure policy. Almost every one of them still worked.

Credential misuse looks like normal authentication, which is why log review has to be paired with continuous identity monitoring. In environments Capstone manages, Adlumin ITDR watches authentication patterns across cloud and identity providers and surfaces logins that do not match a user's established behavior, which is how a valid-but-stolen token gets caught after the initial theft window has closed.

Over the next week, move to the artifacts themselves. Scan container images and build outputs produced since March for unexpected binaries or modified dependencies, and review CI/CD pipeline logs for deployments that do not map to a reviewed commit. Enable MFA on all cloud accounts and on your package registry publishing accounts, since publishing rights are what turn one stolen token into tainted releases carrying your organization's name.

CloudSEK also recommended auditing logging and egress filtering. Build runners rarely need to reach arbitrary internet destinations, so restricting outbound traffic from CI hosts to your registries, artifact stores, and cloud APIs limits where harvested secrets can be sent.

Longer term, three controls reduce the blast radius of the next upstream compromise: secrets scanning that runs on every commit and blocks hardcoded keys, a secrets manager issuing short-lived credentials instead of static keys sitting in environment variables, and least-privilege scoping so a build token can publish one package rather than the whole namespace. Pin dependencies by hash so a swapped version does not install silently.

Detection and Investigation: Finding Compromised Credentials in Your Environment

Start the investigation from the credential, not the host. The stolen material was harvested from process environments, so there is no implant to find on a developer workstation and no malicious binary sitting in a container image. The only evidence of exploitation lives in the authentication and API logs of the services those secrets unlock.

Anchor your search window in March, when the harvest occurred, and run it forward to the present. Marcus Hutchins-adjacent testing by researcher Kevin Beaumont found that credentials dating from March still authenticated successfully at the time of disclosure, after at least one affected organization stated it had rotated everything. That means an exploitation window measured in months, not days, and any log retention shorter than that leaves a blind spot in the timeline you cannot close.

In AWS, pivot on the access key identifier rather than the principal name. aws cloudtrail lookup-events --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIA... returns every recorded call tied to that key. The events worth pulling out first are reconnaissance and persistence calls that a build pipeline would never make:

• GetCallerIdentity, ListBuckets, DescribeInstances, and ListRoles in quick succession, which maps to Cloud Infrastructure Discovery (T1580)
• CreateAccessKey, AttachUserPolicy, or CreateLoginProfile against an identity that has never issued them, indicating Account Manipulation (T1098)
• AssumeRole chains originating from an IP or ASN outside your build infrastructure's known ranges
• Large GetObject or CopyObject volumes to buckets outside the pipeline's normal scope, consistent with Transfer Data to Cloud Account (T1537)

Azure and GCP need the same treatment through different lenses. In Entra ID sign-in logs, filter for service principal sign-ins and compare the source IP and user agent against your CI runners, since a non-interactive identity authenticating from a residential or hosting-provider address is the clearest signal of a stolen secret in use. In GCP, filter Cloud Audit Logs on protoPayload.authenticationInfo.principalEmail for each exposed service account and look for setIamPolicy calls or key creation events that no engineer can account for.

Kubernetes service account tokens produce a distinct pattern. Query the API server audit log for subjects matching system:serviceaccount: where the source IP falls outside cluster and node CIDR ranges, then check for secrets list and get verbs, which is how an attacker converts one token into the rest of the namespace's credentials (T1552.007).

Registry and source control logs close the loop on downstream tampering. Pull image push events and compare digests against what your pipeline recorded as built, look for pulls by unauthenticated or unfamiliar identities, and in GitHub or GitLab audit logs review PAT-authenticated activity for new deploy keys, workflow file edits, force-pushes to protected branches, and workflow runs triggered outside normal commit activity.

Correlate across all three layers by timestamp. A cloud API call from an unfamiliar address, followed within minutes by a registry push and a pipeline execution, describes an attacker moving from harvested secret to build system access. Preserve those log extracts before retention expires, because they determine whether you are reporting a credential exposure or a confirmed intrusion with downstream artifact impact.

Preventing Future Supply Chain Compromise in AI and DevOps Tools

The single highest-value change you can make is pinning dependencies to cryptographic hashes rather than version numbers. A pinned version still resolves to whatever artifact currently sits under that release on the registry, while a hash-pinned requirements file fails the install outright if the bytes do not match what you approved. In Python that means pip install --require-hashes -r requirements.txt with digests generated by pip-compile --generate-hashes, and equivalent lockfile enforcement for npm, Go modules, and container base images.

Pair that with a quarantine window on new releases. Run your builds against an internal proxy repository (Artifactory, Nexus, or a private PyPI mirror) configured so that freshly published versions are not eligible for automatic promotion until they have been public for several days. Compromised builds are typically pulled from the registry within hours of discovery, so a short delay removes the exposure without slowing your release cadence in any way developers will notice.

Close the install-time code execution path next. Source distributions run build scripts on whatever host installs them, which is how a package pull becomes arbitrary code on a build agent or a developer laptop. Restrict installs to pre-built wheels with --only-binary=:all: where your dependency tree allows it, and run dependency resolution inside a container with an egress allowlist covering only your registry and mirror endpoints. A harvesting routine that cannot reach an external collection endpoint gathers nothing.

The credential class matters as much as the pipeline. Long-lived cloud keys and personal access tokens sitting in CI environment variables are valuable precisely because they stay valid. Replace them with OIDC federation, where your pipeline exchanges a short-lived workload identity for a cloud token that expires in minutes, and scope each token to the single repository and action that needs it. That change converts a stolen secret from an open door into a expired artifact by the time anyone reads the dump.

Build provenance into the artifacts you produce and enforce it at deployment:

• Generate an SBOM on every build and store it alongside the artifact so you can answer "did we ship this package version" in minutes instead of days.
• Sign build outputs with Sigstore/cosign and record the attestation, tying each image to the commit and runner that produced it.
• Enforce binary authorization at the Kubernetes admission controller so unsigned or unattested images are rejected before scheduling.
• Require signed commits and two-person review on any change to pipeline configuration, workflow files, or release automation.
• Restrict which registries your runners may reach at the network layer, so a redirected dependency cannot silently resolve elsewhere.

Watch the namespace around the packages you depend on and the ones you publish. Register or monitor near-miss variants of your own package names, and subscribe to registry feeds for new publications that closely resemble the tools embedded in your pipelines. Developer tooling for AI orchestration and vulnerability scanning tends to be installed by convention rather than review, which is what gives a lookalike name reach.

SentinelOne covers the execution side of this across managed environments, flagging outbound connections and child processes spawned by a package installer on build agents and developer workstations, which is the point where a tainted dependency stops being a file and starts being running code.

Velocity and control are compatible here because none of these steps add review gates to ordinary feature work. Hashes, proxy quarantine, and signing enforcement run in automation, and the only manual approval required is for changes to the pipeline itself. Treat build configuration as production code and audit it on the same schedule.

Next Steps: Verify Exposure and Establish Ongoing Monitoring

Two questions decide whether this event touches your organization. Does anything in your build pipelines, container images, or developer workstations pull LiteLLM, and does your release process run Trivy? If either answer is yes, treat every secret that was reachable from those processes as compromised until an audit proves otherwise.

This is where the usual vulnerability workflow stops being useful. Software gets patched and the exposure ends. A credential that left your environment stays valid wherever it is accepted, and the party holding it does not need your infrastructure, your network, or your consent to use it. Alon Gal's summary of the event is the one to carry into your risk register: a single upstream breach now reaches thousands of companies at the same time.

That is why the rotation work belongs under incident management rather than a maintenance ticket. Someone owns it, it has a completion date, and the completion is verified rather than asserted. Leadership should ask for four things in writing:

• Asset inventory confirmation: a definitive list of which repositories, pipelines, and machines pulled the affected packages, and which environments those processes had reach into.
• Rotation completion status: per credential type, showing which old secrets have been explicitly revoked and confirmed dead, not merely superseded.
• Forensic findings: what the authentication and API log review found across the window from March forward, including any use you cannot attribute to your own staff or automation.
• Hardening timeline: named owners and dates for the dependency controls your engineering teams committed to.

Until those four are answered, your exposure is unmeasured rather than closed.