On hindsight and risk assessment
Andrew MacPherson
|Feb 27, 2025
First off, there are a lot of posts from vendors and security people touting “This one trick could have prevented the Bybit hack”. At Privy we hold security in high regard and do not believe in ever “punching down” or using a security event for marketing purposes (that said I am writing THIS piece).
Security incidents WILL happen, everyone makes mistakes – if you don't have any incidents it's usually a red flag for me that indicates that there is a problem with logging, auditing or controls. These needn’t be critical events but there is always an air of alertness and investigation in maintaining a robust security program.
The Bybit hack is a tragedy and a huge loss of confidence in the security of the modern web3 ecosystem. It definitely gives us some food for thought in how we think about security and what real enforcement we should have in place. We should be striving together to implement controls that mean everyone is safer and not rely on individual implementations of security features.
I think there are a number of places where we can improve security and a lot of it comes down to a few security fundamentals in how we handle risk:
We should always try and mitigate single points of failure, it's always going to be the weakest point, whether it's one admin that can force push to production or a discord bot that has full control or security tooling that can execute code on a fleet of devices.
We should strive for least privilege - this goes hand in hand with (1) but having systems in place where the least amount of access is always enforced heavily limits blast radius on vulnerabilities. Does someone need full admin access or can we limit their access to just what they need? Do they need it all the time or only when they are on call? Can we limit a wallet to only be able to transact with a known set of addresses?
Security from the ground up - the lower down your technically enforced security controls are the better secured the systems above that use this. For example rather than have developers have to individually figure out how to deploy s3 buckets and the security controls around them providing a paved road where the safest option is also the easiest option.
Swiss cheese security / assume breach - with enough time and money everything is hackable. Always layer security and approach it with defense in depth as a means of defending where even if attackers completely compromise a layer of defense they can get caught at multiple other layers. For example if someone installs a malicious chrome extension and has their username and password taken you have fallback layers of detecting unusual logins from different locations which locks accounts and enforcing hardware based passkeys for critical systems to prevent attackers from gaining substantial foothold in your organisation.
So lets dig into the Bybit hack and review what we currently know as what happened (this is subject to change as before Wednesday we believed it was a client side compromise on Bybit and that could still ALSO be true).
Bybit were performing a routine transfer from a COLD wallet (SAFE was used) to their company HOT wallet (used for day to day funds)
Bybit used a multisig onchain SAFE wallet (they require multiple approvals to perform an action) Attackers compromised a SAFE developer access (we don't know the exact method, could be account or device access)
Attackers used this access to modify an S3 bucket that was used to serve compiled production javascript (malicious script can be found at https://web[.]archive[.]org/web/20250219172905id_/https://app.safe.global/_next/static/chunks/pages/_app-52c9031bfa03da47.js)
This javascript was effectively modified to say “Are you trying to access the Bybit SAFE? If yes then send back this transaction to sign
Bybit saw a valid transaction for the movement of funds, noticed it passed simulation and approved The ACTUAL transaction was one to upgrade the SAFE contract to be attacker controlled rather than moving funds via a delegate call
I want to make it clear that we are not shilling any product nor pointing fingers of blame and we believe that lashing someone else's security says more about your security maturity rather than anything else. Promoting security and especially that which helps users and the ecosystem as a whole is how we get to a better, safer place. From pcaversaccio’s hard work running Seal911 to the fact that SAFE have removed the ability to do eth_sign. We can make a difference.
Risk assessment & actionable items
If we take a look at the operation and look at it through an objective view we can see a simple risk assessment as follows:
Large value of funds are at risk when we do anything involving a cold wallet ( $1.5B in this case!)
Single point of failure being the SAFE web interface that was used (but somewhat mitigated by having multiple signers) which suggests a transaction and is signed
Large privilege scope where the signers could sign any operation for a cold wallet
Some security controls were at the highest (user) level – users had to manually validate bytecode seen on their ledger wallets
So what actionable items could be used to mitigate some of these risks?
Minimizing funds at risk
Where possible it’s always important to mitigate the risk of worst-case-scenarios where everything goes wrong and total control is lost.
Instead of moving assets from cold to hot wallets where the cold wallet has an enormous amount of funds you can employ one of the following:
Split cold wallets into smaller wallets to minimize the assets at risk
Use an intermediary wallet where there is a fixed amount transferred from the cold wallet to a separate hot wallet which can be staged. Ie the only operation allowed is moving say 50K (or whatever your risk threshold is) from the cold wallet to a specific hot wallet. You can limit destinations, amounts and time (so you cant do 100 transactions at once)
Client diversity
For the same reasons we have client diversity in ethereum ( https://clientdiversity.org/#why ) we want to mitigate some of the risks of a single place where badness can happen that impacts multiple outcomes, some ways of implementing this
Use multiple SAFE interfaces, these could entail running some yourself ( see https://github.com/safe-global/safe-infrastructure )
Using a separate application to build the transaction, this could be something like the safe utils ( https://safeutils.openzeppelin.com/ or https://safehashpreview.com ) or a simple script that executes for some clients
Use multiple hardware and wallet interfaces - having different wallets on the multisig (eg, Ledger, Trezor, GridPlus) means the attackers need to be able to compromise a lot more and increases the cost for attackers
Use multiple software interfaces - rather than everyone using chrome on chromebooks add diversity in software (firefox, windows, safari) also raises the costs for attackers looking to compromise this
Push security down for signers
Having signers have the ability to sign any type of transaction where they only need to sign very specific transactions is like using a sledgehammer for a scalpel, it works but its much higher risk!
Signers should only be able to sign what they need and wallets implementations should technically enforce that they can only transact with known addresses, this can be implemented with policies for TEE wallets (similar to what privy has) or with specific TEE implementations such as the great work the Phala and Flashbots teams have been doing with Dstack
What about the SAFE compromise
At this stage all we know is attackers gained access to the S3 bucket via a single developer compromise. While it doesn't tell us much it does highlight some risks that would be seen from our risk assessment:
Single point of failure - no standard developer credentials should be able to bypass CI/CD and deployment processes. An S3 bucket that serves production should only be writable by the deployment process that requires multiple approvals. A good rule of thumb is only robots in production
Least privilege - No single developer should have access to be able to modify production systems without massive breakglass warnings (these are for total emergency situations). Modifying production environments should not be possible or if it really is required should set off lots of fire alarms
Security from the ground up - The fact that there is a path for a developer to modify a production S3 bucket could indicate that there is no paved road where the safest option is also the easiest option
Actionable Items
Robust CI/CD implementation and testing
Github and Github actions are commonplace and there are multiple blogs outlining good methodologies for implementing them. To keep with the same standards it’s always important to have multiple layers of defense and even GH actions can provide information for attackers. There are a number of awesome (free!) tools to help mitigate these actions:
Zizmor is a fantastic tool that scans your Github actions for common footguns and mistakes
TruffleHog truffle hog allows you to validate no secrets are committed to Github or alerts you if they are (recommend using this as a pre-commit hook)
Socket.dev , while it is a commercial tool its free for open source and small projects, use it to scan for dependency vulnerabilities
AWS Automated Security Helper - this is a neat scripting tool that incorporates a ton of security scanning to give you a great high level overview of what risks look like for a piece of code.
Endpoint detection & response
EDR (endpoint detection & response) is basically a new anti-virus, while its touted as next gen, 0-day preventing tooling in reality you want it for 2 reasons: Stop low level malware (infostealers, clipboard (paste) hijackers, generic repackaged malware) Get indicators of network connections, file writes and process executions and store it somewhere else. This allows you to view what has happened and assess risk as well as write policies and rules that prevent it happening again.
EDR has a lot of push back in crypto because most allow access to remotely connect to devices. My recommendation here is that you disable this functionality or make sure that it requires a second approval to use. Users should be in the loop for what security practices and procedures are and should never feel like security is spying.
Password and Authentication
Use WebAuthN (passkeys if you must, yubikeys if you can), this 100% stops phishing for credentials. Webauthn allows for users to press a button (or biometrically authenticate) when they login to critical surfaces, this should be seamless and the easiest way to authenticate.
Password manager, people are bad at remembering passwords, making good passwords and generally keeping passwords safe. Security teams should optimize to onboarding users to password managers and enabling browser extensions to let users login to sites easier than if they had to remember the password themselves.
Limit credential access to just-in-time or short lived credentials. Make it seamless to revoke and rotate credentials. There are a number of methods that have been used to do this from how cashapp uses short lived generated access from STS ( https://code.cash.app/encryption-using-data-keys ) or the exploration that Andrew Miller and the Flashbots crew are doing on being able to customize what modern authentication will look like inside a TEE
Developer Experience tooling
Developers are employed to be great at what they do… developing. Expecting developers to also understand the entire security landscape and build for it is unacceptable. Modern and mature security teams should always aim to work with developers to make sure that if they have needs that could pose risk (such as modifying production environments or being able to test locally from an external domain). Tooling for this is often custom built but also something that DevX teams can work with security to build. It's always important for security people to remember that if security is forcing developers down a path that's slow or painful to use they are literally taking money away from developers as they can't produce at their maximum ability. That being said there are naturally cases where some friction is required and these should be completely transparent as to why.
Do something for the community
I would like to implore security researchers, developers and anyone who can to contribute to the seal frameworks initiative, a place where anyone can add useful information for improving security for users, developers and customers!
Its also up to us as the security community to uplift people and organisations that do good work in the space. When you see good security work tell people about it! If you’ve done security work tell me and I will promote it!
