MyCrypto’s Security Incident Response 101

An incident will happen at some point. Here’s how to respond.

If you’re reading this, one of the following probably applies to you:

You’ve watched a security incident unfold on the internet.
You’ve experienced a security incident yourself.
You need to prepare for a future security incident.

Either way, you’re in the right place. There are a few distinct steps involved in responding to a security incident.

First: Accept that an incident has occurred or will occur.
Second: Properly respond, both internally and externally.
Last: Deal with the aftermath and learn the best ways to follow-up.

Let’s get started.

Step 1: Acceptance

Accept the following:

It’s not a matter of if you will be hacked, it’s a matter of when.

Don’t delude yourself into thinking you are too smart or too secure. You aren’t. There will always be code that can—and will—be exploited.

You have limited resources. The attackers have limited resources. There are only so many hours in the day and dollars in the bank. However, you must divide those resources between everything you do and protecting against attacks. Attackers only have to dedicate said resources to attacking. They will eventually win.

Now that we’re all on the same page, let’s dive into what you can do about it.

Try to protect against attacks.
Set up systems and a network to detect an attack as quickly as possible.
Respond immediately and properly to an attack.

We sort of touched on the first item in our post on security. However, it should also be noted that every operation is different. Improving opsec is one piece of the puzzle. Reducing the attack surface itself is another.

The second item we will have to save for another post. In brief: How do you know, as soon as possible, that you have been compromised? These may be automated systems or manual (human) ones. In our experience, almost all compromises have been noticed by a team member or the community first, not an automated system. People are your most powerful resource—make sure you are listening to them everywhere, all the time.

We will talk about the third item today, as it seems to be the most commonly ignored. People assume that once you are attacked the world ceases to exist, so they only focus on how to prevent attacks. The opposite is true: once you are attacked, every second and every sentence counts. The most damaging attacks are not the most sophisticated ones; they are the ones that go unnoticed for long periods of time or where the response to the attack creates more confusion and loss.

Why is the crypto space unique?

Why is it that, in the crypto space, an immediate response reduces the amount of loss?

In the “regular” world, it’s very rare that a company keeps their community up-to-date on an attack as it happens. You don’t see Facebook live-tweeting a DNS compromise. You don’t see Home Depot doing minute-by-minute updates of their customer database being stolen. You don’t see Marriott publicly disclosing a bug and asking people to take actions. Why is this space different?

First, it’s decentralized. This means that the attack surfaces are different. Since there are no user details in a database, there is no reason to attack said database. Most attacks obtain the user details or secrets directly from the user or via the application. As a side effect, an attack is now an ongoing situation that will steal data, in real-time, for as long as possible. This is different than a database compromise where an attacker typically enters, dump all the data, and leaves before anyone notices.

Second, it’s decentralized. Unlike Facebook or your credit card details, there is no way to “change” your private key. Once it is compromised it is compromised forever. Facebook can force every user to reset their password, but your crypto wallet can’t.

Thirdly, real money is at stake. While having your Facebook account compromised sucks, it doesn’t result in the loss of your funds. Even if your credit card is compromised, the bank will take care of you. In this space a compromised website results in the loss of funds.

Responding Properly: Internal Communication

Have a Dedicated Channel

As we noted in our previous article, your organization should have a dedicated channel specifically for alerting the team to security incidents and dealing with them. This includes active attacks against your application and infrastructure, or bug reports that come in via support channels.

If there is no dedicated space when an attack occurs or bug is reported, set up a dedicated space immediately and invite anyone who is helping, will be helping, or ends up helping. In a company structure, this will be your team members — all of your team members. Communications and marketing people are just as important as technical engineers.

If you are an open source project or have a less traditional structure, this could also include trusted community members who are helping communicate your message to the outside world.

Get on the Phone

During an urgent incident response, you will be multitasking. You have two eyes and two ears. Being on the phone means you can also talk and listen to your team while viewing and typing elsewhere. Every second you spend looking at the internal communication channel is a second you aren’t on social media, in communication with a host or registrar, or looking into the technical details of the compromise.

When on a call, please be respectful. Even if it’s a ‘video’ call (e.g. Skype, Hangouts), turn off your video out of respect for those with shittier internet connections. Always mute your mic when not talking. Do your best to not clutter any call or chat with irrelevant information or tangents. Write down any information that is not immediately helpful and bring it up later.

Secondly, now is not the time to play the blame game or talk about how horrible the situation is. That can come later. It’s best to focus on what you can do to fix the situation right now.

Use People’s Names

These things move fast. When communicating, always use people’s names. It ensures everything is clear and people don’t assume you are, or are not, talking to them.

Be Concise. Be Specific. Delegate. Have a Clear Ask.

Avoid excessive words, apologies, or beating around the bush, especially if you are taking the lead. Assume that everyone in the group is on board to help and assume that they will understand if you come off a bit blunt. Be sure to appreciate and/or apologize to everyone who helped later, but skip excessive niceties or vague requests now:

Bad: “Hey, John, do you think you could please maybe possibly help look at social media for me for other reports. No worries if you can’t or don’t want to.”
Good: “John, can you search for similar reports across Twitter, Reddit, and our support inbox and put them all in a Google document and share the Google doc link here. Thanks.”

Know when to take the lead and when to follow

There is no room for egos here. If someone else is already leading or the situation seems to be running smoothly, simply drop a, “What can I do to help?” in the channel.

Similarly, it is just as important to recognize when no clear direction is being given. If you are up for the challenge, start giving direction and delegating tasks:

“Based on what I’m hearing, we need to identify compromised addresses. Here’s a spreadsheet (link) we can all add addresses to. Next, can you start drafting a tweet regarding this situation, Marie?”

If you aren’t up for the challenge, gently point it out and encourage someone else to take the initiative:

“I want to help but I’m still catching up. Sandra, you seem to be up to date. Can you list what needs to be done?”

Follow up & follow through

Get in the habit of constantly asking for status updates. Because things move so fast, things have a tendency to be missed on both ends. It sucks to know that a task was completed ten minutes prior but somehow that wasn’t communicated or you didn’t hear.

If you ask someone to do something, ensure that they acknowledge and agree to do it. Don’t assume that they heard you or read the message.

If you were asked to do something, acknowledge and agree to do it and give an approximate ETA up front:

“Yes Griff, I’ll do that now. Should take about 5 mins.”

If you are doing a task:

“Should be ready for the dump in ~15mins”
“Got an error while parsing data — restarting now. ~5mins”
“Spreadsheet with data is complete: [LINK].”

If you delegated a task:

“Marie, you tweeted that right? Can you link here please so we can retweet?”
“John, are you in active communication with Cloudflare? What’s the status?”

Every incident will be different and some situations will be less urgent than others. Regardless of urgency or communication method, stay respectful and focused, and prioritize making the best of this situation for everyone besides yourself. Too often we see people prioritizing their company’s brand or their own ego, which is ultimately harmful to your users, the community, and even your brand/yourself.

Responding Properly: External Communication

While you are sitting in a channel with a bunch of people who are trying to identify and fix the issue at hand, it’s easy to forget about communicating what is happening to the outside world. However, this is one of the most important elements.

How you respond to an attack can literally save people’s funds or cost them. Waiting five minutes or five hours to warn your users could be the difference between $100 stolen or $100,000. A single wrong word in your tweet could result in more funds being stolen… or less. This is why it’s massively important to have a plan, never post a message without having someone else review it, consider the unintended consequences of your message, and don’t let your panic (or ego) overwhelm your common sense.

Where to Communicate

First, realize what you can and can’t do. If you are short-staffed, you only have the resources to update a single channel. Twitter is usually my recommendation as it allows you to be very short, is public, is easily linkable, and is (hopefully) a trusted source of information associated with your company. If this is the case:

Post your initial message / warning / alert / PSA to a single channel. Include that you will be only monitoring and updating this single channel in the short term.
Post across all other channels to this single (public) point of reference.
Provide updates via that single channel as things develop.

If you don’t have the time to post across all the channels, ask others to do it for you. People love being helpful and will step up to the plate if you give them the opportunity. Don’t be scared to ask for help, even in public. That said, you don’t want to ask them to speak on behalf of your company or ask them to communicate a specific update. Instead, use them to direct people to your official communication source:

“We will be posting frequent updates on our Twitter. If anyone sees confusion or misinformation on Reddit, Telegram, Discord, etc. please direct them to our Twitter. Thank you!!”

Ideally, you will have a few people on your team to help monitor and communicate across multiple social media channels. If you don’t have the proper people on your team, think about who you would ask to help out in a situation now. Think about leaders in this space, people with large social followings, people who are security-minded, people who communicate clearly and responsibly, etc. Grow your network and connections today.

When to Communicate

The “when” is going to depend entirely on the situation at hand. The biggest question that needs to be answered is, “Will communicating this now prevent more loss from occurring?”

It doesn’t matter how much loss. It doesn’t matter who is to blame. It doesn’t matter if you discovered a bug that isn’t “in the wild” or if you are under active attack. It doesn’t matter if it will make your brand look bad. It doesn’t matter if you aren’t entirely sure how big the compromise is. If communicating externally could save one person any amount of funds, you should communicate externally immediately.

Examples include:

Your website was compromised.
There was a bug in your code that could result in lost funds.
Your DNS was hijacked.
Your users have reported a phishing email that appears to be sent from your email address.
Your support system was compromised.
An extension unrelated to your company was compromised that targeted your users.
A news article is sending people to a phishing version of your product.

The only time you should hold off on communicating publicly in the short term is if there is nothing any person could do to save their funds or if alerting the public to the issue would in itself cause more loss.

The Parity multisig hack is a good example of the latter. If you knew that the multisigs had been hacked, you knew there was a bug in the code that could be exploited, and it was suddenly trivial for you to recreate the attack. This would have created more loss. So, instead of immediately releasing all the details, the people closest to the issue opted to share information via more private circles and let it ripple outwards from there. That said, the first official public statement about the attack was made about an hour after the discovery, not days or weeks later. The goal was to get a head-start, not to pretend an incident didn’t occur or lie to the community.

Don’t run around and spread fear. Stop, drop, and think before you speak.

How to Communicate

The first public statement you make is probably the most important piece of the equation. The perfect message accomplishes a number of things, but the primary goal is to save users and/or their funds. Therefore, we should look at it from a user’s point of view, not your point of view. A user doesn’t care what, why, or how it happened during this initial message. As a user, I should walk away knowing:

Am I affected by this? How can I know if I am affected by this?
If I am affected by this, what should I do now?
If I am not currently affected by this, what should I do, or not do, to prevent myself from being affected by this?

The message should be very careful not to directly or indirectly cause more loss. Therefore, it’s always good to lead with who is affected rather than what “being affected” results in or why it happened.

Consider the following example:

“URGENT! PSA! Our website was compromised and user funds are being stolen!!!!!!! Do not visit [link to our website]!!!!!!!!!!!!!11!!!!”

Problems:

It doesn’t tell anyone who is affected or not affected by the issue.
It scares people. Scared people act irrationally.
It links to the website that you don’t want them to visit. Pro-tip: scared people will click links before processing instructions, resulting in more loss.
It doesn’t tell people what to do.

Using the words “our website” is better than a link. Even better is purposefully breaking the link (e.g. “mywebsite[.]com”.) This is especially necessary if you have multiple websites or products, but only one is affected by the situation.

Another example:

“WARNING!!! DON’T ENTER YOUR SEED ON OUR WEBSITE TILL THE FURTHER NOTICE!” [huge image of big red warning symbol]

Problems:

It doesn’t tell anyone who is affected or not affected by the issue.
It scares people. Scared people act irrationally.
It doesn’t tell people what to do or where to go for more information.
It’s grammatically incorrect, contributing to confusion.

A last example:

“Urgent! If you have [popular chrome extension name] chrome extension installed AND used [website name] within the last 24 hrs, please transfer your funds immediately to a brand new account. How to do this: [link to knowledge base tutorial]”

Getting better! This clearly identifies who is affected by the issue and gives those people an action to take to prevent further loss. Be prepared to answer clarifying questions in the responses ASAP.

External Communication: Someone publicly reported an issue with your product

Here’s another scenario. Let’s imagine someone discovered a vulnerability or issue with your product and has decided to inform the public, whether or not they informed YOU first. Their report may be inflammatory or it may contain false information. It’s easy to react emotionally and directly to them and their post.

It’s more productive to create a response that stands alone and deals with the security incident directly. Remember, your primary goal is to save users and/or their funds. When responding, remember:

Don’t deflect. Own it. It’s an issue with your product, regardless of who or what is ultimately to blame.
Don’t attack, dox, or belittle the person who reported the issue. Thank them graciously and sincerely.
Determine and explain who is affected, how they can determine if they were affected, what exactly they should do now.
Determine and explain who is not affected.

It is uncomfortable to be informed that something is wrong with your product and that this issue has resulted in users’ information or funds being stolen. It can get especially difficult when the bug reporter has lost funds. They are in a bad place and are not communicating ideally. It’s your job to deal with that graciously:

Accept their state of mind
Dig into their accusations and attacks to see the actual report / issue.
Don’t attack back.
Make the best of the situation for everyone besides yourself.
Don’t blame them for being upset.

Part 3: After the Fact

So you had an incident. You resolved the incident. Everything is back to normal. Now what?

First, don’t delete your tweets or pretend it didn’t happen. Users were affected. Users could still potentially still be affected or discover that they are affected. Don’t hide the incident or hide from the incident.

The end goal here is to ensure users and their funds are safe. This means that anyone who could have possibly been compromised should be able to easily find information. Anyone who wasn’t compromised should learn from this experience and be safer moving forward. Other projects in this space should learn from your experience so maybe it doesn’t happen to them. Take this opportunity to educate your users and the wider community and make everyone safer.

Update (reply to) the original tweet thread / post.

Target audience: everyone who was following along with the incident as it occurred.
What to say: that the situation is resolved, doing X is now safe and that you will be providing further information tomorrow. Let yourself breath. Answer any really common questions that people are actively asking.
When: Within 1–2 hours after the situation is resolved.

Create a new tweet / post with updated information now that it’s over.

Target audience: everyone who learned about the attack after it had been resolved, everyone who wasn’t following along at the time of the attack, everyone who possibly visited your website while it was under attack and could be compromised.
What to communicate: the same as the initial information, but now updated with actual dates / times, not relative dates / times and providing any updated information you may have learned.
When: ~8–12 hours after the initial tweets.
Where to post: wherever you posted initially, but also link to this from the original threads to “connect the dots”.

Using our example above, you would want to let people know:

If they entered their seed phrase between [TIME_1] and [TIME_2] [TIMEZONE] on [DATE], their account could be compromised.
It only affects people who entered their seed phrase, not used a hardware wallet.
It only affects people during those times.
If they are affected, they should create a new wallet and transfer funds. Link to a guide on how to do this.
Be clear that you have regained control of your site and it’s safe to visit now.
Link to your site and tutorial for your site and where people can go for support (now that it’s safe).

Additionally, you also want to do everything you didn’t have time to do during the incident:

Let people know that you are sorry that this happened and are preparing a full post-mortem on the situation to ensure it never happens again.
Thank anyone and everyone who was helpful during the situation. This could be members of your team, community members, people who helped spread the word, the people from your host/registrar who helped resolve the issue, etc. Call them out by name. Reward them.
Encourage your users to take measures to protect themselves if this were to happen again (e.g. get a hardware wallet or use cold storage, or use a block explorer when they want to check their balance instead of entering their seed phrase.)

Put a message on the homepage of your website itself letting people know about the incident and linking to the above post

Be transparent. Not every user visits your Twitter or Telegram or whatever and not every user is up to date. Just because they don’t actively follow you on Twitter doesn’t mean they don’t deserve to know.

Prepare and post a full post-mortem

Target audience: your current users, everyone watching you, everyone who no longer trusts you, security researchers, other projects in this space who could learn from your experience.
What to communicate: What exactly happened. How it happened. Who or what practices were at fault (sometimes you should be humble and take whatever blame you can, and avoid blaming other products or services unless they really screwed up. It is petty and doesn’t help you rebuild trust as it’s essentially saying, “We can’t prevent this in the future because it’s so-and-so’s fault”). What you will be doing to prevent it from happening again. Recommendations for other projects in the space to help them avoid it happening to them. Recommendations for users so they wouldn’t have been affected. Correct any misinformation that came out during the time of the attack.
Where: Publish the post-mortem on whichever platform you use for blog posts and announcements, and disseminate the link to that post on every other outlet you use to communicate.
When: ~1–2 weeks after the attack. This usually gives you enough time to digest everything and gather all the necessary information.

After figuring out the above pieces, take action and follow the proper examples of external communication above.

How you were alerted to the issue, with sincere thanks to the reporter.
The fact that you are taking full responsibility.
All the technical details of the issue.
The steps you are taking to prevent this type of thing from happening again.
End with sincere thanks to reporter again and let people know how to responsibly disclose.

If you’ve made it this far, we commend you. We don’t expect you to memorize all of this — feel free to come back to it, save it, modify it, and share it.

The best thing you can do is communicate, and the worst thing you can do is communicate improperly.