Careful When You Fake Your Twitter Compromise: We Can Detect It

As those who've followed our research know well, during the last four years we have worked on systems that can automatically detect malicious activity on online social networks. In particular, last year we presented COMPA, a system that learns the typical behaviour of social network accounts and raises an alert if an accounts posts a message that does not comply with such behaviour. We showed that COMPA is a reliable tool to detect accounts that have been compromised, and we showed how our behavioural modelling could have helped in preventing high-profile social network compromises, such as the ones against the Skype Twitter account and the associated press Twitter account.

Flash forward by a year or so, Honda reported that their Twitter account had been hacked last week. The alleged culprit was the cartoon villain and Robot Chicken celebrity Skeletor. 

.@PeopleMag I demand a recount for #SexiestManAlive #BoneStructure #Skeletakeover pic.twitter.com/qFMFYZTfiG

— Honda (@Honda) December 1, 2014

Obviously, fictional characters do not hack Twitter accounts, and it was soon clear that this hack had only been simulated for promotional purposes. It is not the first time that such a thing happens, Chipotle did the same a little over a year ago. Apparently Twitter compromises became so mainstream that faking one is an attractive marketing technique. This trick even works, or so it seems. In the day of the simulated compromise, Chipotle collected more than 4,000 followers, an order of magnitude more than what they typically attract.

The clever marketers at these companies did not take into account COMPA though. Our system was able to correctly assess that nothing was anomalous about the malicious tweets sent by Honda, as well as about the ones sent by Chipotle. Basically, our tool is not only useful in detecting messages that are sent by attackers who gained access to social network accounts, but can also detects compromises that are only simulated.

Next time you stage a Twitter compromise, make sure that your messages look anomalous, otherwise we can detect your bluff.

Detecting the Skype Twitter Compromise

On January 1, 2014 Microsoft-owned Skype added itself to the list of high-profile Twitter compromised accounts. The company's Twitter account posted the following tweet:

As it can be seen, the tweet was even "signed" by the #SEA hashtag, which stands for Syrian Electronic Army. SEa is a group of hackers that support the Syrian regime, which has been responsible for previous high-profile Twitter hacks. The tweet states "Don't use Microsoft emails(hotmail,outlook). They are monitoring your accounts and selling the data to the governments. More details soon #SEA". Basically a classical case of political hacktivism.

The tweet looks  anomalous at first sight, not only for the odd content sent out by the account of a Microsoft-owned Twitter account, but also for the hashtag that attributes the tweet to the middle-eastern group. However, Twitter's automated defenses did not block the tweet as anomalous. Even worse, since the compromise happened on a holiday, it took Microsoft hours before the tweet was taken down.

It is to detect and prevent such incidents that we developed COMPA. COMPA learns the typical behavior of a Twitter account, and flags as anomalous any tweet that significantly deviates from the learned behavioral model. My colleague Manuel Egele and I checked the malicious tweet by the Syrian Electronic Army against the behavioral model built for the Skype Twitter account. The result is positive: had it been deployed on Twitter, COMPA would have detected and blocked the tweet, and saved Microsoft some public relations embarrassment. 

More in detail, the Skype twitter account always posts from the Sprinklr Twitter client, while the malicious tweet was sent by the regular Twitter web interface. This fact is already very suspicious by itself. As a second element, the Skype Twitter account never used the #SEA hashtag before. In addition, the malicious tweet did not contain a URL, which is common practice in Skype's tweets. Interestingly, the time of the day at which the tweet was sent matches the typical sending patterns of the Skype Twitter account. However, this was not enough to evade detection by our system.

This result shows that COMPA is effective in detecting and block tweets that are sent by compromised Twitter accounts. We strongly advocate that Twitter and other social networks implement similar techniques to keep their accounts protected, and block malicious tweets before they are posted.

Could the AP Twitter hack have been prevented?

Twitter hacks can cause a lot of damage. It is news of this week that the Associated Press Twitter account got compromised, and sent a tweet announcing that the White House had been hit by a terrorist attack, and that President Obama was injured. The dynamics of the hack are not clear yet, even though some sources claim that the AP people might have been victim of a spearphishing attack.

What is sure is that the hack had a huge, unprecedented effect on the stock market. Right after the malicious tweet was sent, the New York stock exchange suddenly fell more than 150 points. The market recovered short afterwards (after it was clear that the announcement was a hoax), but somebody could definitely have made a lot of money from this event.

This is the first time that people realize that Tweets can have a large effect on financial institutions. The question that people are asking is: could this compromise have been avoided? The answer is maybe. At the last NDSS Symposium we presented a paper titled "COMPA: Detecting Compromised Accounts on Social Networks." The goal of the paper is to detect, and block, messages that are sent by compromised social network accounts, just like the AP one. Our system leverages a simple observation: people develop habits when using social networks. These habits include connecting to the network at specific times, using certain applications / clients to interact with the network, including links to specific domain in their messages, and so on. When an account gets compromised, the malicious messages that are sent are likely to show differences from this behavior. We developed a system, called COMPA, that learns the typical behavior of users on social networks, and raises an anomaly if a user sends a message that does not comply with the learned behavior.

We ran COMPA on the offending Tweet sent by the AP account. More precisely, we learned the historical behavior of the account, and we checked the malicious tweet against it. COMPA detected the tweet as anomalous. In particular, the tweet was sent from the web, while the AP operators typically use the SocialFlow app. In addition, the tweet does not include a URL, which is something that pretty much every news tweet contains. This is not a surprise to us. When the Fox News Politics account got hacked in 2011, COMPA was able to detect the offending tweet as anomalous too.

We think that the type of behavioral modelling that we ran in COMPA is the way in which social networks should implement their detection of compromised accounts algorithms, and we hope to see this type of techniques deployed in the wild in a near future.

Fake followers on Twitter: my two cents

During the last few days, a huge fuss has been made about this report. This article, written by Italian professor Marco Camisani Calzolari, describes a system to detect fake followers on Twitter. The article shows how many of the Twitter followers of corporations and celebrities (up to 45%) are actually fake. Among such celebrities are Italian public persons and politicians such as Beppe Grillo and Nichi Vendola. The news got a lot of attention in Italy, and got reported by foreign press as well (most notably by the the Guardian and the Huffington Post). Of course, a lot of outrage was generated by the supporters of this or that politician, and many people argued that the study wasn't correct. Today, Italian economics professor Francesco Sacco declared that the study actually has an error margin of 1%, and should be considered correct.

Now, I am a researcher, and I am not very interested in flame wars between opposite political factions. However, I am quite disappointed that the Italian press, as well as some foreign newspapers, considered this study as reputable without at least checking with an expert. As of today, a few days after the news was first published, the only person from academia who reviewed the article is an economics professor. With all due respect, I think that somebody with a degree in computer science and some experience in machine learning and artificial intelligence would be a better person to review this article, and judge how reasonable the proposed approach actually is.

I decided to write this blog post because I have been reading a lot of comments on this article, but most of them were just flames, and very few of them analyzed the proposed approach in detail. I decided to analyze it myself. After all, I have been doing research in the field for quite a while now. In the academic world, we have this procedure called peer review. When somebody submits a paper to a journal or to a conference, the paper gets read by two to three other researchers, who value the validity of the proposed approach, and how reasonable the results sound. If the reviewers think the paper is good enough, it will be published. Otherwise, the author will have to make some changes to the paper, and submit elsewhere.

Camisani didn't go through this process, but just uploaded the paper to his website.  For this reason, neither the approach nor the results have been vetted. Let's play what-if, and pretend that this paper actually got submitted to a conference, and that I had been assigned to review it. Here is what I would have written:

The paper proposes a method to detect fake Twitter accounts (i.e., bots) that follow popular accounts, such as the ones belonging to celebrities and corporations. To this end, the author identified a number of features that are typical of "human" activity, as well as ones that are indicative of automatic, "bot-like" activity. For each account taken into consideration, if the account shows features that are typical of a human, it will get "human points". Conversely, if it shows features that are typical of a bot, it will get "bot points". The total of human and bot points gets then passed to a decision software, that decides whether the account is real or not. Here comes the first problem with the article: the decision procedure is not described at all. How many "bot" point does an account need to score to be considered as a bot? Is this compared to "human" points? And how are the accounts that lie in the grey area in the middle calculated? Also, the classification features are not discussed. Why are those typical of human or bot activity? Why is posting from multiple applications a sign of being a human? On the contrary, this could be a sign of being a bot, since Twitter periodically blocks offending applications and miscreants have to create new ones. Moreover, the classification procedure seems to be ad hoc and unverified. Using a classification algorithm and a training phase on labeled data would have helped - a lot.

The second problem with the paper is that it is not clear how the followers for the analysis have been chosen. Only "up to" 10,000 followers per each account were checked, allegedly by using a random algorithm. This has been done, I believe, because Twitter limits the number of queries that can be asked each hour. However, technical details on how the whole process has been performed are missing. Without such details, it is impossible to evaluate how accurate the results are. Stating that half of the followers of somebody are fake just means that, according to the algorithm, 5,000 followers are maybe fake.

A third problem is that it is impossible to check whether the detected accounts are fake or not. The problem is known to be very hard, because it is pretty much impossible to detect a bot from a fairly inactive account. Twitter itself relies on manual analysis to sort this kind of issues. 

The last problem is that this paper doesn't cite any previous research in the field, and there is been a wealth of it. This way, it is impossible to compare how sound the results are, compared to the state of the art. However, this was not the goal of the paper. The goal was to get publicity, and this worked perfectly.

My verdict? REJECT.

Poultry Markets: On the Underground Economy of Twitter Followers

Twitter has become such an important medium that companies and celebrities use it extensively to reach their customers and their fans. Nowadays, creating a large and engaged network of followers can determine the difference between succeeding and failing in marketing. However, creating such a network requires time, especially when the party building it does not have an established reputation among the public.

For this reason, a number of websites to help Twitter users create a large network of followers have emerged. These websites promise their subscribers to provide followers in exchange for a fee. In addition, some of these services offer to spread promotional messages in the network. We call this phenomenon Twitter Account Markets. We study this phenomenon in our paper "Poultry Markets: On the Underground Economy of Twitter Followers", that will appear at the SIGCOMM Workshop on Online Social Networks (WOSN) later this year.

Typically, the services offered by a Twitter Account Market are accessible through a webpage, similar to the one below. Customers can buy followers at a rate that is between $20 and $100 for 1,000 followers. In addition, markets typically offer the possibility of having content sent by a certain number of accounts, again in exchange for a fee. 

All Twitter Account Markets we analyzed offer both "free" and "premium" versions of their services. While premium accounts pay for their services, the free ones gain followers by giving away their Twitter credentials (a clever way of phishing). Once the market administrator gets the credentials for an account, he can follow other Twitter accounts (that are free or premium customers of the market), or send out "promoted" content (typically spam). For convenience, the market administrator typically authorizes an OAUTH application by using his victim's stolen credentials. By doing this, he can easily administer a large number of accounts, by leveraging the Twitter API.

Twitter Account Markets are a big problem on Twitter: first, an account with an inflated number of followers tends to look more trustworthy to the other social network users. Second, these services introduce spam in the network.

Of course, Twitter does not like this behavior. In fact, they introduced a clause in their Terms of Service that specifically forbids to participate in Twitter Account Markets operations. Twitter periodically suspends the OAuth applications that are used by Twitter Account Markets. However, since the market administrator has the credentials to his victims' accounts, he can go and authorize a new application, and continue his operation. 

In our paper, we propose techniques to both detect Twitter Account Market victims and customers. We believe that an effective way of mitigating this problem would be to focus on the customers, rather than on the victims. Since participating in a Twitter Account Market violates the terms of service, Twitter could suspend such accounts, and impact the market from the economic side.

A new strategy for social network spammers?

As is obvious to whoever surfs in the social media world, Spam on social networking sites is on the rise. As we show in our latest paper, malicious users leverage the ease of reaching thousands of users these platforms provide to deliver their messages to as many people as possible. In addition, the personal information that's available through social networks allows spammers to target their campaigns to those users that are more likely to fall for them. In our paper, we show how adult dating spam on Facebook mostly targets male users. However, most spam campaigns, both on Twitter and on Facebook, are pretty easy to detect, and the profiles that carry them out get shut down fast. The reasons are mainly two: the first one is that spammers activity differs quite a lot from human one, and this make detection of spam profiles possible. The second one is that spam bots generally target their victims randomly, and this produces social graphs that are very different from the ones of real people (that, in general, have a very dense "core" of mutual friends in their graph).

Of course, spammers are coming up with smarter solutions, that make detection harder. Since a few months, I noticed spam profiles on Twitter contacting only people that are linked together. This can easily resemble a real social graph, and from one side makes detection harder. On the other hand, this technique attracts more victims, since people are more likely to befriend somebody if this person is already friend with many of their friends (expecially if it is a cute girl... you get it.). Given these observations, I started wondering how much effort it would take for somebody to create a profile, "merge" into a real social network by contacting people that are linked together, and, after having a few hundreds friends, start spamming. 

To check this, I created a fake Facebook profile, and started contacting some people that were connected together. A boost to this process is given by the fact that some people will accept any request they receive, and this would not only increase the number of friends the bot has, but would also attract friend requests, since the fake profile will start appearing in others' profiles as a suggestion. After having collected a decent amount of friends, say 30, it has been enough to sit down and wait for a while. People started asking friendship on their own, and all the bot had to do was accepting these requests. In a bit more than one month, the bot collected 375 friendships on Facebook. This is an order of magnitude higher than the number of friends reached, on average, using random targeting by the spammers we monitored for our previous paper. Needless to say, the spam or automatic activity detection algorithms of Facebook failed. Time for better anti-spam techniques?

Malicious URLs in Twitter

Social networks are for sure good for keeping in touch with friends and staying informed. However, they also are a terrific starting point for attacks. Malware such as koobface used it as a base for spreading, leveraging the fact that users seems to be not to be aware about threats on these platforms as they are for the ones on traditional e-mail or web, as showed on this recent paper. 

Twitter isn't immune from this trend. For malicious users is very easy to create tweets containing bad links, and these messages are very easy to spread, without any need to set up complicated infrastructure such as the ones required for e-mail spam. To fight this trend, the Twitter crew started filtering the links contained in all the tweets, routing them through a wepawet-like service, able to detect the malicious ones and prevent harm to be caused to the users who click on them  (see this blog post by Twitter for details) 
 However, vulnerabilities are hidden everywhere, and sometimes they are so evident that nobody thinks about them. Look at  this status message. There are no links in the tweet, therefore the twitter screening procedure doesn't even get triggered. However, a malicious link might be hidden somewhere else: each tweets contains a "via <someapp>" link, where <someapp> points to a link specified by the application developer. It turns out that these links are not checked, and therefore might point to malicious sites trying to exploit the user's browser. In the example tweet above, the application link has been set to a page identified by wepawet as malicious.
This flaw has been discovered by Manuel Egele and Thorsten Holz, two researchers of our group, during a project on Twitter security. We alerted the Twitter folks, and they reacted quickly. They ensured us that they started queuing the app links to their malicious URL detection system, therefore the issue should be fixed.
Always think twice before clicking on a link on any social network. If you are looking for a tool helping you in detecting malicious links before clicking on them, check longSHORE.