New Insights into Email Spam Operations

Our group has been studying spamming botnets for a while, and our efforts in developing mitigation techniques and taking down botnets have contributed in decreasing the amount of spam on the Internet. During the last couple of years the spam volumes have significantly dropped, but spam still remains a significant burden to the email infrastructure and to email users. Recently, we have been working on gaining a better understanding of spam operations and of the actors involved in this underground economy. We believe that shedding light on these topics can help researchers develop novel mitigation techniques, and identifying which of the already-existing techniques are particularly effective in crippling spam operations, and should therefore be widely deployed. Our efforts produced two papers.

The first paper, which will be presented at AsiaCCS next week, is a longitudinal study of the spam delivery pipeline. Previous research showed that to set up a spam operation a spammer has to interact with multiple specialized actors. In particular, he has to purchase a list of email addresses to target with his spam emails, and he needs a botnet to send the actual spam. Both services are provided by specialized entities that are active on the underground market, which we call "email harvesters" and "botmasters" respectively. In this paper, we studied the relations between the different actors in the spam ecosystem. We want to understand how widely email lists are sold, and to how many spammers, as well as how many botnets each spammer rents to set up their operations.

To perform our study, we proceeded as follows. First, we disseminated fake email addresses under our control on the web. We consider any access to the web pages where these email addresses are hosted as a possible email harvester, and "fingerprint" it by logging its IP address and user agent. By doing this, every time we receive a spam email destined to a certain address, we can track which email harvester collected that address. Similarly, we can fingerprint the botnet that is sending the spam email by using a technique that we presented at USENIX Security in 2012, called SMTP dialects. In a nutshell, this technique leverages the fact that each implementation of the SMTP protocol used by spambots is different, and that it is possible to assess the family that a bot belongs to just by looking at the sequence of SMTP messages that it exchanges with the email server. Finally, we assume that a single spammer is responsible of each spam campaign, and cluster together similar emails.

After collecting the aforementioned information, we can track a spam operation from its beginning to its end: we know which email list spammers used, as well as which botnet they took advantage of. Our results show that spammers develop some sort of "brand loyalty" both to email harvesters and to botmasters: each spammer that we observed used a single botnet over a period of six months, and kept using the same email list for a long period of time.

The second paper, which was presented at the International Workshop on Cyber Crime earlier this month, studies the elements that a spammer needs to set to make his botnet perform well. We studied the statistics of 24 C&C servers belonging to the Cutwail botnet, looking at which element differentiate successful spammers from failed ones. The first element is the number of bots that the spammer uses. Having too many bots connecting to the C&C server saturates its bandwidth and results in bad performance. Another element is the size of the email list used by spammers. "Good" spammers trim their email list from non-existing email addresses, avoiding their bots to waste time sending emails that will never get delivered. A third element consists in having bots retry to send an email multiple times after receiving a server error: since many bots have poor Internet connections, this helps keeping the fraction of emails successfully sent high. The last, surprising finding is that the physical location of bots seems not to influence the performance of a spam campaign. As a side effect of this, successful spammers typically purchase bots located in developing countries, which are typically cheaper.

The findings from this paper show us which elements spammers tune to make their operation perform well. Fortunately, there are a number of systems that have been proposed by the research community that target exactly these elements. We think that widely deploying these proposed techniques could significantly cripple spam operations, to a point that might make these operations not profitable anymore. An example of these techniques is B@BEL, a system that detects whether an email sender is reputable or not, and provides fake feedback on whether an email address exists or not anytime it detects the sender as a bot. Providing fake feedback would make it impossible for spammers to cleanup their lists from non-existing email addresses, compromising the performance of their operations.

Similarly, Beverly et al. proposed a system that flags senders as bots if network errors are too common. Such system can be used as a direct countermeasure to having spammers instruct their bots to keep trying sending emails after receiving errors. Finally, SNARE is a system that, among other features, looks at the geographical distance between sender and recipients to detect spam. Since spammers purchase their bots in countries that are typically far away from their victims (who are mostly located in western countries), this system could be very effective in fighting spam if widely deployed.

We hope that the insights provided in these two papers will provide researchers with new ideas to develop effective anti-spam techniques.

SMTP Dialects, or how to detect bots by looking at SMTP conversations

It is somewhat surprising that, in 2012, we are still struggling fighting spam. In fact, any victory we score against botnets is just temporary, and the spam levels raise again after some time. As an example, the amount of spam received worldwide dropped dramatically when Microsoft shut down the Rustock botnet, but has been rising again since then.

For these reasons, we need new techniques to detect and block spam. Current techniques mostly fall in two categories: content analysis and origin analysis. Content analysis techniques look at what is being sent, and typically analyze the content of an email to see if it is indicative of spam (for example, if it contains words that are frequently linked to spam content). Origin analysis techniques, on the other hand, look at who is sending an email, and flag the email as spam if the sender (for example the IP address the email is coming from) is known to be malicious. Both content analysis and origin analysis techniques fall short and have problems in practice. For instance, content analysis is usually very resource intensive, and cannot be run on every email sent to large, busy mail servers. Also, it can be evaded by carefully crafting the spam email. On the other hand, origin analysis techniques often have coverage problems, and fail to detect as malicious many sources that are actually sending out spam.

In our paper B@BEL: Leveraging Email Delivery for Spam Mitigation, that got presented at the USENIX Security Symposium last August, we propose to look at how emails are sent instead. The idea behind our approach is simple: the SMTP protocol, which is used to send emails on the Internet, follows Postel's Law, which states: "Be liberal in what you accept, but conservative in what you send". As a consequence of this, email software developers can come up with their own interpretation of the SMTP protocol, and still be able to successfully send emails. We call these variations of the protocol SMTP dialects. In the paper we show how it is possible to figure out which software (legitimate of malicious) sent a certain email just by looking at the SMTP messages exchanged between the client and the server. We also show how it is possible to enumerate the dialects spoken by spamming bots, and leverage them for spam mitigation.

Although not perfect, this technique allows, if used in conjunction with existing ones, to catch more spam, and it is a useful advancements in the war against spamming botnets.

Poultry Markets: On the Underground Economy of Twitter Followers

Twitter has become such an important medium that companies and celebrities use it extensively to reach their customers and their fans. Nowadays, creating a large and engaged network of followers can determine the difference between succeeding and failing in marketing. However, creating such a network requires time, especially when the party building it does not have an established reputation among the public.

For this reason, a number of websites to help Twitter users create a large network of followers have emerged. These websites promise their subscribers to provide followers in exchange for a fee. In addition, some of these services offer to spread promotional messages in the network. We call this phenomenon Twitter Account Markets. We study this phenomenon in our paper "Poultry Markets: On the Underground Economy of Twitter Followers", that will appear at the SIGCOMM Workshop on Online Social Networks (WOSN) later this year.

Typically, the services offered by a Twitter Account Market are accessible through a webpage, similar to the one below. Customers can buy followers at a rate that is between $20 and $100 for 1,000 followers. In addition, markets typically offer the possibility of having content sent by a certain number of accounts, again in exchange for a fee. 

All Twitter Account Markets we analyzed offer both "free" and "premium" versions of their services. While premium accounts pay for their services, the free ones gain followers by giving away their Twitter credentials (a clever way of phishing). Once the market administrator gets the credentials for an account, he can follow other Twitter accounts (that are free or premium customers of the market), or send out "promoted" content (typically spam). For convenience, the market administrator typically authorizes an OAUTH application by using his victim's stolen credentials. By doing this, he can easily administer a large number of accounts, by leveraging the Twitter API.

Twitter Account Markets are a big problem on Twitter: first, an account with an inflated number of followers tends to look more trustworthy to the other social network users. Second, these services introduce spam in the network.

Of course, Twitter does not like this behavior. In fact, they introduced a clause in their Terms of Service that specifically forbids to participate in Twitter Account Markets operations. Twitter periodically suspends the OAuth applications that are used by Twitter Account Markets. However, since the market administrator has the credentials to his victims' accounts, he can go and authorize a new application, and continue his operation. 

In our paper, we propose techniques to both detect Twitter Account Market victims and customers. We believe that an effective way of mitigating this problem would be to focus on the customers, rather than on the victims. Since participating in a Twitter Account Market violates the terms of service, Twitter could suspend such accounts, and impact the market from the economic side.

Knowing a Bot’s true name, or how to find interesting malware samples

Folklore tells us that by knowing a creature’s true name one obtains great power over her. This is the reason why daemons and such usually don’t tell you their true name, and popstars often times go under pseudonyms.

In a lot more prosaic fashion, security researchers often times struggle in finding malware samples to run for their experiments. The reason is that, most of the time, antivirus companies don’t agree on a name for a malware family.

The way antivirus companies come up with names for malware is funny by itself, and it often generates laughter in the cybercrime community. An example is the Cutwail botnet, whose real name is “Psyche Evolution”. How people came up with Cutwail is a mistery.

Fact is that I was looking for samples for the “Donbot” bot to validate some novel research. According to m86, this botnet is responsible for about 20% of worldwide spam. I looked on anubis, our honeypot system that collects thousands of malware samples, with no results. I even started wondering if that bot really existed, or it was just a legend.

After losing hope, I was told that Donbot is also known as Buzus. No idea what the true name of the bot is, but by using the second name I was suddenly able to find working samples. Which, for a poor grad student struggling with experiments, is good enough.

BotMagnifier: Locating Spambots on the Internet

During the 20th USENIX Security Symposium, which will take place in San Francisco starting August 8, we will present our paper BotMagnifier: Locating Spambots on the Internet.

This paper tries to tackle the problem of detecting bot infected machines from a new perspective: the idea behind BotMagnifier is that bots belonging to the same botnet will share the same codebase and will take orders from the same set of C&C servers. Based on this insight, it should be possible to detect bot infected machines by learning the spamming behavior of a subset of known bots, and look in a network traffic dataset for more machines (i.e., IP addresses) that behaved in the same way.

Having an extensive list of bot infected machines is useful for many purposes: it helps tracking the size of the world's largest spamming botnets, and it can be used by ISPs to clean up their networks, by removing or sanitizing the infected machines.

We developed a system, called BotMagnifier, which is able to grow bot populations from a subset of known spamming bots. In particular, our system builds, for each day of analysis, a collection of IP addresses, called seed pools, that are known to have carried out a specific campaign. To do this, we take advantage of a large spam trap, set up by a US provider. On the side, we also run malware samples, and, when possible, label the campaigns we observe in the spam trap with the botnet that generated them.

After we have the seed pools, we learn the spamming behavior of these IP addresses using a transaction log. A transaction log is a record of transactions carried out on the Internet during the same time period used for the generation of the seed pools. It gives information on which IP address sent an email to which destination at what time. We used the logs from our Spamhaus mirror at UCSB for building the transaction log. There are many feature that can characterize the spamming behavior of a bot. Unfortunately, our transaction log is very partial, and show us only a small part of the email transactions that actually happened. For this reason, we only characterize the behavior of the botnet based on the destinations (i.e., mail servers) its bots contacted during a certain time frame. First, we list the destination each seed pool contacted on the transaction logs. As a second step, we look for more IPs that contacted a certain number of those destination (more than a threshold N), and no others. By doing this, we obtain a magnified pool of IP address, that we believe belong to the same botnet.

To validate our approach, we used the data contained in the Cutwail C&C servers we captured last summer. We extracted a subset of IPs, we grew them as described, and we checked how many of those actually connected to the C&C servers during the time of the experiment. Our results show that, with good confidence, our approach is able to effectively track botnets.

We also ran BotMagnifier in the wild for four months. During this period, we were able to track the activity of the world's largest spamming botnets (Rustock, Lethic, Cutwail), and we detected important events, such as the comeback of the Waledack botnet, or the takedown of MegaD.

Despite our choice of datasets for building seed pools and for the transaction logs, we also show that the approach can work on any other dataset, by tweaking some parameters.

Why timing patterns and botnet detection don't work together

Last year I started working on a project whose goal was to spot behavioral patterns in the bots belonging to different botnets. The basic idea was that bots belonging to a botnet will periodically get orders from a botmaster. The orders will include an e-mail template, and a list of addresses to send those e-mails to. After having carried out its task, a bot would have waited for the next chunk of orders from the botmaster. From a traffic point of view, this behavior would have been reflected in periods of high activity, followed by periods of idleness by the bots.

The first dataset we used to study the bot behavior is a spam trap set up by a large ISP. This spam trap is composed by 150k e-mail addresses, all belonging to the same domain. We logged the e-mails these addresses received for a while, and clustered them in campaigns. Our assumption is that each campaign will be carry out by a single botnet (but, of course, the same botnet can carry out different campaigns). By analyzing the different campaigns, we found that the spam trap addresses received the e-mails during specific times, which reflected in spikes and long idle periods. We were happy about this discovery, that would have made it possible to detect bots just by observing their e-mail sending behavior.

We then moved to the logs from our Spamhaus mirror. By looking at the queries mail server ask to our server, we can infer which IP address sent an e-mail to which server at a given point in time. By looking at these logs, we found out that the same IP addresses that showed nice timing patterns in the spam trap data appeared to be active all the time on this dataset.

To find out what was going on, we decided to run malware samples from the largest spamming botnets at the time. We actually found out that the bots are active all the time, with no meaningful idle periods. We then made another interesting discovery: usually bots get a chunk of a large e-mail list to send their spam to, and often this list is alphabetically ordered by domain. The timing patterns we were seeing are caused by the bots reaching the letter our spam trap domain starts with, and the idle periods were caused by the bots sending mails to other domains! 

Mystery solved, and a good lesson learned.

The Underground Economy of Spam: A Botmaster's Perspective of Coordinating Large-Scale Spam Campaigns

During the 4th USENIX Symposium on Large-Scale Exploits and Emerging Threats, which will take place in Boston next week, we will present our paper The Underground Economy of Spam: A Botmaster's Perspective of Coordinating Large-Scale Spam Campaigns.

It all started last August, when we identified several Command and Control servers responsible of the activities of the Cutwail spamming botnet. Thanks to our contacts with different Internet Service Providers, we managed to take down 16 of these servers and we obtained access to the data stored on them. At first, this operation had a big impact on the botnet's activity, reducing the amount of spam sent by Cutwail by a lot (from our estimates, the servers we took down accounted for half to two thirds of the overall ones). Unfortunately, after a few weeks, the Cutwail crew set up new Command and Control servers, and the spam activity began again. Currently, after the Rustock takedown from last week, Cutwail is the second largest spamming botnet after Lethic.

Even though this takedown has not been very successful in the big picture of fighting world's spam, the data we obtained gave us a unique insight on the modus operandi of the botmasters of a large botnet, as well as on the challenges involved in sending millions of junk emails throughout the Internet. Cutwail is sold as a software package under the name of 0bulk Psyche Evolution. This package provides a web interface that aids the spammer in all the tasks involved in organizing the spam campaign: building a template for the emails, making sure that it doesn't get detected as spam by spamassassin, picking a list of email addresses to send the spam to, and selecting the bots that will carry out that campaign. 0bulk Psyche Evolution comes with a user manual that provides useful instruction on how to correct dimension all the parameters involved in the spam campaign to make it as effective as possible.

The servers we obtained access to had all been set up by the same people, who rented them to different organization to run their spam campaigns. All the most common types of spam were sent using these servers, from phishing, to pharmaceuticals, to malware attachments. In the paper, we provide detailed statistics on how effective these operations were, and how many victims and bots were involved. During a period of one year, the part of the botnet controlled by those servers was able to deliver more than 500 billion messages.

In the paper, we also analyze spamdot.biz, an online forum where it was possible to trade illicit goods such as renting a botnet, or buying a list of email addresses to send spam to. The analysis of this forum gave us an overview on how much these goods are worth. This also gave us the possibility to estimate how much money the Cutwail crew made by running the botnet. Approximately, this should be between 1.7 million dollars to 4.2 million dollars over one year.

Where do all those bots come from?

Many studies periodically tell us where the worldwide spam comes from. One of the latest identified the United States as the country carrying out the most spam, followed by India and Brazil. What is not usually mentioned is whether differences exist between the country distribution of the bots belonging to the main botnets. During one of our recent research projects, we had the possibility to track some of the world's major botnets. We collected IPs belonging to these botnets for four months, from September to the beginning of February, so that we could have a good overview of the whole botnet populations. 

Interestingly, we found out that these populations are not uniform, but vary a lot from botnet to botnet. Below is the country distribution for Rustock:

For this botnet, the majority of the bots we tracked is located in the United States (7.9%), followed by Brazil (7.4%), Vietnam (6.4%) and Germany (6.2%). Rustock is considered to be the most active botnet at the time, and the fact that most of its bots are located in the US corroborates the general statistics we cited.

Other botnets have very different country distributions than Rustock. Here is the worldwide location of Lethic bots:

The country where most Lethic bots are located is Brazil (6.8%), followed by India (6.64%), Russia (6.2%) and Vietnam (5.9%). The United States only host 4.6% of the Lethic bots.

Another interesting country distribution to look at is the one of the Cutwail botnet:

For this botnet, 9.7% of the bots are in Brazil, 7% are in India, 6.7% are in Russia, and only 3.1% are in the US.

The last interesting botnet we tracked is the so-called Waledac 2.0, that started spamming again at the end of December 2010. Here is its country distribution:

For this botnet, the most bots are in Brazil (15%), while the US accounts only for 2.4% of them.

The reason for this big differences might be found in how malware is spread. A common case is a legitimate website that gets compromised, and tries to make user machines install the malicious software with a drive-by-download attack. Of course, if the legitimate site was, for example, a Brazilian site, the majority of the machines that will get infected by visiting it will be from Brazil.

Also, in the underground economy, bots located in different countries are sold at different prices. Those located in Europe or in the United States are the most expensive, mainly because they can ensure higher bandwidth and send out more spam messages.

A new strategy for social network spammers?

As is obvious to whoever surfs in the social media world, Spam on social networking sites is on the rise. As we show in our latest paper, malicious users leverage the ease of reaching thousands of users these platforms provide to deliver their messages to as many people as possible. In addition, the personal information that's available through social networks allows spammers to target their campaigns to those users that are more likely to fall for them. In our paper, we show how adult dating spam on Facebook mostly targets male users. However, most spam campaigns, both on Twitter and on Facebook, are pretty easy to detect, and the profiles that carry them out get shut down fast. The reasons are mainly two: the first one is that spammers activity differs quite a lot from human one, and this make detection of spam profiles possible. The second one is that spam bots generally target their victims randomly, and this produces social graphs that are very different from the ones of real people (that, in general, have a very dense "core" of mutual friends in their graph).

Of course, spammers are coming up with smarter solutions, that make detection harder. Since a few months, I noticed spam profiles on Twitter contacting only people that are linked together. This can easily resemble a real social graph, and from one side makes detection harder. On the other hand, this technique attracts more victims, since people are more likely to befriend somebody if this person is already friend with many of their friends (expecially if it is a cute girl... you get it.). Given these observations, I started wondering how much effort it would take for somebody to create a profile, "merge" into a real social network by contacting people that are linked together, and, after having a few hundreds friends, start spamming. 

To check this, I created a fake Facebook profile, and started contacting some people that were connected together. A boost to this process is given by the fact that some people will accept any request they receive, and this would not only increase the number of friends the bot has, but would also attract friend requests, since the fake profile will start appearing in others' profiles as a suggestion. After having collected a decent amount of friends, say 30, it has been enough to sit down and wait for a while. People started asking friendship on their own, and all the bot had to do was accepting these requests. In a bit more than one month, the bot collected 375 friendships on Facebook. This is an order of magnitude higher than the number of friends reached, on average, using random targeting by the spammers we monitored for our previous paper. Needless to say, the spam or automatic activity detection algorithms of Facebook failed. Time for better anti-spam techniques?

What Spam is out there?

More than 90% of world's email traffic is spam, or at least so they say. We don't see most of it, because it gets filtered by services such as spamassassin, spamhaus, or gmail itself. However, sometimes a message makes it to our inbox. Normal people just delete these messages (some of them actually fail for the scams that are contained in them and buy the advertised stuff), but for security researchers it is also intersting to understand where there messages are coming from. Nowadays, most of the spam is sent by botnets. Infected computers from all over the world get orders from their Command and Control servers, and start sending unwanted emails. Compared to traditional spam, which is carried out by dedicated servers, this approach has the advantage to make it hard the blacklisting of the spamming IPs.

Currently, 4 botnets are responsible for the majority of the worldwide spam: rustock, lethic, megaD and cutwail. Let's see what kind of spam these four botnets send out.
Rustock is suposedly the largest, or at least the most active, spamming botnet out there. A recent research shows that it is responsible for 39% of worldwide spam. This botnet is one of the most sophisticated regarding to packing its e-mails to avoid detection. For example, at the end of the subject line they always put some random text fetched from wikipedia. This botnet is mainly sending out Viagra-related spam, with subject lines of the form dear xys, get 80% off all prices. The body of the email usually contains a fake newsletter, with a link pointing to an online pharmacy. Below is an example of such an email.

 

Lethic is quite large too. This botnet has a fairly specific behavior while packaging its e-mails. It merges together different words, or it mispells common spam keywords to avoid detection. Lethic seems to be running three campaigns. The first one is a drug campaign.  The subjects of the mails belonging to this campaign contain words such as QualityMedications, MaleEnhancement, WithoutPrescription.  The second campaign run by lethic is a replica watch one. The subjects of the mails in this case contain words such as Rep1caWatch and Bvlgari. The third campaign aims to sell proprietary sotware for cheap. Common keywords here are OemSoftwares and Adobe.
Cutwail is another large botnet. As for lethic, they run a cheap drugs / Viagra campaign, but this botnet seems to be focused on phishing scams. In our test setup we observed malware samples belonging to this botnet to send out fake Facebook requests, Amazon and UPS confirmation e-mails, and, more recently, LinkedIn related e-mails. All these e-mails contain links to a fake login page, aiming to steal victims' login credentials.
Below is an example of a phishing e-mail sent by cutwail.

The last large spamming botnet is MegaD. This botnet is sending out male enhancement related emails, but it is easily recognizable from the other ones because of the funny subjects the emails are shipped with (Immense manhood without problems, Eldorado of ero-cures).