At work we recently had a security incident on one of our websites. Here is a REDACTED version of the internal report. It’s unusual to see people sharing reports like this, so I had to write it from scratch. Hopefully it will be useful to someone as a template, or maybe someone will tell me how I could have done better.

I know that the incident wasn’t very serious, and the action points are really rather basic. But sometimes you need to use an incident to raise awareness, and to get agreement for doing the basic things!

There will be a follow-up to this post in the near future describing in detail how we have made a uwsgi jail for fail2ban. (Edit: see uWSGI and fail2ban.)

How did we find out?

On 3rd October 2016, COWORKER spotted a new error reported via Sentry that referred to /etc/passwd, which (quite correctly) caused him some concern. LINK TO TICKET

InvalidSchema: No connection adapters were found for 'file:///etc/passwd'

What happened, when, and on which servers?

The incident was, essentially, a scan intended to discover vulnerabilities. We consider this an attack because we have not authorised any scan, and because this scan was reckless as to its consequences. The scan would only have discovered vulnerabilities by successfully damaging the server or by successfully making unauthorised access to data on the server. On this occasion our existing security prevented these things from happening, but the scan showed clear intent.

Timeline (all 2016-10-03, local time [+0100])

09:52:42 First request on FRONTEND
10:40:21 Sentry error generated on BACKEND (09:40:21 UTC)
13:02:47 Alarm raised (TICKET)
13:46:10 Last request before traffic was blocked on FRONTEND

The attack was mounted against WEBSITE 1 and WEBSITE 2, which are hosted on the server FRONTEND. This server also hosts multiple other websites which were not attacked, including our own company website.

The attacked websites on FRONTEND are implemented via reverse proxies which redirect certain requests to a second server BACKEND and a third server DEVBACKEND. The attack on these websites therefore involved three servers, and records of the attack were preserved on all of them.

What was our immediate response?

I checked the logs on FRONTEND and BACKEND, and saw that IP address IP.v4.ADDR.246 was generating a large amount of intentionally malformed http requests. Then I used the Linux iptables utility on FRONTEND to add a rule that dropped (discarded) all traffic from IP.v4.ADDR.246.

iptables -A INPUT -p tcp --source IP.v4.ADDR.246 -j DROP

Afterwards I could see the attack continuing for ~30 more minutes (the firewall statistics showed an increasing dropped packet count for that rule). Then the attack ceased, and has not been resumed at the time of writing.

Was there any unauthorised disclosure?

No. The scan spidered many of the public pages of each website. A few static pages were retrieved many, many times, in what may have been some kind of throughput stress probe.

Other requests were in the nature of vulnerability probes and did not result in any information disclosures (returning 403, 404 or 500). Even if /etc/passwd had been disclosed, it would only have revealed usernames, not passwords. Unix systems store password information elsewhere: https://en.wikipedia.org/wiki/Passwd#History. It’s probable that this particular request was intended, like the thousands of other invalid requests, as a probe rather than a serious attempt to retrieve that file (there was, for example, another attack on /etc/password using directory traversal).

Forensics (what else have we found out since then?)

Nature of the attack

Some clues to the toolkit being used were found in the uwsgi log; it seems to have been Acunetix (www.acunetix.com) – a commercial vulnerability toolkit with legitimate uses, except that we don’t appreciate it being deployed against our server without consent. Other resources referenced were bxss.me/redirToFile and vulnweb.com, which both seem to be associated with Acunetix. Despite its innocuous appearance, the user agent signature in our logs seems to be specific to Acunetix:

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.21 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.21

Identity of the attacker

According to various databases (Maxmind, RIPE etc), the IP address netblock IP.v4.ADDR.0/24 is located in COUNTRY and associated with COMPANY, which appears to be a small local ISP. Note that this does not mean that the attacker was at that location; it’s just a good guess where the system is that launched the scan. The attacker could be anywhere. Netblock ASredacted is registered to OFFSHORE but indirectly associated with COMPANY.

Pattern of behaviour

The IP address IP.v4.ADDR.246 has regularly been reported for abusive behaviour (indeed, the whole netblock, so the IP may well be dynamically allocated). This includes very recent occurrences, with similar behaviour and with specific reference to the Acunetix tool I previously mentioned. www.abuseipdb.com

Attacks are often preceded by reconnaissance visits. I examined older log files and other systems to see if I could identify any such visits. Gratifyingly, although only one of our websites was attacked on 3rd October, various of our systems were each subjected to a single probing request during a five minute window on 1st October by an attacker at IP address IP.v4.ADDR.21 (that is, in the same netblock as IP.v4.ADDR.246).

BOX1/other_vhosts_access.log.1:HOSTNAME:80 IP.v4.ADDR.21 - - [01/Oct/2016:10:02:59 +0100] "\x16\x03\x02" 400 0 "-" "-"
BOX2/other_vhosts_access.log.1:HOSTNAME:80 IP.v4.ADDR.21 - - [01/Oct/2016:10:04:23 +0100] "\x16\x03\x02" 400 0 "-" "-"
BACKEND/access.log.1:IP.v4.ADDR.21 - - [01/Oct/2016:10:07:03 +0100] "\x16\x03\x02" 400 0 "-" "-"
DEVBACKEND/access.log.1:IP.v4.ADDR.21 - - [01/Oct/2016:10:06:58 +0100] "\x16\x03\x02" 400 0 "-" "-"
BOX3/other_vhosts_access.log.1:HOSTNAME:80 IP.v4.ADDR.21 - - [01/Oct/2016:10:05:49 +0100] "\x16\x03\x02" 400 0 "-" "-"
GREENBOX/other_vhosts_access.log.1:HOSTNAME:80 IP.v4.ADDR.21 - - [01/Oct/2016:10:04:57 +0100] "\x16\x03\x02" 400 0 "-" "-"
FRONTEND/access.log.1:IP.v4.ADDR.21 - - [01/Oct/2016:10:07:06 +0100] "\x16\x03\x02" 400 0 "-" "-"

(The string “\x16\x03\x02” indicates an attempt to find OpenSSL instances that are vulnerable to the POODLE attack: https://en.wikipedia.org/wiki/POODLE)

Interestingly, not all our systems were probed on 1st October. The fact that GREENBOX was probed, but BLUEBOX was not probed, seems to indicate systematic probing of DATA CENTRE EAST addresses (BLUEBOX being at DATA CENTRE WEST).

Conclusion

The pattern of behaviour indicates that this was not a specific targetted attack on CUSTOMER (with their links to COUNTRY). It was most likely an untargetted and rather humdrum attempt at server takeover or defacement.

Learning points (what can we do better?)

Evidently it’s possible for an unsophisticated attack to proceed for quite some time without triggering an error condition, and even then we could potentially miss the significance of what we are seeing. Our response time to the Sentry report was appropriate to Sentry’s purpose (automatic application bug reporting), but inadequate for a security incident that was inadvertently detected by Sentry. [action] We should look at putting some form of intrusion detection in place on all our servers.

We could usefully combine intrusion detection with improved automated response. For example we already use fail2ban to drop malicious ssh traffic. [action] We could extend fail2ban's config to drop malicious http traffic, and explicitly notify us that it has done so. (See below.)

We have been developing an enhanced firewall config for OTHER PROJECT, and [action] we could roll out the firewall config to all the other servers.

[action] We have taken the opportunity to raise awareness (gently) with coworkers and customers that we could be targetted.

The attacker has not yet returned (as far as we know) so we don’t need to reassess our priorities but [action] we need to maintain enhanced vigilance for a few weeks.

Making fail2ban work with uwsgi

Although fail2ban is usually thought of as watching for failed login attempts (including failed attempts to log in to a website via apache authentication), we could configure it to look in the uwsgi log for (e.g.) ten consecutive 403/404/500/etc errors within ten minutes from the same IP address. This is not a complete defence but might protect us from an unsophisticated attacker getting lucky.

But because we make extensive use of Apache reverse proxying into uwsgi, this is not simple.

  • Apache knows where each http request originated, but doesn’t know the final status of the request. However, it already adds an X-Forwarded-For header (containing the originating ip address) to each request.
  • uwsgi (possibly on a different server) knows what the final status of each request is, but doesn’t know where the request originated. However, we can [action] configure uwsgi with the option 'log-x-forwarded-for' to log the IP address from X-Forwarded-For header in place of the IP address of the Apache reverse proxy. This configuration change is trivial (and a welcome improvement in its own right).

We then need to [action] teach fail2ban to watch the uwsgi log, and what action to take when a ban is triggered. This is more complex than simply banning the X-Forwarded-For address, since uwsgi and its associated fail2ban may be running on another server. The following rule scans all incoming traffic for the banned ip address in the X-Forwarded-For header and drops the traffic:

actionban = iptables -I fail2ban-<name> 1 -p tcp -m string --algo bm --string 'X-Forwarded-For: <ip>' -j DROP

It’s nontrivial to share fail2ban addresses between multiple hosts (crazy solutions exist that involve databases and cron jobs), so the Apache server will continue to forward requests to the uwsgi host, unless its own fail2ban instance is triggered.

We should also take the opportunity to [action] review other fail2ban options. I’d suggest ban for three hours if more than 10 failures in 10 minutes (bantime=10800, findtime=600, maxretry=10), and bans should be applied to all ports, not just the service that triggered the ban. This is based on actual observed attempts at compromising ssh on our servers, one of which lasted over six hours of multiple ban/unban cycles (currently 10 mins) from the same address. [action] We could also consider using mod_security with Apache and fail2ban.

Finally I’d suggest we should [action] set up a security@ email address, which fail2ban should mail whenever a ban is triggered, and maybe it should spam us on irc too.