i’ve spent most of the last week investigating spooky HTTP request that we’ve found in access logs of few production servers. very likely that traffic was replayed by mail content scanners used by messagelabs and mimecast.
all started with java exception popping up in our access logs:
[2020-10-23 08:19:04,037] [qtp306206744-300170] bla.servlet.ServletName - 400: Unable to parse URI query org.eclipse.jetty.http.BadMessageException: 400: Unable to parse URI query [..] Caused by: java.lang.IllegalArgumentException: Not valid encoding '%2S'
surely enough there was accompanying HTTP call that included the unexpected %2S:
69.174.91.34 - - [23/Oct/2020:08:19:04 +0000] "GET /whatever/SomeServlet?param1=npgvba¶mt2=nopqrnopqr¶m3=%2Sfbzr%2Scngu%2S HTTP/1.1" 503 5 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" 0.067 0.064 TLSv1.2/ECDHE-RSA-AES256-GCM-SHA384
SomeServlet is our valid servlet; param1, param2, param3 – are all valid parameter names for that servlet. But values of all 3 parameters were nonsense, and – in case of param3 – not even valid url-encoding.
grepping through logs showed more similar requests arriving from few dozens of IPs. all of those IPs were in 6 /24 subnets: 173.205.33.0/24, 4.53.111.0/24, 69.174.91.0/24, 70.39.129.0/24, 77.67.20.0/24, 77.67.54.0/24 – most hosted by GTT, one – by Level3. needless to say their abuse e-mails did not respond to us in any way.
with help from colleagues we’ve found that most of the requests from those IPs had time correlation with mails coming to us or being sent by us; those mails contained urls that were requested. some of the URLs were called in the original form, including parameters; other had parameter values ROT13’ed – as in the example above.
SomeServlet?param1=npgvba¶mt2=nopqrnopqr¶m3=%2Sfbzr%2Scngu%2S after decrypting becomes SomeServlet?param1=action¶mt2=abcdeabcde¶m3=%2Fsome%2Fpath%2F – and that was very familiar to us.
we’ve found a common pattern for most of mails that correlated with strange HTTP calls. for mails that arrived via messagelabs mail security service we’ve seen mostly this order of events:
- for incoming mails: HTTP request arrives to our servers, few seconds later our MX receives the message
- for outgoing mails: our outgoing server hands over message to MX at message labs, few seconds later HTTP request arrives to our servers
in case of mimecast – it’s more bizzare – we usually first received the message and few seconds later got HTTP call referring to URL from the e-mail.
note that it was e-mails from multiple different clients that traveled via the same 3rd party mail inspectors.
mentions of similar discoveries made by others:
- https://medium.com/@nickbaum/all-bugs-lead-to-rome-505a1205bb9
- https://serverfault.com/questions/573081/does-anyone-recognize-this-e-mail-sniffer-or-malware-using-rot13-encoding
- https://meta.discourse.org/t/invite-tokens-accepted-and-rot13-transformed/90316
- https://stackoverflow.com/questions/32486730/mangled-url-parameters-in-ie9
- https://security.stackexchange.com/questions/48684/help-investigating-potential-website-attack-url-rewriting-and-rot-13-obfuscatio