What's really important is that you never should keep a compromised system like this running, even if you think you found all modifications the attacker did.
You probably didn't.
So save your configs and set this machine up from scratch.
I was gonna say much the same thing. Once a machine has been compromised, you've got to format and re-image it. The machine in the article could well still have a rootkit installed. Not to mention he doesn't seem to know the initial attack vector, so any security updates are meaningless if someones credentials have been stolen.
Finally, if you're really paranoid, you also have to flash the bios in the machine too as there are rootkits that can potentially survive formatting...
I'd add that you don't even need to have the machine actually be compromised.
If you think it might be compromised, if you have reason to believe that perhaps something bad happened, you probably should re-image it. There may be smoke without a fire, but it's not worth the risk, is it?
True, but even this isn't always perfect. I was told recently that Xen shares the kernel memory space among VMs so if once is compromised at the kernel level all of them could be compromised. This person dislikes using Amazon for this reason because he can't be sure his box isn't popped through another VM.
Generally speaking, Xen-style virtualization is where each VM has their own copy of the kernel and their own memory space. There are new-ish features like transcendent memory which can blur the lines, but I don't think they are widely deployed.
Systems like docker, LXC, and virtuozzo are where you run into the issue you describe.
This is correct, of course, in principle, but in the real world its often not feasible. As a freelancer, the author likely spends much of his time working on small business blogs and brochure sites.
These are frequently set up on a single server, containing code, database, and any file storage necessary. They're frequently written by cloning and modifying a framework, making reinstalling from a known good copy difficult to impossible. They often have no version control and are frequently configured to allow the webserver to modify code, so there is rarely a known good copy of the site code. In short, what's running on the compromised server is often the only usable iteration of the site's code available.
I hope nobody on HN would willingly set up a site like this, but the fact remains that they exist and need to be maintained. Attempting to clean a system in place is nowhere near 100% effective, but it's much more effective than doing nothing, which is exactly what will happen if you try to tell a client that they have to be offline for days, lose several months of data, and pay someone to rewrite half their site from scratch because you want to format their server and attempt to rebuild it.
Definitely. Also, I don't know anything about their relations, but if this is a client machine, first thing to do would be to contact the client telling him that his box is compromised and if he wants me to investigate. Otherwise, from a non-technical person view, the whole (awesome) story looks like somebody attempted to do something with e-mails and he failed.
At first I felt bad that I didn't know all the tricks the author knew, but then I realized I would never, ever try to clean a machine. I would copy (and verify) and config files I needed and just spin up a new vm. Its not worth the time and there's a slim chance you'll actually catch everything. Funny how he didn't just run rootkit hunter. I wonder if it would have detected this hack.
I'm also starting to think that the move from Linux being a sysadmin-only OS to being something that can casually be spun up by even the most jr of developers, means we're having a Microsoft circa 1997 moment where security needs to be commoditized. I wonder if any of the few commercial AV's for Linux are updated frequently enough to catch hacks like these.
I think he's too clever for his own good. This machine should have been considered beyond salvage, remove any precious data (you did have a backup, didn't you?) and re-image. I've had to recover data from hacked boxes a few times for 'brand new customers' and the first thing I do with a system like that is to make sure I get a console wired up and the uplink disconnected. No point in taking chances.
Save any data you need and consider the machine totally compromised.
By that, I mean you need to consider any IPMI processor compromised as well, since these can be attacked (or their secrets stolen). Basically rebuild the machine from as low a level as you can. Depending on the brand of server hardware, you may need physical access.
If you restore data from backup you're fine. Configuration should be re-applied. Unfortunately you need to check every file with an executable bit that comes out of your backups - review shell scripts, recompile executables.
It's rare that attackers tamper with data, usually they just leak it. Source code may be targeted though.
It's complicated to just do something you were not supposed to and then go with: "look I know I had to do that but instead I did that so please give me [amount you never aggreed to pay]".
What he could have done on the other hand is: inform the client of the attack and propose him to get rid of it for a certain fee...
Still, he provided a service that was important for the customer. And, the customer wanted him to configure exim4 because a lot of the emails sent were marked as spam. The compromise was most likely the biggest reason for that.
Sometimes it's like that. He could try and get the customer to pay him for this but he might lose on the long term if they've been a recurring customer for a long time.
The moment he discovered an attacker he should have been on the phone with the client explaining the situation and asking how to proceed. If the client wanted him to shut down and clean up the hack, he should be charging for that. If not, he should have done nothing.
Exactly. Does your business model care if your server is running apache or nginx? No? Then why the hell isn't that automated, baked, and tamper-proofed? We'll know we've arrived when the only feasible attack is sneaking malicious code into a git repo and hoping no one notices.
By regularly redeploying or re-imaging. Exploits can only live for a certain timescale + you can lock down nearly every entry point (no need for SSH access for instance).
I rather prefer these to splash screens, they evoke the opposing page to the beginning of a magazine article; splash screens feel more like studio logo animations before a film, only much less rewarding.
Your source's UX qualifications also seem a bit... dubious.
> Your source's UX qualifications also seem a bit... dubious.
Just a developer and sys admin, my qualifications come purely from usage. I only noticed the trend of web development going back to what it was in the early 2000's where landing pages & splash screens were all the rage. I hate reading any blog on medium or that follows this format to me it detracts from the content and is visually irritating.
Oh god, I didn't realize the linked site was your own. So, in the interest of explaining why I was such an ass, allow me to donate some un-asked for constructive criticism: your site suffers from a lack of contrast/color and also lack of "breathing room" (line-heights and margins are too small). It's not just a matter of taste; it actually taxes the eye and makes it harder to absorb your content. Which isn't bad by the way. Just a bit hard to focus on.
It's all good, you should have seen it before this. I think real programmers should only have websites that render in lynx. :D I want bare bones, hard to read, detail oriented, static website for myself.
We were actually thinking about going away from medium-like splashes. Would love to get more feedback though, but I feel like this could be a right direction.
