Unix and Linux Help, Resources and information for Unix/Linux, Mac OS X. Articles on blogging, web site mechanics, and self employment. Mostly techy, Unix/Linux related, but we don't really try to stay tightly focused. If you've never been here before, there's a lot to explore.
A customer reported that a Linux machine used for ssh access (to in turn give telnet access to an ancient SCO machine) was refusing logins. I asked him to try logging in as root at the console; he was unable to do so.
When I arrived on site, I found that I could not login as he had said. I rebooted to single use mode and started peeking around. The machine had been hacked; there was little doubt about that. It's HOW it was hacked that bothers me,
First, there was no attempt to hide any evidence. I could see in wtmp and the secure logs that someone had logged in from a German ISP address, attained su status, and created a new su user for himself. He then changed root's password.
Fine so far, right? But then he did something very strange. He hand edited /etc/passwd and added "/nologin" at the end of each line except root and his own. This was what was preventing people from logging in.
Why do that?
My first thought was that this was just a disgruntled employee doing minor mischief. But when I went multi-user and started checking more, I found this:
That looks like the machine has been put into a botnet. I ran rkhunter but didn't find anything else unusual.
This is very odd. If you want the machine for a botnet, why disable the user logins, which only serves to immediately call attention to the machine?
Another oddity: this same issue happened several months earlier. That is, users could not login and the root password was changed. That time, the user access came back before I could get there and I had them boot to single user mode to change the root password. I wish I knew if an irc daemon was running then, but I attributed all of that to user error or a router glitch.
Could it be just an inept hacker? A "kiddie script" that disables logins? But why undo its work? And why redo it now?
And he DID redo it. The time stamps are plain: he did all this just days ago. It makes no sense.
I suspect that this person got in because someone's home machine is already part of the botnet. I don't know how he attained escalated permission, but once you have physical access, all bets are off. We'll have to reinstall the machine, but if I can't identify the source, what's the point?
I don't know. I'm really not sure what to do. For the moment, I've locked down ssh so that only I can get on - I want to see if he does have another back door. But I'm also concerned about other machines in the network - any of these could be compromised also. So where do we go from here? I don't want to put this customer to a lot of expense for nothing, but the whole situation is disquieting.
It does offer a lesson though: when something odd like that happens, we should take the time to look more deeply. If I had spotted that ircd months ago, I'd have... what? I don't know. But still, I should have looked deeper then.
/Linux/strange-hack.html copyright and reprint notice
Yesterday a customer called because he needed to be able to write into a certain share on his Samba server. I ssh'd right in, made the change to the config file, restarted Samba and shot him off an "All set!" email.
Such confidence had I that immediately after hitting send, I left my house to do some errands and when I realized I had forgotten my phone, I didn't even bother to go back for it: everything is under control, all pigs are fed and ready to fly.
Yeah. When I got home, I found both phone and email messages from my customer. Such a nice guy he is - he was APOLOGIZING to me because it didn't work. "Maybe I'm doing something wrong?", he asked.
I ssh'ed in again and saw my "mistake". I had written "writeable" rather than "writable" in the config file. I quickly fixed that, restarted Samba, snapped off another email explaining my error and took a break for lunch.
Unfortunately the pigs seemed to still be having a little trouble with the flying stuff. I had barely bitten into my sandwich before he called again. Permission denied. Can't do it. Was he doing something wrong?, he begged to know? Of course not, I assured him. The damn pigs were just being stubborn.
I double checked. Yes, he had write permission in the directory. What the heck? Here's part of the config file for your amusement:
Those pigs have wings, dammit! So exactly what happens, I asked?
"I choose Save As. I navigate down to CPONLINE..."
Ooops. Magic word. He said "Down", didn't he? The pigs all perked up and started tentatively fluttering their wings. I asked the $64,000 question: "Are you going through the Syn75 share or the CPONLINE share?"
NO, he was not using CPONLINE. He was navigating down through the Syn75 share. THAT share has no write permission - it doesn't matter that CPONLINE is under it, that only is writable if you come to it through the CPONLINE share! I had him map a network drive to CPONLINE and the pigs lifted off into the sky and everybody was happy.
Because he's such a nice guy ("Maybe I'm doing something wrong?) and because I should have paid more attention when he asked that, I'm not even sending him a bill for any of it.
Pigs: to your stations! Fly, you pink porkers, fly!
/Linux/samba-pigs.html copyright and reprint notice
[Sharename]
path = /path/ShareName
valid users = @somegroup
write list = @somegroup
read only = No
create mask = 0660
directory mask = 0770
force create mode = 0660
force directory mode = 0770
BTW, whomever made the CPONLINE share a subshare of syn75 should be forced to slop the hogs for a week for making such a newbie mistake. Oink! Oink!
Thu Nov 5 19:03:04 2009 Michiel
I didn't even know that you could nest shares.
Thu Nov 5 21:11:31 2009 URL works for me - weird Ed
Apologies to those of you complaining that the URL is broken... I just tried it again, and it works for me... strange. I wonder what browsers you're using? I'm using Firefox 3.5.3.
Sorry you had to fetch it from Google cache, Tony. Hope the info was new, though.
Cheers,
Ed.
Add your comments
Let me first apologize to the people who don't yet have Wave accounts - it must be very boring and also frustrating listening to people wax on about things you can't yet experience.
On that note, I do have a few invitations left for regular contributors, customers, and so on.. Just drop me an email.
For those who do have accounts, I'm "pcunix@googlewave.com" and you can find my public waves by searching "with:public creator:pcunix".
Now what I really wanted to talk about.
Last night, I presented Google Wave to our little retirement community Computer Club. They are a mixed crowd: we have a guy who programmed systems in the early 50's for the Department of Defense, another man who managed programmers before he retired, other people who used computers extensively at their jobs... and people who are struggling to understand email.
I can't make everybody happy, but I do try to keep things basic enough for the newbies and deep enough not to bore the old hands. Obviously that can never be entirely successful, but I thought the reactions last night were interesting.
Some people "got it" very quickly. They understood it so much that they were arguing with the people who didn't get it, saying things, like "No, really, this would have been a fantastic tool to have for the work I used to do!" Others were obviously confused.
One man in particular almost seemed angry. "It's confusing", he said. "Why do I need all this stuff that it does? I don't - I do fine with email!"
I was momentarily tempted to ask why he comes to Computer Club if he doesn't want to learn anything new, but I realized that wasn't what he meant: he just doesn't want to learn a new way of doing email. Email as it exists now meets his needs, he doesn't need to combine it with IM, doesn't need in-line replies, doesn't need Yes/No gadgets - doesn't need the confusion, thank you very much. If Google or anybody else is going to try to drag him into using Wave, he'll be kicking and screaming all the way.
I offered my argument that as we start to use Wave for some things, we'll realize that we ought to start with a Wave just because we might need it to be a Wave later. Someone else agreed, pointing out that if you have two ways of doing something, you'll naturally settle in to using one, probably the more powerful way, even if you don't use all of its features all the time. Our Mr. Confused was having none of that, though. As I said, he seemed a little angry - perhaps he had the idea that Google was somehow going to force this upon him.
That's not how its going to happen. Nor are people like him ever going to ask for a Wave account just to kick the tires. Too confusing, no perceived value, not going to happen.
I think that what WILL happen is that Mr. Confused and Mrs. I Never Heard of It Anyway are going to get softly dragged in. That is, they'll click on a Web page that invites them to join a discussion group or to get more information about some subject they are interested in. When they access that link, they'll need a Wave account. By that time, that will be an instant or near instant process - much like getting a Gmail account now. To these people, this will just be another web page, something they need this "Wave account" to access, but that's simple.
They may not even realize at first that they have a new tool. It's just a discussion about local tax rates with their neighbors, or a list of resources and information about something else. They may not realize that they now have the ability to create their own waves. They may know nothing about Yes/No gadgets until they see one and use it. But over time, as they have joined more Waves, they'll start to "get it". Maybe a more technical friend will show them a few tricks, maybe they'll read a little how-to at another web page - or at another Wave!
That's probably how it's going to happen for a lot of people. No kicking and screaming, just a gentle slide into something new.
/Web/gentle-wave.html copyright and reprint notice
There are people who don't understand Google Wave. There are people who don't like Google Wave. There are people who do understand Wave, and people who do like it. Most of those who don't like it just don't understand it, but even a few who do understand still don't like it.
Most (maybe even all) of the complaints you'll hear about Wave are gripes about problems that obviously will be fixed as this progresses out if its current "Preview" stage. Most of the complaints are also client-side implementation issues that have nothing whatsoever to do with the underlying concepts. You need to keep that in mind when listening to negative comments.
One of the most important things to understand about Wave is that anyone can create a Wave server or a Wave client. You can go to http://google.com/wave to use Google's web based client, but I and many other Mac users use Waveboard, a third party Wave client. If you Google for "google wave client", you'll find many other clients. Doing a search for "google wave servers" doesn't yield quite so much, but I did spot at least one, and as time goes on there will be more. Although I and others often refer to "Google Wave", in fact the idea is that Wave servers will be like SMTP servers: anybody can run one and your Wave server will happily talk to any other Wave server.
Let's have a look-see at some of the griping.
Google Wave is IM squared. Not only do you see what the other person is typing, but if there are many people involved in the Wave, you see all of them typing, back-spacing, correcting things - the screen jumps around and it can all be very annoying.
The biggest problem there is the jittery screen. That is, of course, a client side issue - nothing says you HAVE to have the screen updated in real time. And nothing says you have to participate in a Wave that has dozens of people actively typing. One of the truly beautiful things about Wave is the "replay" ability - you can come back when all the excitement has died down and run through the whole thing step by step at your own pace.
I have confidence that client-side issues like this will be fixed, and soon.
If a lot of people are in a Wave, it does get slow. That feels like a client side issue to me - just don't try to show all that activity at the same time. Buffer it up and display it when things calm down a bit.
Early adopters are finding that large Waves crash and burn. It's not hard to split off and start another Wave, but that needs to be fixed. I don't know if that's client side, server side or a general weakness in the protocol, but it needs fixing. Again: Preview release.
When I first got Wave, I found my contacts list populated with people who already have Wave accounts. These people were apparently people I know, or at least have had email correspondence with. I use Gmail, so Google probably pulled them from there.
I don't recognize half the people on that list.
The reason is simple enough: their Wave account doesn't match whatever I know them as. Google knows them, and knows the connection between that account and whatever email I know them as, but Google doesn't let me see that connection.
Presently, you can't organize your Wave contacts into groups. Obviously that's a necessary and useful feature and justas obviously it WILL be added. But right now? Nope.
Remember, this is a "Preview". Sometimes things that are supposed to work get balky. Sometimes your Client loses contact with its server. Sometimes just plain weird stuff happens. For example, I had marked a Wave as "public", which means that anyone can see it and add to it. The darn thing kept losing its public status. It seems to be OK now, but that's annoying.
You can find my public waves by searching (in Wave) for "with:public creator:pcunix"
It's unclear how the problem of misbehaving people will be dealt with. Right now, if you add someone to a Wave, you can't take them off easily or even just block their messages from your view. In a public Wave, any idiot can join the conversation and you can't filter them out. People can add objectionable 'bots to your Waves - somebody added Eliza Robot to one of my public Waves. I was able to delete that, but this kind of nonsense does happen and we will need ways to prevent it.
The matter of you being added to Waves you don't want to be part of is a common complaint, but there's a simple fix - just "mute" the Wave and it won't bother you again. The mute function moves the Wave out of your inbox and ignores any updates that would bring it back to your attention. If you ever change your mind, you can drag it back to your inbox, but otherwise it has been gagged and silenced.
I've talked to people who think Wave is IM. Others think it is email. Still others think it's a Wiki.
It is all of that, and more. Some people, stuck in their false perceptions, may never see the reality. I think as more of us start using Wave, the confused nay-sayers like Robert Scoble will eventually understand.
Just this morning one of my Wave contacts posted a new Wave about using a Wave as a Technical information log. He says:
Trying to visualize how the different elements stitch together is often almost as hard as starting from scratch. The more I play with waves the more it really seems like a one stop shop for interfacing to information.
He has the right idea. Robert Scoble may not understand yet, but others do. The Waves are coming!
/Web/misunderstandig-wave.html copyright and reprint notice
Thanks to one of our readers (thanks again, Donal) I got a Google Wave invitation Friday morning. By the way - if someone says they have sent you an invitation, you may have to wait: Donal said that he had sent that invitation on Monday. I don't know if Google is just slow in processing these or if they are deliberately doling them out slowly (probably the latter), but once you actually get your invite, you can be up and working in minutes.
I started out using Wave in my Firefox browser, but quickly switched to Waveboard, a Mac Wave client. It's not that it's all that much better than running Wave in a browser; it's just that I like having it in its own Dock icon.
I created a few waves and soon had a few conversations going with other Wave users who I added to the Waves. My contacts as supplied by Google seem to be people from my Gmail contacts who also have Wave accounts. I recognize only about half of them, though: probably because they used a different name in email than they do in Wave.
My first conversation was with Donal, thanking him for the invitation. That could have just as easily been done in email or chat, of course; there was no specific reason to use Wave. However, in the next conversation, Wave was useful.
I had started a Wave titled "Until everyone can use this, sure is useless :-)", in which I lamented my inability to bring in people who don't currently have Wave accounts. I have uses in mind for Wave, but without being able to add in non-wave users, I can't do anything useful. I added in everyone in my Contacts list and a few comments soon came.
One of the people happened to be someone I do business with and his appearance reminded me that there was something I wanted to talk to him about. We started doing that in a "private" conversation within the existing wave, but then realized that it was better to spawn it off to a new Wave. That's very easy to do and is an advantage of Wave over Mail and Chat - not that you can't peel off from either, but it's easier in Wave.
I also started a "public" Wave. That's a Wave that anyone can join (assuming you have already been blessed with a Wave account). You create a public Wave by adding "public@a.googlewave.com" to the list of people you want to be able to read the Wave. With that, it's now open to the world. Presently, there is no way to post a link to a public Wave; you have to search within Wave to find them. For example, to find my Wave, you'd search for "with:public Tony Lawrence's Unix, Linux and Mac OS X Tips".
Note that it's perfectly possible to insert advertising into your Waves. I did that here with simple text links, but you could put in Javascript with a Wave Gadget. How long before there's an Adsense Gadget? I'd guess not very long.
I started another Wave called "Will Wave replace Email?". I opined:
For SOME email conversations, Wave is much better. The problem (for mail) is that you don't necessarily know ahead of time that a conversation would be better in Wave. So... once this is ubiquitous and we all understand that, we might just start using Wave instead of email at the beginning of a conversation?
Two people have commented on that so far. One said:
If Google is going to succeed with this they're going to have to do two things.
1. Make it so user@googlewave.com is a real e-mail address that can be used by anyone. Regular e-mails sent to these addresses will automatically turn into Waves for the Wave user.
2. Make it so that a Wave user can create a wave and add people to it that use regular e-mail. Waves will get sent to standard e-mail users as regular messages and get sorted into threads the way their e-mail client chooses to do it.
I suspect that's exactly what will happen. A Wave isn't always better than email, but sometimes it is and you don't necessarily know at the beginning of a conversation that it would be better as a Wave. For example, suppose that halfway through a long back and forth email discussion you need to bring someone else in. With email, you'd have to forward all the prior messages - that can be tough for the recipient. With Wave, you just bring them in and they can replay the previous messages step by step if they want to. That's a powerful advantage over email and it's why, if this does become ubiquitous, many of us will probably just use Wave instead of email.
That could really change things, couldn't it?
/Web/first-google-waves.html copyright and reprint notice
When we first moved here, our address didn't exist in Google Maps, nor could anything but dirt be seen in the satellite images. That changed soon enough and for some time now I've been able to plug in my home address as a starting point for directions.
A year or so ago, I started having a little trouble with that. The reason was because someone I do business with added me to a user generated map. From that point on, Google saw my address as a business address, and would react by asking "Did you mean A.P. Lawrence?". I'd just click on that and everything would be fine.
The other day I noticed that I couldn't do that. Google Maps insists that my address simply does not exist. Indeed, if I summon up a map of the town, the whole street has disappeared! It's still visible in the satellite view, as is my house and even the golf cart parked in my driveway, but the map view shows empty space.
Yahoo still knows we are here. I don't LIKE Yahoo maps, but I'm stuck with them for now. It's not all that critical; I only use these as a failsafe for my car's GPS, but it is a bit unsettling to have disappeared.
Interestingly, one of my neighbors ( who also runs a business from his home) does appear in the business listings. He's at the top of the page, even though our street no longer exists. Why is he still there but I am not? Who knows? Why does google show addresses in other States when I have specifically given both a town, a State and a zip code? How can Google have maps of a street for several years and not have them now? I do not know.
Perhaps Google knows something I do not. We are on the amorphous edges of the supposedly spooky Bridgewater Triangle ; perhaps we are slowly being sucked into another dimension? If so, it's been grand and I will write if I can.
There is a "Report a problem" link at the bottom of the Google page. I clicked on that and explained the vanishing of my street. We'll see how long it takes for Google to find us again.
/Web/google-maps-loses-me.html copyright and reprint notice
Earlier this week I got email from VMware announcing that I could now order Fusion 3.0 for my Mac. Because I had been a beta tester of this, they offered me a coupon code that was supposed to give me a 25% discount.
To my annoyance, it did not. The order page insisted the code was invalid.
In spite of diminished income in this recession, I still have more money than patience, so I just ordered the upgrade without the discount. Of course that ticks me off, so I returned to the email that promised this boon and noted that it didn't say a word about "No reply possible", so I hit reply and (politely) expressed my disappointment.
Yeah, I know - I should not hold my breath.
I don't want to single out VMware here. Yeah, it's really dumb to send out coupon codes that you aren't honoring, but never mind that. Where VMware really fails is that I can't send them email.
That's hardly unusual. At far too many large companies today, email from outside is discouraged or blocked outright. You usually can't hit "Reply" and if you visit their web sites, you are more likely to be forced fill out contact forms that may confine you to certain subjects - your particular concern may not be among the choices.
VMware has such a system. In addition to finding nothing that matched my needs, all of their forms request extraneous information that I don't feel like providing, thank you very much anyway.
Suggestion to companies implementing such things: have a "I think you screwed me" form and DON'T have any required fields other than one of email or phone.
It's possible that someone from VMware may eventually reply, but I have little confidence of that. I could try calling them, but large company voice mail systems aren't fun to navigate. I SHOULD be able to send email. That is the most convenient way to provide everything that they'd need to either redress my complaint or tell me to go stuff it. Nobody has to write down who I am, why I got the code - it's all there, because I'm replying to their promise!
Oh, right: they'd need a lot of people to handle customer emails. Oh, boo-hoo: how much would it truly cost? How much happier would those annoying customers be if they could communicate this way?
As noted, VMware is hardly the only sinner. I'm just ticked at them because they promised me $15.00 off and didn't give it to me. I'll get over it.
/Opinion/answering-email.html copyright and reprint notice
I admit that I have done a "chmod 777" when I should not have. Almost always that came from haste or frustration. Not frustration with Unix permissions, but frustration with whoever had daily care responsibility for the system - their inability to understand permissions might have driven me to this.
Sometimes it hardly matters. At many small businesses, everybody has "got root" anyway and has learned that this magic incantation will "fix" problems. Well, until it breaks a setuid program, of course. Nobody, NOBODY ever learns "chmod +w", do they?
Another rare breakage is /tmp. It's supposed to have the "t" bit set so that only the owners of files can delete, but I've had folks "777" it. Why? Who knows?
More usually the open permissions are applied to some common set of data. All goes well until someone removes (or just moves!) something that is needed by someone else, and then the crying starts.
The most horribly wrong things that can be done with permissions come from people who have learned about "-R" (recursive) or wild cards. Two or three times a year I find a system where someone has done a chmod to ".*". That's bad enough by itself, but when combined with -R, the results can be spectacular.
What you should understand is that this is all unnecessary. Most of us who have to deal with the results of misunderstood chmod's wish that users didn't even know that the numeric form exists. There would be far fewer errors if users only knew the symbolic modes.
The symbolic form is also much more powerful. Consider this :
ONLY the file that was already executable had full execution bits added - try THAT with Windows!
(If you did want to change all the files, you'd use "chmod a+x a b c")
But I'm being silly. People will continue to "chmod 777" anytime they have a problem. Program doesn't work? Chmod 777. Unexpected error? Chmod 777. Grinding noise inside the computer? Chmod 777. Too hot in here? Chmod 777.
Feeling frustrated by people changing permissions for no reason? By now you should know the cure. Say it with me: Chmod 777.
Don't you feel better now?
/Unixart/chmod-777.html copyright and reprint notice
You need to let users create files in a common directory, but you don't want them to be able to delete other's files. Or you've put certain files, directories or symlinks into a user's home directory and don't want them to be able to mess with any of those. What can you do?
If you create /foo and do "chmod 1777 /foo", you'll have a world-writeable directory with the "text bit" set. Any user can create files here, but they can only delete files that they own (root can still rm anything). That's ownership as listed in the "owner" column of an "ls -l". Group ownership doesn't come into play here although it does change responses a bit.
Let's see what happens when Sam tries to remove Pete's files in a directory with the text bit set:
Now watch what happens when a user in the "wheel" group does the same thing:
Having write permission makes rm proceed without caution, only to be brought up short by the restrictions of the "t" bit.
If the problem is removal of a directory and it is not terribly inconvenient for you to have that directory actually be on a separate filesystem, then "mount" can make the directory safe from removal. You can read more at mount --bind, but it's not very complicated. Let's say we have /dev/foo mounted at /foo and I want a "link" to that under /home/fred. All I have to do is:
mount --bind /foo /home/fred/foo
Fred can have full write permissions on /foo if he needs it, but he will not be able to remove /home/fred/foo. Not even root can:
# rm -rf /home/fred/foo
rm: cannot remove directory '/home/fred/foo': Device or resource busy
Now THAT is removal protection!
Typically, ACL's let you avoid complicated groups by setting specific permissions for specific users. Other than setting a file as "immutable" (chattr +i filename on some Linuxes), you really can't prevent removal of a file. Of course setting it that way may also make it useless, as even the owner can't modify or remove it either without doing "chattr -i" first.. See ACL's for more on that.
I've been working out of my home office since 1983, but had no idea I had so much company: The Rise of the 'Homepreneur' (Business Week) says that there are over six and a half million "homepreneurs" and that we are responsible for 10% of private sector employment.
Wow. When I tell people I work from a home office, the reaction is often a bit negative. "How long have you been doing that?", they'll ask, with a strong implication that I must just be "between" jobs. As I've said in other places, you could work for yourself for 50 years, but if you took a job at the Mall just before you died, your obituary will probably mention that: most people don't think of self employment, and especially self employment from a home office, as "real" employment.
When I first started this business, I did rent an office. It didn't take me long to realize that was a silly and unneeded expense. Yes, I had a few clients that had visited me there, but the bulk of my business was at clients offices or done remotely (dial up modems in those days). Many a day my office sat dark and unused - but the rent was still due at the end of the month. I was a "tenant at will", no lease, so after enlisting my father's help to build a workspace desk in my cellar, I moved "home" and stopped paying rent for space I wasn't using.
I actually wasn't paying rent. I had made a Faustian deal with the building owner: free consulting services in exchange for rent. His over-use of those privileges and his annoying business advice were also large factors in my moving to my home.
Admittedly there are times that a home office is inconvenient. Every now and then I get a client who needs or wants to visit. As my office isn't separated from my home, that means making everything presentable for guests - even if someone is supposedly just dropping something off, the beds must be made, there must be fresh towels in the guest bathroom and neither the dishwasher nor the laundry can be running when they arrive.
We also have to get dressed. We would have made the beds anyway, and as the guest bathroom isn't used al that much, the towels might pass, but ordinarily we might not dress until it's time to go get the mail. I'm also accustomed to showering and shaving when I feel like it: 7:00 AM, sure, but there's nothing wrong with 11:00 either. And shaving? Well, maybe, maybe not...
The link above points out that working at home makes financial sense:
Indeed, the most obvious financial benefit for home-based entrepreneurs is lower operating costs. A 2006 SBA study compared tax returns of sole proprietors who deducted home-office expenses with those who deducted commercial rent. That analysis found that home businesses, on average, had lower sales and net profits than companies in commercial spaces. But profitable home-based ventures retained a greater share of their total receipts as net income: 36%, vs. 21% for non-home-based businesses.
That's a fifteen percent difference - a good chunk of extra cash in your pocket. Remember, it's not just rent that you save. There are commuting costs, incidentals like coffee and furniture and often you are duplicating things you may already own at home. When I closed up my office, I ended up with having two of many things I really only needed one of. I could have avoided a fair amount of expense if I had never had that office.
I've often said that although our Congress critters fawn over the big companies who fill their campaign chests, small companies are much more important to our overall economy. I did not realize that the smallest of the small - people very much like me - are such a big part of that.
/Employment/homepreneurs.html copyright and reprint noticeWhen we purchased our present home, I went through some trouble and not-inconsiderable expense to avoid that type of situation. The BCS Technology World Headquarters are in the (finished) basement, and consist of two offices and a shop. There is even separate power for the offices and shop, completely isolated from the upstairs, except for the meter of course.
Should a client visit, s/he can go directly from the front door to the downstairs without having to pass through any of the living space on the main floor. Well, almost...you have to walk past the kitchen to get to the stairs. That's usually not a distraction unless, say, a roast is in the slow cooker and starting to produce that make-you-hungry aroma...
Anyhow, I and others have long maintained that if you intend to work out of your home, you need to be careful to create a clear demarcation between the business space and the living space. I did that by virtue of turning most of the downstairs into business space and limiting downstairs use for non-business purposes to about 30 percent of the available floor space (my wife has a spacious and well-lighted craft area--I may be a Big Dumb Dinosaur, but I'm not stupid). When I step into the BCS Technology space there is nothing that suggests home--well, almost nothing...my wife yelling down to me that the aforementioned roast is ready makes it clear that home is always right above my head.
The separation of business and living space reduces the "bleed over" problem that often afflicts home businesses and occasionally results in domestic difficulties. If you are going to convey and maintain a professional demeanor and appearance during work hours, you cannot have home stuff intruding on the business, e.g., screaming kids vying for your attention as you attempt to talk business with a client. Similarly, once you turn off the office lights and retire to watch TV or do other things with your spouse, you should not have anything about the business present and/or visible--unless, of course, you want your spouse to feel that s/he is competing with the business for your time and attention. That means not having a business phone line ringing the bedroom phone or a workstation set up in a corner of the living room. You do that and I can almost guarantee that your spouse and you will be experiencing some tense moments.
Tue Oct 27 15:10:13 2009 TonyLawrence
We used to have everything separate - the office was in the basement, had its own phone lines, own refrigerator, own door... but in our new home, it's just a room and I didn't bother with separate phone lines.
It would be 'better" to have it as BDD recommends, but my wife and I have been together in the same room for many, many years, so we're fine. I strongly discourage clients from visiting and don't work al that much anyway :-)
Tue Oct 27 18:09:33 2009 BigDumbDinosaur
http://bcstechnology.net
I don't have the refrigerator, but do have a fairly high-end stereo to provide music as I sit in my chair and scratch my (rapidly balding) head. My office is also big enough that I have room for a test bench on which to put PCs for test and repair of either the hardware or (more likely) the OS. Since the development of Windows XP, that bench has seen a lot of use.
Add your comments
I was listening to an NPR show about internet privacy and the "worrysome" fact that internet information lasts forever. The very forgettable guest being interviewed was harping on "forgetting" - he apparently wants us to be able to set retention dates for things the Internet knows about us. Callers chimed in with stories of real and potential embarrassment from discretions and more serious actions that their boss, their children or their spouses might accidentally discover while bumbling about the Internet. SOMETHING MUST BE DONE!
I don't agree.
If you want to go digging around, you can find some "embarrassing" stuff about me on the Internet. That is, you'd find stuff that you might THINK would embarrass me and probably would embarrass whoever that "let's forget it" guy is and apparently could upset some of the people who called in all worried about something they said or did in 1994. As for me, I don't care. If you aren't smart enough to realize that EVERYBODY has skeletons in their closet, that EVERYBODY has been petty, vain, jealous, stupid, dishonest, and worse, why would I care what you think about me?
Wouldn't we all be better off if we stopped pretending that we are perfect or even close to it? I'm not saying we shouldn't strive toward not being jackasses, not doing dumb things. I'm saying we should accept that we are human, we do screw up and we and everyone else just need to get over it.
Maybe if everyone's "dirt" was always easily dredged up we could dispense with this fantasy of saintly people passing through their oh-so-perfect lives without any stain of error. Maybe if nobody could hide their indiscretions and mistakes, our children would better know how to avoid or mitigate their own?
Say it with me now: I can be a jackass and so can everyone else. I have done stupid things, cruel things, idiotic things and so has everyone else. Anyone who presents a perfect facade to the world has dirt behind the curtain and is lying to us overtly or by omission.
If I'm trying to find out what kind of person you are and I come up with nothing, what have I learned? What are the possibilities? Either you've been really sneaky and secretive or you are such a timid, inactive and uninvolved person that you've never had an opportunity to screw up. Do I really like either of those? No.
Let's stop being phony. People screw up. Maybe there are a few untainted people somewhere, but most of us wouldn't like them because they probably have no fire, no spark, nothing to make them interesting. They walk through life so carefully, so fearful of error - what clods!
So no, I don't want the internet to forget anything about me. I want it all preserved forever. I want my future relatives to be able to learn things about me that I can't learn about my ancestors. I want future historians to have a treasure trove of data that will tell them societal secrets that are almost never known about past generations. I don't WANT the Internet to forget!
/Web/forget.html copyright and reprint noticeYes, but the Internet has facilitated them in ways that were heretofore impossible.
As far as whether data retention should be indefinite, that's ultimately a matter of capacity. While I don't foresee us running out of capacity in the immediate future, I suspect older information will disappear simply because someone somewhere at sometime will deem said information to no longer be of sufficient value to retain. Or, some bean counter will say we don't need to spend money and put more disks into the server. Just get rid of some data that is of limited value/interest.
Meanwhile, be careful of what you say and do in public (or "private" in some cases). Don't videotape that orgiastic sexual encounter you had with the neighbor's wife. Odds are it'll end up on U-Toobe, Facebook, etc., and come back to haunt you.
Sat Oct 24 17:00:58 2009 BrettLegree
http://6weeks.ca
"Yes, but the Internet has facilitated them in ways that were heretofore impossible."
Of course - my angle on this (which I didn't expand upon) was more along the lines of "just because it is easy to do something i.e. run a 'morality check' on someone via Google, doesn't mean we should".
Another example - where I work (and where many people work no doubt), management hits employees over the head about "wasting time on the internet using company resources".
Let's face it, people have been wasting time on the job long before computers existed. Reading newspapers, chatting, etc. - by and large, though, the work got done.
But now that it's easy to track one particular "time wasting method"... they pound it into our heads, over and over again.
I get paid to think, sometimes I need a break. I used to surf to tech sites that would - OMG! - expand my knowledge.
But the groovy new system they're using is becoming more and more restrictive. So you know what I do?
Get up from my desk, take a walk. Get a coffee. Chat with someone.
Still gives my brain a break. But I don't learn as much. Oh well, their loss.
Back to the privacy thing - yes, be careful what you do in private lest it end up online and you lose your job.
You know what I say to that, then?
Bring on the mind reading equipment.
Thoughtcrime.
Let's read everyone's minds, so that we're all on an even keel.
I wonder how many politicians and C-level executives would be willing to let us in on *their* dirty little secrets?
:)
Mon Oct 26 13:53:55 2009 RickBrandfass
It is amazing that so many people who post things on the internet don't take longevity into consideration. This can be seen in the available articles, blogs, etc. which do not have a visible post date. Sometimes it is impossible to tell if an article is one week or one decade old. Most of the time, it makes a difference.
Mon Oct 26 14:25:30 2009 TonyLawrence
Sometimes it is impossible to tell if an article is one week or one decade old. Most of the time, it makes a difference.
True. I was guilty of that here. I did have a link at the bottom that would get the original date, but that was inconvenient. A few years back I started putting the date right under the title and have updated many older articles to the same standard. There are still a few old posts out there without those dates, but I fix 'em as I catch 'em.
Add your comments
Comments /Linux/strange-hack.html
Fri Nov 6 03:05:46 2009 anonymous
intrusion detection and eradication?
Fri Nov 6 03:26:49 2009 TonyLawrence
I don't know that you can ever trust eradication.
Fri Nov 6 05:57:59 2009 anonymous
Im new to Linux but why not just set up DenyHosts to work after one attempt? I know that an average hacker would be able to try from IP's all around the world and one attempt for them would not be that big of deal but for the script kids out there I think they would not have the patients.
Fri Nov 6 07:31:52 2009 drag
> Im new to Linux but why not just set up DenyHosts to work after one attempt?
Waste of time.
The correct action is really to just nuke the box from orbit and restore from backup. But since it seems likely that the machine has been hacked for a long time now then that is probably not a good approach...
So the solution is to nuke the box from orbit and then rebuild it from scratch. That is about it.
Really you shouldn't even let it shutdown. The way your suppose to do things is that as soon as you discovered the intrusion to literally just pull the plug from the device. Your only chance to really understand what is going on is by examine evidence on the file system. I don't know how common it is, but it is known that some rootkits look for certain commands and whatnot and sticking a script in runlevel 6 is a pretty simple way to go about covering your tracks. So you just pull the plug from the machine, image the drive using DD and then carry out your investigation using the image you created.
That would of been the right thing to do.
Of course the worst thing you could do would be to try and 'restore' the machine by running anti-rootkits and anti-virus and things like that. Anti-virus is snake oil and can't be trusted to ever make your machine safe again after a successful attack, regardless of what OS your running. The time, money, and effort you put into trying to 'save' a install is being wasted and you'll never know for certain. From a economic perspective it is just going to save time and effort to start over again.
So sorry. That is why we use secure passwords and keep our desktops secure, so we don't have to put up with this sort of shit.
-----------------------------------------------
As far as what to do about the rest of the network of machines I think that if the owner uses similar passwords on multiple machines then those machines are probably hacked. Or are going to get hacked. Even if it is not necessarily true it is a safe assumption.
If the owner refuses to do the sensible and correct thing to pretty much wipe out his network then what still you can do is install a Network-based IDS like snort.
Build a passive ethernet tap and setup a nice machine that will be able keep up with all the network traffic and stick it on the line leading to the router out of the network. If it is just a script kiddie running IRC and doing file swapping it will be pretty easy to spot any other infected machine. Then you can setup email alerts and that sort of thing.
Fri Nov 6 12:23:28 2009 TonyLawrence
Ayuop - Drag is exactly right.
But - it's difficult to convince non-technical people to do that. A botnet compromised machine is generally not obviously harmful to them. As I noted above, if this guy hadn't disabled other people's logins, this might never have been noticed.
Fri Nov 6 13:33:46 2009 turn it into a VM and study it with time....... Petem
i'd turn the machine into a VM before nuking it.. that way you can study all aspects while keeping it segregated..
Fri Nov 6 13:38:05 2009 BruceGarlock
http://bgarlock.com
Some people still think you can get rid of a virus or Trojan. The only remedy is the nuke/pave option. I'd almost make an image of that machine, and turn it into a honeypot, and watch the bad guys. Maybe get enough evidence to actually prosecute them? Of course, if they are in a different country, US law probably is not enforceable.
I have not researched international law regarding computer break-ins, so what is the law? Can anything be done?
Fri Nov 6 13:41:52 2009 TonyLawrence
I was just talking to the owner of the hacked machine.
He thinks this might be another computer consultant he had a falling out with. That could explain the disabling of logins if the guys intent was to be annoying without really being vicious. But what is the IRC channel for? And of course, you still can't assume that other machines aren't compromised.
So...
Fri Nov 6 14:00:40 2009 two reasons Marc
I can think of two reasons for the hacker to block user login:
1.- The hacker needs to guarantee some time on the machine for whatever reason, he knows there's no tech guy on site so he prevents remote login. He wins some hours before someone comes and has phisical access. long shot but who knows.
2.- the hacker is testing the company for future hacks. He may just want to know how long does it take for them to repair it.
The irc server, just something to draw attention.
Fri Nov 6 14:18:00 2009 TonyLawrence
But he left it disabled for days. He killed access on October 30th, and I didn't get there until November 4th. I see that he did login again on November 2d, also.
The "mildly ticked off consultant theory" gains credibility from the previous incident where it was disabled for a few weeks and then miraculously fixed itself.
The possible perp travels overseas a lot (not that he couldn't have used the German ISP from anywhere, of course).
The client's theory is that he just wants to be mildly annoying. He may have been surprised that I didn't get there to fix it quickly the last time (I had the flu!) and he may have just been busy with other things until now or maybe whatever ticked him off just crossed his mind again...
Great theories, but unless the guy left a message saying who he is (rather unlikely!), I don't feel comfortable with the "he just wants to be annoying" idea.
But... it's their system, their security, their choice.
Fri Nov 6 14:38:25 2009 meanasspenguin
http://meanasspenguin.org
This machine sounds like a standard infrastructure access point, where not a whole lot changes. I think that a baseline security scanner would have found this problem and alerted you to exactly what had been done -- much better than after-the-fact forensics. I prefer a product like samhain, but others also swear by tripwire.
Fri Nov 6 14:40:21 2009 anonymous
Recovery:
1) Pull the plug. Now.
2) Image the hard drive for later analysis.
3) Nuke the box.
4) Carefully restore from backups. Not the OS, just data. Install all apps from scratch.
4) Lock down ssh. No one needs to access an ssh box with a password. Use keys.
5) Install an IDS - Tripwire/snort, etc. Use them. Try to find out if he comes back.
6) Check every other system on the network for similar hacks.
Fri Nov 6 14:48:57 2009 Been hacked too, in 1998 anonymous
One of my servers was hacked in 1998. There was a perl script in /tmp, owned by named, running in a tight loop trying to gain access to root. The system was already patched against that vulnerability, but every time it ran, the break-in attempt was noted with an email. After 2 hours, I had over 2400 emails.
I unplugged the network, killed the main process that didn't make sense (a perl script) and began researching what had changed from an off-line disk mirror taken the prior weekend (7days earlier). Only files under /tmp were altered that weren't clearly user modified. I disabled bind, updated the version - bind was just 3 months behind on patches - redeployed it into a chroot'd environment, then wiped all the /tmp/named files. Plugged the box back into the network. No reboot.
Having a recent enough off-line mirror that you can use as validation is important. We retain 30 days of backups for every server. Further, any internet facing server is **expected to be hacked** and CxO folks understand it. We concentrate on what our recovery steps are post-hack for those boxes - push a static web site out ASAP from a read-only NAS mount while we perform the necessary "what happened" research. Dynamic content isn't available, but much of our content is static. Steps are different based on the services impacted, obviously.
Fri Nov 6 15:46:54 2009 Don't forget to check your Winboxes... RevEggplant
Don't forget to question any other boxes running Windows that login to that box. Think keyloggers. They tend to hide once installed. I agree with what the other gents are saying about pulling the plug and stopping everything immediately, then rebuilding from scratch with backups of data added. It's the only way to be sure.
Fri Nov 6 15:54:49 2009 TonyLawrence
Don't forget to question any other boxes running Windows t
Yeah. I know. I said "I'm also concerned about other machines in the network - any of these could be compromised also"
But take it from the clients perspective: We want him to immediately stop all business activity, and then reinstall ALL operating systems and software. That's a major and very expensive undertaking.
And if he's right that this is just a mildly annoyed ex-consultant, he could do all that for nothing. If he's wrong... he NEEDS to do this, and the sooner the better.
Imagine the expense. Some older machines won't have OS install disks - new hardware, new software. The old SCO machine is very troubling because of hardware. Because you would lose a lot of business income otherwise, you'd need to do all this very quickly - bringing in a team to sweep through the place identifying needs and taking action.
NOT pretty.
Fri Nov 6 16:17:28 2009 BruceGarlock
http://bgarlock.com
I need to dig out my copy of "The Cuckoo's Egg: Tracking a Spy Through the Maze of Computer Espionage" by Cliff Stoll - have not read it in a while, and this break-in reminds me of that book.
If you have not read it, I *HIGHLY* recommend it - my wife even enjoyed it, and she's really not into computers - at all..
Fri Nov 6 16:36:16 2009 drag
Ya.. this is a extremely difficult situation.
Ideally they should clean house and just get a fresh start on the computer infrastructure.
But you may be able to figure out what is going on on the via a investigation of the drive contents. If you could figure out if it is a consultant or not then that is one thing, but I find that extremely unlikely. For all we know that the guy cleaned up the machine because he realized he was committing a felony then later on decided to post login information to some script kiddie channel to let the 13-15 year olds commit the crimes.
Regardless: Running a IRC channel is NOT revenge behavior.
Whatever compromise that you come up with make sure that it's a good one. You'll have to impress on him the seriousness of the issues here... For example do they handle customer information on any of the machines? Any credit card information? Then they could lose their ability to take credit cards or at least get a fine if information that they store ever gets traced back to them.
It is common for people to get blackmailed by hackers. It usually works; send some information about the company in a email and threaten to go public if they don't pay a couple thousand dollars. Businesses invariably pay the hacker, and then a few months later they come back for more blackmail money when they get broke again.
Any sort of government contraction? Do they handle any medical information? Theoretically they could be liable for hiding a information loss from people.
If they refuse to work with you then I'd say tell them they are on their own. Your a professional and you have a responsibility to do a job right... a foreman building a building wouldn't just leave out a structural member because the customer demanded that it would be to expensive to install, right? They would just walk away from a job. Tell them they can deal with somebody with no morals or ethics if they don't want to go through at least some of the correct steps.
Fri Nov 6 16:50:15 2009 TonyLawrence
This is outside of my scope anyway. I don't have the resources to handle it. Whatever they decide, someone else is going to have to handle it.
I can assist in specific areas (the SCO box, the mailserver) but I'm not able to handle a system reconstruction because I'm just one person (and no, I don't hire subs).
Also, this really should be overseen by someone who specializes in security. Thar's not me. I don't have the knowledge or the experience.
Fri Nov 6 17:07:20 2009 anonymous
Use rsync from another box every few minutes to return the computer to pristine condition every couple of minutes. That way even if the box does get compromised it is fixed again immediately. You could also block all IP's into the box and only allow people to come in from their home IP adress ranges.
Fri Nov 6 17:18:39 2009 TonyLawrence
Most home users have dynamic ranges. The better solution would be to force them into using ssh keys - http://aplawrence.com/Security/sshpassphrases.html - but IF THE HOME MACHINE IS COMPROMISED, that doesn't help.
The "rsync" idea would be trivial for a hacker to stop. Moreover, it's not much more work to put up a fake rsync responder that would make the other machine think it had successfully copied.
Fri Nov 6 17:26:24 2009 Live CD TroyTruchon
So the customer has some telnet only boxes on an insecure but private network that he remote accesses through this box... well if thats all it does why does it have anything else installed? A customized version of microcore Linux set to reboot once a day with pretty much just sshd, telnet, and GNU Screen would be practically unhackable.
Fri Nov 6 17:29:41 2009 TonyLawrence
Check every other system on the network for similar hacks.
Having been through this before, I can explain the problem with that. It's the same issue as with fast propagating network worms: to kill the infection, you have to take EVERY machine off-network and keep them off until they are clean.
When dealing with a simple problem, you might be able to clean. People do that often with virus attacks - it's a lot better than reimaging every box. But with a hack, how do you know how clever the s.o.b. is? You don't, so your only real choice is full sanitation - horribly expensive and time consuming. Consider that old software may be lost, unavailable, operating systems like their ancient SCO may not work with new hardware... important data files have to be examined and proved free of intrusion... it's a major mess!
Fri Nov 6 17:35:53 2009 TonyLawrence
A customized version of microcore Linux set to reboot once a day with pretty much just sshd, telnet, and GNU Screen would be practically unhackable.
No, not if the source is a compromised home user.
I've said before that VPN's from home users are dangerous. I don't care WHAT they are connecting to, what security is in place - if the home machine has been compromised, everything is at risk.
Fri Nov 6 17:38:20 2009 TonyLawrence
I was reminded of this: http://aplawrence.com/Blog/B371.html
Too many people are too trusting in letting people use VPN's. Great convenience, but a potential risk.
Fri Nov 6 17:38:22 2009 anonymous
Well true, but if you set it to reboot every three hours or so, thus giving you effectively a fresh install each time the Hacker will get tired of rehacking and installing everything three-four times a day.
Fri Nov 6 17:39:29 2009 mario
If you reboot the machine, a running trojan process might not be there anymore. The only solution is to have the server always keep a running terminal on the serial console. A running root shell on the serial console can not be exploited remotely, just with physical access. But this way, you could still login in such a case, where the local accounts or passwords have been tempered with. (As long as the intruder doesn't detect the running serial console.)
Fri Nov 6 17:48:56 2009 TonyLawrence
As long as the intruder doesn't detect the running serial console.)
No different than leaving root logged in on ALT-F3
And a simple "w" or "last" discovers that instantly. To hide that, you'd need to install your own root kit - spy vs. spy :-)
Fri Nov 6 18:02:00 2009 BigDumbDinosaur
http://bcstechnology.net
Of course, my first question would be why wasn't the disgruntled ex-consultant's credentials immediately removed from the system when things went sour between him and the client? After all, the majority of security breaks that occur on UNIX-like systems are made possible by careless username and password policing. Is this another case of having an easy-to-remember root password because "it's convenient?"
Something I've hammered into my clients' heads is the need to keep administrative access to any machine under tight control and keep it limited only to those who have a compelling need. Nevermore so is this requirement than when a machine is exposed to the Internet. As someone above said, it should be assumed that any exposed machine will eventually be subject to a hack attack.
As for the Windows boxes...I believe Rent-A-Dumpster offers a solution... <Grin>
Fri Nov 6 18:38:58 2009 drag
"""
And a simple "w" or "last" discovers that instantly. To hide that, you'd need to install your own root kit - spy vs. spy :-)
"""
Yes. That is why playing hacker games is not going to get you anywhere. The _only_ correct action is to simply pull the power and make a image of the drive and you use that for forensics.
Think about a crime scene. Does the police start just tearing into everything, crawling in and out of all the windows, and trying to recreate the crime to try to see if they can find the criminal by accident?
NO.. They secure the scene and make sure that evidence is preserved.
And not only is it your job to preserve evidence, it is your job to do things like establish chains of custody and other things to make sure that you can prove that you have not tampered with the evidence.
It's such a simple thing to do... stick a drive into a external adapter and use dd to pull a image without mounting or otherwise touching any of the data on disk it's just a 'duh' to do it. Otherwise your just stomping all over everything with the digital equivalent of muddy boots in a ham fisted attempted to outsmart a unknown person.
Fri Nov 6 18:44:42 2009 A possible reason for the hack... DaemonZOGG
Here is an exerpt from "en.wikipedia.org/wiki/IRC" :
"..as a way of obtaining a bouncer-like effect, an IRC client (typically text-based, for example Irssi) may be run on an always-on server to which the user connects via ssh. This also allows devices that only have ssh functionality, but no actual IRC client installed themselves, to connect to the IRC and allows sharing of IRC sessions.[68]
To prevent the IRC client to be closed on termination of the ssh connection, it can be run inside a piece of screen-detaching software (e.g. GNU Screen or tmux), thus staying connected to the IRC network(s) at all time, being able to log channels the user is interested in, etc. Modelled after this setup[69], an IRC client following the client-server model, called Quassel IRC, has been developed. "
Who knows? Right? .. ;)
Fri Nov 6 18:46:28 2009 TonyLawrence
would be why wasn't the disgruntled ex-consultant's credentials immediately removed from the system when things went sour between him and the client?
I don't know. The first I heard about any unpleasant relations was today. It's quite possible that nobody but the owner knew about this and he may not have thought about hacking.
That assumes it WAS that guy. I think a compromised home machine is still in the running.
Fri Nov 6 18:50:21 2009 TonyLawrence
Right, who knows?
I keep cycling back to "Why call attention to yourself by disabling logins? Why leave an easily seen trail in /var/log/secure and wtmp?"
Whatever. As I said, it's not something I can handle.
Fri Nov 6 20:32:24 2009 A suggestion in general. Albinootje
What I really like is to use only Virtualization or sophisticated chrooted environments for the production servers, and then on the Hardware Node (The host) you would run process accounting (acct).
Having that will give you quite a bit history of what commands any intruder has used.
Fri Nov 6 20:36:23 2009 Hmmm DaemonZOGG
What about the old SCO box that the linux box telneted into? Were there any intrusions, modifications, suspect log entries, copied files, etc? (It's difficult to say or type the word "sco" considering what they have done to the open-source community). But, let's press on...
Fri Nov 6 20:42:49 2009 TonyLawrence
What about the old SCO box that the linux box telneted into?
Nothing obvious. But again, who knows? I can't guarantee that.
Fri Nov 6 21:58:01 2009 drag
"""What I really like is to use only Virtualization or sophisticated chrooted environments for the production servers, and then on the Hardware Node (The host) you would run process accounting (acct).
Having that will give you quite a bit history of what commands any intruder has used. """
I am not necessarily disagreeing with you, but I just like to point the following out when this subject gets brought up.
Chroot != Security mechanism. It is simply a way to isolate one environment from another in a fairly weak manner. It is designed pretty much for developmental or software compatibility purposes and is really quite worthless at increasing system security.
If a user gains root access in a chroot environment then it's trivial to break out of it. So trivial that is is just laughable.
So if you think about it...
In a normal system if a user breaks into a web-facing applications they are restricted by the rights of the user of that application. If that application is running as 'root' then your system is compromised. If the application is not running as root then to fully take over a system the attacker must exploit a local privilege escalation vulnerability. So that means that they need a local kernel exploit or a insecure 'setuid root' program and that sort of thing.
If the attacker hacks a 'chroot'd program then they are still facing the same barriers. If the program is running with root privileges then they have your machine.. breaking out of chroot for root is a feature, not a bug. If the application is running as a regular user then the chroot environment may present something of a barrier by isolating the attacker from other applications, but seeing how a hacked network-facing application has the ability to download and execute any arbitrary code then most kernel-level problems should be exploitable.
So chroot really gains you nothing that a properly secured system without chroot can. Maybe a little bit, but not much. You'd be better off checking your file system permissions and that sort of thing. It's not really a slam-dunk that some people think it is.
Now virtualization can actually help somewhat... but the thing to remember with virtualization that it is a cost-saving feature rather then a security one.
Ideally you should isolate different services on separate physical machines. If you can't afford to do that then you can achieve similar levels of isolation by running multiple VMs... but it is still possible to break out of VMs.
So say you have 30 different guest operating systems running in their own virtual machines. If one of the VMs are hacked and the attacker is able to exploit a flaw in the virtual machine then they can get access to the host operating system. Once they get access to the host then they can get full access to all the guests automatically. The host operates in a 'trusted' fashion that the security of all the guests depend on. Lose the host and you lose everything. Having separate physical machines is better because that is not really possible.
Of course since we are constrained by a budget then having virtual machines is a very nice option.
Something that I would recommend to use other then chroot is to use OpenVZ for Linux, BSD Jails, or Solaris's Containers/Zones. These are forms of virtualization that have similar low-overhead when compared to chroot, but unlike chroot they are designed for security in mind. One of the good ways to look at them is that they create a environment were 'root' is a non-privileged user.. were as with chroot root retains all the same privileges and was never originally intended for increased security. That way you get most of both worlds; convenience and low overhead with reduced cost compared to running separate hardware.
Containers are my friend. <3
Fri Nov 6 23:30:14 2009 Well, in the end... DaemonZOGG
Although it is worth looking in to, I don't think you'll find issues on the other workstations. It all seems centered around the Linux ssh box. Regardless if the intrusion came from the outside or not, I strongly believe that the intruder was using the box as either a file transfer relay over IRC, or utilizing the cpu resources on the box itself in order to assist in processing something they had. Why bog-down your own resources when you could utilize someone elses. Keeping all of the other users off of the system would free up most of the hardware resources just for you. No need to cover your tracks if your IP & MAC are spoofed through anonymous servers.
Stranger things have happened.
A few years ago, a hacker was arrested for breaking into the servers at Sandia National Labs for the sole purpose of obtaining extra drive space for his movie collection. Of course, I don't think he realized at the time exactly what type of organization he had broken into. ;)
Anywayz, if your client has the time, set up all of the usual tools (tripwire,wireshark,snort,etc). As Drag mentioned.. it is very important to know what commands they used or passed. Only then, will it give you some insight into the "why". :)
Best of luck!
- Daemon_ZOGG
Add your comments