Hi, I wrote about kernel errors which somebody pointed out was because the server was running out of memory. Now I found the following which makes me think that that server may have been compromized. Here's what I get when I issued: netstat -nap tcp 0 0 131.x.x.x:38423 72.x.x.x:80 ESTABLISHED 5226/ps x tcp 0 0 131.x.x.x:38420 72.x.x.x:80 ESTABLISHED 5365/ps x About a hundred instances of that program 'ps x' running. Also here's what ps -ef produced: apache 6323 1 0 10:30 ? 00:00:00 ps x apache 6324 1 0 10:30 ? 00:00:00 ps x apache 6326 1 0 10:30 ? 00:00:00 ps x apache 6328 1 0 10:30 ? 00:00:00 ps x apache 6330 1 0 10:30 ? 00:00:00 ps x

> On Thu, 2006-11-16 at 10:26 -0600, olga(a)urbantimes.net wrote: > >> Hi, >> >> >> I wrote about kernel errors which somebody pointed out was because the >> server was running out of memory. >> >> Now I found the following which makes me think that that server may have >> been compromized. >> >> Here's what I get when I issued: netstat -nap >> >> >> tcp 0 0 131.x.x.x:38423 72.x.x.x:80 ESTABLISHED 5226/ps x >> tcp 0 0 131.x.x.x:38420 72.x.x.x:80 ESTABLISHED 5365/ps x >> >> >> About a hundred instances of that program 'ps x' running. >> >> >> Also here's what ps -ef produced: >> >> >> apache 6323 1 0 10:30 ? 00:00:00 ps x apache 6324 1 0 10:30 ? >> 00:00:00 ps x >> apache 6326 1 0 10:30 ? 00:00:00 ps x apache 6328 1 0 10:30 ? >> 00:00:00 ps x >> apache 6330 1 0 10:30 ? 00:00:00 ps x > > What does ls -l /proc/6323/exe say? That would be a symlink to the > executable for that process. Normal ps lives in /bin so the link should point at /bin/ps. If it > is connecting out to a remote host, it's likely not the normal ps, just something that's masking > itself to make it less likely to get picked up. > > -- > David Hollis <dhollis(a)davehollis.com&gt; > > apache 3102 1 0 15:53 ? 00:00:00 httpd apache 3104 1 0 15:53 ? 00:00:00 httpd apache 3106 1 0 15:53 ? 00:00:00 httpd apache 3108 1 0 15:53 ? 00:00:00 httpd apache 3110 1 0 15:53 ? 00:00:00 httpd apache 3112 1 0 15:53 ? 00:00:00 httpd apache 3114 1 0 15:53 ? 00:00:00 httpd apache 3116 1 0 15:53 ? 00:00:00 httpd apache 3118 1 0 15:53 ? 00:00:00 httpd apache 3120 1 0 15:53 ? 00:00:00 httpd apache 3122 1 0 15:53 ? 00:00:00 httpd apache 3125 1 0 15:54 ? 00:00:00 httpd apache 3127 1 0 15:54 ? 00:00:00 httpd apache 3129 1 0 15:54 ? 00:00:00 httpd apache 3131 1 0 15:54 ? 00:00:00 httpd apache 3133 1 0 15:54 ? 00:00:00 httpd apache 3135 1 0 15:54 ? 00:00:00 httpd apache 3137 1 0 15:54 ? 00:00:00 httpd apache 3139 1 0 15:54 ? 00:00:00 httpd apache 3141 1 0 15:54 ? 00:00:00 httpd apache 3143 1 0 15:54 ? 00:00:00 httpd apache 3145 1 0 15:54 ? 00:00:00 httpd apache 3639 1 0 15:57 ? 00:00:00 ps x apache 3642 1 0 15:57 ? 00:00:00 ps x apache 3645 1 0 15:58 ? 00:00:00 ps x apache 3647 1 0 15:58 ? 00:00:00 ps x I am getting a ton of these... Here's what ls -l /proc/3147/exe says lrwxrwxrwx 1 apache apache 0 Nov 16 15:56 /proc/3147/exe -> /usr/bin/perl When I do netstat -nap I get: tcp 0 0 131.x.x.x:44160 72.14.x.x:80 ESTABLISHED - tcp 0 0 131.x.x.x:44161 72.14.x.x:80 ESTABLISHED - tcp 0 0 131.x.x.x:44162 72.14.x.x:80 ESTABLISHED - The ip points to google... And these appeared in the /tmp folder: drwxrwxrwt 8 root root 4096 Nov 16 16:00 . drwxr-xr-x 23 root root 4096 Nov 16 14:35 .. srwx------ 1 root nobody 0 Nov 16 14:36 .fam_socket drwxrwxrwt 2 xfs xfs 4096 Nov 16 14:35 .font-unix srw-rw-rw- 1 root root 0 Nov 16 14:36 .gdm_socket -rw-r--r-- 1 apache apache 0 Nov 15 15:20 .httpd drwxrwxrwt 2 root root 4096 Nov 16 14:36 .ICE-unix drwx------ 2 root root 4096 Nov 16 14:59 mc-root drwx------ 2 root root 12288 Nov 16 15:16 orbit-root -rw-r--r-- 1 apache apache 0 Nov 16 15:58 sess_azx3a4wq3x1f2aad4a34sxx1w2o52a45 -rw-r--r-- 1 apache apache 11669 Nov 16 15:43 sess_rdav631df3a1ddfaa34s1x1wwo521459 -r--r--r-- 1 root root 11 Nov 16 14:36 .X0-lock drwxrwxrwt 2 root root 4096 Nov 16 14:36 .X11-unix What is going on?

El Jueves, 16 de Noviembre de 2006 22:56, olga(a)urbantimes.net escribió: > > On Thu, 2006-11-16 at 10:26 -0600, olga(a)urbantimes.net wrote: > >> Hi, > >> > >> I wrote about kernel errors which somebody pointed out was because > the > >> server was running out of memory. > >> > >> Now I found the following which makes me think that that server may > have > >> been compromized. > >> > >> Here's what I get when I issued: netstat -nap > >> > >> tcp 0 0 131.x.x.x:38423 72.x.x.x:80 ESTABLISHED > >> 5226/ps x > >> tcp 0 0 131.x.x.x:38420 72.x.x.x:80 ESTABLISHED > >> 5365/ps x > >> > >> About a hundred instances of that program 'ps x' running. > >> > >> Also here's what ps -ef produced: > >> > >> apache 6323 1 0 10:30 ? 00:00:00 ps x > >> apache 6324 1 0 10:30 ? 00:00:00 ps x > >> apache 6326 1 0 10:30 ? 00:00:00 ps x > >> apache 6328 1 0 10:30 ? 00:00:00 ps x > >> apache 6330 1 0 10:30 ? 00:00:00 ps x > > > > What does ls -l /proc/6323/exe say? That would be a symlink to the > > executable for that process. Normal ps lives in /bin so the link > should > > point at /bin/ps. If it is connecting out to a remote host, it's > likely > > not the normal ps, just something that's masking itself to make it > less > > likely to get picked up. > > > > -- > > David Hollis <dhollis(a)davehollis.com&gt; > > apache 3102 1 0 15:53 ? 00:00:00 httpd > apache 3104 1 0 15:53 ? 00:00:00 httpd > apache 3106 1 0 15:53 ? 00:00:00 httpd > apache 3108 1 0 15:53 ? 00:00:00 httpd > apache 3110 1 0 15:53 ? 00:00:00 httpd > apache 3112 1 0 15:53 ? 00:00:00 httpd > apache 3114 1 0 15:53 ? 00:00:00 httpd > apache 3116 1 0 15:53 ? 00:00:00 httpd > apache 3118 1 0 15:53 ? 00:00:00 httpd > apache 3120 1 0 15:53 ? 00:00:00 httpd > apache 3122 1 0 15:53 ? 00:00:00 httpd > apache 3125 1 0 15:54 ? 00:00:00 httpd > apache 3127 1 0 15:54 ? 00:00:00 httpd > apache 3129 1 0 15:54 ? 00:00:00 httpd > apache 3131 1 0 15:54 ? 00:00:00 httpd > apache 3133 1 0 15:54 ? 00:00:00 httpd > apache 3135 1 0 15:54 ? 00:00:00 httpd > apache 3137 1 0 15:54 ? 00:00:00 httpd > apache 3139 1 0 15:54 ? 00:00:00 httpd > apache 3141 1 0 15:54 ? 00:00:00 httpd > apache 3143 1 0 15:54 ? 00:00:00 httpd > apache 3145 1 0 15:54 ? 00:00:00 httpd > apache 3639 1 0 15:57 ? 00:00:00 ps x > apache 3642 1 0 15:57 ? 00:00:00 ps x > apache 3645 1 0 15:58 ? 00:00:00 ps x > apache 3647 1 0 15:58 ? 00:00:00 ps x > > > I am getting a ton of these... > Here's what ls -l /proc/3147/exe says > lrwxrwxrwx 1 apache apache 0 Nov 16 15:56 /proc/3147/exe > -> > /usr/bin/perl > > When I do netstat -nap I get: > tcp 0 0 131.x.x.x:44160 72.14.x.x:80 ESTABLISHED - > tcp 0 0 131.x.x.x:44161 72.14.x.x:80 ESTABLISHED - > tcp 0 0 131.x.x.x:44162 72.14.x.x:80 ESTABLISHED - > > The ip points to google... > > And these appeared in the /tmp folder: > > drwxrwxrwt 8 root root 4096 Nov 16 16:00 . > drwxr-xr-x 23 root root 4096 Nov 16 14:35 .. > srwx------ 1 root nobody 0 Nov 16 14:36 .fam_socket > drwxrwxrwt 2 xfs xfs 4096 Nov 16 14:35 .font-unix > srw-rw-rw- 1 root root 0 Nov 16 14:36 .gdm_socket > -rw-r--r-- 1 apache apache 0 Nov 15 15:20 .httpd > drwxrwxrwt 2 root root 4096 Nov 16 14:36 .ICE-unix > drwx------ 2 root root 4096 Nov 16 14:59 mc-root > drwx------ 2 root root 12288 Nov 16 15:16 orbit-root > -rw-r--r-- 1 apache apache 0 Nov 16 15:58 > sess_azx3a4wq3x1f2aad4a34sxx1w2o52a45 > -rw-r--r-- 1 apache apache 11669 Nov 16 15:43 > sess_rdav631df3a1ddfaa34s1x1wwo521459 > -r--r--r-- 1 root root 11 Nov 16 14:36 .X0-lock > drwxrwxrwt 2 root root 4096 Nov 16 14:36 .X11-unix > > What is going on? > Finally...did they break into your system? Did you find something strange on the logs? I wonder what happened, give us some information this thread is quite interesting and will help other folks in a near future ;-) One way or another, if they got shell access (even remote text shell, you know...) you should think about reinstalling your system, as far as i know, if the left a rootkit you must not trust your system anymore. By the way, let me give you and advice, installing Babel Enterprise could be a nice idea, ( http://babel.sourceforge.net/en/ ), yeah yeah, it's GPL ;-) Babel is an enterprise-grade auditing system to manage a consistency on security policy between different systems in a non-homogeneus architecture. Babel allows to manage very different operating systems, like AIX, Solaris, Windows 2000, Windows XP, Linux, *BSD or HPUX. Babel allows administrator team to monitor the hardening level of their systems and keep constantly monitored, using periodic policy polling, and of course, a WEB Based, graphical reporting, and of course, a centralized management for all systems There's a demo online, try it. Hope this helps.

A bunch of these with other file names instead of borek.txt and other ips as well.

> El Jueves, 16 de Noviembre de 2006 22:56, olga(a)urbantimes.net escribió: >> > On Thu, 2006-11-16 at 10:26 -0600, olga(a)urbantimes.net wrote: >> >> Hi, >> >> >> >> I wrote about kernel errors which somebody pointed out was because >> the >> >> server was running out of memory. >> >> >> >> Now I found the following which makes me think that that server may >> have >> >> been compromized. >> >> >> >> Here's what I get when I issued: netstat -nap >> >> >> >> tcp 0 0 131.x.x.x:38423 72.x.x.x:80 ESTABLISHED >> >> 5226/ps x >> >> tcp 0 0 131.x.x.x:38420 72.x.x.x:80 ESTABLISHED >> >> 5365/ps x >> >> >> >> About a hundred instances of that program 'ps x' running. >> >> >> >> Also here's what ps -ef produced: >> >> >> >> apache 6323 1 0 10:30 ? 00:00:00 ps x >> >> apache 6324 1 0 10:30 ? 00:00:00 ps x >> >> apache 6326 1 0 10:30 ? 00:00:00 ps x >> >> apache 6328 1 0 10:30 ? 00:00:00 ps x >> >> apache 6330 1 0 10:30 ? 00:00:00 ps x >> > >> > What does ls -l /proc/6323/exe say? That would be a symlink to the >> > executable for that process. Normal ps lives in /bin so the link >> should >> > point at /bin/ps. If it is connecting out to a remote host, it's >> likely >> > not the normal ps, just something that's masking itself to make it >> less >> > likely to get picked up. >> > >> > -- >> > David Hollis <dhollis(a)davehollis.com&gt; >> >> apache 3102 1 0 15:53 ? 00:00:00 httpd >> apache 3104 1 0 15:53 ? 00:00:00 httpd >> apache 3106 1 0 15:53 ? 00:00:00 httpd >> apache 3108 1 0 15:53 ? 00:00:00 httpd >> apache 3110 1 0 15:53 ? 00:00:00 httpd >> apache 3112 1 0 15:53 ? 00:00:00 httpd >> apache 3114 1 0 15:53 ? 00:00:00 httpd >> apache 3116 1 0 15:53 ? 00:00:00 httpd >> apache 3118 1 0 15:53 ? 00:00:00 httpd >> apache 3120 1 0 15:53 ? 00:00:00 httpd >> apache 3122 1 0 15:53 ? 00:00:00 httpd >> apache 3125 1 0 15:54 ? 00:00:00 httpd >> apache 3127 1 0 15:54 ? 00:00:00 httpd >> apache 3129 1 0 15:54 ? 00:00:00 httpd >> apache 3131 1 0 15:54 ? 00:00:00 httpd >> apache 3133 1 0 15:54 ? 00:00:00 httpd >> apache 3135 1 0 15:54 ? 00:00:00 httpd >> apache 3137 1 0 15:54 ? 00:00:00 httpd >> apache 3139 1 0 15:54 ? 00:00:00 httpd >> apache 3141 1 0 15:54 ? 00:00:00 httpd >> apache 3143 1 0 15:54 ? 00:00:00 httpd >> apache 3145 1 0 15:54 ? 00:00:00 httpd >> apache 3639 1 0 15:57 ? 00:00:00 ps x >> apache 3642 1 0 15:57 ? 00:00:00 ps x >> apache 3645 1 0 15:58 ? 00:00:00 ps x >> apache 3647 1 0 15:58 ? 00:00:00 ps x >> >> >> I am getting a ton of these... >> Here's what ls -l /proc/3147/exe says >> lrwxrwxrwx 1 apache apache 0 Nov 16 15:56 /proc/3147/exe >> -> >> /usr/bin/perl >> >> When I do netstat -nap I get: >> tcp 0 0 131.x.x.x:44160 72.14.x.x:80 ESTABLISHED - >> tcp 0 0 131.x.x.x:44161 72.14.x.x:80 ESTABLISHED - >> tcp 0 0 131.x.x.x:44162 72.14.x.x:80 ESTABLISHED - >> >> The ip points to google... >> >> And these appeared in the /tmp folder: >> >> drwxrwxrwt 8 root root 4096 Nov 16 16:00 . >> drwxr-xr-x 23 root root 4096 Nov 16 14:35 .. >> srwx------ 1 root nobody 0 Nov 16 14:36 .fam_socket >> drwxrwxrwt 2 xfs xfs 4096 Nov 16 14:35 .font-unix >> srw-rw-rw- 1 root root 0 Nov 16 14:36 .gdm_socket >> -rw-r--r-- 1 apache apache 0 Nov 15 15:20 .httpd >> drwxrwxrwt 2 root root 4096 Nov 16 14:36 .ICE-unix >> drwx------ 2 root root 4096 Nov 16 14:59 mc-root >> drwx------ 2 root root 12288 Nov 16 15:16 orbit-root >> -rw-r--r-- 1 apache apache 0 Nov 16 15:58 >> sess_azx3a4wq3x1f2aad4a34sxx1w2o52a45 >> -rw-r--r-- 1 apache apache 11669 Nov 16 15:43 >> sess_rdav631df3a1ddfaa34s1x1wwo521459 >> -r--r--r-- 1 root root 11 Nov 16 14:36 .X0-lock >> drwxrwxrwt 2 root root 4096 Nov 16 14:36 .X11-unix >> >> What is going on? >> > > Finally...did they break into your system? Did you find something strange > on > the logs? I wonder what happened, give us some information this thread is > quite interesting and will help other folks in a near future ;-) > One way or another, if they got shell access (even remote text shell, you > know...) you should think about reinstalling your system, as far as i > know, > if the left a rootkit you must not trust your system anymore. > > By the way, let me give you and advice, installing Babel Enterprise could > be a > nice idea, ( http://babel.sourceforge.net/en/ ), yeah yeah, it's GPL ;-) > > Babel is an enterprise-grade auditing system to manage a consistency on > security policy between different systems in a non-homogeneus > architecture. > Babel allows to manage very different operating systems, like AIX, > Solaris, > Windows 2000, Windows XP, Linux, *BSD or HPUX. > > Babel allows administrator team to monitor the hardening level of their > systems and keep constantly monitored, using periodic policy polling, and > of > course, a WEB Based, graphical reporting, and of course, a centralized > management for all systems > > There's a demo online, try it. > > Hope this helps. It does appear that there has been a break-in. Some kind of script was running that was consuming all system resourses. At the time it was running, it was also deleting log entries, so if I looked at the log and searched for the time we brough the server up on the network, logs would show no activity at that time. And that 72.x.x.x IP was probably bogus as well. Here's what I found in the httpd error log: --06:31:56-- http://autocoutureinc.com/borek.txt => `borek.txt' Resolving autocoutureinc.com... 208.67.181.244 Connecting to autocoutureinc.com|208.67.181.244|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 11,666 (11K) [text/plain] 0K .......... . 100% 169.99 KB/s 06:31:56 (169.99 KB/s) - `borek.txt' saved [11666/11666] Died at sess_rdav631df3a1ddfaa34s1x1w2o521459 line 24. Died at sess_rdav631df3a1ddfaa34s1x1w2o521459 line 24. rm: cannot remove `borek.txt*': No such file or directory % Total % Received % Xferd Average Speed Time Curr. Dload Upload Total Current Left Speed 100 11666 100 11666 0 0 23100 0 0:00:00 0:00:00 0:00:00 156k Died at sess_dda2631df3a1ddfaa34s1x1wwo521459 line 24. Died at sess_dda2631df3a1ddfaa34s1x1wwo521459 line 24. rm: cannot remove `borek.txt*': No such file or directory Died at sess_edav631df3a15dfaa34s1x1wwo521459 line 24. Died at sess_edav631df3a15dfaa34s1x1wwo521459 line 24. sh: line 1: lynx: command not found sh: line 1: fetch: command not found Died at sess_tdx4d3td33a1ddfaa34s1x11x2521459 line 24. Died at sess_tdx4d3td33a1ddfaa34s1x11x2521459 line 24. --06:32:02-- http://autocoutureinc.com/borek.txt => `borek.txt' Resolving autocoutureinc.com... 208.67.181.244 Connecting to autocoutureinc.com|208.67.181.244|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 11,666 (11K) [text/plain] 0K .......... . 100% 166.39 KB/s A bunch of these with other file names instead of borek.txt and other ips as well.

On Thu, 16 Nov 2006 olga(a)urbantimes.net wrote: > Here's what I get when I issued: netstat -nap > > tcp 0 0 131.x.x.x:38423 72.x.x.x:80 ESTABLISHED > 5226/ps x > tcp 0 0 131.x.x.x:38420 72.x.x.x:80 ESTABLISHED > 5365/ps x > > About a hundred instances of that program 'ps x' running. > > Also here's what ps -ef produced: > > apache 6323 1 0 10:30 ? 00:00:00 ps x > apache 6324 1 0 10:30 ? 00:00:00 ps x > apache 6326 1 0 10:30 ? 00:00:00 ps x > apache 6328 1 0 10:30 ? 00:00:00 ps x > apache 6330 1 0 10:30 ? 00:00:00 ps x The processes are owned by apache, so the chances are that there is a security hole in the version of apache/httpd that you are using, or perhaps more likely there are exploitable web pages somewhere on your server (maybe a bulletin board like phpbb, or phpmyadmin). I see that these things are talking to port 80, ie. probably a web server on the remote site, so it could be getting commands from there or attempting to spread further. Your web logs may be able to tell you more. Michael Young -- fedora-list mailing list fedora-list(a)redhat.com To unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-list

On Thu, 16 Nov 2006 10:26:20 -0600, olga wrote: >Hi, > > I wrote about kernel errors which somebody pointed out was because the >server was running out of memory. > >Now I found the following which makes me think that that server may have >been compromized.

If you can, unplug the network wire (though if they know what they are doing, your hard drive might be wiped off when their scripts detect that the network is down. It's your call.). Run rpm -V from a rescue cd (not the one in /usr/bin) on procps, net-tools, and the other essential system utilities (including rpm itself). Then you'll know for sure.

On Thu, 16 Nov 2006 16:16:34 -0700, Robin Laing wrote: >Amadeus W. M. wrote: > >>On Thu, 16 Nov 2006 10:26:20 -0600, olga wrote: >> >> >> >>>Hi, >>> >>>I wrote about kernel errors which somebody pointed out was because the >>>server was running out of memory. >>> >>>Now I found the following which makes me think that that server may have >>>been compromized. > >snip > > >>If you can, unplug the network wire (though if they know what they are >>doing, your hard drive might be wiped off when their scripts detect that >>the network is down. It's your call.). Run rpm -V from a rescue cd (not the >>one in /usr/bin) on procps, net-tools, and the other essential system >>utilities (including rpm itself). Then you'll know for sure. >> > >Just posting a question in regards to this statement. > >How about pulling the plug and fscking the drive using the rescue CD? >Not the best idea but could save a total wipe. > fsck checks the integrity of the file system (orphan inodes and such), not what's on it. I meant the hackers might have left some program behind to wipe off the drive to remove all their traces when the network goes down. The actions the victim will take are a different story, and might take depend on what's at stake. When I was hacked (a vulnerability in rpc.statd in RH6.2), there wasn't any sensitive data on the drive, so pulling the plug was not high risk. Script kiddies usually don't care to clean up after themselves, they leave behind a load of hacking tools. The former KGB agent, or the former FBI agent, on the other hand... If the victim is an important server, it might not even be possible (or easy) to take it off the network without some prior notice to the users. So it's the administrator's decision. It must be swift though, as a hacked machine is being used to scan and break into other machines. In fact that's how I new I was hacked, I started to receive emails from various universities that my machine was trying to break in into theirs.

Well, we got also infected with this "bastard" ok, we´running Mandrake 10.2 (the good old one) but same probbs. How i found it? i was looking what is running on this MDK... uuuuuuhhhhh whats that => APACHE -DSSL ??? hmmm with high CPU Load.... i was wondering. Also o had lately lags in our bandwidth.... alot spam Mails and a few other strange things. Ok.. time to do smth...... In our case this is bastard tells you i am "APACHE -DSSL" WRONG!!!! this is a Perl Deamon connecting to the Irc Network and spreading all infos of ur sys, AND!!!! gives them full access to ur Server....... What to do???? Where the heck does it load from? Well.... it is a Exploit used by hackers to hijack Boards, no matter if phpBB, Joomla or other.. its Code injection and execution !! once u got infected u r having a probb we DONT know at time a solution to kick this lil baby off, not yet..... What we did? well... this exploid needds to load external code to execute.... we found where and how it starts up, in our case it is the file "borek.txt" (search for it by google etc. and you will find similar probbs;) ) OK... we saw where this bastard tryed to load it´s code... so we blocked this IP. This will give us now the time and chance to search how it works and maybe find a solution to fix it and close this backdoor/bug When u deny/drop/reject access to the IP where the code is placed, the deamon cant start up.. simple? yes, but no solution..... We´ll finger out how and what it is and by chance bring u all (and us) a solution ti fix it cheers from Germany, Schnulli By the way, when still someone has a solution feel free to post it here or leave me a note