How To Make a Password Cracking Cluster

[NeuTron]

How to Build a Password Auditing Cluster

This is a really easy project that can be a lot of fun. All you will need to make this work is a cd burner, a router, some ethernet cables and some old PCs you have lying around. You can very easily turn them into a home-made password cracking super-computer.

For convenience purposes, we will be using a bootable linux cd instead of actually installing linux. This is very handy because you don’t have to wipe out the current configuration of the machines you will be using in the cluster. The live linux cd I chose is called ClusterKnoppix. This was because it runs the KDE window manager and makes mounting cds and hard drives very easy. Another live linux cluster cd I really liked was called CHAOS (http://itsecurity.mq.edu.au/chaos/tyd/) . The .iso file for CHAOS is only 6 megs and the whole operating system can fit on one of those business card sized cdrs. “The super computer for your wallet” they call it.

Step One: Creating the cluster

Download the ClusterKnoppix .iso file from this link (http://bofh.be/clusterknoppix/). Here’s were the cd burner comes into play…you have to burn one cd for each machine in the cluster. I used four in my experiment but if you have access to a classroom of computers, you are in for a lot of cd burning.

When ClusterKnoppix boots up, it will try and get its network configuration via DHCP. To make life easier on yourself, have a DHCP server running. This is why I used a broadband router with DHCP enabled to connect my machines instead of a switch. If you use a hub or switch and don’t have a DHCP server, configure static IP addresses for all the nodes.

If everything went well, you now have a functioning OpenMosix cluster. OpenMosix is the clustering software that comes loaded on ClusterKnoppix that makes all of this possible. In order to see the status of your cluster, click on the openmosixViewer icon on the bar at the bottom of the screen. This should be displaying all of the nodes in your cluster. It also shows the speed/power of each node with a number next its address. This number is going to determine how much of the processing load will be placed on each of the computers.

Step Two: The password cracker/auditor

Everybody knows that JtR (John the Ripper) is the best password cracker ever written. Unfortunately it is a huge pain to distribute on a cluster. After fighting with it for a few days, I looked for other options and found a project called Cisilia. Cisilia is a password cracker that is meant to be used on OpenMosix clusters. Here is the easiest way to install and use it while doing the fewest reboots as possible:
1. Boot up your windows box that contains the passwords you want to audit.
2. First download Pwdump2 from here (http://www.cs.ucsb.edu/~vigna/archive/security/pwdump2/) (an application used to dump Windows 2000/XP SAM file)
3. at the command line, type “pwdump2 > c:\passwd“. This will create a copy of your password file called passwd on the root of your c:\ drive.
4. Next download the cisilia .tar.gz file from here (http://www.cisiar.org/proyectos/cisilia/home_en.php) . Put the .tar.gz file on the root of your c:\ drive.
5. Put in your ClusterKnoppix cd and boot this machine into your cluster.
6. On the desktop of KDE, you will see an icon for your hardrive. Double click it to mount it.
7. Copy the passwd file and cisilia file onto your desktop.
8. Double click the cisilia file and extract it to a folder on your desktop.
9. Open that new folder and right click in the window on some white space. Then choose “open a terminal here”
10. type “ ./configure”
11. after that process is finished, type “make” which will create the executable files in the /src directory of the cisilia folder.
12. copy the passwd file into the /src directory of the cisilia folder.
13. Now here is the syntax for the cisilia command: [#/ cisilia –l “file to log results” –n <number of cisilia instances to run> ./passwd

The number of cisilia instances to run can vary. On my 4 node cluster, I ran 6 instances. The faster two machines were meant to take 2 instances each and the slower computers could take one instance each. So I ran this command= cisilia –l ./finished.txt –n 6 ./passwd

This started cisilia cracking the passwd file with 6 different processes. As soon as the other machines in the cluster saw that one of the nodes was running 6 processor intensive programs, they each started to take instances to lower the burden on that one single machine. It worked just like I wanted because 2 of the P4 machines each claimed 2 instances and the P3 machines new to only take one for each of them. This distributed the 6 processes in the most efficient way. You can see that it is working with the openmosixviewer program because all the nodes will now be under heavy load and you only ran the program from one computer. If you click on the process migration button in the openmosixviewer you can see your processes running on the other computers. It is VERY cool to watch and feel the massive processing power.

After only three minutes I had brute forced a moderately complex 6 digit password.

If anyone tries this, post your result here. I would love to hear from someone who can run this on a big cluster, like a computer classroom for instance.

-NeuTron
Password Cracking Cluster (http://www.obsessedwithsecurity.com/community/viewtopic.php?t=3)

[hexadecimal]

im definitly going to have to do this with my xbox...
one question though....i heard from my friend that clusters are INtel only....is this true or can you have AMD clusters....or a mix of both

[Relyt]

NeuTron,

This sound like a real hoot! I have 3 boxes within arm's reach and my wife's and son's are available via the wireless router, So there's 5. Guess I better get to buring CD's this coming weekend so I can give it a whirl. Thanks for the info.


cheers

[NumbGreenThumb]

This method is much easier than using Jon the Ripper, good tutorial.

[NeuTron]

hexadecimal - I know that openMosix will run on AMD. I am not sure if you can have a mix of intel and amd but Id bet it would work.

Just as a side note, you can also use this cluster to run Blender for rendering 3d animations. Or, with the CHAOS distro, you can make a cluster that works on the SETI@home project. Its built into the CHAOS distribution so that you can start it with a single command. Maybe Ill write something about those clusters some other time.

[thehorse13]

Cluster Hosts
===============
5 dual Xeon Dell PowerEdge 2650s 2 GIG of RAM each

Results
===============
All passwords (6 - 8 alpha numeric chars) seven accounts in total, cracked in eleven seconds. Wow.

Happy Admins
===============
0

:)

[NeuTron]

10 Xeons and 10 Gigs of RAM...11 seconds...holy crap. My hat is off to you, sir. Where you using cisilia and ClusterKnoppix?

[Maestr0]

Very cool, Neutron.


-Maestr0

[AxessTerminated]

I have a few free classes at school where I have access to the computer labs.
I'm going to test it tomorrow with like two...to make sure i can do it right. If it all works out, I'm gonna try it next week with a ton.

Great tutorial!

A_T

[Shugart]

Awsome! I have got access to 45 machines on the lab... This could be fun...

Shugart

[AxessTerminated]

dammit. my burner is horrible. so assuming i get it to burn me a cd tonight...ill give you guys the results within a week or two.

[Shugart]

too bad on the burner...

Looks like I can't do it for Win NT due to server upgrade that I might do.. so till then...

Shugart

[hypronix]

/me thinks what the disciplinary actions would be for trying to do this with as many computers on the Univ network as possible...

Maybe if I come in one night... I like how for you P3's are older computers... and I'm thinking of building some old-ass P1's and P2's for distcc [distributed compiling for Gentoo].

The best I can do is that plus my P4 laptop and a P4 desktop... I'll see if I can get my Linux password[s] cracked below 24 hours [last I tried on just the laptop after a day JtR was pretty far from getting it...]

[VictorKaum]

Yep, Hypronix, I know the feeling, clustering P1 boxes...

[(V)/\><]

i just wanted to put this link here,

http://www.blackaltar.org/beowulf/index.html

[hexadecimal]

how exactly does this work?...do you only need one computer with clusterKNOPPIX or all of them that you plan on clustering

[NeuTron]

You need to boot all of the nodes with ClusterKnoppix. This automatically creates an OpenMosix cluster. You only have to start the password cracker on one node but you run it will several instances. The other nodes in the cluster will see that one node has too many processor intensive programs running and they will autmatically lessesn the burden by migrating the extra processes over to themselves.

[Und3ertak3r]

The other nodes in the cluster will see that one node has too many processor intensive programs running and they will autmatically lessesn the burden by migrating the extra processes over to themselves.
geez if only ppl were like this...

NeuTron thanks for your tutorial.. it is great to see the interest it has raised.. gotta play with another clustering os...

Cheers

[hexadecimal]

so what your saying is that a 50 node cluster needs 50 copies of clusterknoppix.....its gonna be a LONG night.....is their a way to get bios to boot off a network copy?....similiar to how linux can install from a network

[AO-Protected]

Just wondering... I have 3 systems with Red Hat 8.0 installed on each of them (along with WinXP, through dual boot); and I am thinking to permentaly install a cluster on the RH boxes, so I don't have to make or boot from a CD each time I want to use the cluster. That said, what tool that is very close or equivelant to ClusterKnoppix can I use and install it on the Redhat nodes ?? (I used Knoppix 3.6 itelf, very cool OS)

And also: Can the PWD cracker program mentioned be used for other applications or usages, for testing purposes of course, such as the self decrypting PGP program, and the like ? What can it be applied to other than SAM files ? Thanks for any feedback.

[SDK]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=262750#post797870) by NeuTron
You need to boot all of the nodes with ClusterKnoppix. This automatically creates an OpenMosix cluster. You only have to start the password cracker on one node but you run it will several instances. The other nodes in the cluster will see that one node has too many processor intensive programs running and they will autmatically lessesn the burden by migrating the extra processes over to themselves.

You still need a program that is written to run a on cluster, right? You were unable to run John.

[NeuTron]

You need a program that you can run multiple instances of while it is processing from one source. It is tough to make john do this, but not impossible. Cisilia makes it incredibly simpler.
-NeuTron

[AO-Protected]

NeuTron, quick Q: pwdump2 is not working for me getting the following:

Unable to adjust token privileges: 5
Failed enabling Debug privilege. Proceeding anyway
Failed to open lsass: 5. Exiting.


So my Q is: If using knoppix I get the SAM file, through C:\windows\system32\config\SAM

and rename it to lets say passwd.txt, would that do the job ?

Second issue: You mentioned that the work load is automatically synchronized among all the nodes of the cluster, is that true for all applications ? do they have all to support this specific OpenMosis kernel ??

Thanks for any feedback you may provide :)

[Und3ertak3r]

so what your saying is that a 50 node cluster needs 50 copies of clusterknoppix.....its gonna be a LONG night

Only if you have one burner.. if you have 5 or more burners.. you have a choice as to how you go about it.. did 5 copies using 3 machines in half an hour this afternoon .. with delays.. will play with the disks 2morrow.. saturday..yay.. I dont care abot the pwd cracking.. it is the simple clustering.. I got a nu toy..

[Maestr0]

Actually you only need ONE cd. The clusterknoppix distro comes with a DHCP and PXE server which will allow you to net boot all your clients into the cluster after the PXEserver is started. Worked great for me on about 10 machines I used for a test run, I plan to do about 90 tonight, :)


-Maestr0


PS. Ao-Protected, you need to use an account that has Admin and Debug privileges to dump the LSA secrets. The SAM file itself is not what you want as it contains much more info than just the usernames and pwd hashes.

[hexadecimal]

ive burned 5 copies already and plan on using them on my net in my room...gonna get my dad's sam and crack it hehehe

right now what me and my friend are trying to do is get our school to let us use the science computer lab during lunch time and use live distros to mess around....theres more to it but 50 P4 w 256 DDR all networked over a 4mbs connection....gonna be fun :D

<edit>

sorry forgot to mention my dad forgot his password and wants on his computer with access to his files...did not mean any bad display...my error...yes i deserve the negs

</edit>

[Maestr0]

Cluster night went well, had lots of fun with OpenMosix. I was able to crack 15 passwords between 7 and 15 characters alphanumeric, using Cisilia all were cracked within 2 hours. Unfortunately I was unable to get Cisilia to spawn more than 13 processes (I believe this is a problem in her code), so many of the 90 nodes were set on POVray and other valuable processes wasting tasks :). I had some limited success getting JtR to run parallel, there are some tricks you can use for dictionary attacks but the secret of brute forcing eludes me for now. There are two programs to aid in this 'pljohn' (a perl script that wraps multiple JtR processes and uses a pipe) or its C counterpart 'forkjohn'. Some of the nodes were booted into X, some booted using the CHAOS kernel. The 2.6 Kernel on the clusterknoopix CD does not appear to support OpenMosix correctly but the 2.4 worked fine. All nodes were booted from the OpenMosicTerminalServer (DHCP,PXE). There's nothing quite like seeing the 90GB of available RAM. Thanks for the fun tip NeuTron, I'd like to hear if anyone else experiments with this. Also credit goes to my 2 lab monkeys appearing in the photos for their help in booting 90 machines repeatedly. :)

-Maestr0


All nodes were running on DELL GX-270's, running 2.8Ghz P4's with 1 GB RAM and e1000 gigabit network cards.

[BigDick]

Maestr0,
You just wish you could live the life of a lab monkey for ever...

BigDick (you know who) :D

[w0ob]

Now this was a very nice post.. However i have a couple of questions that i hope somone could help me with :)

I have recently installed a w-lan at home and i want to test the security on my router.
Im using a 128 bit wep encryption.. is there anyway to make a cluster that could help me with trying to crack that wep key?

thnx for a nice post :)

[NeuTron]

Actually, there is a tool which was designed to audit WEP keys using multiple machines(like a cluster). Its called SCOUT.
the cracker part of the tool can be comfortably parallelized and run on a cluster or a grid.
http://www.security.iitk.ac.in/IITKHACK04/papers/cp01.pdf

[cleanbash]

For those of us who have some more time, check this out:

http://www.mini-itx.com/projects/cluster/

[Irongeek]

Great tutorial, please write more. I also wanted to point out that if pwdump does not work because you are not an admin just copy off the SAM and SYSTEM files and use SamInside to crack Syskey and export a pwdump format file. See if you can find version 2.1.3 of SamInside as newer versions don't let you dump pwdump files with just the demo version.

[djscribble]

IronGeek: when you say that if you aren't an admin to copy the sam and then use saminside are you talking about using linux w/ ntfs read ability?? i thought the file was "protected" by windows so that you couldn't do that...

[Irongeek]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=262750#post799015) by djscribble
IronGeek: when you say that if you aren't an admin to copy the sam and then use saminside are you talking about using linux w/ ntfs read ability?? i thought the file was "protected" by windows so that you couldn't do that...

Point taken, I forgot to mention that you would have to boot from a CD (Knoppix, Bart’s PE builder, ETC) or put the drive in another computer to get the files off.

[w0ob]

Hey neutron thnx ALOT for helping me out :)

Hope i can help you some time :)

And by the way.. next term (spring) i will get access to a lab with about 20 comps with different perfomance of course.. on a 100 mbit fiber connection.. so im gonna talk to my teacher and see if its possible do play around some :)

[plinius]

thanks a lot!! Very nice tutorial.
(do you happen to know something about the load-balancing capabilities/algorithms used, this domain interests me a lot).

plinius.

[Jareds411]

Can someone help me out with a couple things?

Does anyone know of a good tutorial explaining SAM files?
Does anyone know of a free DHCP server for windows or linux?
Would it be possible to cluster windows computers, or windows and linux computers together?
Does anyone know of a system like OpenMosix that acutally devides individual processes?

Thanks for any help, this whole concept is very interesting to me. :)

[Irongeek]

Do a search of the tutorials section and you will find a lot about SAM files.

[Phonedog911]

they were working on a p2p style password cracking cluster at www.hbx.us apparently users would offer up a percent of their processing power to the group and in turn gain access to the pool of processing power to crack what they want. they might still be working on it but i dunno, the site owner got arrested or something cuz their shell server was full of cc#'s. i thought it was a cool sounding idea though

[johnnymier]

First of all, Great tutorial NeuTron!

I setup the cluster in 4 P4 boxes 128 ram each. Passwd file had 3 accounts with 6-8 characters (alphanumeric). The first password took about 35 seconds. I left the cluster running for about 50 minutes and couldn't get the other two accounts. Is that normal behavior?? After that time I had to leave the computer lab at uni because some other class needed to use those boxes. I am keen to try it again next week with maybe two more machines in the cluster. I will let you all know of my results.

cheers

-j

[djscribble]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=262750#post799015) by djscribble
IronGeek: when you say that if you aren't an admin to copy the sam and then use saminside are you talking about using linux w/ ntfs read ability?? i thought the file was "protected" by windows so that you couldn't do that...

iron geek: another thought on the sam and system file -- you can get it from c:/winnt/repair or c:/windows/repair so i guess we are both kinda right...

[NeuTron]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=262750#post800295) by johnnymier
First of all, Great tutorial NeuTron!

I setup the cluster in 4 P4 boxes 128 ram each. Passwd file had 3 accounts with 6-8 characters (alphanumeric). The first password took about 35 seconds. I left the cluster running for about 50 minutes and couldn't get the other two accounts. Is that normal behavior?? After that time I had to leave the computer lab at uni because some other class needed to use those boxes. I am keen to try it again next week with maybe two more machines in the cluster. I will let you all know of my results.

cheers

-j

Johnnymier, my cluster didn't have enough processing power to crack all my passwords either. Only 2 P4's and 2 P3's. Thats why I want EVERYBODY who tries this to post the size of the cluster and the time it took to crack their list. Also mention the compexity and number of the passwords. This way we can benchmark how much processing power it takes to crack certain types and lengths of PWs. Thanks to everyone who has replies so far, it has motivated me to write more tuts like this one.

[sec_ware]

Hi.

Cisilia looks like a sophisticated project. Just out of curiosity:
Has anyone looked at the scalability?
Has anyone tried to use it on opteron/itaniumII-processors (I got a
segmentation fault doing ./configure, make. However, I did not try
to change any cisilia-default flags)?


I'd like to add a reference to a "poor man"-version:
the "distributed john the ripper"-project [1], which
is based on john the ripper [2]. Maybe some of you
already have looked at it, since it was mentioned in [3].


Description: The way it works is different than cisilia's -
as much as I understood of the latter. The server "djohnd"
simply splits the password-file, here called passwd,
into smaller pieces and "distributes" the task of cracking
onto several nodes (in a client-server manner). The clients themselves
use "john the ripper" for that purpose.
I guess, after that description you might know, why I call it
a "poor man"-version ;).


Application: For administrators of large systems with many users,
this might be an interesting option. Use idle-time of your machines
to check for weak passwords, coordinated by "djohnd".


Security issue: djohnd is in its current version highly sensible
to DOS attacks. So, and not only because of that,
use it in a trusted environment only.
Hmm, or not at all :D




[1] http://freshmeat.net/projects/djohn/
[2] http://www.openwall.com/john/
[3] http://www.antionline.com/showthread.php?s=&threadid=258149&perpage=10&highlight=distributed%20john%20the%20ripper&pagenumber=3

[Jareds411]

Does anyone know of a program like cisilia to crack other types of things, or does anyone know of a clustering system that splits up individual processes?

[AxessTerminated]

Jareds411. Cisilia is open source. You could modify the source, adding algorithms yourself. You could even start a project to do so on something like SourceForge. I think it would be a great idea if Cisilia had the capabilities to crack passwords, encryption schemes, EFS, and even RSA and PGP style encryption.

A_T

[Jareds411]

I'm sure I could do that pretty easily, seeing how I'm 15 and barely know any C++. :p

Back on topic, when I'm running clusterknoppix, it won't let me write to the harddrive, only read. Does anyone know how to change that?

[mgomespt]

Has anyone successfully changed the alphabet being tested by cisilia??? by default it only tests uppercase chars and numbers, i wanted it to include lowercase also...i've tried building a alphabet file like in cisilia man page, but it seems upper and lowercase get recognized only as being uppercase...(i see this on with verbose on (-V) in cisilia)...as far as i know windows XP passwords can get both upper and lowercase (of course they can, my login password uses both!)
can anyone help me?

[Mad_Whitehat]

hmm i tried your technique in my computer class, but nodes are not automaticly added, it could be i did something wrong, or i am using a different version of clusterknoppix, i downloaded 3.6.

Which version did you download and got it working that nice :S?

[Mad_Whitehat]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=262750#post805458) by mgomespt
Has anyone successfully changed the alphabet being tested by cisilia??? by default it only tests uppercase chars and numbers, i wanted it to include lowercase also...i've tried building a alphabet file like in cisilia man page, but it seems upper and lowercase get recognized only as being uppercase...(i see this on with verbose on (-V) in cisilia)...as far as i know windows XP passwords can get both upper and lowercase (of course they can, my login password uses both!)
can anyone help me?

probably it tries to crack LanMan hashes, these are the easiest cause they can only be uppercase chars, while NTLM can be Upper and Lower case chars. I think this is the problem you are discribing

[NeuTron]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=262750#post807143) by Mad_Whitehat
hmm i tried your technique in my computer class, but nodes are not automaticly added, it could be i did something wrong, or i am using a different version of clusterknoppix, i downloaded 3.6.

Which version did you download and got it working that nice :S?

Do you have a DHCP server available? Are the nodes x86 architecture? Oh, and 3.6 is the most recent.

[Mad_Whitehat]

hmm well they are all AMD Processors

i have no DHCP server, but i manually set the ip's and they worked (i could contact the internet)

//edit:

note i didn't try to crack anything, i just opened openmosixviewer and there was nothing there :S, did it miss any options i needed to set ? or am i just acting like a retard ;)

[NeuTron]

Put up a DHCP server...it makes life a whole lot easier. The problem you're having is that when clusterKnoppix boots, it first tries to get an address, then it joins itself to the cluster. Because you are assigning an IP afterward, it never tries to join the cluster. Im not sure what the openMosix command is to join it to the cluster. If you put up a DHCP server, it'll work...I gaurantee it!

*edit* The best way to do it would be to configure a DHCP server on one of your cluster nodes

-NeuTron

[NtrlZr]

I manage to boot the clusterknoppix, but how do i run cisilia?

[NtrlZr]

No forget it.... I didn't read the post very well.

[NtrlZr]

And to the one which had some questions about getting the hashes, you can use NTFSDOS Professional. Here's (http://http://home.no/neutralized/NTFSDOSPro.zip) the link.
Now, extract the archives to each diskette. Archive 1, to diskette number one (remember to format disk 1 as a ms-dos bootdisk) and archive 2 to diskette number two. Then put in disk1 in the target machine's floppy and boot. When you have booted the diskette, type "ntfspro" in the dos window and follow the short instructions. Then type C: and after that; "cd windows\system32\config". ("cd winnt\system32\config" if you have win2k.)



Now you should be in the config folder. Now type "copy sam A:"
The SAM should now be on your diskette. (if the sam is syskeyed you want the system file to though it's too big for a diskette, you'll have to copy it on a usb pen or something like that. It WILL NOT work to copy it directly to the disk.)

When your in windows again, put in your diskette and copy the sam over to a more secure place so you where you won't lose it. You'll never know.

If the sam is syskeyed then download SamInside 2.2.6.0 full version (http://home.no/neutralized/Saminside2.2.6.zip), load the sam and the system file, and then extract the hashes to a file. Now you have unlocked the sam!

msg me if you need help with the sam file.

[NtrlZr]

I have a seriously problem; When i boot clusterknoppix and open the mosixview panel, there are no nodes there. :S

Yes i have a DHCP server on the network, and i have set to recieve IP adress automatically.

Can someone help me?

[NeuTron]

Originally posted here (http://www.AntiOnline.com/showthread.php?threadid=262750#post819879) by NtrlZr
I have a seriously problem; When i boot clusterknoppix and open the mosixview panel, there are no nodes there. :S

Yes i have a DHCP server on the network, and i have set to recieve IP adress automatically.

Can someone help me?

I need more information about your setup before I can help you. PM when you get a chance and we can discuss it on IRC because I hate writing long posts :)

-NeuTron

[mattadata]

I have Cluster Knoppix up and running on 4 machines. Everything works great, except I can't give Cisilia more than 9 threads. 10 or greater causes it to fail. With 9 processes, all of my machines are only running at about 70%. Any suggestions?

[NeuTron]

You only want to load one process per processor. Each process tries to take 100% of the processor, so loading more than one per machine doesnt make sense. I know this is contrary to my tutorial but Ive learned a little more about this since then.

[floridabull]

Well after a few spindles of cd's and a 200 box cluster (Thanks local junior college), This cracks the passwords before my I-pod will play a mp3. Thanks.

[Egaladeist]

Hi floridabull,

You might want to consider making your second post in a thread that isn't already almost four months old...the flashing dates mean the thread is old...and unless you have something ' relevant to add ' to the discussion...please don't revive it.

Eg ;)