Crack files (bases) commands and utilities

Here in this post spimple notes (OSX):
#1 Near to crack a binary file (bases) and a simple script
#2 Use RamDisk in order to speed-up I/O bound tasks

#1 Near to Crack a binary file

Crack  firmware, a binary file format ... or hack whatever thing is an activity that can drive you crazy.
You can hope to complete crack/hack if you have :

• a lot of time
• a lot of motivation
• a lot of inventive and fantasy
• you haven't a job

#1.1 command 'file'

Usually we can start with shell command "file" ( in order figure out the file type: the file extension isn't sufficient). The command file tries to classify the file using heuristics such as filesystem tests, magic tests, and language tests (man file for further details).
The output of this command is the type of the input file (i.e. JPEG image data, JFIF standard 1.01) or simply "data" if binary file can't be classified.
Useful options are  --mime-type and --mime-encoding  in order to have mime type string of the binary file.
Example :

#root> file dat.bin
dat.bin: JPEG image data, JFIF standard 1.01

#root> file --mime-type --mime-encoding dat.bin
dat.bin: image/jpeg; charset=binary

#1.2 command 'strings'

#root> strings /bin/bash | more
...
GNU bash, version %s-(%s)
x86_64-apple-darwin11
Usage:
%s [GNU long option] [option] ...
%s [GNU long option] [option] script-file ...
GNU long options:
debug
debugger
...

#1.3 HexEditor

After a quick overview using commands file and strings we have to drill down the binary using an HexEditor. An HexEditor  has some usefull features in order to search (binary) data values, move to a specific offest, edit data, cut&paste binary data and in order to create a new binary file. Moreover 010Editor (a commercial HexEditor) has a template feature that allows you to map the binary data to C-like data structure (template). In fact each file format has an header with information like version, data length, resolution, bit colors (as example a JPG image header) ... all those information are mapped automatically on the templete easy to read and change. Another useful HexEditor is A.X.E. (Advanced Hex Editor) which can define templates, grammars ... and much more.
A.X.E. can attach a running process in order to explore allocated memory and/or provides a graphical view of the (file)  binary. In graphical view each byte is coded with a color:  this feature is useful in order to discover pattern or (in some cases) bitmap images (below images is from my mp3 player firmware).

#1.4 Script

#!/bin/bash
 
if [ $# -le 0 ]; then
 echo "`basename $0` <file-to-analyze> [verbose]"
 exit 1
fi
 
FILE_NAME=$1
FILE_TMP_NAME=$FILE_NAME.tmp
FILE_LOG=$FILE_NAME-offsets.log
BYTES=`stat -f "%z" $FILE_NAME`
 
echo "---$FILE_NAME File length : $BYTES---" >>$FILE_LOG
echo "File length : $BYTES"
 
SFCMD="file $FILE_TMP_NAME | grep -v \"$FILE_TMP_NAME: data\""
if [ "x$2" == "xverbose" ]; then
 echo "verbose mode ON"
 echo "verbose mode ON" >>$FILE_LOG
 SFCMD="file $FILE_TMP_NAME"
fi
 
for (( BLOBPOS=0; BLOBPOS<=$BYTES; BLOBPOS++ )); do
 # dd bs=1 if=$FILE_NAME of=$FILE_TMP_NAME skip=$BLOBPOS 2>/dev/null
 tail -c+$BLOBPOS $FILE_NAME >$FILE_TMP_NAME
 RESULT=`eval $SFCMD`
 if [ "x$RESULT" != "x" ]; then
 HEXBLOBPOS=`echo "ibase=10;obase=16;$BLOBPOS" | bc`
 echo "Start by offset dec '$BLOBPOS' / hex '$HEXBLOBPOS' $RESULT" >>$FILE_LOG
 fi
 
 if [ "$(($BLOBPOS % 5))" -eq "0" ]; then
 echo "Processed $BLOBPOS of $BYTES"
 fi
done
 
rm $FILE_TMP_NAME

Script usage is :

 ./search.sh <file-to-analyze> [verbose]

output of script is the log file <file-to-analyze>-offsets.log
Basically script moves the offset into binary file and try to figure out the type using the command file.
As output you have a log file  candidate offset with resources :

...
Start by offset dec '290' / hex '122'  firmware.bin: TrueType font data
Start by offset dec '553' / hex '229'  firmware.bin: Targa image data - RGB
...

(for a total of 500KB of binary data)
the script has discovered correctly all the 4 resources .... but also more than 100 wrong contents. Obviously some of wrongly recognised formats are clearly incorrect (because the meta information printed by the script are inconsistently).
Example :
... hex '122' firmware.bin: Sendmail frozen configuration  - version 9929."
(I mean : version  9929." at position  can't be true)
Btw the script is pretty useful if you have an idea about what you are looking for  but I easily guess that in most cases it can't be used as is. Actually it is enough discover a sequence like FFD8 in order to detect a JPG (and it is just an example of huge  sequences of bytes that represent well know resource format)

#2 Use  RamDisk in order to speedup some tasks

Analyze binary file using script is a task mainly I/O bound. In this case (and cases like this) in order to speed up the task we can use RamDisk (use RAM instead HardDisk if you have enough RAM). Following one-line shell command (for mac OSX) can help to create a volume mounted on RAM.

VOLUME_SIZE_IN_MB=500;VOLUME_NAME="RamDisk";RDDIM="$(($VOLUME_SIZE_IN_MB*2048))";diskutil erasevolume HFS+ "$VOLUME_NAME" `hdiutil attach -nomount ram://$RDDIM`

Disk of 500MB is mounted on Desktop as disk labelled "RamDisk" (by shell "cd /Volumes/RamDisk").
This is a comparison between "HardDisk Volume" Vs. "RamDisk Volume"
How to create a big file (1GB file) :

 #root> dd if=/dev/zero of=test.bin  count=1000000000

or

 #root> head -c 1000000000 /dev/random >test.bin

#root> time cp test.bin test.bin.cpy
real 0m54.919s
user 0m0.004s
sys 0m1.043s
(18 MB/s)

#root> time cp test.bin test.bin.cpy
real 0m7.166s
user 0m0.004s
sys 0m1.194s
(139MB/s)

Sure this is a copy pf sequential bytes ... random access is another thing