Firmware Modification Kit
Giving YOU the power to customize your firmware
documentation version 0.51 beta
all scripts, untrx, wrt_vx_imgtool, and docs (c) 2006,2007 Jeremy Collake <jeremy.collake@gmail.com> http://www.bitsum.com
other portions (c) their respective holders

Contents

I. Description
II. Why is this useful?
III. Where to get the kit
IV. Requirements
V. Compatibility
VI. Instructions
A. extracting firmware
B. modifying firmware
1. installing packages
2. removing packages
C. re-building firmware
VII. Caveats
A. binary compatibility issues
B. maximum firmware size
C. solutions to file system incompatibilities
VIII. Acknowledgements and more
IX. Links
X. Revision history
XI. About the author

I. Description
This kit gives the user the ability to make changes to a firmware image without recompiling the firmware sources. It works by extracting the firmware into its component parts, then extracting the file system image (assuming its in one the supported formats). The user can then make modifications to the extracted file system, and rebuild the firmware image.

Although this kit completely automates the process, it is also useful for those who need a handy collection of utilities to manipulate firmware images. The following utilities are included in this kit, along with scripts to automate their use:

ASUSTRX | An extended version of ASUSTRX that can build both 'normal' TRX files and, optionally, those with an ASUS addver style header appended. It can also, uniquely, force segment offsets in the TRX (with -b switch) for compatibility with Marvell ASUS devices like the WL-530g. This tool replaces both 'normal' trx tool and addver. Current versions included are: 0.90 beta.
ADDPATTERN | Utility to pre-pend Linksys style HDR0 header to a TRX.
ADDVER | ASUS utility to append a header to a TRX image that contains version information. ASUSTRX includes this capability. Current version: unversioned.
CRAMFSCK | CRAMFS file system image checker and extractor. Current versions included are: 2.4x.
MKSQUASHFS | Builds a squashfs file system image. Current versions included are: 2.1-r2, 3.0.
MKCRAMFS | Builds a cramfs file system image. Coming in next version. Current versions included are: 2.4x.
MOTOROLA_BIN | A utility that prepends 8 byte headers to TRX images for Motorola devices WR850G, WA840G, WE800G. Current version: unversioned.
UNCRAMFS | Alternate tool to extracts a cramfs file system image. Use cramfsck instead whenever possible as it seems to be more reliable. Current versions included are: 0.7 (for cramfs v2.x).
UNSQUASHFS | Extracts a zlib squashfs file system image. Current versions included are 1.0 for 3.0 images and 1.0 for 2.x images (my own blend).
UNSQUASHFS-LZMA | Extracts an lzma squashfs file system image. Current versions included are 1.0 for 3.0 images and 1.0 for 2.x images (my own blend). Note: Not all squashfs-lzma patches are compatible with one another. I'm working on adding support for all common squashfs-lzma variations.
UNTRX | Disassembles TRX style firmwares into their component parts. Also supports pre-pended addpattern HDR0 style headers. This was developed exclusively for this kit. Current versions included are: 0.45.
WRT_VX_IMGTOOL | Utility to generate VxWorks compatible firmware images for the WRT54G(S) v5 series.

Tools/changes unique to this kit

This kit doesn't just collect existing tools, it also offers new ones and modifications to existing ones. Custom code currently is:

ASUSTRX - Modified to allow '-b' switch to force segment offsets (needed for some devices, like the WL-530g).
UNTRX - A new tool to extract the contents of a TRX image into its component parts.
UNSQUASHFS - Custom blend that supports extraction of 2.x squashfs images. This tool wasn't officially made available until squashfs 3.0, so up until now it wasn't available for 2.x images.
WRT-VX-IMGTOOL - A new tool to view, extract, build, and fix VxWorks compatible firmware images for the WRT54G(S) v5 series.

II. Why is this useful?
A list of some of the many reasons this might be desired are:

- add initialization scripts
- install new packages
- extend/change HTTP managed console
- remove un-needed packages
- mix-and-match packages from various flavors of the firmware image.

III. Where to get the kit

The best way to download and use this kit is to check out the sources from the repository. The automated scripts all automatically build the executables, and the sources are the same as you'll find in the tar.gz archive.

To anonymously check out the firmware-mod-kit:

mkdir firmware_mod_kit cd firmware_mod_kit svn checkout svn://svn.berlios.de/firmwaremodkit/trunk

Kit downloads and links:

Project hosting and repository: https://developer.berlios.de/projects/firmwaremodkit/
Anonymous svn checkout: svn checkout svn://svn.berlios.de/firmwaremodkit/trunk
Latest official build: http://www.bitsum.com/files/firmware_mod_tools.tar.gz
/w pre-built linux x86 binaries: http://www.bitsum.com/files/firmware_mod_tools_prebuilt.tar.gz
Sometimes newer snapshots are posted for testing. When available, they are here.
User forums can be found at http://www.bitsum.com/smf.

Packages:

All of the many OpenWrt packages should work with OpenWrt images and many other firmware images.

A specific collection of DD-WRT packages for use with the kit: http://www.bitsum.com/files/ddwrt_packages.tar.gz .

IV. Requirements

I've worked to make sure the requirements are low enough for anyone to use this kit. However, to reduce distribution costs and better represent the multi-platform capabilities of this kit, the binary tools in the kit do automatically build themselves the first time you run one of the scripts. Any machine set up to build anything C or C++ on the machine will probably be ready. The few requirements are all basic items that everyone has easy access to:

- A compatible platform. The following are specifically supported (or specifically unsupported):
  - LINUX (recommended)
  - OS X **warning: I don't test every build under OS X. It's up to you to email me if you have problems under it.**
  - CYGWIN **alpha stage testing (!! built firmwares may not boot !!)**
- GNU C (gcc)
- GNU C++ (g++)
- GNU make
- zlib development files (zlib1g-dev)
- Standard C runtime libraries
- Standard C++ runtime libraries
- TAR and GZIP (optional, used by ipkg_install.sh)
- And maybe others..

For OS X, installing the Mac OS X Developer Tools will supply the needed GNU C and C++ compilers and make utility.

V. Compatibility

Any TRX style firmware image format should that uses squashfs-lzma as a root file system work fine. Firmwares that use regular zlib squashfs or unsupported file systems will work if the steps that extract the file system and rebuild the file system are tweaked.

The kit can build firmware images for the following devices, amongst many others:

- WRT54G v1 - v6
- WRT54GS v1 - v6
- WRTSL54G v?
- ASUS WL-330g
- ASUS WL-500g/p
- ASUS WL-520g
- ASUS WL-530g
- Belkin 7230-4 (some versions)
- Belkin 7231-4 (?)
- Buffalo WHR-G54S
- Buffalo WHR-HP-G54
- Siemens SE505
- ... lots of others* ...

The kit is known (or thought) to be compatible with the following firmwares:

DD-WRT | tested - versions v23 SP1 and later are compatible (soon older versions too). WARNING: Builds of DD-WRT later than 08/04/2006 (v23 SP2) have some protections against changing the web UI.
OpenWrt White Russian | untested (probably not atm) - not really necessary, based on OpenWrt has its Image Builder.
OpenWrt Kamikaze | untested (should work) - not really necessary, based on OpenWrt has its Image Builder.
FreeWrt | untested - should work ok
HyperWrt | untested
Ewrt | untested
Sveasoft Alchemy | untested
Sveasoft Talisman | untested
Linksys / other vendor | not supported by scripts yet - haven't added cramfs handling
WL-330G (vendor) | untested - should work ok
WL-520G (vendor) | untested - should work ok
WL-530G (vendor) | supported
WL-550G (vendor) | untested - should work ok
many others* | untested

* Even if the kit doesn't automatically support the firmware or device you want, you can manually use the tools inside to manipulate many firmware images, or request support be added.

VI. Instructions
I've scripts that simply and abstract the firmware modification process. Basically, it's a one-step process to extract the firmware, and a one-step process to rebuild the firmware. There are also scripts to install or remove individual packages or entire directories of packages.

The procedure of calls is:

1. extract_firmware.sh ....
2. (optional) ipkg_install[_all]).sh ...
3. (optional) ipkg_remove[_all].sh ...
4. build_firmware.sh

A. Extracting the firmware (extract_firmware.sh)

> Simply run extract_firmware.sh with the following parameters. You must run this tool from inside the directory it exists in.

> $ ./extract_firmware.sh firmware.bin working_directory/

> - firmware.bin is the firmware image you want to extract, i.e. DD-WRT.v23_MICRO_GENERIC.BIN.
> - working_directory is the working directory you want to use. This is where the intermediate files and the extracted file system will be stored. Note that files in the working directory may be deleted!
> It doesn't matter which firmware image you supply, i.e. for a WRT54G or an ASUS WL-500G. These images are usually all the same and differ only in the header format. The rebuilding process will create images for the various models.
> An extraction log is created as extract.log.

B. Making changes to the firmware's file system

> Modifying the firmware is simple. The file system is stored in the working directory you supplied to extract_firmware.sh. Here are the subfolders of this directory:
> rootfs/
> This is where the file system is. Make modifications here.
> image_parts/
> This is where intermediate files are stored. If you need to replace the kernel (not at all recommended), you can do so by replacing the appropriate file here (usually segment2).
> installed_packages/
> If you use the ipkg_install.sh tool, this is where packages you've installed after firmware extraction will reside.
> One can manually change the files in rootfs and/or use the automated package scripts.

> Installing packages (.IPK)

> > In compliment to manually making changes, one can use pre-built IPKG format package files with the ipkg_install/remove scripts. Although not all will work and a repository for packages verified to work has not yet been produced, many packages work just fine.

> > Packages are pre-built collections of files pertaining to a set of software. OpenWrt and DD-WRT both use packages, and most are cross-compatible. These packages are stored in a tar/gzip archive of a pre-defined structure that includes some control files. You can extract and copy the files manually, or use the ipkg_install.sh script included with this kit. For a list of some of the available packages (not all may work), see http://downloads.openwrt.org/whiterussian/packages/ and http://ftp.selfnet.de/pub/openwrt/backports/rc5/ . We are currently in the process of compiling a set of packages known to be good for DD-WRT ... to help in this matter please visit here.

> > For information on how to create IPKs, see the IPK Creation Kit.

> > Example use:

> > $ ./ipkg_install.sh some_package-1.2.5.ipk working_directory/

> > some_package-1.2.5.ipkg would be the filename of the package.
> > working_directory is the same working directory you supplied to the extract_firmware.sh script.

> > Notes:

> > A script, ipkg_install_all.sh is also available to install an entire folder of packages.

> > The collection of installed packages will be stored in working_directory/installed_packages .

> Removing packages (.IPK)

> > To remove packages there is the ipkg_remove.sh script.

> > Example use:

> > $ ./ipkg_remove.sh some_package-1.2.5.ipk working_directory/

> > some_package-1.2.5.ipkg would be the filename of the package.
> > working_directory is the same working directory you supplied to the extract_firmware.sh script.

> > Notes:

> > A script, ipkg_remove_all.sh is also available to install an entire folder of packages.

> > The collection of installed packages stored working_directory/installed_packages will be updated to reflect the change if successful.

C. Re-building the firmware (build_firmware.sh)

> Rebuilding the firmware is as easy as extracting it. Use the build_firmware.sh script to automate the process. You must run this tool from inside the directory it exists in.

> Example use:

> $ ./build_firmware.sh output_directory/ working_directory/

> - output_directory is the path to which the created firmware images should be stored. If images already exist here, they will be over-written. Firmware images for various models will be emitted (these images are all the same but have different header patterns so they are accepted by the target models).
> - working_directory is the working directory supplied to extract_firmware.sh.

> A building log is created as build.log.

VIII. Caveats

There are a few things to watch out for when modifying firmware images. If you make reasonable changes these problems will probably never affect you. Be warned, this section may be technical in parts.

A. Binary compatibility issues

- If installing a binary package:
  - It is possible the uclibc or glibc library was pruned (unused objects removed). In the unlikely event the binary package requires some of these previously unused (and now missing) functions, you may need to install to the firmware file system a new uclibc or glibc library. If you are installing another pruned uclibc or glibc library, then it is remotely possible some pre-existing package requires objects not found in this new library. In such a very rare case you can re-link the library or find a full copy of the library to install.
  - Different firmwares may use different kernel versions, or have different kernel modules installed. If you get too wild with your mix-and-matching (i.e. use older versions with newer, or mixtures of different firmwares all-together), problems can result. To install a new kernel (not at all recommended), replace the segment2 (or other) file found found in the intermediate directory with an appropriate one (can be from another firmware as long as lzma compression is used).
  - Make sure any dependencies are also installed to the file system, along with initialization scripts.
- If making web UI changes:
  - WARNING: DD-WRT builds dated later than 08/04/2006 have some protections against modifying the web UI.

B. Maximum firmware size

Currently, the maximum firmware image size supported by this kit is 5,898,240 (0x5A0000) bytes, but can be adjusted through the "-m [size]" parameter of the asustrx tool.

C. Solutions to file system incompatibilities

! This section is very much a work in progress !

The squashfs file system is case sensitive (i.e. teSt.bin ! = test.bin) and adheres to EXT2 naming rules. To support proper extraction of the file system, the working directory should be on a case sensitive file system and support symbolic links. The scripts will warn you if you attempt to use as working directory on a case INsensitive file system. You'll see a number of errors if you extract to a file system that doesn't support symbolic links.

Note: Case insensitivity doesn't always cause problems, especially if the file system preserves case ok. It depends on the firmware and version. Most of the time for DD-WRT as long as a case preserving file system is used as a working directory, all should work fine.

These solutions apply to those using CYGWIN, or some other environment that may not have a case sensitive file system by default. Linux users typically do not have this problem since EXT2 and EXT3 file systems are case sensitive.

WARNING: THESE ARE IN EARLY STAGES OF TESTING AND MAY NOT WORK.

| Applies to | Description
SOLUTION 1 | CYGWIN, OTHER | It's reported this does not work under CYGWIN due to the CYGWIN utilities all being case INsensitive (thanks pagedude). Use a network share to a case sensitive file system. This is, by far, the easiest solution. Simply use an appropriate network share for the working directory.
SOLUTION 2 | CYGWIN | For NT partitions it's reported you can enable case sensitivity through the following registry key, but I do not know if this works or not. \HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\kernel\obcaseinsensitive [REG_DWORD] = 0

SOLUTION 3

OS X (not usually needed, at least for DD-WRT)

One can either Turn on HSFX for the boot disk, or (more easily) create a temporary HSFX case sensitive file system to use as a working directory.

Example of how to create a temporary HSFX case sensitive file system:
to create:
$ hdiutil create -size 20m -fs HFSX -volname db90h -nouuid test-image
to mount:
$ hdiutil attach test-image.dmg
to un-mount:
$ hdiutil detach /Volumes/db90h

(thanks solarflare for this information)

SOLUTION 4 | CYGWIN | Mount an ext2 files system. Eko, of DD-WRT forums, contributes this tip. Make a partition on a hard drive or USB thumb drive, then install "Ext2 Installable File System For Windows". 10MB should generally be plenty of working directory space, but 20MB is suggested for more optimal file system performance.

IV. Acknowledgments and more

The author and maintainer of this kit is Jeremy Collake (db90h). Special thanks to Solarflare, TexHex <floh@fbesser.de>, and others.

If YOU would like to contribute to this kit, send me an email at jeremy.collake@gmail.com or visit the current project hosting at http://code.google.com/p/firmware-mod-kit/ . Developers are welcome.

X. Links

- IPK Creation Kit (alpha testing)

X. Revision history

0.51

- Increased maximum firmware image size for TRX images. Now supports DD-WRT 'mega' images.

0.50

- Removed unnecessary -lz parameter (zlib) from gcc.

0.49

- unsquashfs: created first ever (afaik) blend of unsquashfs for squashfs 2.x images.
- mksquashfs: added 2-1.r2. Script support remains todo.
- other misc. changes.

0.48

- scripts: now use cramfsck -x to extract cramfs images instead of uncramfs. uncramfs proved to fail to extract some images.
- dirs: renamed cramfs-1.1 to cramfs-2.x to reflect the accurate version number.

0.471

- motorola_bin: added source code.

0.47

- wrt_vx_imgtool: added this utility, which can build, extract, view, and fix firmware images for WRT54G(s) v5/v6 series running VxWorks.
- motorola_bin: added this binary only (linux) utility. Build script doesn't invoke it yet, but it's easy to use for those who need it.
- untrx: minor memory leak fixed.
- docs: added some device compatibility notes and other edits.

0.46:

- scripts: added support for cramfs (testing)
- scripts: added support for WL-520/530/550G images (only 530g tested).
- sources: added cramfs 1.1 tools (mkcramfs, cramfsck)
- images: a symlink is now used between the TRX and generic image. Previously, the intermediate TRX image was just renamed to the generic image.
- scripts: no longer show untrx output
- docs: minor additions

0.45:

- untrx: added support for attempted detection of file system and version and emission of appropriately named segment.
- untrx: can now handle trx's that have less than 3 segments.
- untrx: complete overhauled (needed it). many changes.
- sources: moved squashfs 3.0 stuff to a dedicated directory.
- packages: added dd-wrt package for Samba. thanks TexHex <floh@fbesser.de>.
- asustrx: updated to add -b switch for padding segments (used by ASUS).
- asustrx: other misc. enhancements.
- scripts: added check_for_upgrade.sh which automatically checks for a newer version of the kit. It's launched when you use extract or build firmware.
- scripts: many other fixes and enhancements.

0.43:

- scripts: ipkg_remove.sh - rewritten core templated file removal. It actually works now.
- scripts: moved build stuff to shared.inc, as appropriate.
- packages: added dd-wrt package KAID, mainly for removal of this gargantuan (nearly 1MB uncompressed) package. Note: you can easily create your own packages for easier removal of things you don't want by using the IPK template.

0.42:

- ipk_template: added new ipk_template directory and make_ipk.sh script to create IPKG format packages (.IPK).
- packages: added my newly created ssmtp package with nvram->conf file startup script. Installs completely into the DD-WRT file system.
- packages: added ddwrt_packages subdirectory which will include pre-built packages custom tailored or known to work
- scripts: ipkg_install.sh numerous improvements and fixes.
- scripts: ipkg_remove.sh numerous improvements and fixes.
- scripts: ipkg_install.sh has new untested/alpha stage capabilities to handle IPKs with remote sources.
- docs: several updates

0.40 beta (md5: 786472cfa03fb1cd70e1b025d091168b)

- scripts: ipkg_remove.sh - new script for basic IPK package removal
- scripts: ipkg_install.sh possible OS X fixes
- scripts: ipkg_install.sh changed technique of recording installed packages
- scripts: ipkg_install.sh outputs control file so user can see dependencies and more
- scripts: several improvements
- docs: hopefully more readable format in places
- docs: many mods and extensions

0.33 beta

- structure: moved tool source codes to src subfolder
- scripts: added ipkg_install_all.sh to install an entire folder of packages.
- scripts: consolidated shared code into an 'include' script
- scripts: added forgotten parameter count check to ipkg_install.sh
- scripts other minor changes

0.31 beta

- new tests indicate OS X with native HFS+ file system results in firmware images that work just fine! This build should be fully compatible with OS X without even necessitating HSFX use for most firmwares.
- mksquashfs: fix for big endian machines (specifically ppc / OS X) resulting from change to support dd-wrt 08/10/06+ builds.

0.30 beta

- scripts: created symlinks that omit the .sh extension
- scripts: expanded build success check
- scripts: fixed failure to exit if tools build failed
- scripts: other minor cosmetic improvements
- untrx: fixed a file handle leak in v0.28 beta
- docs: added example procedure to install SSMTP into DD-WRT image
- docs: added table of contents
- docs: misc. additions and revisions

0.29 beta

- scripts: added ipkg_install.sh to install .IPK format packages.

0.28 beta

- the v0.26 fix for changed signature of 08/10/06 builds was incomplete (test was flawed), resulting in builds that wouldn't run. fixed properly now.
- mksquashfs: added -magic [file] parameter to embed whichever is used in the original firmware
- untrx: added extraction of squashfs signature/magic for passing to mksquashfs
- docs: added more stuff

0.27 beta

- scripts: changed warning about file system.

0.26 beta

- Fixed compatibility with DD-WRT builds 08/10/06 and higher. Brainslayer played a joke on me.

0.25 beta

- OS X now works. When using case insensitive file system though built firmwares might have errata.
- trx: removed this tool due to odd command line parsing bug on (big endian?) OS X systems. asustrx now used in its place.
- scripts: file system case insensitive now warning, can be continued in default settings.
- untrx: fixed some endian problems.
- untrx: fixed a problem with U2ND headers (previous version only worked on trx/generic firmwares).
- docs: minor additions.

0.23 beta

- package: removed linux executables, download now half size. Surely everyone can install basic build tools..
- scripts: added working directory file system test for case insensitivity and write access
- scripts: added option to continue on after warning when OS X or CYGWIN present
- scripts: removed pre-fixed script name, much cleaner looking output now
- scripts: added variable to choose whether or not to abort if file system found incompatible
- scripts: added cleanup.sh script to clean tools and logs.
- untrx: renamed extfirmware to untrx, much more intuitive name

0.22 beta

- scripts: added support for OS X
- scripts: added support for CYGWIN
- scripts: minor cosmetic work
- extfirmware: added endian neutrality
- trx: fixed OS X support
- asustrx: fixed OS X support

0.20 beta

- scripts: added check for Linux and OS X and appropriate compatibility messages
- extfirmware: added GNU license header
- extfirmware: removed some unnecessary header includes
- extfirmware: switched to u_int32_t for wider compatibility
- extfirmware: other minor improvements
- trx: increased maximum firmware size to 0x5A0000 (OpenWrt) from 0x3A0000 (DD-WRT).
- trx: switched to u_int32_t for wider compatibility
- makefile: uses gcc instead of cc now
- docs: several updates

v0.15 beta

- Reworked scripts so they have cleaner, clearer output and use a log file (build.log and extract.log)
- Added more sanity checks in the scripts
- Documentation additions
- Misc. polishing

v0.13 alpha: Documentation updates, re-included pre-built executables
v0.10 alpha: First decent public release.

XI. About the author

I've been doing R&D on for the WRT community for a few months now. Notable accomplishments include developing a method to flash linux onto the WRT54G(S) v5 series without hardware modifications, an assortment of utilities including one to manipulate VxWorks compatible firmware images for the WRT54G(S) v5 series, one to manipulate default nvram variables embedded in CFEs, one to encode and decode sdram parameters, and many more. I've been a commercial software developer for over 10 years, specializing in low-level programming for the x86 architecture, primarily for Windows. My web site is at http://www.bitsum.com, where you'll find links to some of my software, including PECompact, a popular Win32 executable compressor.

Contact me at jeremy@bitsum.com if you need work done or or donate money to encourage more works like this by donating via Paypal to jeremy@bitsum.com. There are also other ways to help. Feel free to contact me if interested.

This document (c)2006,2007,2008 Jeremy Collake.
All Rights reserved. This document may be freely republished in its unaltered and whole form only. Alterations or partial publishing requires approval of Jeremy Collake <jeremy@bitsum.com>.

This kit is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA