Home > Software > Soul Spiral
Soul Spiral
article by Ben Lincoln, software by Ben Lincoln and Andrew Fradley
Soul Spiral is a utility for unpacking the contents of the bigfile.dat format used by many Crystal Dynamics games based on the engine originally created for Gex: Enter the Gecko. It can also read the pill.big data file from the PC and PlayStation versions of Blood Omen, and has support for the .bgx / .big data files used in Blood Omen 2 (except the GameCube version) and Mad Dash Racing.
Soul Spiral's only function is to unpack content from a monolithic data file. It is not an image/model viewer or a sound file player. Once the files are extracted, other tools (such as ModelEx, The Eye of RAW, The Sound of Her Wings, or the outstanding Blood Pill) are necessary to actually do anything with any of the files. Because the game assets are usually in a proprietary format, not all of them have been reverse-engineered well enough to do anything with yet.
Soul Spiral requires the .NET Framework to be installed on your PC. It can be obtained from the Microsoft website or Windows Update, but should be installed by default if you are using Windows 7 or later.
Usage:
- Open bigfile.dat (or one of the other supported files).
- Soul Spiral will attempt to automatically determine the game it's from. If it is successful, you should probably just click Use the automatic settings. If it's not successful, pick the best match from the list and click Use the selected file type. If it works, please email me and let me know which game it worked with, and the contents of the "BigFile Information" view in Soul Spiral.
- Browse the contents and either export a single item or all of them using the Export menu.
- Make use of the contents in another utility.
Supported Formats:
- Akuji: The Heartless (PlayStation) - fully supported, including hashed name lookups.
- Blood Omen (all known PlayStation and PC versions) - fully supported, including complete hashed name lookups for most versions of the game.
- Blood Omen 2 (PC, PlayStation 2, and Xbox) - full support for .bgx / .big files.
- Gex (PlayStation and Sega Saturn) - uncompressed files only, no hashed name lookups.
- Gex: Enter the Gecko (PlayStation) - fully supported, including hashed name lookups.
- Gex 3: Deep Cover Gecko (PlayStation) - fully supported, including hashed name lookups.
- Defiance (all known PlayStation 2 and PC releases) - fully supported, including hashed name lookups for most files.
- Mad Dash Racing (Xbox) - full support for .bgx files.
- Soul Reaver Lighthouse Demo (PlayStation) - fully supported, including hashed name lookups for most files.
- Soul Reaver (all known PlayStation, PC, and Dreamcast versions created from January 23rd, 1999 onward) - fully supported, including hashed name lookups for most files.
- Soul Reaver 2 Air Forge Demo (PlayStation 2) - fully supported, including hashed name lookups for most files.
- Soul Reaver 2 (all PlayStation 2 and PC releases) - fully supported, including hashed name lookups for most files.
- Tomb Raider: Legend Demo (PlayStation 2) - supported, but without hashed name lookups.
- Tomb Raider: Legend (PlayStation 2 and PSP) - supported, but without hashed name lookups.
- Walt Disney World Quest: Magical Racing Tour (PlayStation) - fully supported, including hashed name lookups.
Currently Unsupported Formats:
- Blood Omen 2 (all platforms) - .rkv files are not yet supported.
- Blood Omen 2 (GameCube) - Not yet supported due to use of compressed files.
- Legacy of Kain: Defiance (all Xbox releases) - uses an unsupported variation of the bigfile format.
- Legacy of Kain: Defiance (downloadable PC release) - uses an encrypted bigfile format.
- Mad Dash Racing (Xbox) - .rkv files are not yet supported.
- Tomb Raider: Legend (all Xbox/Xbox 360/PC releases) - uses an unsupported variation of the bigfile format.
- Whiplash (PlayStation 2 / Xbox) - This game uses a later revision of the Mad Dash Racing engine, but the file format for .bg2, .mxb, and .rkv files is currently unsupported.
Known Issues:
- The save/discard buttons in the hex editor are not very good (this is my fault, not Andrew's). The green one saves changes, and the red one discards them.
- When unpacking files from PC versions of Blood Omen, files whose actual name has a .HTM extension are given a .HTM.TIM extension. This is intentional, so that they can be correctly recognized as PlayStation TIM images by the user and the operating system, as well as to avoid confusion (.HTM is more commonly-associated with HTML files). They cannot simply be given a .TIM extension, because there are also lower-resolution files with identical names that have .TIM extensions.
- There will be a noticeable pause when loading larger files, such as those from Defiance. This is due to the SHA256 fingerprinting. I couldn't think of a good way to avoid this while still supporting all of the known files.
See also the GitHub repository for this tool, maintained by Andrew Fradley.
Updates in Version 1.18:
- Fixed an old-but-undiscovered bug which results in files exported from the PlayStation release and beta versions of Soul Reaver to have the incorrect extension .pcm instead of .drm. Thank you to Gh0stBlade and Andrew Fradley for reporting the issue.
Updates in Version 1.17:
- Added support for the 1999-04-14 prototype of Soul Reaver. Thank you to justz00t for saving this previously-unknown build of the game!
- Added support for the 2001-07-17 prototype of Soul Reaver 2. Thank you to Project Deluge for saving such an interesting piece of history!
- Added support for several other bigfiles from different variations of existing games.
Updates in Version 1.14:
- Added support for prototypes of Soul Reaver from early 1999.
- Added support for the Japanese PlayStation 2 demo version of Soul Reaver 2, courtesy of Nex.
Updates in Version 1.13:
Updates in Version 1.12:
- Added complete filename-lookup data for the recently-discovered prerelease version of Gex 3 from 1998-12-30.
- Added fingerprint data for the 1.02 patch version and Premiere Collection version of Soul Reaver 2. Thanks Andrew!
Updates in Version 1.11:
- Added nearly-complete filename-lookup data for the Japanese PlayStation 2 version of Soul Reaver 2, as well as the PC demo version of the same game.
- Corrected numerous incorrect filename entries for Soul Reaver 2 which had become corrupt in version 1.8, and added quite a few additional entries.
- Additional filename-lookup data for Defiance.
- Added additional filename-lookup data for some uncommon versions of several games.
- Fixed the program's icon.
Updates in Version 1.10:
- Added full filename-lookup data for all known Dreamcast versions of Soul Reaver.
- Added about 66%-complete filename-lookup data for the PC demo version of Soul Reaver 2.
- Additional filename-lookup data for Defiance, thanks to Andrew.
- Corrected a bug which resulted in files not being identified correctly in some cases.
- Additions/corrections to file fingerprint data.
Updates in Version 1.9:
- Added proper credit to John Doom for his role in Soul Spiral having .bgx / .big file support.
- Additional filename-lookup data for Defiance.
- Added some additional fingerprints for files that are not currently handled, but should be in a future version.
Updates in Version 1.8:
- Andrew Fradley spent a bunch of time adding support for the .bgx / .big files used in Blood Omen 2 and Mad Dash Racing, and that support has been merged into this release.
- Added decompression support for the compression used for some files in Akuji: The Heartless, Gex: Enter the Gecko, Gex 3: Deep Cover Gecko, The Lighthouse Demo / proto1 early demo versions of Soul Reaver, and Walt Disney World Magical Racing Tour. It should now be possible to successfully unpack all content from the data files for these games. This will probably work with alpha builds of Soul Reaver as well.
- Added complete hashed-name lookup tables for Akuji: The Heartless, Gex: Enter the Gecko, Gex 3: Deep Cover Gecko, and Walt Disney World Magical Racing Tour.
- Added nearly-complete hashed-name lookup tables for The Lighthouse Demo early demo versions of Soul Reaver. This will also probably work with alpha builds of Soul Reaver.
- Added most of the remaining hashed-name lookup table data for all known versions of Blood Omen, Soul Reaver (except Dreamcast versions), and Defiance.
- Merged in some additional bug fixes from Andrew.
- Added support for certain PAL PlayStation beta builds of Soul Reaver, which use a different XOR encryption key than other PlayStation versions of the game.
- Completely rewrote the file fingerprinting mechanism to also use SHA256 hashes of known files. This should help prevent false-positive matches, but was also necessary in order to differentiate between incompatible bigfiles with identical filesizes, such as the PAL beta builds of Soul Reaver mentioned above.
- Added fingerprints for an enormous list of additional files which were previously unrecognized.
- Rewrote the filename hash-lookup code to use ASCII hexadecimal representations of the hashes instead of integer representations.
- Rewrote some (but not all) of the really bad code that I wrote 10-15 years ago to make it easier to support additional games. There's still a lot of work to be done.
Show changes in previous releases
Updates in Version 1.7:
- Can now correctly identify most of the content in the NTSC version of the Soul Reaver 2 Air Forge demo.
Updates in Version 1.6:
- Can now correctly identify most of the content in the PAL version of the Soul Reaver 2 Air Forge demo, and some of the content in the NTSC version.
Updates in Version 1.5:
- Replaced the old JET (MS Access) hash database system with flat text files so that Soul Spiral now works on Windows® 7.
- Added preliminary support for bigfiles from big-endian platforms (e.g. the Sega Saturn).
- Added very basic support for the Sega Saturn version of Gex as a test of the big-endian functionality.
- Added an ASCII view of the index data for bigfile contents to the information panel when files are selected. The ASCII view and decimal views are also displayed in "opposite-endian" format, which is whichever endianness the bigfile was not read as. E.g. if the bigfile is from a little-endian platform, "opposite-endian" is big-endian.
- Really fixed the bug regarding file counts that was only sort of fixed in version 1.4.
- Some minor cleanup for the source code release (see Source Code Collections).
Updates in Version 1.4:
- Added most of the remaining hashes for the retail PC and PlayStation versions of Blood Omen. Thanks to Andrey Grachev for providing some of the file information.
- Corrected two old-but-unnoticed bugs: selecting the root node of the bigfile in the left pane now displays the correct total number of files, instead of twice the total. Also corrected a capitalization issue with the hash lookup table for the English PC version of Soul Reaver 2.
- Included the correct (release) version of the ohrainBOWS.mdb file this time as it is much smaller than the dev version.
Updates in Version 1.3:
- Incorporated most of the reverse hash lookups for Blood Omen (both PC and PlayStation versions). This is an initial attempt, so there may be some errors. An update with the rest should be released in the near future.
- Incorporated an experimental hex editor feature is, courtesy of Andrew Fradley. This allows basic changes to be made to bigfile contents without extracting them and then doing a replace. The save/revert icons I used when I initially integrated it with Soul Spiral (in the unreleased 1.2 version) are not very intuitive, but I didn't have time to change them for this official release. The green icon saves changes, and the red icon discards them.
Updates in Version 1.1:
- Incorporated some additional reverse hash lookups for Defiance and SR2 based on information from Alexey Kragin and Andrew Fradley.
- Made the progress dialogue modal.
Updates in Version 1.0:
- This a complete rewrite from the ground up.
- Added support for reading the compressed file format used in the PS2 versions of Soul Reaver 2 (this was REALLY hard to reverse-engineer), so the RAW images from those games can now be exported.
- Brand new UI with fancy buttons and progress bars that is much nicer to work with.
- Added the ability to export a single directory and its subdirectories, in addition to the old export single/all functions.
- Added the ability to replace a file (unless it is compressed in the BigFile) with another file of identical size. This allows files to be exported, modified, and then re-imported.
- Added reverse hash lookup functionality for many more files in Soul Reaver and Defiance.
- Added preliminary support for a couple of other games, including Gex 2.
- Added full support for the XOR'd index variation used in the PAL retail release of Soul Reaver, instead of the workaround in previous versions.
- Changed the duplicate filename behaviour to be more intuitive and less messy.
- Added the ability to choose to view proper directory names or the raw index organization.
- All I/O operations are much faster.
- Added the ability to export raw index data instead of looking at messy log files.
- Removed the Options menu because at least for now there is no need of one.
Updates in Version 0.9:
- Andrew Fradley reverse-engineered the filename-hashing algorithm from the PC and PlayStation versions of Soul Reaver. I've added support in the ohrainBOWS database for the vast majority of the files in the PC version. The PlayStation version is partially complete.
- Fixed a bug that prevented Soul Spiral from opening the German PC release of Soul Reaver 2.
- Added support for reverse hash lookups for the German PC release of Soul Reaver 2.
- Implemented nested directories properly.
- Directories and their contents are now sorted in alphabetical order.
- Modified the handling of duplicate references to the same file in the bigfile. They will now all be listed, with a .dupe extension for multiple references to the same file. A future release will have a more elegant solution.
- Added identification abilities for a few other variants of the games, including the PAL release of Legend (thanks Dan) and Kain the Vampire.
Updates in Version 0.8:
- Used information from Andrew Fradley to add reverse hash lookups for the PC version of Soul Reaver 2.
- Added a few more hash names to both versions of Soul Reaver 2.
Updates in Version 0.7:
- Migrated to .NET 2.0.
- Added recognition for the PSP release of Tomb Raider: Legend.
- Fixed a nasty bug related to the PS2 versions of Soul Reaver 2. The bug still applies to the Air Forge demo - it will be fixed completely in a future release.
- Added most of the other hash names to the ohrainBOWS database for the PS2 versions of Soul Reaver 2. Recognition of proper filenames in those bigfiles is now over 94%.
Updates in Version 0.6:
- Added support for Tomb Raider: Legend and the prerelease demo (considerable thanks for this goes to Ben from Tomb Raider Central for sending me the PAL coverdisc for analysis!). This only includes the PS2 versions. The PAL retail version will not be identified correctly (I don't have a copy to fingerprint yet), but it should open if the TR: Legend file type is selected.
- Added support for reverse hash lookups to determine proper filenames. This currently only works with the North American and PAL releases of Soul Reaver 2 for the PS2, and not all names are present. As I have time to research the other games, support will be added for them as well. This feature depends on the presence of the file "ohrainBOWS.mdb" in the same folder as Soul Spiral.
- Changed the UI to represent folders differently to work better with reverse hash lookups.
Updates in Version 0.5:
- The file format used in the Soul Reaver lighthouse demo, Akuji, and Gex 3 is now interpreted correctly. However, there are still some oddball entries in their indices for items with a length of 0 and an offset of 0. I believe these are pointers of some kind to external files.
- The Options window now includes a checkbox for turning off the prepending of object offsets to their names if their names can be determined based on their contents (like the "*.SRObj" files. When exporting, this prevents duplicate items from being exported (and makes individual objects easier to find), but if two different objects have the same internal name, only one will be exported, and there is no guarantee as to which one it will be.
- A few additional item types are recognized, notably the sound clip collections in Soul Reaver 2.
Updates in Version 0.4:
- All versions of Soul Reaver 2 and Defiance are now interpreted correctly. 0.3 read the right locations of their files, but not the right file sizes.
- Added support for the Soul Reaver 2 Air Forge demo.
- Added support for Blood Omen.
- Added a few additional file types that Soul Spiral can recognize.
Updates in Version 0.3:
- Can now read the bigfile from the PAL PlayStation version of Soul Reaver.
- Has an Options window with a debug logging option that will probably only be useful to me.
- Only displays read errors once instead of over and over and over.
- TIM-recognizing for the PlayStation version of Soul Reaver has been tweaked.
Note: Releases prior to 1.1 are no longer publicly available.
|
The main Soul Spiral UI
|
|
|
The file type selection dialogue
|
|
|
The automatic recognition dialogue
|
|
|
|
Related Articles:
The Great Bigfile Hunt
Source Code Collections
Blood Omen Decompressor
The Eye of RAW
ModelEx
Only Skin Deep
The Sound of Her Wings
Extract the Music from Defiance for the PC (How To)