From 85003a40c2f89693faed8af9185571e0b9a8194f Mon Sep 17 00:00:00 2001 From: lethosor Date: Wed, 8 Jul 2020 01:03:58 -0400 Subject: [PATCH] Split memory research into separate doc and expand --- docs/Binpatches.rst | 2 +- docs/Compile.rst | 2 + docs/Contributing.rst | 20 ------ docs/Memory-research.rst | 149 +++++++++++++++++++++++++++++++++++++++ index.rst | 1 + 5 files changed, 153 insertions(+), 21 deletions(-) create mode 100644 docs/Memory-research.rst diff --git a/docs/Binpatches.rst b/docs/Binpatches.rst index a0fcc6630..40ae8af32 100644 --- a/docs/Binpatches.rst +++ b/docs/Binpatches.rst @@ -24,7 +24,7 @@ There are no binary patches available for Dwarf Fortress versions after 0.34.11. This system is kept for the chance that someone will find it useful, so some hints on how to write your own follow. This will require disassembly and -decent skill in `memory research `. +decent skill in `memory research `. * The patches are expected to be encoded in text format used by IDA. diff --git a/docs/Compile.rst b/docs/Compile.rst index 5bb51a01f..c8c51e88c 100644 --- a/docs/Compile.rst +++ b/docs/Compile.rst @@ -161,6 +161,8 @@ in any case. Note that the scripts in the "build" folder on Windows will set the architecture automatically. +.. _compile-build-options: + Other settings -------------- There are a variety of other settings which you can find in CMakeCache.txt in diff --git a/docs/Contributing.rst b/docs/Contributing.rst index cb9e87a0e..6231b6f04 100644 --- a/docs/Contributing.rst +++ b/docs/Contributing.rst @@ -41,26 +41,6 @@ How to get new code into DFHack * Work on :issue:`reported problems ` will take priority over ideas or suggestions. -.. _contributing-memory-research: - -Memory research ---------------- -If you want to do memory research, you'll need some tools and some knowledge. -In general, you'll need a good memory viewer and optionally something -to look at machine code without getting crazy :) -Using publicly known information and analyzing the game's data is preferred. - -Good Windows tools include: - -* IDA Freeware 7.0 (for non-commercial use, supports 32-bit and 64-bit) -* Cheat Engine - -Good Linux tools: - -* angavrilov's df-structures gui (32-bit only, visit us on IRC for details) -* IDA Freeware 7.0 (see above) -* edb (Evan's Debugger) -* Some of the tools residing in the ``legacy`` dfhack branch. Using the library as a developer ================================ diff --git a/docs/Memory-research.rst b/docs/Memory-research.rst new file mode 100644 index 000000000..08c0c9573 --- /dev/null +++ b/docs/Memory-research.rst @@ -0,0 +1,149 @@ +.. _memory-research: + +############### +Memory research +############### + +There are a variety of tools that can be used to analyze DF memory - some are +listed here. Note that some of these may be old and unmaintained. If you aren't +sure what tool would be best for your purposes, feel free to ask for advice (on +IRC, Bay12, etc.). + +.. contents:: + :local: + + +Cross-platform tools +==================== + +Ghidra +------ + +Ghidra is a cross-platform reverse-engineering framework (written in Java) +available at https://ghidra-sre.org. It supports analyzing both 32-bit and +64-bit executables for all supported DF platforms. There are some custom DFHack +Ghidra scripts available in the `df_misc`_ repo (look for ``.java`` files). + + +IDA Freeware 7.0 +---------------- + +Available from `Hex-Rays `_. +Supports analyzing both 32-bit and 64-bit executables for all supported DF platforms. +Some ``.idc`` scripts for IDA are available in the `df_misc`_ repo. + +.. _df_misc: https://github.com/DFHack/df_misc + + +Hopper +------ + +Runs on macOS and some Linux distributions; available from https://www.hopperapp.com/. +`TWBT `_ uses this to produce some patches. + + +DFHack tools +------------ + +Plugins +~~~~~~~ + +There are a few development plugins useful for low-level memory research. They +are not built by default, but can be built by setting the ``BUILD_DEVEL`` +`CMake option `. These include: + +- ``check-structures-sanity``, which performs sanity checks on the given DF + object. Note that this will crash in several cases, some intentional, so using + this with `GDB ` is recommended. +- ``memview``, which produces a hex dump of a given memory range. It also + highlights valid pointers, and can be configured to work with `sizecheck` + to auto-detect object sizes. +- ``vectors``, which can identify instances of ``std::vector`` in a given memory range. + +Scripts +~~~~~~~ + +Several `development scripts ` can be useful for memory research. +These include (but are not limited to): + +- `devel/dump-offsets` +- `devel/find-offsets` +- `devel/lsmem` +- `devel/sc` (requires `sizecheck`) +- `devel/visualize-structure` +- Generally, any script starting with ``devel/find`` + +.. _sizecheck: + +Sizecheck +~~~~~~~~~ + +Sizecheck is a custom tool that hooks into the memory allocator and inserts a +header indicating the size of every object. The corresponding logic to check for +this header when freeing memory usually works, but is inherently not foolproof. +You should not count on DF being stable when using this. + +DFHack's implementation of sizecheck is currently only tested on Linux, although +it probably also works on macOS. It can be built with the ``BUILD_SIZECHECK`` +`CMake option `, which produces a ``libsizecheck`` +library installed in the ``hack`` folder. You will need to preload this library +manually, by setting ``PRELOAD_LIB`` on Linux (or ``LD_PRELOAD`` if editing +the ``dfhack`` launcher script directly), or by editing the ``dfhack`` +launcher script and adding the library to ``DYLD_INSERT_LIBRARIES`` on macOS. + +There is also an older sizecheck implementation by Mifki available on +`GitHub `__ (``b.cpp`` is the main +sizecheck library, and ``win_patch.cpp`` is used for Windows support). To use +this with other DFHack tools, you will likely need to edit the header's +magic number to match what is used in `devel/sc` (search for a hexadecimal +constant starting with ``0x``). + +Legacy tools +~~~~~~~~~~~~ + +Some very old DFHack tools are available in the `legacy branch on GitHub `_. +No attempt is made to support these. + + +Linux-specific tools +==================== + +.. _linux-gdb: + +GDB +--- + +`GDB `_ is technically cross-platform, but +tends to work best on Linux, and DFHack currently only offers support for using +GDB on 64-bit Linux. To start with GDB, pass ``-g`` to the DFHack launcher +script:: + + ./dfhack -g + +Some basic GDB commands: + +- ``run``: starts DF from the GDB prompt. Any arguments will be passed as + command-line arguments to DF (e.g. `load-save` may be useful). +- ``bt`` will produce a backtrace if DF crashes. + +See the `official GDB documentation `_ +for more details. + + +df-structures GUI +----------------- + +This is a tool written by Angavrilov and available on `GitHub `__. +It only supports 32-bit DF. Some assistance may be available on IRC. + + +EDB (Evan's debugger) +--------------------- + +Available on `GitHub `__. + + +Windows-specific tools +====================== + +Some people have used `Cheat Engine `__ for research in the past. diff --git a/index.rst b/index.rst index b3498d432..4c5b2e6e4 100644 --- a/index.rst +++ b/index.rst @@ -65,4 +65,5 @@ For Developers /docs/Documentation /library/xml/SYNTAX /library/xml/how-to-update + /docs/Memory-research /docs/Binpatches