Merge remote-tracking branch 'lethosor/docs-reorg' into develop

develop
lethosor 2020-07-08 23:47:46 -04:00
commit cd46c30914
27 changed files with 858 additions and 597 deletions

@ -0,0 +1,6 @@
Thank you for your interest in contributing to DFHack! If you're reading this
document, you're probably viewing it on GitHub. The DFHack docs are hosted
on [ReadTheDocs](https://dfhack.readthedocs.io/) - in particular, contributing
guidelines are [here](https://docs.dfhack.org/en/latest/docs/Contributing.html).
Double-checking the style guidelines before submitting a pull request is
always appreciated.

@ -1,229 +0,0 @@
###########################
How to contribute to DFHack
###########################
.. contents::
.. _contributing-code:
Contributing Code
=================
Several things should be kept in mind when contributing code to DFHack.
Code Format
-----------
* Four space indents for C++. Never use tabs for indentation in any language.
* LF (Unix style) line terminators
* Avoid trailing whitespace
* UTF-8 encoding
* For C++:
* Opening and closing braces on their own lines or opening brace at the end of the previous line
* Braces placed at original indent level if on their own lines
* #includes should be sorted. C++ libraries first, then dfhack modules, then df structures,
then local includes. Within each category they should be sorted alphabetically.
How to get new code into DFHack
-------------------------------
* Submit pull requests to the ``develop`` branch, not the ``master`` branch.
(The ``master`` branch always points at the most recent release)
* Use a new branch for each feature or bugfix so that your changes can be merged independently
(i.e. not the master or develop branch of your fork).
* If possible, compile on multiple platforms when changing anything that compiles
* It must pass CI - run ``python travis/all.py`` to check this.
* Update ``changelog.txt`` and ``docs/Authors.rst`` when applicable. See
`build-changelog` for more information on the changelog format.
* Create a GitHub pull request once finished
* Submit ideas and bug reports as :issue:`issues on GitHub <>`.
Posts in the forum thread can easily get missed or forgotten.
* Work on :issue:`reported problems <?q=is:open+-label:idea>`
will take priority over ideas or suggestions.
.. _contributing-memory-research:
Memory research
---------------
If you want to do memory research, you'll need some tools and some knowledge.
In general, you'll need a good memory viewer and optionally something
to look at machine code without getting crazy :)
Using publicly known information and analyzing the game's data is preferred.
Good Windows tools include:
* IDA Freeware 7.0 (for non-commercial use, supports 32-bit and 64-bit)
* Cheat Engine
Good Linux tools:
* angavrilov's df-structures gui (32-bit only, visit us on IRC for details)
* IDA Freeware 7.0 (see above)
* edb (Evan's Debugger)
* Some of the tools residing in the ``legacy`` dfhack branch.
Using the library as a developer
================================
Currently, the most direct way to use the library is to write a script or plugin that can be loaded by it.
All the plugins can be found in the 'plugins' folder. There's no in-depth documentation
on how to write one yet, but it should be easy enough to copy one and just follow the pattern.
``plugins/skeleton/skeleton.cpp`` is provided for this purpose.
Other than through plugins, it is possible to use DFHack via remote access interface,
or by writing scripts in Lua or Ruby. There are plenty of examples in the scripts folder.
The `lua-api` is quite well documented.
The most important parts of DFHack are the Core, Console, Modules and Plugins.
* Core acts as the centerpiece of DFHack - it acts as a filter between DF and
SDL and synchronizes the various plugins with DF.
* Console is a thread-safe console that can be used to invoke commands exported by Plugins.
* Modules actually describe the way to access information in DF's memory. You
can get them from the Core. Most modules are split into two parts: high-level
and low-level. High-level is mostly method calls, low-level publicly visible
pointers to DF's data structures.
* Plugins are the tools that use all the other stuff to make things happen.
A plugin can have a list of commands that it exports and an onupdate function
that will be called each DF game tick.
Rudimentary API documentation can be built using doxygen (see build options
in ``CMakeCache.txt`` or with ``ccmake`` or ``cmake-gui``). The full DFHack
documentation is built with Sphinx_, which runs automatically at compile time.
.. _Sphinx: http://www.sphinx-doc.org
DFHack consists of variously licensed code, but invariably weak copyleft.
The main license is zlib/libpng, some bits are MIT licensed, and some are
BSD licensed. See the `license` for more information.
Feel free to add your own extensions and plugins. Contributing back to
the DFHack repository is welcome and the right thing to do :)
DF data structure definitions
-----------------------------
DFHack uses information about the game data structures, represented via xml files
in the ``library/xml/`` submodule.
See https://github.com/DFHack/df-structures, and the documentation linked in the index.
Data structure layouts are described in files following the ``df.*.xml`` name pattern.
This information is transformed by a perl script into C++ headers describing the
structures, and associated metadata for the Lua wrapper. These headers and data
are then compiled into the DFHack libraries, thus necessitating a compatibility
break every time layouts change; in return it significantly boosts the efficiency
and capabilities of DFHack code.
Global object addresses are stored in :file:`symbols.xml`, which is copied to the dfhack
release package and loaded as data at runtime.
Remote access interface
-----------------------
DFHack supports remote access by exchanging Google protobuf messages via a TCP
socket. Both the core and plugins can define remotely accessible methods. The
``dfhack-run`` command uses this interface to invoke ordinary console commands.
Currently the supported set of requests is limited, because the developers don't
know what exactly is most useful. `remotefortressreader` provides a fairly
comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`.
Documentation Standards
=======================
DFHack documentation is built with Sphinx_, and configured automatically
through CMake. If you want to build the docs *only*, use this command::
sphinx-build . docs/html
Whether you're adding new code or just fixing old documentation (and there's plenty),
there are a few important standards for completeness and consistent style. Treat
this section as a guide rather than iron law, match the surrounding text, and you'll
be fine.
Each command should have a short (~54 character) help string, which is shown
by the `ls` command. For scripts, this is a comment on the first line
(the comment marker and whitespace is stripped). For plugins it's the second
argument to ``PluginCommand``. Please make this brief but descriptive!
Everything should be documented! If it's not clear *where* a particular
thing should be documented, ask on IRC or in the DFHack thread on Bay12 -
as well as getting help, you'll be providing valuable feedback that
makes it easier for future readers!
Scripts can use a custom autodoc function, based on the Sphinx ``include``
directive - anything between the tokens is copied into the appropriate scripts
documentation page. For Ruby, we follow the built-in docstring convention
(``=begin`` and ``=end``). For Lua, the tokens are ``[====[`` and ``]====]``
- ordinary multi-line strings. It is highly encouraged to reuse this string
as the in-console documentation by (eg.) printing it when a ``-help`` argument
is given.
The docs **must** have a heading which exactly matches the command, underlined
with ``=====`` to the same length. For example, a lua file would have::
local helpstr = [====[
add-thought
===========
Adds a thought or emotion to the selected unit. Can be used by other scripts,
or the gui invoked by running ``add-thought gui`` with a unit selected.
]====]
Where the heading for a section is also the name of a command, the spelling
and case should exactly match the command to enter in the DFHack command line.
Try to keep lines within 80-100 characters, so it's readable in plain text
in the terminal - Sphinx (our documentation system) will make sure
paragraphs flow.
If there aren't many options or examples to show, they can go in a paragraph of
text. Use double-backticks to put commands in monospaced font, like this::
You can use ``cleanowned scattered x`` to dump tattered or abandoned items.
If the command takes more than three arguments, format the list as a table
called Usage. The table *only* lists arguments, not full commands.
Input values are specified in angle brackets. Example::
Usage:
:arg1: A simple argument.
:arg2 <input>: Does something based on the input value.
:Very long argument:
Is very specific.
To demonstrate usage - useful mainly when the syntax is complicated, list the
full command with arguments in monospaced font, then indent the next line and
describe the effect::
``resume all``
Resumes all suspended constructions.
If it would be helpful to mention another DFHack command, don't just type the
name - add a hyperlink! Specify the link target in backticks, and it will be
replaced with the corresponding title and linked: eg ```autolabor```
=> `autolabor`. Link targets should be equivalent to the command
described (without file extension), and placed above the heading of that
section like this::
.. _autolabor:
autolabor
=========
Add link targets if you need them, but otherwise plain headings are preferred.
Scripts have link targets created automatically.
Other ways to help
==================
DFHack is a software project, but there's a lot more to it than programming.
If you're not comfortable programming, you can help by:
* reporting bugs and incomplete documentation
* improving the documentation
* finding third-party scripts to add
* writing tutorials for newbies
All those things are crucial, and often under-represented. So if that's
your thing, go get started!

@ -22,8 +22,6 @@ import re
import shlex # pylint:disable=unused-import
import sys
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'docs'))
from gen_changelog import generate_changelog
# -- Support :dfhack-keybind:`command` ------------------------------------
# this is a custom directive that pulls info from dfhack.init-example
@ -153,9 +151,10 @@ def write_script_docs():
'gui': 'GUI Scripts',
'modtools': 'Scripts for Modders'}
for k in head:
title = ('.. _{k}:\n\n{l}\n{t}\n{l}\n\n'
title = ('.. _scripts-{k}:\n\n{l}\n{t}\n{l}\n\n'
'.. include:: /scripts/{a}about.txt\n\n'
'.. contents::\n\n').format(
'.. contents:: Contents\n'
' :local:\n\n').format(
k=k, t=head[k],
l=len(head[k])*'#',
a=('' if k == 'base' else k + '/')
@ -180,19 +179,23 @@ def all_keybinds_documented():
# Actually call the docs generator and run test
generate_changelog()
write_script_docs()
all_keybinds_documented()
# -- General configuration ------------------------------------------------
sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'docs', 'sphinx_extensions'))
# If your documentation needs a minimal Sphinx version, state it here.
needs_sphinx = '1.8'
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.extlinks']
extensions = [
'sphinx.ext.extlinks',
'dfhack.changelog',
]
# This config value must be a dictionary of external sites, mapping unique
# short alias names to a base URL and a prefix.

@ -15,7 +15,8 @@ the `binpatch` command.
at all possible - that way your work will work for many versions
across multiple operating systems.
.. contents::
.. contents:: Contents
:local:
Getting a patch
@ -24,7 +25,7 @@ There are no binary patches available for Dwarf Fortress versions after 0.34.11.
This system is kept for the chance that someone will find it useful, so some
hints on how to write your own follow. This will require disassembly and
decent skill in `memory research <contributing-memory-research>`.
decent skill in `memory research <memory-research>`.
* The patches are expected to be encoded in text format used by IDA.

@ -13,8 +13,9 @@ DFHack from source, this document will walk you through the build process. Note
that some steps may be unconventional compared to other projects, so be sure to
pay close attention if this is your first time compiling DFHack.
.. contents::
:depth: 2
.. contents:: Contents
:local:
:depth: 1
.. _compile-how-to-get-the-code:
@ -161,6 +162,8 @@ in any case.
Note that the scripts in the "build" folder on Windows will set the architecture
automatically.
.. _compile-build-options:
Other settings
--------------
There are a variety of other settings which you can find in CMakeCache.txt in

@ -0,0 +1,59 @@
.. _contributing:
###########################
How to contribute to DFHack
###########################
.. contents:: Contents
:local:
.. _contributing-code:
Contributing Code
=================
Several things should be kept in mind when contributing code to DFHack.
Code Format
-----------
* Four space indents for C++. Never use tabs for indentation in any language.
* LF (Unix style) line terminators
* Avoid trailing whitespace
* UTF-8 encoding
* For C++:
* Opening and closing braces on their own lines or opening brace at the end of the previous line
* Braces placed at original indent level if on their own lines
* #includes should be sorted. C++ libraries first, then dfhack modules, then df structures,
then local includes. Within each category they should be sorted alphabetically.
How to get new code into DFHack
-------------------------------
* Submit pull requests to the ``develop`` branch, not the ``master`` branch.
(The ``master`` branch always points at the most recent release)
* Use a new branch for each feature or bugfix so that your changes can be merged independently
(i.e. not the master or develop branch of your fork).
* If possible, compile on multiple platforms when changing anything that compiles
* It must pass CI - run ``python travis/all.py`` to check this.
* Update documentation when applicable - see `docs-standards` for details.
* Update ``changelog.txt`` and ``docs/Authors.rst`` when applicable. See
`build-changelog` for more information on the changelog format.
* Create a GitHub pull request once finished
* Submit ideas and bug reports as :issue:`issues on GitHub <>`.
Posts in the forum thread can easily get missed or forgotten.
* Work on :issue:`reported problems <?q=is:open+-label:idea>`
will take priority over ideas or suggestions.
Other ways to help
==================
DFHack is a software project, but there's a lot more to it than programming.
If you're not comfortable programming, you can help by:
* reporting bugs and incomplete documentation
* improving the documentation
* finding third-party scripts to add
* writing tutorials for newbies
All those things are crucial, and often under-represented. So if that's
your thing, go get started!

@ -4,7 +4,8 @@
DFHack Core
###########
.. contents::
.. contents:: Contents
:local:
:depth: 2
@ -165,10 +166,13 @@ right place to do it.
Most such plugins or scripts support the built-in ``enable`` and ``disable``
commands. Calling them at any time without arguments prints a list
of enabled and disabled plugins, and shows whether that can be changed
through the same commands.
through the same commands. Passing plugin names to these commands will enable
or disable the specified plugins. For example, to enable the `manipulator`
plugin::
To enable or disable plugins that support this, use their names as
arguments for the command::
enable manipulator
It is also possible to enable or disable multiple plugins at once::
enable manipulator search
@ -271,6 +275,9 @@ something. Usage::
Allows dealing with plugins individually by name, or all at once.
Note that plugins do not maintain their enabled state if they are reloaded, so
you may need to use `enable` to re-enable a plugin after reloading it.
.. _ls:
@ -408,7 +415,7 @@ All matching init files will be executed in alphebetical order.
A world being loaded can mean a fortress, an adventurer, or legends mode.
These files are best used for non-persistent commands, such as setting
a `fix <fix>` script to run on `repeat`.
a `fix <scripts-fix>` script to run on `repeat`.
.. _onUnload.init:

@ -0,0 +1,87 @@
===========================
DFHack development overview
===========================
DFHack has various components; this page provides an overview of some. If you
are looking to develop a tool for DFHack, developing a script or plugin is
likely the most straightforward choice.
Other pages that may be relevant include:
- `contributing`
- `documentation`
- `license`
.. contents:: Contents
:local:
Plugins
-------
DFHack plugins are written in C++ and located in the ``plugins`` folder.
Currently, documentation on how to write plugins is somewhat sparse. There are
templates that you can get use to get started in the ``plugins/skeleton``
folder, and the source code of existing plugins can also be helpful.
If you want to compile a plugin that you have just added, you will need to add a
call to ``DFHACK_PLUGIN`` in ``plugins/CMakeLists.txt``.
Plugins have the ability to make one or more commands available to users of the
DFHack console. Examples include `3dveins` (which implements the ``3dveins``
command) and `reveal` (which implements ``reveal``, ``unreveal``, and several
other commands).
Plugins can also register handlers to run on every tick, and can interface with
the built-in `enable` and `disable` commands. For the full plugin API, see the
skeleton plugins or ``PluginManager.cpp``.
Installed plugins live in the ``hack/plugins`` folder of a DFHack installation,
and the `load` family of commands can be used to load a recompiled plugin
without restarting DF.
See `plugins-index` for a list of all plugins included in DFHack.
Scripts
-------
DFHack scripts can currently be written in Lua or Ruby. The `Lua API <lua-api>`
is more complete and currently better-documented, however. Referring to existing
scripts as well as the API documentation can be helpful when developing new
scripts.
`Scripts included in DFHack <scripts-index>` live in a separate `scripts repository <https://github.com/dfhack/scripts>`_.
This can be found in the ``scripts`` submodule if you have
`cloned DFHack <compile-how-to-get-the-code>`, or the ``hack/scripts`` folder
of an installed copy of DFHack.
Core
----
The `DFHack core <dfhack-core>` has a variety of low-level functions. It is
responsible for hooking into DF (via SDL), providing a console, and providing an
interface for plugins and scripts to interact with DF.
Modules
-------
A lot of shared code to interact with DF in more complicated ways is contained
in **modules**. For example, the Units module contains functions for checking
various traits of units, changing nicknames properly, and more. Generally, code
that is useful to multiple plugins and scripts should go in the appropriate
module, if there is one.
Several modules are also `exposed to Lua <lua-cpp-func-wrappers>`, although
some functions (and some entire modules) are currently only available in C++.
Remote access interface
-----------------------
DFHack supports remote access by exchanging Google protobuf messages via a TCP
socket. Both the core and plugins can define remotely accessible methods. The
``dfhack-run`` command uses this interface to invoke ordinary console commands.
Currently the supported set of requests is limited, because the developers don't
know what exactly is most useful. `remotefortressreader` provides a fairly
comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`.

@ -1,8 +1,8 @@
.. _documentation:
####################
DFHack Documentation
####################
###########################
DFHack Documentation System
###########################
DFHack documentation, like the file you are reading now, is created as ``.rst`` files,
@ -25,24 +25,122 @@ compile the documentation in order to read it. Like Markdown, reST documents are
designed to be just as readable in a plain-text editor as they are in HTML format.
The main thing you lose in plain text format is hyperlinking.)
.. contents::
.. contents:: Contents
:local:
.. _docs-standards:
Documentation standards
=======================
Whether you're adding new code or just fixing old documentation (and there's plenty),
there are a few important standards for completeness and consistent style. Treat
this section as a guide rather than iron law, match the surrounding text, and you'll
be fine.
Each command should have a short (~54 character) help string, which is shown
by the `ls` command. For scripts, this is a comment on the first line
(the comment marker and whitespace is stripped). For plugins it's the second
argument to ``PluginCommand``. Please make this brief but descriptive!
Everything should be documented! If it's not clear *where* a particular
thing should be documented, ask on IRC or in the DFHack thread on Bay12 -
as well as getting help, you'll be providing valuable feedback that
makes it easier for future readers!
Scripts can use a custom autodoc function, based on the Sphinx ``include``
directive - anything between the tokens is copied into the appropriate scripts
documentation page. For Ruby, we follow the built-in docstring convention
(``=begin`` and ``=end``). For Lua, the tokens are ``[====[`` and ``]====]``
- ordinary multi-line strings. It is highly encouraged to reuse this string
as the in-console documentation by (eg.) printing it when a ``-help`` argument
is given.
The docs **must** have a heading which exactly matches the command, underlined
with ``=====`` to the same length. For example, a lua file would have::
local helpstr = [====[
add-thought
===========
Adds a thought or emotion to the selected unit. Can be used by other scripts,
or the gui invoked by running ``add-thought gui`` with a unit selected.
]====]
Where the heading for a section is also the name of a command, the spelling
and case should exactly match the command to enter in the DFHack command line.
Try to keep lines within 80-100 characters, so it's readable in plain text
in the terminal - Sphinx (our documentation system) will make sure
paragraphs flow.
If there aren't many options or examples to show, they can go in a paragraph of
text. Use double-backticks to put commands in monospaced font, like this::
You can use ``cleanowned scattered x`` to dump tattered or abandoned items.
If the command takes more than three arguments, format the list as a table
called Usage. The table *only* lists arguments, not full commands.
Input values are specified in angle brackets. Example::
Usage:
:arg1: A simple argument.
:arg2 <input>: Does something based on the input value.
:Very long argument:
Is very specific.
To demonstrate usage - useful mainly when the syntax is complicated, list the
full command with arguments in monospaced font, then indent the next line and
describe the effect::
``resume all``
Resumes all suspended constructions.
If it would be helpful to mention another DFHack command, don't just type the
name - add a hyperlink! Specify the link target in backticks, and it will be
replaced with the corresponding title and linked: eg ```autolabor```
=> `autolabor`. Link targets should be equivalent to the command
described (without file extension), and placed above the heading of that
section like this::
.. _autolabor:
autolabor
=========
Add link targets if you need them, but otherwise plain headings are preferred.
Scripts have link targets created automatically.
Building the documentation
==========================
Required dependencies
=====================
---------------------
In order to build the documentation, you must have Python with Sphinx
version 1.3.1 or later. Both Python 2.x and 3.x are supported.
version 1.8 or later. Both Python 2.x and 3.x are supported.
When installing Sphinx from OS package managers, be aware that there is
another program called Sphinx, completely unrelated to documentation management.
Be sure you are installing the right Sphinx; it may be called ``python-sphinx``,
for example. To avoid doubt, ``pip`` can be used instead as detailed below.
Once you have installed Sphinx, ``sphinx-build --version`` should report the
version of Sphinx that you have installed. If this works, CMake should also be
able to find Sphinx.
For more detailed platform-specific instructions, see the sections below:
.. contents::
:local:
:backlinks: none
Linux
-----
~~~~~
Most Linux distributions will include Python as standard.
Check your package manager to see if Sphinx 1.3.1 or later is available,
@ -66,7 +164,7 @@ install, find ``sphinx-build`` and ensure its directory is in your local ``$PATH
macOS
-----
~~~~~
macOS has Python 2.7 installed by default, but it does not have the pip package manager.
You can install Homebrew's Python 3, which includes pip, and then install the
@ -87,7 +185,7 @@ If not, just installing sphinx-doc for macOS's system Python 2.7 is fine.
Windows
-------
~~~~~~~
Python for Windows can be downloaded `from python.org <https://www.python.org/downloads/>`_.
The latest version of Python 3 is recommended, as it includes pip already.
@ -105,13 +203,6 @@ Note that this may require opening a new (admin) command prompt if you just
installed pip from the same command prompt.
Building the documentation
==========================
Once you have installed Sphinx, ``sphinx-build --version`` should report the
version of Sphinx that you have installed. If this works, CMake should also be
able to find Sphinx.
Using CMake
-----------
@ -155,6 +246,7 @@ the root DFHack folder::
Sphinx has many options to enable clean builds, parallel builds, logging, and
more - run ``sphinx-build --help`` for details.
.. _build-changelog:
Building the changelogs

@ -1,16 +1,19 @@
:orphan:
.. _HISTORY:
.. _History:
########################
HISTORY - old changelogs
########################
#####################
Historical changelogs
#####################
This file is where old changelogs live, so the current `changelog`
in ``NEWS.rst`` doesn't get too long.
This file is where old changelogs live, so the `current changelog <changelog>`
doesn't get too long. Some of these changelogs are also formatted differently
from current changelogs and would be difficult for the current `changelog
generation system <build-changelog>` to handle.
.. contents::
:depth: 2
.. contents:: Contents
:local:
:depth: 1
DFHack 0.43.05-r3
=================

@ -20,7 +20,7 @@ enhancements by default, and more can be enabled. There are also many tools
You can even add third-party scripts and plugins to do almost anything!
For modders, DFHack makes many things possible. Custom reactions, new
interactions, magic creature abilities, and more can be set through `modtools`
interactions, magic creature abilities, and more can be set through `scripts-modtools`
and custom raws. Non-standard DFHack scripts and inits can be stored in the
raw directory, making raws or saves fully self-contained for distribution -
or for coexistence in a single DF install, even with incompatible components.
@ -30,7 +30,8 @@ allows easier development of new tools. As an open-source project under
`various copyleft licences <license>`, contributions are welcome.
.. contents::
.. contents:: Contents
:local:
.. _installing:

@ -24,8 +24,9 @@ implemented by Lua files located in :file:`hack/lua/*`
(:file:`library/lua/*` in the git repo).
.. contents::
:depth: 3
.. contents:: Contents
:local:
:depth: 2
=========================
@ -801,6 +802,8 @@ Random number generation
Dimension may be 1, 2 or 3 (default).
.. _lua-cpp-func-wrappers:
C++ function wrappers
=====================

@ -0,0 +1,149 @@
.. _memory-research:
###############
Memory research
###############
There are a variety of tools that can be used to analyze DF memory - some are
listed here. Note that some of these may be old and unmaintained. If you aren't
sure what tool would be best for your purposes, feel free to ask for advice (on
IRC, Bay12, etc.).
.. contents:: Contents
:local:
Cross-platform tools
====================
Ghidra
------
Ghidra is a cross-platform reverse-engineering framework (written in Java)
available at https://ghidra-sre.org. It supports analyzing both 32-bit and
64-bit executables for all supported DF platforms. There are some custom DFHack
Ghidra scripts available in the `df_misc`_ repo (look for ``.java`` files).
IDA Freeware 7.0
----------------
Available from `Hex-Rays <https://www.hex-rays.com/products/ida/support/download_freeware/>`_.
Supports analyzing both 32-bit and 64-bit executables for all supported DF platforms.
Some ``.idc`` scripts for IDA are available in the `df_misc`_ repo.
.. _df_misc: https://github.com/DFHack/df_misc
Hopper
------
Runs on macOS and some Linux distributions; available from https://www.hopperapp.com/.
`TWBT <https://github.com/mifki/df-twbt/blob/master/PATCHES.md>`_ uses this to produce some patches.
DFHack tools
------------
Plugins
~~~~~~~
There are a few development plugins useful for low-level memory research. They
are not built by default, but can be built by setting the ``BUILD_DEVEL``
`CMake option <compile-build-options>`. These include:
- ``check-structures-sanity``, which performs sanity checks on the given DF
object. Note that this will crash in several cases, some intentional, so using
this with `GDB <linux-gdb>` is recommended.
- ``memview``, which produces a hex dump of a given memory range. It also
highlights valid pointers, and can be configured to work with `sizecheck`
to auto-detect object sizes.
- ``vectors``, which can identify instances of ``std::vector`` in a given memory range.
Scripts
~~~~~~~
Several `development scripts <scripts-devel>` can be useful for memory research.
These include (but are not limited to):
- `devel/dump-offsets`
- `devel/find-offsets`
- `devel/lsmem`
- `devel/sc` (requires `sizecheck`)
- `devel/visualize-structure`
- Generally, any script starting with ``devel/find``
.. _sizecheck:
Sizecheck
~~~~~~~~~
Sizecheck is a custom tool that hooks into the memory allocator and inserts a
header indicating the size of every object. The corresponding logic to check for
this header when freeing memory usually works, but is inherently not foolproof.
You should not count on DF being stable when using this.
DFHack's implementation of sizecheck is currently only tested on Linux, although
it probably also works on macOS. It can be built with the ``BUILD_SIZECHECK``
`CMake option <compile-build-options>`, which produces a ``libsizecheck``
library installed in the ``hack`` folder. You will need to preload this library
manually, by setting ``PRELOAD_LIB`` on Linux (or ``LD_PRELOAD`` if editing
the ``dfhack`` launcher script directly), or by editing the ``dfhack``
launcher script and adding the library to ``DYLD_INSERT_LIBRARIES`` on macOS.
There is also an older sizecheck implementation by Mifki available on
`GitHub <https://github.com/mifki/df-sizecheck>`__ (``b.cpp`` is the main
sizecheck library, and ``win_patch.cpp`` is used for Windows support). To use
this with other DFHack tools, you will likely need to edit the header's
magic number to match what is used in `devel/sc` (search for a hexadecimal
constant starting with ``0x``).
Legacy tools
~~~~~~~~~~~~
Some very old DFHack tools are available in the `legacy branch on GitHub <https://github.com/dfhack/dfhack/tree/legacy/tools>`_.
No attempt is made to support these.
Linux-specific tools
====================
.. _linux-gdb:
GDB
---
`GDB <https://www.gnu.org/software/gdb/>`_ is technically cross-platform, but
tends to work best on Linux, and DFHack currently only offers support for using
GDB on 64-bit Linux. To start with GDB, pass ``-g`` to the DFHack launcher
script::
./dfhack -g
Some basic GDB commands:
- ``run``: starts DF from the GDB prompt. Any arguments will be passed as
command-line arguments to DF (e.g. `load-save` may be useful).
- ``bt`` will produce a backtrace if DF crashes.
See the `official GDB documentation <https://www.gnu.org/software/gdb/documentation/>`_
for more details.
df-structures GUI
-----------------
This is a tool written by Angavrilov and available on `GitHub <https://github.com/angavrilov/cl-linux-debug>`__.
It only supports 32-bit DF. Some assistance may be available on IRC.
EDB (Evan's debugger)
---------------------
Available on `GitHub <https://github.com/eteran/edb-debugger>`__.
Windows-specific tools
======================
Some people have used `Cheat Engine <https://www.cheatengine.org/>`__ for research in the past.

@ -13,7 +13,8 @@ which they first appeared. See `build-changelog` for more information.
See `changelog` for a list of changes grouped by stable releases.
.. contents::
:depth: 2
.. contents:: Contents
:local:
:depth: 1
.. include:: /docs/_auto/news-dev.rst

@ -13,14 +13,15 @@ appeared. See `build-changelog` for more information.
See `dev-changelog` for a list of changes grouped by development releases.
.. contents::
:depth: 2
.. contents:: Contents
:local:
:depth: 1
.. include:: /docs/_auto/news.rst
Older Changelogs
================
Are kept in a seperate file: `HISTORY`
Are kept in a seperate file: `History`
.. that's ``docs/history.rst``, if you're reading the raw text.
.. that's ``docs/History.rst``, if you're reading the raw text.

@ -1,3 +1,5 @@
.. _plugins-index:
##############
DFHack Plugins
##############
@ -9,8 +11,9 @@ game subsystems or the entire renderer.
Most commands offered by plugins are listed here,
hopefully organised in a way you will find useful.
.. contents::
:depth: 3
.. contents:: Contents
:local:
:depth: 2
===============================
Data inspection and visualizers

@ -0,0 +1,17 @@
#############
Removed tools
#############
This page lists tools (plugins or scripts) that were previously included in
DFHack but have been removed. It exists primarily so that internal links still
work (e.g. links from the `changelog`).
.. contents:: Contents
:local:
:depth: 1
.. _warn-stuck-trees:
warn-stuck-trees
================
The corresponding DF bug, :bug:`9252` was fixed in DF 0.44.01.

@ -1,14 +0,0 @@
###############
Removed scripts
###############
The following scripts were removed for various reasons.
.. contents::
:depth: 2
.. _warn-stuck-trees:
warn-stuck-trees
================
The corresponding DF bug, :bug:`9252` was fixed in DF 0.44.01.

@ -1,3 +1,5 @@
.. _scripts-index:
##############
DFHack Scripts
##############

@ -0,0 +1,34 @@
===================================
DF data definitions (DF-structures)
===================================
DFHack's information about DF's data structures is stored in XML files in the
`df-structures repository <https://github.com/DFHack/df-structures>`_. If you
have `obtained a local copy of the DFHack source <compile-how-to-get-the-code>`,
DF-structures is included as a submodule in ``library/xml``.
Data structure layouts are described in files named with the ``df.*.xml``
pattern. This information is transformed by a Perl script (``codegen.pl``) into
C++ headers, as well as metadata for the Lua wrapper. This ultimately allows
DFHack code to access DF data directly, with the same speed and capabilities as
DF itself, which is an advantage over the older out-of-process approach (used
by debuggers and utilities like Dwarf Therapist). The main disadvantage of this
approach is that any compiled code relying on these layouts will break when DF's
layout changes, and will need to be recompiled for every new DF version.
Addresses of DF global objects and vtables are stored in a separate file,
:file:`symbols.xml`. Since these are only absolute addresses, they do not need
to be compiled in to DFHack code, and are instead loaded at runtime. This makes
fixes and additions to global addresses possible without recompiling DFHack.
In an installed copy of DFHack, this file can be found at the root of the
``hack`` folder.
The following pages contain more detailed information about various aspects
of DF-structures:
.. toctree::
:titlesonly:
/library/xml/SYNTAX
/library/xml/how-to-update

@ -1,285 +1,5 @@
import collections
import copy
import itertools
import os
import sys
import os, sys
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'sphinx_extensions'))
CHANGELOG_PATHS = (
'docs/changelog.txt',
'scripts/changelog.txt',
'library/xml/changelog.txt',
)
CHANGELOG_SECTIONS = [
'New Plugins',
'New Scripts',
'New Tweaks',
'New Features',
'New Internal Commands',
'Fixes',
'Misc Improvements',
'Removed',
'API',
'Internals',
'Lua',
'Ruby',
'Structures',
]
REPLACEMENTS = {
'`search`': '`search-plugin`',
}
def to_title_case(word):
if word == word.upper():
# Preserve acronyms
return word
return word[0].upper() + word[1:].lower()
def find_all_indices(string, substr):
start = 0
while True:
i = string.find(substr, start)
if i == -1:
return
yield i
start = i + 1
def replace_text(string, replacements):
for old_text, new_text in replacements.items():
new_string = ''
new_string_end = 0 # number of characters from string in new_string
for i in find_all_indices(string, old_text):
if i > 0 and string[i - 1] == '!':
# exempt if preceded by '!'
new_string += string[new_string_end:i - 1]
new_string += old_text
else:
# copy until this occurrence
new_string += string[new_string_end:i]
new_string += new_text
new_string_end = i + len(old_text)
new_string += string[new_string_end:]
string = new_string
return string
class ChangelogEntry(object):
def __init__(self, text, section, stable_version, dev_version):
text = text.lstrip('- ')
# normalize section to title case
self.section = ' '.join(map(to_title_case, section.strip().split()))
self.stable_version = stable_version
self.dev_version = dev_version
self.dev_only = text.startswith('@')
text = text.lstrip('@ ')
self.children = []
split_index = text.find(': ')
if split_index != -1:
self.feature, description = text[:split_index], text[split_index+1:]
if description.strip():
self.children.insert(0, description.strip())
else:
self.feature = text
self.feature = self.feature.replace(':\\', ':').rstrip(':')
self.sort_key = self.feature.upper()
def __repr__(self):
return 'ChangelogEntry(%r, %r)' % (self.feature, self.children)
def parse_changelog():
entries = []
for fpath in CHANGELOG_PATHS:
if not os.path.isfile(fpath):
continue
with open(fpath) as f:
cur_stable = None
cur_dev = None
cur_section = None
last_entry = None
multiline = ''
for line_id, line in enumerate(f.readlines()):
line_id += 1
if multiline:
multiline += line
elif '[[[' in line:
multiline = line.replace('[[[', '')
if ']]]' in multiline:
line = multiline.replace(']]]', '')
multiline = ''
elif multiline:
continue
if not line.strip() or line.startswith('==='):
continue
if line.startswith('##'):
cur_section = line.lstrip('#').strip()
elif line.startswith('#'):
cur_dev = line.lstrip('#').strip().lower()
if ('alpha' not in cur_dev and 'beta' not in cur_dev and
'rc' not in cur_dev):
cur_stable = cur_dev
elif line.startswith('-'):
if not cur_stable or not cur_dev or not cur_section:
raise ValueError(
'%s:%i: Entry without section' % (fpath, line_id))
last_entry = ChangelogEntry(line.strip(), cur_section,
cur_stable, cur_dev)
entries.append(last_entry)
elif line.lstrip().startswith('-'):
if not cur_stable or not cur_dev:
raise ValueError(
'%s:%i: Sub-entry without section' % (fpath, line_id))
if not last_entry:
raise ValueError(
'%s:%i: Sub-entry without parent' % (fpath, line_id))
last_entry.children.append(line.strip('- \n'))
else:
raise ValueError('Invalid line: ' + line)
if not entries:
raise RuntimeError('No changelog files with contents found')
return entries
def consolidate_changelog(all_entries):
for sections in all_entries.values():
for section, entries in sections.items():
# sort() is stable, so reverse entries so that older entries for the
# same feature are on top
entries.reverse()
entries.sort(key=lambda entry: entry.sort_key)
new_entries = []
for feature, group in itertools.groupby(entries,
lambda e: e.feature):
old_entries = list(group)
children = list(itertools.chain(*[entry.children
for entry in old_entries]))
new_entry = copy.deepcopy(old_entries[0])
new_entry.children = children
new_entries.append(new_entry)
entries[:] = new_entries
def print_changelog(versions, all_entries, path, replace=True, prefix=''):
# all_entries: version -> section -> entry
with open(path, 'w') as f:
def write(line):
if replace:
line = replace_text(line, REPLACEMENTS)
f.write(prefix + line + '\n')
for version in versions:
sections = all_entries[version]
if not sections:
continue
version = 'DFHack ' + version
write(version)
write('=' * len(version))
write('')
for section in CHANGELOG_SECTIONS:
entries = sections[section]
if not entries:
continue
write(section)
write('-' * len(section))
for entry in entries:
if len(entry.children) == 1:
write('- ' + entry.feature + ': ' +
entry.children[0].strip('- '))
continue
elif entry.children:
write('- ' + entry.feature + ':')
write('')
for child in entry.children:
write(' - ' + child)
write('')
else:
write('- ' + entry.feature)
write('')
write('')
def generate_changelog(all=False):
entries = parse_changelog()
# scan for unrecognized sections
for entry in entries:
if entry.section not in CHANGELOG_SECTIONS:
raise RuntimeWarning('Unknown section: ' + entry.section)
# ordered versions
versions = ['future']
# map versions to stable versions
stable_version_map = {}
# version -> section -> entry
stable_entries = collections.defaultdict(lambda:
collections.defaultdict(list))
dev_entries = collections.defaultdict(lambda:
collections.defaultdict(list))
for entry in entries:
# build list of all versions
if entry.dev_version not in versions:
versions.append(entry.dev_version)
stable_version_map.setdefault(entry.dev_version, entry.stable_version)
if not entry.dev_only:
# add non-dev-only entries to both changelogs
stable_entries[entry.stable_version][entry.section].append(entry)
dev_entries[entry.dev_version][entry.section].append(entry)
consolidate_changelog(stable_entries)
consolidate_changelog(dev_entries)
print_changelog(versions, stable_entries, 'docs/_auto/news.rst')
print_changelog(versions, dev_entries, 'docs/_auto/news-dev.rst')
if all:
for version in versions:
if version not in stable_version_map:
print('warn: skipping ' + version)
continue
if stable_version_map[version] == version:
version_entries = {version: stable_entries[version]}
else:
version_entries = {version: dev_entries[version]}
print_changelog([version], version_entries,
'docs/_changelogs/%s-github.txt' % version,
replace=False)
print_changelog([version], version_entries,
'docs/_changelogs/%s-reddit.txt' % version,
replace=False,
prefix='> ')
return entries
if __name__ == '__main__':
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', '--all', action='store_true',
help='Print changelogs for all versions to docs/_changelogs')
parser.add_argument('-c', '--check', action='store_true',
help='Check that all entries are printed')
args = parser.parse_args()
os.chdir(os.path.abspath(os.path.dirname(__file__)))
os.chdir('..')
entries = generate_changelog(all=args.all)
if args.check:
with open('docs/_auto/news.rst') as f:
content_stable = f.read()
with open('docs/_auto/news-dev.rst') as f:
content_dev = f.read()
for entry in entries:
for description in entry.children:
if not entry.dev_only and description not in content_stable:
print('stable missing: ' + description)
if description not in content_dev:
print('dev missing: ' + description)
from dfhack.changelog import cli_entrypoint
cli_entrypoint()

@ -0,0 +1,19 @@
========================
DFHack Development Guide
========================
These are pages relevant to people developing for DFHack.
.. toctree::
:maxdepth: 1
/docs/Dev-intro
/docs/Contributing
/docs/Compile
/docs/NEWS-dev
/docs/Lua API
/docs/Documentation
/docs/Structures-intro
/docs/Memory-research
/docs/Binpatches

@ -0,0 +1,300 @@
import collections
import copy
import itertools
import os
import sys
from dfhack.util import DFHACK_ROOT, DOCS_ROOT
CHANGELOG_PATHS = (
'docs/changelog.txt',
'scripts/changelog.txt',
'library/xml/changelog.txt',
)
CHANGELOG_PATHS = (os.path.join(DFHACK_ROOT, p) for p in CHANGELOG_PATHS)
CHANGELOG_SECTIONS = [
'New Plugins',
'New Scripts',
'New Tweaks',
'New Features',
'New Internal Commands',
'Fixes',
'Misc Improvements',
'Removed',
'API',
'Internals',
'Lua',
'Ruby',
'Structures',
]
REPLACEMENTS = {
'`search`': '`search-plugin`',
}
def to_title_case(word):
if word == word.upper():
# Preserve acronyms
return word
return word[0].upper() + word[1:].lower()
def find_all_indices(string, substr):
start = 0
while True:
i = string.find(substr, start)
if i == -1:
return
yield i
start = i + 1
def replace_text(string, replacements):
for old_text, new_text in replacements.items():
new_string = ''
new_string_end = 0 # number of characters from string in new_string
for i in find_all_indices(string, old_text):
if i > 0 and string[i - 1] == '!':
# exempt if preceded by '!'
new_string += string[new_string_end:i - 1]
new_string += old_text
else:
# copy until this occurrence
new_string += string[new_string_end:i]
new_string += new_text
new_string_end = i + len(old_text)
new_string += string[new_string_end:]
string = new_string
return string
class ChangelogEntry(object):
def __init__(self, text, section, stable_version, dev_version):
text = text.lstrip('- ')
# normalize section to title case
self.section = ' '.join(map(to_title_case, section.strip().split()))
self.stable_version = stable_version
self.dev_version = dev_version
self.dev_only = text.startswith('@')
text = text.lstrip('@ ')
self.children = []
split_index = text.find(': ')
if split_index != -1:
self.feature, description = text[:split_index], text[split_index+1:]
if description.strip():
self.children.insert(0, description.strip())
else:
self.feature = text
self.feature = self.feature.replace(':\\', ':').rstrip(':')
self.sort_key = self.feature.upper()
def __repr__(self):
return 'ChangelogEntry(%r, %r)' % (self.feature, self.children)
def parse_changelog():
entries = []
for fpath in CHANGELOG_PATHS:
if not os.path.isfile(fpath):
continue
with open(fpath) as f:
cur_stable = None
cur_dev = None
cur_section = None
last_entry = None
multiline = ''
for line_id, line in enumerate(f.readlines()):
line_id += 1
if multiline:
multiline += line
elif '[[[' in line:
multiline = line.replace('[[[', '')
if ']]]' in multiline:
line = multiline.replace(']]]', '')
multiline = ''
elif multiline:
continue
if not line.strip() or line.startswith('==='):
continue
if line.startswith('##'):
cur_section = line.lstrip('#').strip()
elif line.startswith('#'):
cur_dev = line.lstrip('#').strip().lower()
if ('alpha' not in cur_dev and 'beta' not in cur_dev and
'rc' not in cur_dev):
cur_stable = cur_dev
elif line.startswith('-'):
if not cur_stable or not cur_dev or not cur_section:
raise ValueError(
'%s:%i: Entry without section' % (fpath, line_id))
last_entry = ChangelogEntry(line.strip(), cur_section,
cur_stable, cur_dev)
entries.append(last_entry)
elif line.lstrip().startswith('-'):
if not cur_stable or not cur_dev:
raise ValueError(
'%s:%i: Sub-entry without section' % (fpath, line_id))
if not last_entry:
raise ValueError(
'%s:%i: Sub-entry without parent' % (fpath, line_id))
last_entry.children.append(line.strip('- \n'))
else:
raise ValueError('Invalid line: ' + line)
if not entries:
raise RuntimeError('No changelog files with contents found')
return entries
def consolidate_changelog(all_entries):
for sections in all_entries.values():
for section, entries in sections.items():
# sort() is stable, so reverse entries so that older entries for the
# same feature are on top
entries.reverse()
entries.sort(key=lambda entry: entry.sort_key)
new_entries = []
for feature, group in itertools.groupby(entries,
lambda e: e.feature):
old_entries = list(group)
children = list(itertools.chain(*[entry.children
for entry in old_entries]))
new_entry = copy.deepcopy(old_entries[0])
new_entry.children = children
new_entries.append(new_entry)
entries[:] = new_entries
def print_changelog(versions, all_entries, path, replace=True, prefix=''):
# all_entries: version -> section -> entry
with open(path, 'w') as f:
def write(line):
if replace:
line = replace_text(line, REPLACEMENTS)
f.write(prefix + line + '\n')
for version in versions:
sections = all_entries[version]
if not sections:
continue
version = 'DFHack ' + version
write(version)
write('=' * len(version))
write('')
for section in CHANGELOG_SECTIONS:
entries = sections[section]
if not entries:
continue
write(section)
write('-' * len(section))
for entry in entries:
if len(entry.children) == 1:
write('- ' + entry.feature + ': ' +
entry.children[0].strip('- '))
continue
elif entry.children:
write('- ' + entry.feature + ':')
write('')
for child in entry.children:
write(' - ' + child)
write('')
else:
write('- ' + entry.feature)
write('')
write('')
def generate_changelog(all=False):
entries = parse_changelog()
# scan for unrecognized sections
for entry in entries:
if entry.section not in CHANGELOG_SECTIONS:
raise RuntimeWarning('Unknown section: ' + entry.section)
# ordered versions
versions = ['future']
# map versions to stable versions
stable_version_map = {}
# version -> section -> entry
stable_entries = collections.defaultdict(lambda:
collections.defaultdict(list))
dev_entries = collections.defaultdict(lambda:
collections.defaultdict(list))
for entry in entries:
# build list of all versions
if entry.dev_version not in versions:
versions.append(entry.dev_version)
stable_version_map.setdefault(entry.dev_version, entry.stable_version)
if not entry.dev_only:
# add non-dev-only entries to both changelogs
stable_entries[entry.stable_version][entry.section].append(entry)
dev_entries[entry.dev_version][entry.section].append(entry)
consolidate_changelog(stable_entries)
consolidate_changelog(dev_entries)
print_changelog(versions, stable_entries, os.path.join(DOCS_ROOT, '_auto/news.rst'))
print_changelog(versions, dev_entries, os.path.join(DOCS_ROOT, '_auto/news-dev.rst'))
if all:
for version in versions:
if version not in stable_version_map:
print('warn: skipping ' + version)
continue
if stable_version_map[version] == version:
version_entries = {version: stable_entries[version]}
else:
version_entries = {version: dev_entries[version]}
print_changelog([version], version_entries,
os.path.join(DOCS_ROOT, '_changelogs/%s-github.txt' % version),
replace=False)
print_changelog([version], version_entries,
os.path.join(DOCS_ROOT, '_changelogs/%s-reddit.txt' % version),
replace=False,
prefix='> ')
return entries
def cli_entrypoint():
import argparse
parser = argparse.ArgumentParser()
parser.add_argument('-a', '--all', action='store_true',
help='Print changelogs for all versions to docs/_changelogs')
parser.add_argument('-c', '--check', action='store_true',
help='Check that all entries are printed')
args = parser.parse_args()
entries = generate_changelog(all=args.all)
if args.check:
with open(os.path.join(DOCS_ROOT, '_auto/news.rst')) as f:
content_stable = f.read()
with open(os.path.join(DOCS_ROOT, '_auto/news-dev.rst')) as f:
content_dev = f.read()
for entry in entries:
for description in entry.children:
if not entry.dev_only and description not in content_stable:
print('stable missing: ' + description)
if description not in content_dev:
print('dev missing: ' + description)
def sphinx_entrypoint(app, config):
generate_changelog()
def setup(app):
app.connect('config-inited', sphinx_entrypoint)
return {
'version': '0.1',
'parallel_read_safe': True,
'parallel_write_safe': True,
}

@ -0,0 +1,7 @@
import os
DFHACK_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
DOCS_ROOT = os.path.join(DFHACK_ROOT, 'docs')
if not os.path.isdir(DOCS_ROOT):
raise ReferenceError('docs root not found: %s' % DOCS_ROOT)

@ -20,7 +20,7 @@ enhancements by default, and more can be enabled. There are also many tools
You can even add third-party scripts and plugins to do almost anything!
For modders, DFHack makes many things possible. Custom reactions, new
interactions, magic creature abilities, and more can be set through `modtools`
interactions, magic creature abilities, and more can be set through `scripts-modtools`
and custom raws. Non-standard DFHack scripts and inits can be stored in the
raw directory, making raws or saves fully self-contained for distribution -
or for coexistence in a single DF install, even with incompatible components.
@ -40,6 +40,7 @@ User Manual
/docs/Core
/docs/Plugins
/docs/Scripts
/docs/index-dev
Other Contents
==============
@ -50,19 +51,4 @@ Other Contents
/docs/Authors
/LICENSE
/docs/NEWS
/docs/Scripts-removed
For Developers
==============
.. toctree::
:maxdepth: 1
/Contributing
/docs/Compile
/docs/NEWS-dev
/docs/Lua API
/docs/Documentation
/library/xml/SYNTAX
/library/xml/how-to-update
/docs/Binpatches
/docs/Removed

@ -1 +1 @@
Subproject commit 2d67b03a1c5c701e65fb483b9d6c1d3067f990cb
Subproject commit 847ec0c2d4dc74a887e1f980eeed6f97345c195e