Merge remote-tracking branch 'lethosor/docs-reorg' into develop
lethosor
commit
cd46c30914
@ -0,0 +1,6 @@
|
|||||||
|
Thank you for your interest in contributing to DFHack! If you're reading this
|
||||||
|
document, you're probably viewing it on GitHub. The DFHack docs are hosted
|
||||||
|
on [ReadTheDocs](https://dfhack.readthedocs.io/) - in particular, contributing
|
||||||
|
guidelines are [here](https://docs.dfhack.org/en/latest/docs/Contributing.html).
|
||||||
|
Double-checking the style guidelines before submitting a pull request is
|
||||||
|
always appreciated.
|
||||||
@ -1,229 +0,0 @@
|
|||||||
###########################
|
|
||||||
How to contribute to DFHack
|
|
||||||
###########################
|
|
||||||
|
|
||||||
.. contents::
|
|
||||||
|
|
||||||
|
|
||||||
.. _contributing-code:
|
|
||||||
|
|
||||||
Contributing Code
|
|
||||||
=================
|
|
||||||
Several things should be kept in mind when contributing code to DFHack.
|
|
||||||
|
|
||||||
Code Format
|
|
||||||
-----------
|
|
||||||
* Four space indents for C++. Never use tabs for indentation in any language.
|
|
||||||
* LF (Unix style) line terminators
|
|
||||||
* Avoid trailing whitespace
|
|
||||||
* UTF-8 encoding
|
|
||||||
* For C++:
|
|
||||||
|
|
||||||
* Opening and closing braces on their own lines or opening brace at the end of the previous line
|
|
||||||
* Braces placed at original indent level if on their own lines
|
|
||||||
* #includes should be sorted. C++ libraries first, then dfhack modules, then df structures,
|
|
||||||
then local includes. Within each category they should be sorted alphabetically.
|
|
||||||
|
|
||||||
How to get new code into DFHack
|
|
||||||
-------------------------------
|
|
||||||
* Submit pull requests to the ``develop`` branch, not the ``master`` branch.
|
|
||||||
(The ``master`` branch always points at the most recent release)
|
|
||||||
* Use a new branch for each feature or bugfix so that your changes can be merged independently
|
|
||||||
(i.e. not the master or develop branch of your fork).
|
|
||||||
* If possible, compile on multiple platforms when changing anything that compiles
|
|
||||||
* It must pass CI - run ``python travis/all.py`` to check this.
|
|
||||||
* Update ``changelog.txt`` and ``docs/Authors.rst`` when applicable. See
|
|
||||||
`build-changelog` for more information on the changelog format.
|
|
||||||
* Create a GitHub pull request once finished
|
|
||||||
* Submit ideas and bug reports as :issue:`issues on GitHub <>`.
|
|
||||||
Posts in the forum thread can easily get missed or forgotten.
|
|
||||||
* Work on :issue:`reported problems <?q=is:open+-label:idea>`
|
|
||||||
will take priority over ideas or suggestions.
|
|
||||||
|
|
||||||
.. _contributing-memory-research:
|
|
||||||
|
|
||||||
Memory research
|
|
||||||
---------------
|
|
||||||
If you want to do memory research, you'll need some tools and some knowledge.
|
|
||||||
In general, you'll need a good memory viewer and optionally something
|
|
||||||
to look at machine code without getting crazy :)
|
|
||||||
Using publicly known information and analyzing the game's data is preferred.
|
|
||||||
|
|
||||||
Good Windows tools include:
|
|
||||||
|
|
||||||
* IDA Freeware 7.0 (for non-commercial use, supports 32-bit and 64-bit)
|
|
||||||
* Cheat Engine
|
|
||||||
|
|
||||||
Good Linux tools:
|
|
||||||
|
|
||||||
* angavrilov's df-structures gui (32-bit only, visit us on IRC for details)
|
|
||||||
* IDA Freeware 7.0 (see above)
|
|
||||||
* edb (Evan's Debugger)
|
|
||||||
* Some of the tools residing in the ``legacy`` dfhack branch.
|
|
||||||
|
|
||||||
Using the library as a developer
|
|
||||||
================================
|
|
||||||
Currently, the most direct way to use the library is to write a script or plugin that can be loaded by it.
|
|
||||||
All the plugins can be found in the 'plugins' folder. There's no in-depth documentation
|
|
||||||
on how to write one yet, but it should be easy enough to copy one and just follow the pattern.
|
|
||||||
``plugins/skeleton/skeleton.cpp`` is provided for this purpose.
|
|
||||||
|
|
||||||
Other than through plugins, it is possible to use DFHack via remote access interface,
|
|
||||||
or by writing scripts in Lua or Ruby. There are plenty of examples in the scripts folder.
|
|
||||||
The `lua-api` is quite well documented.
|
|
||||||
|
|
||||||
The most important parts of DFHack are the Core, Console, Modules and Plugins.
|
|
||||||
|
|
||||||
* Core acts as the centerpiece of DFHack - it acts as a filter between DF and
|
|
||||||
SDL and synchronizes the various plugins with DF.
|
|
||||||
* Console is a thread-safe console that can be used to invoke commands exported by Plugins.
|
|
||||||
* Modules actually describe the way to access information in DF's memory. You
|
|
||||||
can get them from the Core. Most modules are split into two parts: high-level
|
|
||||||
and low-level. High-level is mostly method calls, low-level publicly visible
|
|
||||||
pointers to DF's data structures.
|
|
||||||
* Plugins are the tools that use all the other stuff to make things happen.
|
|
||||||
A plugin can have a list of commands that it exports and an onupdate function
|
|
||||||
that will be called each DF game tick.
|
|
||||||
|
|
||||||
Rudimentary API documentation can be built using doxygen (see build options
|
|
||||||
in ``CMakeCache.txt`` or with ``ccmake`` or ``cmake-gui``). The full DFHack
|
|
||||||
documentation is built with Sphinx_, which runs automatically at compile time.
|
|
||||||
|
|
||||||
.. _Sphinx: http://www.sphinx-doc.org
|
|
||||||
|
|
||||||
DFHack consists of variously licensed code, but invariably weak copyleft.
|
|
||||||
The main license is zlib/libpng, some bits are MIT licensed, and some are
|
|
||||||
BSD licensed. See the `license` for more information.
|
|
||||||
|
|
||||||
Feel free to add your own extensions and plugins. Contributing back to
|
|
||||||
the DFHack repository is welcome and the right thing to do :)
|
|
||||||
|
|
||||||
DF data structure definitions
|
|
||||||
-----------------------------
|
|
||||||
DFHack uses information about the game data structures, represented via xml files
|
|
||||||
in the ``library/xml/`` submodule.
|
|
||||||
|
|
||||||
See https://github.com/DFHack/df-structures, and the documentation linked in the index.
|
|
||||||
|
|
||||||
Data structure layouts are described in files following the ``df.*.xml`` name pattern.
|
|
||||||
This information is transformed by a perl script into C++ headers describing the
|
|
||||||
structures, and associated metadata for the Lua wrapper. These headers and data
|
|
||||||
are then compiled into the DFHack libraries, thus necessitating a compatibility
|
|
||||||
break every time layouts change; in return it significantly boosts the efficiency
|
|
||||||
and capabilities of DFHack code.
|
|
||||||
|
|
||||||
Global object addresses are stored in :file:`symbols.xml`, which is copied to the dfhack
|
|
||||||
release package and loaded as data at runtime.
|
|
||||||
|
|
||||||
Remote access interface
|
|
||||||
-----------------------
|
|
||||||
DFHack supports remote access by exchanging Google protobuf messages via a TCP
|
|
||||||
socket. Both the core and plugins can define remotely accessible methods. The
|
|
||||||
``dfhack-run`` command uses this interface to invoke ordinary console commands.
|
|
||||||
|
|
||||||
Currently the supported set of requests is limited, because the developers don't
|
|
||||||
know what exactly is most useful. `remotefortressreader` provides a fairly
|
|
||||||
comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`.
|
|
||||||
|
|
||||||
|
|
||||||
Documentation Standards
|
|
||||||
=======================
|
|
||||||
DFHack documentation is built with Sphinx_, and configured automatically
|
|
||||||
through CMake. If you want to build the docs *only*, use this command::
|
|
||||||
|
|
||||||
sphinx-build . docs/html
|
|
||||||
|
|
||||||
Whether you're adding new code or just fixing old documentation (and there's plenty),
|
|
||||||
there are a few important standards for completeness and consistent style. Treat
|
|
||||||
this section as a guide rather than iron law, match the surrounding text, and you'll
|
|
||||||
be fine.
|
|
||||||
|
|
||||||
Each command should have a short (~54 character) help string, which is shown
|
|
||||||
by the `ls` command. For scripts, this is a comment on the first line
|
|
||||||
(the comment marker and whitespace is stripped). For plugins it's the second
|
|
||||||
argument to ``PluginCommand``. Please make this brief but descriptive!
|
|
||||||
|
|
||||||
Everything should be documented! If it's not clear *where* a particular
|
|
||||||
thing should be documented, ask on IRC or in the DFHack thread on Bay12 -
|
|
||||||
as well as getting help, you'll be providing valuable feedback that
|
|
||||||
makes it easier for future readers!
|
|
||||||
|
|
||||||
Scripts can use a custom autodoc function, based on the Sphinx ``include``
|
|
||||||
directive - anything between the tokens is copied into the appropriate scripts
|
|
||||||
documentation page. For Ruby, we follow the built-in docstring convention
|
|
||||||
(``=begin`` and ``=end``). For Lua, the tokens are ``[====[`` and ``]====]``
|
|
||||||
- ordinary multi-line strings. It is highly encouraged to reuse this string
|
|
||||||
as the in-console documentation by (eg.) printing it when a ``-help`` argument
|
|
||||||
is given.
|
|
||||||
|
|
||||||
The docs **must** have a heading which exactly matches the command, underlined
|
|
||||||
with ``=====`` to the same length. For example, a lua file would have::
|
|
||||||
|
|
||||||
local helpstr = [====[
|
|
||||||
|
|
||||||
add-thought
|
|
||||||
===========
|
|
||||||
Adds a thought or emotion to the selected unit. Can be used by other scripts,
|
|
||||||
or the gui invoked by running ``add-thought gui`` with a unit selected.
|
|
||||||
|
|
||||||
]====]
|
|
||||||
|
|
||||||
|
|
||||||
Where the heading for a section is also the name of a command, the spelling
|
|
||||||
and case should exactly match the command to enter in the DFHack command line.
|
|
||||||
|
|
||||||
Try to keep lines within 80-100 characters, so it's readable in plain text
|
|
||||||
in the terminal - Sphinx (our documentation system) will make sure
|
|
||||||
paragraphs flow.
|
|
||||||
|
|
||||||
If there aren't many options or examples to show, they can go in a paragraph of
|
|
||||||
text. Use double-backticks to put commands in monospaced font, like this::
|
|
||||||
|
|
||||||
You can use ``cleanowned scattered x`` to dump tattered or abandoned items.
|
|
||||||
|
|
||||||
If the command takes more than three arguments, format the list as a table
|
|
||||||
called Usage. The table *only* lists arguments, not full commands.
|
|
||||||
Input values are specified in angle brackets. Example::
|
|
||||||
|
|
||||||
Usage:
|
|
||||||
|
|
||||||
:arg1: A simple argument.
|
|
||||||
:arg2 <input>: Does something based on the input value.
|
|
||||||
:Very long argument:
|
|
||||||
Is very specific.
|
|
||||||
|
|
||||||
To demonstrate usage - useful mainly when the syntax is complicated, list the
|
|
||||||
full command with arguments in monospaced font, then indent the next line and
|
|
||||||
describe the effect::
|
|
||||||
|
|
||||||
``resume all``
|
|
||||||
Resumes all suspended constructions.
|
|
||||||
|
|
||||||
If it would be helpful to mention another DFHack command, don't just type the
|
|
||||||
name - add a hyperlink! Specify the link target in backticks, and it will be
|
|
||||||
replaced with the corresponding title and linked: eg ```autolabor```
|
|
||||||
=> `autolabor`. Link targets should be equivalent to the command
|
|
||||||
described (without file extension), and placed above the heading of that
|
|
||||||
section like this::
|
|
||||||
|
|
||||||
.. _autolabor:
|
|
||||||
|
|
||||||
autolabor
|
|
||||||
=========
|
|
||||||
|
|
||||||
Add link targets if you need them, but otherwise plain headings are preferred.
|
|
||||||
Scripts have link targets created automatically.
|
|
||||||
|
|
||||||
Other ways to help
|
|
||||||
==================
|
|
||||||
DFHack is a software project, but there's a lot more to it than programming.
|
|
||||||
If you're not comfortable programming, you can help by:
|
|
||||||
|
|
||||||
* reporting bugs and incomplete documentation
|
|
||||||
* improving the documentation
|
|
||||||
* finding third-party scripts to add
|
|
||||||
* writing tutorials for newbies
|
|
||||||
|
|
||||||
All those things are crucial, and often under-represented. So if that's
|
|
||||||
your thing, go get started!
|
|
||||||
|
|
||||||
@ -0,0 +1,59 @@
|
|||||||
|
.. _contributing:
|
||||||
|
|
||||||
|
###########################
|
||||||
|
How to contribute to DFHack
|
||||||
|
###########################
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
|
||||||
|
.. _contributing-code:
|
||||||
|
|
||||||
|
Contributing Code
|
||||||
|
=================
|
||||||
|
Several things should be kept in mind when contributing code to DFHack.
|
||||||
|
|
||||||
|
Code Format
|
||||||
|
-----------
|
||||||
|
* Four space indents for C++. Never use tabs for indentation in any language.
|
||||||
|
* LF (Unix style) line terminators
|
||||||
|
* Avoid trailing whitespace
|
||||||
|
* UTF-8 encoding
|
||||||
|
* For C++:
|
||||||
|
|
||||||
|
* Opening and closing braces on their own lines or opening brace at the end of the previous line
|
||||||
|
* Braces placed at original indent level if on their own lines
|
||||||
|
* #includes should be sorted. C++ libraries first, then dfhack modules, then df structures,
|
||||||
|
then local includes. Within each category they should be sorted alphabetically.
|
||||||
|
|
||||||
|
How to get new code into DFHack
|
||||||
|
-------------------------------
|
||||||
|
* Submit pull requests to the ``develop`` branch, not the ``master`` branch.
|
||||||
|
(The ``master`` branch always points at the most recent release)
|
||||||
|
* Use a new branch for each feature or bugfix so that your changes can be merged independently
|
||||||
|
(i.e. not the master or develop branch of your fork).
|
||||||
|
* If possible, compile on multiple platforms when changing anything that compiles
|
||||||
|
* It must pass CI - run ``python travis/all.py`` to check this.
|
||||||
|
* Update documentation when applicable - see `docs-standards` for details.
|
||||||
|
* Update ``changelog.txt`` and ``docs/Authors.rst`` when applicable. See
|
||||||
|
`build-changelog` for more information on the changelog format.
|
||||||
|
* Create a GitHub pull request once finished
|
||||||
|
* Submit ideas and bug reports as :issue:`issues on GitHub <>`.
|
||||||
|
Posts in the forum thread can easily get missed or forgotten.
|
||||||
|
* Work on :issue:`reported problems <?q=is:open+-label:idea>`
|
||||||
|
will take priority over ideas or suggestions.
|
||||||
|
|
||||||
|
|
||||||
|
Other ways to help
|
||||||
|
==================
|
||||||
|
DFHack is a software project, but there's a lot more to it than programming.
|
||||||
|
If you're not comfortable programming, you can help by:
|
||||||
|
|
||||||
|
* reporting bugs and incomplete documentation
|
||||||
|
* improving the documentation
|
||||||
|
* finding third-party scripts to add
|
||||||
|
* writing tutorials for newbies
|
||||||
|
|
||||||
|
All those things are crucial, and often under-represented. So if that's
|
||||||
|
your thing, go get started!
|
||||||
|
|
||||||
@ -0,0 +1,87 @@
|
|||||||
|
===========================
|
||||||
|
DFHack development overview
|
||||||
|
===========================
|
||||||
|
|
||||||
|
DFHack has various components; this page provides an overview of some. If you
|
||||||
|
are looking to develop a tool for DFHack, developing a script or plugin is
|
||||||
|
likely the most straightforward choice.
|
||||||
|
|
||||||
|
Other pages that may be relevant include:
|
||||||
|
|
||||||
|
- `contributing`
|
||||||
|
- `documentation`
|
||||||
|
- `license`
|
||||||
|
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
|
||||||
|
|
||||||
|
Plugins
|
||||||
|
-------
|
||||||
|
|
||||||
|
DFHack plugins are written in C++ and located in the ``plugins`` folder.
|
||||||
|
Currently, documentation on how to write plugins is somewhat sparse. There are
|
||||||
|
templates that you can get use to get started in the ``plugins/skeleton``
|
||||||
|
folder, and the source code of existing plugins can also be helpful.
|
||||||
|
|
||||||
|
If you want to compile a plugin that you have just added, you will need to add a
|
||||||
|
call to ``DFHACK_PLUGIN`` in ``plugins/CMakeLists.txt``.
|
||||||
|
|
||||||
|
Plugins have the ability to make one or more commands available to users of the
|
||||||
|
DFHack console. Examples include `3dveins` (which implements the ``3dveins``
|
||||||
|
command) and `reveal` (which implements ``reveal``, ``unreveal``, and several
|
||||||
|
other commands).
|
||||||
|
|
||||||
|
Plugins can also register handlers to run on every tick, and can interface with
|
||||||
|
the built-in `enable` and `disable` commands. For the full plugin API, see the
|
||||||
|
skeleton plugins or ``PluginManager.cpp``.
|
||||||
|
|
||||||
|
Installed plugins live in the ``hack/plugins`` folder of a DFHack installation,
|
||||||
|
and the `load` family of commands can be used to load a recompiled plugin
|
||||||
|
without restarting DF.
|
||||||
|
|
||||||
|
See `plugins-index` for a list of all plugins included in DFHack.
|
||||||
|
|
||||||
|
Scripts
|
||||||
|
-------
|
||||||
|
|
||||||
|
DFHack scripts can currently be written in Lua or Ruby. The `Lua API <lua-api>`
|
||||||
|
is more complete and currently better-documented, however. Referring to existing
|
||||||
|
scripts as well as the API documentation can be helpful when developing new
|
||||||
|
scripts.
|
||||||
|
|
||||||
|
`Scripts included in DFHack <scripts-index>` live in a separate `scripts repository <https://github.com/dfhack/scripts>`_.
|
||||||
|
This can be found in the ``scripts`` submodule if you have
|
||||||
|
`cloned DFHack <compile-how-to-get-the-code>`, or the ``hack/scripts`` folder
|
||||||
|
of an installed copy of DFHack.
|
||||||
|
|
||||||
|
Core
|
||||||
|
----
|
||||||
|
|
||||||
|
The `DFHack core <dfhack-core>` has a variety of low-level functions. It is
|
||||||
|
responsible for hooking into DF (via SDL), providing a console, and providing an
|
||||||
|
interface for plugins and scripts to interact with DF.
|
||||||
|
|
||||||
|
Modules
|
||||||
|
-------
|
||||||
|
|
||||||
|
A lot of shared code to interact with DF in more complicated ways is contained
|
||||||
|
in **modules**. For example, the Units module contains functions for checking
|
||||||
|
various traits of units, changing nicknames properly, and more. Generally, code
|
||||||
|
that is useful to multiple plugins and scripts should go in the appropriate
|
||||||
|
module, if there is one.
|
||||||
|
|
||||||
|
Several modules are also `exposed to Lua <lua-cpp-func-wrappers>`, although
|
||||||
|
some functions (and some entire modules) are currently only available in C++.
|
||||||
|
|
||||||
|
Remote access interface
|
||||||
|
-----------------------
|
||||||
|
DFHack supports remote access by exchanging Google protobuf messages via a TCP
|
||||||
|
socket. Both the core and plugins can define remotely accessible methods. The
|
||||||
|
``dfhack-run`` command uses this interface to invoke ordinary console commands.
|
||||||
|
|
||||||
|
Currently the supported set of requests is limited, because the developers don't
|
||||||
|
know what exactly is most useful. `remotefortressreader` provides a fairly
|
||||||
|
comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`.
|
||||||
|
|
||||||
@ -0,0 +1,149 @@
|
|||||||
|
.. _memory-research:
|
||||||
|
|
||||||
|
###############
|
||||||
|
Memory research
|
||||||
|
###############
|
||||||
|
|
||||||
|
There are a variety of tools that can be used to analyze DF memory - some are
|
||||||
|
listed here. Note that some of these may be old and unmaintained. If you aren't
|
||||||
|
sure what tool would be best for your purposes, feel free to ask for advice (on
|
||||||
|
IRC, Bay12, etc.).
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
|
||||||
|
|
||||||
|
Cross-platform tools
|
||||||
|
====================
|
||||||
|
|
||||||
|
Ghidra
|
||||||
|
------
|
||||||
|
|
||||||
|
Ghidra is a cross-platform reverse-engineering framework (written in Java)
|
||||||
|
available at https://ghidra-sre.org. It supports analyzing both 32-bit and
|
||||||
|
64-bit executables for all supported DF platforms. There are some custom DFHack
|
||||||
|
Ghidra scripts available in the `df_misc`_ repo (look for ``.java`` files).
|
||||||
|
|
||||||
|
|
||||||
|
IDA Freeware 7.0
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Available from `Hex-Rays <https://www.hex-rays.com/products/ida/support/download_freeware/>`_.
|
||||||
|
Supports analyzing both 32-bit and 64-bit executables for all supported DF platforms.
|
||||||
|
Some ``.idc`` scripts for IDA are available in the `df_misc`_ repo.
|
||||||
|
|
||||||
|
.. _df_misc: https://github.com/DFHack/df_misc
|
||||||
|
|
||||||
|
|
||||||
|
Hopper
|
||||||
|
------
|
||||||
|
|
||||||
|
Runs on macOS and some Linux distributions; available from https://www.hopperapp.com/.
|
||||||
|
`TWBT <https://github.com/mifki/df-twbt/blob/master/PATCHES.md>`_ uses this to produce some patches.
|
||||||
|
|
||||||
|
|
||||||
|
DFHack tools
|
||||||
|
------------
|
||||||
|
|
||||||
|
Plugins
|
||||||
|
~~~~~~~
|
||||||
|
|
||||||
|
There are a few development plugins useful for low-level memory research. They
|
||||||
|
are not built by default, but can be built by setting the ``BUILD_DEVEL``
|
||||||
|
`CMake option <compile-build-options>`. These include:
|
||||||
|
|
||||||
|
- ``check-structures-sanity``, which performs sanity checks on the given DF
|
||||||
|
object. Note that this will crash in several cases, some intentional, so using
|
||||||
|
this with `GDB <linux-gdb>` is recommended.
|
||||||
|
- ``memview``, which produces a hex dump of a given memory range. It also
|
||||||
|
highlights valid pointers, and can be configured to work with `sizecheck`
|
||||||
|
to auto-detect object sizes.
|
||||||
|
- ``vectors``, which can identify instances of ``std::vector`` in a given memory range.
|
||||||
|
|
||||||
|
Scripts
|
||||||
|
~~~~~~~
|
||||||
|
|
||||||
|
Several `development scripts <scripts-devel>` can be useful for memory research.
|
||||||
|
These include (but are not limited to):
|
||||||
|
|
||||||
|
- `devel/dump-offsets`
|
||||||
|
- `devel/find-offsets`
|
||||||
|
- `devel/lsmem`
|
||||||
|
- `devel/sc` (requires `sizecheck`)
|
||||||
|
- `devel/visualize-structure`
|
||||||
|
- Generally, any script starting with ``devel/find``
|
||||||
|
|
||||||
|
.. _sizecheck:
|
||||||
|
|
||||||
|
Sizecheck
|
||||||
|
~~~~~~~~~
|
||||||
|
|
||||||
|
Sizecheck is a custom tool that hooks into the memory allocator and inserts a
|
||||||
|
header indicating the size of every object. The corresponding logic to check for
|
||||||
|
this header when freeing memory usually works, but is inherently not foolproof.
|
||||||
|
You should not count on DF being stable when using this.
|
||||||
|
|
||||||
|
DFHack's implementation of sizecheck is currently only tested on Linux, although
|
||||||
|
it probably also works on macOS. It can be built with the ``BUILD_SIZECHECK``
|
||||||
|
`CMake option <compile-build-options>`, which produces a ``libsizecheck``
|
||||||
|
library installed in the ``hack`` folder. You will need to preload this library
|
||||||
|
manually, by setting ``PRELOAD_LIB`` on Linux (or ``LD_PRELOAD`` if editing
|
||||||
|
the ``dfhack`` launcher script directly), or by editing the ``dfhack``
|
||||||
|
launcher script and adding the library to ``DYLD_INSERT_LIBRARIES`` on macOS.
|
||||||
|
|
||||||
|
There is also an older sizecheck implementation by Mifki available on
|
||||||
|
`GitHub <https://github.com/mifki/df-sizecheck>`__ (``b.cpp`` is the main
|
||||||
|
sizecheck library, and ``win_patch.cpp`` is used for Windows support). To use
|
||||||
|
this with other DFHack tools, you will likely need to edit the header's
|
||||||
|
magic number to match what is used in `devel/sc` (search for a hexadecimal
|
||||||
|
constant starting with ``0x``).
|
||||||
|
|
||||||
|
Legacy tools
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Some very old DFHack tools are available in the `legacy branch on GitHub <https://github.com/dfhack/dfhack/tree/legacy/tools>`_.
|
||||||
|
No attempt is made to support these.
|
||||||
|
|
||||||
|
|
||||||
|
Linux-specific tools
|
||||||
|
====================
|
||||||
|
|
||||||
|
.. _linux-gdb:
|
||||||
|
|
||||||
|
GDB
|
||||||
|
---
|
||||||
|
|
||||||
|
`GDB <https://www.gnu.org/software/gdb/>`_ is technically cross-platform, but
|
||||||
|
tends to work best on Linux, and DFHack currently only offers support for using
|
||||||
|
GDB on 64-bit Linux. To start with GDB, pass ``-g`` to the DFHack launcher
|
||||||
|
script::
|
||||||
|
|
||||||
|
./dfhack -g
|
||||||
|
|
||||||
|
Some basic GDB commands:
|
||||||
|
|
||||||
|
- ``run``: starts DF from the GDB prompt. Any arguments will be passed as
|
||||||
|
command-line arguments to DF (e.g. `load-save` may be useful).
|
||||||
|
- ``bt`` will produce a backtrace if DF crashes.
|
||||||
|
|
||||||
|
See the `official GDB documentation <https://www.gnu.org/software/gdb/documentation/>`_
|
||||||
|
for more details.
|
||||||
|
|
||||||
|
|
||||||
|
df-structures GUI
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
This is a tool written by Angavrilov and available on `GitHub <https://github.com/angavrilov/cl-linux-debug>`__.
|
||||||
|
It only supports 32-bit DF. Some assistance may be available on IRC.
|
||||||
|
|
||||||
|
|
||||||
|
EDB (Evan's debugger)
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
Available on `GitHub <https://github.com/eteran/edb-debugger>`__.
|
||||||
|
|
||||||
|
|
||||||
|
Windows-specific tools
|
||||||
|
======================
|
||||||
|
|
||||||
|
Some people have used `Cheat Engine <https://www.cheatengine.org/>`__ for research in the past.
|
||||||
@ -0,0 +1,17 @@
|
|||||||
|
#############
|
||||||
|
Removed tools
|
||||||
|
#############
|
||||||
|
|
||||||
|
This page lists tools (plugins or scripts) that were previously included in
|
||||||
|
DFHack but have been removed. It exists primarily so that internal links still
|
||||||
|
work (e.g. links from the `changelog`).
|
||||||
|
|
||||||
|
.. contents:: Contents
|
||||||
|
:local:
|
||||||
|
:depth: 1
|
||||||
|
|
||||||
|
.. _warn-stuck-trees:
|
||||||
|
|
||||||
|
warn-stuck-trees
|
||||||
|
================
|
||||||
|
The corresponding DF bug, :bug:`9252` was fixed in DF 0.44.01.
|
||||||
@ -1,14 +0,0 @@
|
|||||||
###############
|
|
||||||
Removed scripts
|
|
||||||
###############
|
|
||||||
|
|
||||||
The following scripts were removed for various reasons.
|
|
||||||
|
|
||||||
.. contents::
|
|
||||||
:depth: 2
|
|
||||||
|
|
||||||
.. _warn-stuck-trees:
|
|
||||||
|
|
||||||
warn-stuck-trees
|
|
||||||
================
|
|
||||||
The corresponding DF bug, :bug:`9252` was fixed in DF 0.44.01.
|
|
||||||
@ -0,0 +1,34 @@
|
|||||||
|
===================================
|
||||||
|
DF data definitions (DF-structures)
|
||||||
|
===================================
|
||||||
|
|
||||||
|
DFHack's information about DF's data structures is stored in XML files in the
|
||||||
|
`df-structures repository <https://github.com/DFHack/df-structures>`_. If you
|
||||||
|
have `obtained a local copy of the DFHack source <compile-how-to-get-the-code>`,
|
||||||
|
DF-structures is included as a submodule in ``library/xml``.
|
||||||
|
|
||||||
|
Data structure layouts are described in files named with the ``df.*.xml``
|
||||||
|
pattern. This information is transformed by a Perl script (``codegen.pl``) into
|
||||||
|
C++ headers, as well as metadata for the Lua wrapper. This ultimately allows
|
||||||
|
DFHack code to access DF data directly, with the same speed and capabilities as
|
||||||
|
DF itself, which is an advantage over the older out-of-process approach (used
|
||||||
|
by debuggers and utilities like Dwarf Therapist). The main disadvantage of this
|
||||||
|
approach is that any compiled code relying on these layouts will break when DF's
|
||||||
|
layout changes, and will need to be recompiled for every new DF version.
|
||||||
|
|
||||||
|
Addresses of DF global objects and vtables are stored in a separate file,
|
||||||
|
:file:`symbols.xml`. Since these are only absolute addresses, they do not need
|
||||||
|
to be compiled in to DFHack code, and are instead loaded at runtime. This makes
|
||||||
|
fixes and additions to global addresses possible without recompiling DFHack.
|
||||||
|
In an installed copy of DFHack, this file can be found at the root of the
|
||||||
|
``hack`` folder.
|
||||||
|
|
||||||
|
The following pages contain more detailed information about various aspects
|
||||||
|
of DF-structures:
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:titlesonly:
|
||||||
|
|
||||||
|
/library/xml/SYNTAX
|
||||||
|
/library/xml/how-to-update
|
||||||
|
|
||||||
@ -1,285 +1,5 @@
|
|||||||
import collections
|
import os, sys
|
||||||
import copy
|
sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'sphinx_extensions'))
|
||||||
import itertools
|
|
||||||
import os
|
|
||||||
import sys
|
|
||||||
|
|
||||||
CHANGELOG_PATHS = (
|
from dfhack.changelog import cli_entrypoint
|
||||||
'docs/changelog.txt',
|
cli_entrypoint()
|
||||||
'scripts/changelog.txt',
|
|
||||||
'library/xml/changelog.txt',
|
|
||||||
)
|
|
||||||
|
|
||||||
CHANGELOG_SECTIONS = [
|
|
||||||
'New Plugins',
|
|
||||||
'New Scripts',
|
|
||||||
'New Tweaks',
|
|
||||||
'New Features',
|
|
||||||
'New Internal Commands',
|
|
||||||
'Fixes',
|
|
||||||
'Misc Improvements',
|
|
||||||
'Removed',
|
|
||||||
'API',
|
|
||||||
'Internals',
|
|
||||||
'Lua',
|
|
||||||
'Ruby',
|
|
||||||
'Structures',
|
|
||||||
]
|
|
||||||
|
|
||||||
REPLACEMENTS = {
|
|
||||||
'`search`': '`search-plugin`',
|
|
||||||
}
|
|
||||||
|
|
||||||
def to_title_case(word):
|
|
||||||
if word == word.upper():
|
|
||||||
# Preserve acronyms
|
|
||||||
return word
|
|
||||||
return word[0].upper() + word[1:].lower()
|
|
||||||
|
|
||||||
def find_all_indices(string, substr):
|
|
||||||
start = 0
|
|
||||||
while True:
|
|
||||||
i = string.find(substr, start)
|
|
||||||
if i == -1:
|
|
||||||
return
|
|
||||||
yield i
|
|
||||||
start = i + 1
|
|
||||||
|
|
||||||
def replace_text(string, replacements):
|
|
||||||
for old_text, new_text in replacements.items():
|
|
||||||
new_string = ''
|
|
||||||
new_string_end = 0 # number of characters from string in new_string
|
|
||||||
for i in find_all_indices(string, old_text):
|
|
||||||
if i > 0 and string[i - 1] == '!':
|
|
||||||
# exempt if preceded by '!'
|
|
||||||
new_string += string[new_string_end:i - 1]
|
|
||||||
new_string += old_text
|
|
||||||
else:
|
|
||||||
# copy until this occurrence
|
|
||||||
new_string += string[new_string_end:i]
|
|
||||||
new_string += new_text
|
|
||||||
new_string_end = i + len(old_text)
|
|
||||||
new_string += string[new_string_end:]
|
|
||||||
string = new_string
|
|
||||||
return string
|
|
||||||
|
|
||||||
class ChangelogEntry(object):
|
|
||||||
def __init__(self, text, section, stable_version, dev_version):
|
|
||||||
text = text.lstrip('- ')
|
|
||||||
# normalize section to title case
|
|
||||||
self.section = ' '.join(map(to_title_case, section.strip().split()))
|
|
||||||
self.stable_version = stable_version
|
|
||||||
self.dev_version = dev_version
|
|
||||||
self.dev_only = text.startswith('@')
|
|
||||||
text = text.lstrip('@ ')
|
|
||||||
self.children = []
|
|
||||||
|
|
||||||
split_index = text.find(': ')
|
|
||||||
if split_index != -1:
|
|
||||||
self.feature, description = text[:split_index], text[split_index+1:]
|
|
||||||
if description.strip():
|
|
||||||
self.children.insert(0, description.strip())
|
|
||||||
else:
|
|
||||||
self.feature = text
|
|
||||||
self.feature = self.feature.replace(':\\', ':').rstrip(':')
|
|
||||||
|
|
||||||
self.sort_key = self.feature.upper()
|
|
||||||
|
|
||||||
def __repr__(self):
|
|
||||||
return 'ChangelogEntry(%r, %r)' % (self.feature, self.children)
|
|
||||||
|
|
||||||
def parse_changelog():
|
|
||||||
entries = []
|
|
||||||
|
|
||||||
for fpath in CHANGELOG_PATHS:
|
|
||||||
if not os.path.isfile(fpath):
|
|
||||||
continue
|
|
||||||
with open(fpath) as f:
|
|
||||||
cur_stable = None
|
|
||||||
cur_dev = None
|
|
||||||
cur_section = None
|
|
||||||
last_entry = None
|
|
||||||
multiline = ''
|
|
||||||
for line_id, line in enumerate(f.readlines()):
|
|
||||||
line_id += 1
|
|
||||||
|
|
||||||
if multiline:
|
|
||||||
multiline += line
|
|
||||||
elif '[[[' in line:
|
|
||||||
multiline = line.replace('[[[', '')
|
|
||||||
|
|
||||||
if ']]]' in multiline:
|
|
||||||
line = multiline.replace(']]]', '')
|
|
||||||
multiline = ''
|
|
||||||
elif multiline:
|
|
||||||
continue
|
|
||||||
|
|
||||||
if not line.strip() or line.startswith('==='):
|
|
||||||
continue
|
|
||||||
|
|
||||||
if line.startswith('##'):
|
|
||||||
cur_section = line.lstrip('#').strip()
|
|
||||||
elif line.startswith('#'):
|
|
||||||
cur_dev = line.lstrip('#').strip().lower()
|
|
||||||
if ('alpha' not in cur_dev and 'beta' not in cur_dev and
|
|
||||||
'rc' not in cur_dev):
|
|
||||||
cur_stable = cur_dev
|
|
||||||
elif line.startswith('-'):
|
|
||||||
if not cur_stable or not cur_dev or not cur_section:
|
|
||||||
raise ValueError(
|
|
||||||
'%s:%i: Entry without section' % (fpath, line_id))
|
|
||||||
last_entry = ChangelogEntry(line.strip(), cur_section,
|
|
||||||
cur_stable, cur_dev)
|
|
||||||
entries.append(last_entry)
|
|
||||||
elif line.lstrip().startswith('-'):
|
|
||||||
if not cur_stable or not cur_dev:
|
|
||||||
raise ValueError(
|
|
||||||
'%s:%i: Sub-entry without section' % (fpath, line_id))
|
|
||||||
if not last_entry:
|
|
||||||
raise ValueError(
|
|
||||||
'%s:%i: Sub-entry without parent' % (fpath, line_id))
|
|
||||||
last_entry.children.append(line.strip('- \n'))
|
|
||||||
else:
|
|
||||||
raise ValueError('Invalid line: ' + line)
|
|
||||||
|
|
||||||
if not entries:
|
|
||||||
raise RuntimeError('No changelog files with contents found')
|
|
||||||
|
|
||||||
return entries
|
|
||||||
|
|
||||||
def consolidate_changelog(all_entries):
|
|
||||||
for sections in all_entries.values():
|
|
||||||
for section, entries in sections.items():
|
|
||||||
# sort() is stable, so reverse entries so that older entries for the
|
|
||||||
# same feature are on top
|
|
||||||
entries.reverse()
|
|
||||||
entries.sort(key=lambda entry: entry.sort_key)
|
|
||||||
new_entries = []
|
|
||||||
for feature, group in itertools.groupby(entries,
|
|
||||||
lambda e: e.feature):
|
|
||||||
old_entries = list(group)
|
|
||||||
children = list(itertools.chain(*[entry.children
|
|
||||||
for entry in old_entries]))
|
|
||||||
new_entry = copy.deepcopy(old_entries[0])
|
|
||||||
new_entry.children = children
|
|
||||||
new_entries.append(new_entry)
|
|
||||||
entries[:] = new_entries
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
def print_changelog(versions, all_entries, path, replace=True, prefix=''):
|
|
||||||
# all_entries: version -> section -> entry
|
|
||||||
with open(path, 'w') as f:
|
|
||||||
def write(line):
|
|
||||||
if replace:
|
|
||||||
line = replace_text(line, REPLACEMENTS)
|
|
||||||
f.write(prefix + line + '\n')
|
|
||||||
for version in versions:
|
|
||||||
sections = all_entries[version]
|
|
||||||
if not sections:
|
|
||||||
continue
|
|
||||||
version = 'DFHack ' + version
|
|
||||||
write(version)
|
|
||||||
write('=' * len(version))
|
|
||||||
write('')
|
|
||||||
for section in CHANGELOG_SECTIONS:
|
|
||||||
entries = sections[section]
|
|
||||||
if not entries:
|
|
||||||
continue
|
|
||||||
write(section)
|
|
||||||
write('-' * len(section))
|
|
||||||
for entry in entries:
|
|
||||||
if len(entry.children) == 1:
|
|
||||||
write('- ' + entry.feature + ': ' +
|
|
||||||
entry.children[0].strip('- '))
|
|
||||||
continue
|
|
||||||
elif entry.children:
|
|
||||||
write('- ' + entry.feature + ':')
|
|
||||||
write('')
|
|
||||||
for child in entry.children:
|
|
||||||
write(' - ' + child)
|
|
||||||
write('')
|
|
||||||
else:
|
|
||||||
write('- ' + entry.feature)
|
|
||||||
write('')
|
|
||||||
write('')
|
|
||||||
|
|
||||||
|
|
||||||
def generate_changelog(all=False):
|
|
||||||
entries = parse_changelog()
|
|
||||||
|
|
||||||
# scan for unrecognized sections
|
|
||||||
for entry in entries:
|
|
||||||
if entry.section not in CHANGELOG_SECTIONS:
|
|
||||||
raise RuntimeWarning('Unknown section: ' + entry.section)
|
|
||||||
|
|
||||||
# ordered versions
|
|
||||||
versions = ['future']
|
|
||||||
# map versions to stable versions
|
|
||||||
stable_version_map = {}
|
|
||||||
# version -> section -> entry
|
|
||||||
stable_entries = collections.defaultdict(lambda:
|
|
||||||
collections.defaultdict(list))
|
|
||||||
dev_entries = collections.defaultdict(lambda:
|
|
||||||
collections.defaultdict(list))
|
|
||||||
|
|
||||||
for entry in entries:
|
|
||||||
# build list of all versions
|
|
||||||
if entry.dev_version not in versions:
|
|
||||||
versions.append(entry.dev_version)
|
|
||||||
stable_version_map.setdefault(entry.dev_version, entry.stable_version)
|
|
||||||
|
|
||||||
if not entry.dev_only:
|
|
||||||
# add non-dev-only entries to both changelogs
|
|
||||||
stable_entries[entry.stable_version][entry.section].append(entry)
|
|
||||||
dev_entries[entry.dev_version][entry.section].append(entry)
|
|
||||||
|
|
||||||
consolidate_changelog(stable_entries)
|
|
||||||
consolidate_changelog(dev_entries)
|
|
||||||
|
|
||||||
print_changelog(versions, stable_entries, 'docs/_auto/news.rst')
|
|
||||||
print_changelog(versions, dev_entries, 'docs/_auto/news-dev.rst')
|
|
||||||
|
|
||||||
if all:
|
|
||||||
for version in versions:
|
|
||||||
if version not in stable_version_map:
|
|
||||||
print('warn: skipping ' + version)
|
|
||||||
continue
|
|
||||||
if stable_version_map[version] == version:
|
|
||||||
version_entries = {version: stable_entries[version]}
|
|
||||||
else:
|
|
||||||
version_entries = {version: dev_entries[version]}
|
|
||||||
print_changelog([version], version_entries,
|
|
||||||
'docs/_changelogs/%s-github.txt' % version,
|
|
||||||
replace=False)
|
|
||||||
print_changelog([version], version_entries,
|
|
||||||
'docs/_changelogs/%s-reddit.txt' % version,
|
|
||||||
replace=False,
|
|
||||||
prefix='> ')
|
|
||||||
|
|
||||||
return entries
|
|
||||||
|
|
||||||
if __name__ == '__main__':
|
|
||||||
import argparse
|
|
||||||
parser = argparse.ArgumentParser()
|
|
||||||
parser.add_argument('-a', '--all', action='store_true',
|
|
||||||
help='Print changelogs for all versions to docs/_changelogs')
|
|
||||||
parser.add_argument('-c', '--check', action='store_true',
|
|
||||||
help='Check that all entries are printed')
|
|
||||||
args = parser.parse_args()
|
|
||||||
|
|
||||||
os.chdir(os.path.abspath(os.path.dirname(__file__)))
|
|
||||||
os.chdir('..')
|
|
||||||
entries = generate_changelog(all=args.all)
|
|
||||||
|
|
||||||
if args.check:
|
|
||||||
with open('docs/_auto/news.rst') as f:
|
|
||||||
content_stable = f.read()
|
|
||||||
with open('docs/_auto/news-dev.rst') as f:
|
|
||||||
content_dev = f.read()
|
|
||||||
for entry in entries:
|
|
||||||
for description in entry.children:
|
|
||||||
if not entry.dev_only and description not in content_stable:
|
|
||||||
print('stable missing: ' + description)
|
|
||||||
if description not in content_dev:
|
|
||||||
print('dev missing: ' + description)
|
|
||||||
|
|||||||
@ -0,0 +1,19 @@
|
|||||||
|
========================
|
||||||
|
DFHack Development Guide
|
||||||
|
========================
|
||||||
|
|
||||||
|
These are pages relevant to people developing for DFHack.
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:maxdepth: 1
|
||||||
|
|
||||||
|
/docs/Dev-intro
|
||||||
|
/docs/Contributing
|
||||||
|
/docs/Compile
|
||||||
|
/docs/NEWS-dev
|
||||||
|
/docs/Lua API
|
||||||
|
/docs/Documentation
|
||||||
|
/docs/Structures-intro
|
||||||
|
/docs/Memory-research
|
||||||
|
/docs/Binpatches
|
||||||
|
|
||||||
@ -0,0 +1,300 @@
|
|||||||
|
import collections
|
||||||
|
import copy
|
||||||
|
import itertools
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from dfhack.util import DFHACK_ROOT, DOCS_ROOT
|
||||||
|
|
||||||
|
CHANGELOG_PATHS = (
|
||||||
|
'docs/changelog.txt',
|
||||||
|
'scripts/changelog.txt',
|
||||||
|
'library/xml/changelog.txt',
|
||||||
|
)
|
||||||
|
CHANGELOG_PATHS = (os.path.join(DFHACK_ROOT, p) for p in CHANGELOG_PATHS)
|
||||||
|
|
||||||
|
CHANGELOG_SECTIONS = [
|
||||||
|
'New Plugins',
|
||||||
|
'New Scripts',
|
||||||
|
'New Tweaks',
|
||||||
|
'New Features',
|
||||||
|
'New Internal Commands',
|
||||||
|
'Fixes',
|
||||||
|
'Misc Improvements',
|
||||||
|
'Removed',
|
||||||
|
'API',
|
||||||
|
'Internals',
|
||||||
|
'Lua',
|
||||||
|
'Ruby',
|
||||||
|
'Structures',
|
||||||
|
]
|
||||||
|
|
||||||
|
REPLACEMENTS = {
|
||||||
|
'`search`': '`search-plugin`',
|
||||||
|
}
|
||||||
|
|
||||||
|
def to_title_case(word):
|
||||||
|
if word == word.upper():
|
||||||
|
# Preserve acronyms
|
||||||
|
return word
|
||||||
|
return word[0].upper() + word[1:].lower()
|
||||||
|
|
||||||
|
def find_all_indices(string, substr):
|
||||||
|
start = 0
|
||||||
|
while True:
|
||||||
|
i = string.find(substr, start)
|
||||||
|
if i == -1:
|
||||||
|
return
|
||||||
|
yield i
|
||||||
|
start = i + 1
|
||||||
|
|
||||||
|
def replace_text(string, replacements):
|
||||||
|
for old_text, new_text in replacements.items():
|
||||||
|
new_string = ''
|
||||||
|
new_string_end = 0 # number of characters from string in new_string
|
||||||
|
for i in find_all_indices(string, old_text):
|
||||||
|
if i > 0 and string[i - 1] == '!':
|
||||||
|
# exempt if preceded by '!'
|
||||||
|
new_string += string[new_string_end:i - 1]
|
||||||
|
new_string += old_text
|
||||||
|
else:
|
||||||
|
# copy until this occurrence
|
||||||
|
new_string += string[new_string_end:i]
|
||||||
|
new_string += new_text
|
||||||
|
new_string_end = i + len(old_text)
|
||||||
|
new_string += string[new_string_end:]
|
||||||
|
string = new_string
|
||||||
|
return string
|
||||||
|
|
||||||
|
class ChangelogEntry(object):
|
||||||
|
def __init__(self, text, section, stable_version, dev_version):
|
||||||
|
text = text.lstrip('- ')
|
||||||
|
# normalize section to title case
|
||||||
|
self.section = ' '.join(map(to_title_case, section.strip().split()))
|
||||||
|
self.stable_version = stable_version
|
||||||
|
self.dev_version = dev_version
|
||||||
|
self.dev_only = text.startswith('@')
|
||||||
|
text = text.lstrip('@ ')
|
||||||
|
self.children = []
|
||||||
|
|
||||||
|
split_index = text.find(': ')
|
||||||
|
if split_index != -1:
|
||||||
|
self.feature, description = text[:split_index], text[split_index+1:]
|
||||||
|
if description.strip():
|
||||||
|
self.children.insert(0, description.strip())
|
||||||
|
else:
|
||||||
|
self.feature = text
|
||||||
|
self.feature = self.feature.replace(':\\', ':').rstrip(':')
|
||||||
|
|
||||||
|
self.sort_key = self.feature.upper()
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
return 'ChangelogEntry(%r, %r)' % (self.feature, self.children)
|
||||||
|
|
||||||
|
def parse_changelog():
|
||||||
|
entries = []
|
||||||
|
|
||||||
|
for fpath in CHANGELOG_PATHS:
|
||||||
|
if not os.path.isfile(fpath):
|
||||||
|
continue
|
||||||
|
with open(fpath) as f:
|
||||||
|
cur_stable = None
|
||||||
|
cur_dev = None
|
||||||
|
cur_section = None
|
||||||
|
last_entry = None
|
||||||
|
multiline = ''
|
||||||
|
for line_id, line in enumerate(f.readlines()):
|
||||||
|
line_id += 1
|
||||||
|
|
||||||
|
if multiline:
|
||||||
|
multiline += line
|
||||||
|
elif '[[[' in line:
|
||||||
|
multiline = line.replace('[[[', '')
|
||||||
|
|
||||||
|
if ']]]' in multiline:
|
||||||
|
line = multiline.replace(']]]', '')
|
||||||
|
multiline = ''
|
||||||
|
elif multiline:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not line.strip() or line.startswith('==='):
|
||||||
|
continue
|
||||||
|
|
||||||
|
if line.startswith('##'):
|
||||||
|
cur_section = line.lstrip('#').strip()
|
||||||
|
elif line.startswith('#'):
|
||||||
|
cur_dev = line.lstrip('#').strip().lower()
|
||||||
|
if ('alpha' not in cur_dev and 'beta' not in cur_dev and
|
||||||
|
'rc' not in cur_dev):
|
||||||
|
cur_stable = cur_dev
|
||||||
|
elif line.startswith('-'):
|
||||||
|
if not cur_stable or not cur_dev or not cur_section:
|
||||||
|
raise ValueError(
|
||||||
|
'%s:%i: Entry without section' % (fpath, line_id))
|
||||||
|
last_entry = ChangelogEntry(line.strip(), cur_section,
|
||||||
|
cur_stable, cur_dev)
|
||||||
|
entries.append(last_entry)
|
||||||
|
elif line.lstrip().startswith('-'):
|
||||||
|
if not cur_stable or not cur_dev:
|
||||||
|
raise ValueError(
|
||||||
|
'%s:%i: Sub-entry without section' % (fpath, line_id))
|
||||||
|
if not last_entry:
|
||||||
|
raise ValueError(
|
||||||
|
'%s:%i: Sub-entry without parent' % (fpath, line_id))
|
||||||
|
last_entry.children.append(line.strip('- \n'))
|
||||||
|
else:
|
||||||
|
raise ValueError('Invalid line: ' + line)
|
||||||
|
|
||||||
|
if not entries:
|
||||||
|
raise RuntimeError('No changelog files with contents found')
|
||||||
|
|
||||||
|
return entries
|
||||||
|
|
||||||
|
def consolidate_changelog(all_entries):
|
||||||
|
for sections in all_entries.values():
|
||||||
|
for section, entries in sections.items():
|
||||||
|
# sort() is stable, so reverse entries so that older entries for the
|
||||||
|
# same feature are on top
|
||||||
|
entries.reverse()
|
||||||
|
entries.sort(key=lambda entry: entry.sort_key)
|
||||||
|
new_entries = []
|
||||||
|
for feature, group in itertools.groupby(entries,
|
||||||
|
lambda e: e.feature):
|
||||||
|
old_entries = list(group)
|
||||||
|
children = list(itertools.chain(*[entry.children
|
||||||
|
for entry in old_entries]))
|
||||||
|
new_entry = copy.deepcopy(old_entries[0])
|
||||||
|
new_entry.children = children
|
||||||
|
new_entries.append(new_entry)
|
||||||
|
entries[:] = new_entries
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def print_changelog(versions, all_entries, path, replace=True, prefix=''):
|
||||||
|
# all_entries: version -> section -> entry
|
||||||
|
with open(path, 'w') as f:
|
||||||
|
def write(line):
|
||||||
|
if replace:
|
||||||
|
line = replace_text(line, REPLACEMENTS)
|
||||||
|
f.write(prefix + line + '\n')
|
||||||
|
for version in versions:
|
||||||
|
sections = all_entries[version]
|
||||||
|
if not sections:
|
||||||
|
continue
|
||||||
|
version = 'DFHack ' + version
|
||||||
|
write(version)
|
||||||
|
write('=' * len(version))
|
||||||
|
write('')
|
||||||
|
for section in CHANGELOG_SECTIONS:
|
||||||
|
entries = sections[section]
|
||||||
|
if not entries:
|
||||||
|
continue
|
||||||
|
write(section)
|
||||||
|
write('-' * len(section))
|
||||||
|
for entry in entries:
|
||||||
|
if len(entry.children) == 1:
|
||||||
|
write('- ' + entry.feature + ': ' +
|
||||||
|
entry.children[0].strip('- '))
|
||||||
|
continue
|
||||||
|
elif entry.children:
|
||||||
|
write('- ' + entry.feature + ':')
|
||||||
|
write('')
|
||||||
|
for child in entry.children:
|
||||||
|
write(' - ' + child)
|
||||||
|
write('')
|
||||||
|
else:
|
||||||
|
write('- ' + entry.feature)
|
||||||
|
write('')
|
||||||
|
write('')
|
||||||
|
|
||||||
|
|
||||||
|
def generate_changelog(all=False):
|
||||||
|
entries = parse_changelog()
|
||||||
|
|
||||||
|
# scan for unrecognized sections
|
||||||
|
for entry in entries:
|
||||||
|
if entry.section not in CHANGELOG_SECTIONS:
|
||||||
|
raise RuntimeWarning('Unknown section: ' + entry.section)
|
||||||
|
|
||||||
|
# ordered versions
|
||||||
|
versions = ['future']
|
||||||
|
# map versions to stable versions
|
||||||
|
stable_version_map = {}
|
||||||
|
# version -> section -> entry
|
||||||
|
stable_entries = collections.defaultdict(lambda:
|
||||||
|
collections.defaultdict(list))
|
||||||
|
dev_entries = collections.defaultdict(lambda:
|
||||||
|
collections.defaultdict(list))
|
||||||
|
|
||||||
|
for entry in entries:
|
||||||
|
# build list of all versions
|
||||||
|
if entry.dev_version not in versions:
|
||||||
|
versions.append(entry.dev_version)
|
||||||
|
stable_version_map.setdefault(entry.dev_version, entry.stable_version)
|
||||||
|
|
||||||
|
if not entry.dev_only:
|
||||||
|
# add non-dev-only entries to both changelogs
|
||||||
|
stable_entries[entry.stable_version][entry.section].append(entry)
|
||||||
|
dev_entries[entry.dev_version][entry.section].append(entry)
|
||||||
|
|
||||||
|
consolidate_changelog(stable_entries)
|
||||||
|
consolidate_changelog(dev_entries)
|
||||||
|
|
||||||
|
print_changelog(versions, stable_entries, os.path.join(DOCS_ROOT, '_auto/news.rst'))
|
||||||
|
print_changelog(versions, dev_entries, os.path.join(DOCS_ROOT, '_auto/news-dev.rst'))
|
||||||
|
|
||||||
|
if all:
|
||||||
|
for version in versions:
|
||||||
|
if version not in stable_version_map:
|
||||||
|
print('warn: skipping ' + version)
|
||||||
|
continue
|
||||||
|
if stable_version_map[version] == version:
|
||||||
|
version_entries = {version: stable_entries[version]}
|
||||||
|
else:
|
||||||
|
version_entries = {version: dev_entries[version]}
|
||||||
|
print_changelog([version], version_entries,
|
||||||
|
os.path.join(DOCS_ROOT, '_changelogs/%s-github.txt' % version),
|
||||||
|
replace=False)
|
||||||
|
print_changelog([version], version_entries,
|
||||||
|
os.path.join(DOCS_ROOT, '_changelogs/%s-reddit.txt' % version),
|
||||||
|
replace=False,
|
||||||
|
prefix='> ')
|
||||||
|
|
||||||
|
return entries
|
||||||
|
|
||||||
|
def cli_entrypoint():
|
||||||
|
import argparse
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument('-a', '--all', action='store_true',
|
||||||
|
help='Print changelogs for all versions to docs/_changelogs')
|
||||||
|
parser.add_argument('-c', '--check', action='store_true',
|
||||||
|
help='Check that all entries are printed')
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
entries = generate_changelog(all=args.all)
|
||||||
|
|
||||||
|
if args.check:
|
||||||
|
with open(os.path.join(DOCS_ROOT, '_auto/news.rst')) as f:
|
||||||
|
content_stable = f.read()
|
||||||
|
with open(os.path.join(DOCS_ROOT, '_auto/news-dev.rst')) as f:
|
||||||
|
content_dev = f.read()
|
||||||
|
for entry in entries:
|
||||||
|
for description in entry.children:
|
||||||
|
if not entry.dev_only and description not in content_stable:
|
||||||
|
print('stable missing: ' + description)
|
||||||
|
if description not in content_dev:
|
||||||
|
print('dev missing: ' + description)
|
||||||
|
|
||||||
|
|
||||||
|
def sphinx_entrypoint(app, config):
|
||||||
|
generate_changelog()
|
||||||
|
|
||||||
|
|
||||||
|
def setup(app):
|
||||||
|
app.connect('config-inited', sphinx_entrypoint)
|
||||||
|
|
||||||
|
return {
|
||||||
|
'version': '0.1',
|
||||||
|
'parallel_read_safe': True,
|
||||||
|
'parallel_write_safe': True,
|
||||||
|
}
|
||||||
@ -0,0 +1,7 @@
|
|||||||
|
import os
|
||||||
|
|
||||||
|
DFHACK_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))))
|
||||||
|
DOCS_ROOT = os.path.join(DFHACK_ROOT, 'docs')
|
||||||
|
|
||||||
|
if not os.path.isdir(DOCS_ROOT):
|
||||||
|
raise ReferenceError('docs root not found: %s' % DOCS_ROOT)
|
||||||
@ -1 +1 @@
|
|||||||
Subproject commit 2d67b03a1c5c701e65fb483b9d6c1d3067f990cb
|
Subproject commit 847ec0c2d4dc74a887e1f980eeed6f97345c195e
|
||||||
Loading…
Reference in New Issue