diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md new file mode 100644 index 000000000..2367db678 --- /dev/null +++ b/.github/CONTRIBUTING.md @@ -0,0 +1,6 @@ +Thank you for your interest in contributing to DFHack! If you're reading this +document, you're probably viewing it on GitHub. The DFHack docs are hosted +on [ReadTheDocs](https://dfhack.readthedocs.io/) - in particular, contributing +guidelines are [here](https://docs.dfhack.org/en/latest/docs/Contributing.html). +Double-checking the style guidelines before submitting a pull request is +always appreciated. diff --git a/Contributing.rst b/Contributing.rst deleted file mode 100644 index e358e66b3..000000000 --- a/Contributing.rst +++ /dev/null @@ -1,229 +0,0 @@ -########################### -How to contribute to DFHack -########################### - -.. contents:: - - -.. _contributing-code: - -Contributing Code -================= -Several things should be kept in mind when contributing code to DFHack. - -Code Format ------------ -* Four space indents for C++. Never use tabs for indentation in any language. -* LF (Unix style) line terminators -* Avoid trailing whitespace -* UTF-8 encoding -* For C++: - - * Opening and closing braces on their own lines or opening brace at the end of the previous line - * Braces placed at original indent level if on their own lines - * #includes should be sorted. C++ libraries first, then dfhack modules, then df structures, - then local includes. Within each category they should be sorted alphabetically. - -How to get new code into DFHack -------------------------------- -* Submit pull requests to the ``develop`` branch, not the ``master`` branch. - (The ``master`` branch always points at the most recent release) -* Use a new branch for each feature or bugfix so that your changes can be merged independently - (i.e. not the master or develop branch of your fork). -* If possible, compile on multiple platforms when changing anything that compiles -* It must pass CI - run ``python travis/all.py`` to check this. -* Update ``changelog.txt`` and ``docs/Authors.rst`` when applicable. See - `build-changelog` for more information on the changelog format. -* Create a GitHub pull request once finished -* Submit ideas and bug reports as :issue:`issues on GitHub <>`. - Posts in the forum thread can easily get missed or forgotten. -* Work on :issue:`reported problems ` - will take priority over ideas or suggestions. - -.. _contributing-memory-research: - -Memory research ---------------- -If you want to do memory research, you'll need some tools and some knowledge. -In general, you'll need a good memory viewer and optionally something -to look at machine code without getting crazy :) -Using publicly known information and analyzing the game's data is preferred. - -Good Windows tools include: - -* IDA Freeware 7.0 (for non-commercial use, supports 32-bit and 64-bit) -* Cheat Engine - -Good Linux tools: - -* angavrilov's df-structures gui (32-bit only, visit us on IRC for details) -* IDA Freeware 7.0 (see above) -* edb (Evan's Debugger) -* Some of the tools residing in the ``legacy`` dfhack branch. - -Using the library as a developer -================================ -Currently, the most direct way to use the library is to write a script or plugin that can be loaded by it. -All the plugins can be found in the 'plugins' folder. There's no in-depth documentation -on how to write one yet, but it should be easy enough to copy one and just follow the pattern. -``plugins/skeleton/skeleton.cpp`` is provided for this purpose. - -Other than through plugins, it is possible to use DFHack via remote access interface, -or by writing scripts in Lua or Ruby. There are plenty of examples in the scripts folder. -The `lua-api` is quite well documented. - -The most important parts of DFHack are the Core, Console, Modules and Plugins. - -* Core acts as the centerpiece of DFHack - it acts as a filter between DF and - SDL and synchronizes the various plugins with DF. -* Console is a thread-safe console that can be used to invoke commands exported by Plugins. -* Modules actually describe the way to access information in DF's memory. You - can get them from the Core. Most modules are split into two parts: high-level - and low-level. High-level is mostly method calls, low-level publicly visible - pointers to DF's data structures. -* Plugins are the tools that use all the other stuff to make things happen. - A plugin can have a list of commands that it exports and an onupdate function - that will be called each DF game tick. - -Rudimentary API documentation can be built using doxygen (see build options -in ``CMakeCache.txt`` or with ``ccmake`` or ``cmake-gui``). The full DFHack -documentation is built with Sphinx_, which runs automatically at compile time. - -.. _Sphinx: http://www.sphinx-doc.org - -DFHack consists of variously licensed code, but invariably weak copyleft. -The main license is zlib/libpng, some bits are MIT licensed, and some are -BSD licensed. See the `license` for more information. - -Feel free to add your own extensions and plugins. Contributing back to -the DFHack repository is welcome and the right thing to do :) - -DF data structure definitions ------------------------------ -DFHack uses information about the game data structures, represented via xml files -in the ``library/xml/`` submodule. - -See https://github.com/DFHack/df-structures, and the documentation linked in the index. - -Data structure layouts are described in files following the ``df.*.xml`` name pattern. -This information is transformed by a perl script into C++ headers describing the -structures, and associated metadata for the Lua wrapper. These headers and data -are then compiled into the DFHack libraries, thus necessitating a compatibility -break every time layouts change; in return it significantly boosts the efficiency -and capabilities of DFHack code. - -Global object addresses are stored in :file:`symbols.xml`, which is copied to the dfhack -release package and loaded as data at runtime. - -Remote access interface ------------------------ -DFHack supports remote access by exchanging Google protobuf messages via a TCP -socket. Both the core and plugins can define remotely accessible methods. The -``dfhack-run`` command uses this interface to invoke ordinary console commands. - -Currently the supported set of requests is limited, because the developers don't -know what exactly is most useful. `remotefortressreader` provides a fairly -comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`. - - -Documentation Standards -======================= -DFHack documentation is built with Sphinx_, and configured automatically -through CMake. If you want to build the docs *only*, use this command:: - - sphinx-build . docs/html - -Whether you're adding new code or just fixing old documentation (and there's plenty), -there are a few important standards for completeness and consistent style. Treat -this section as a guide rather than iron law, match the surrounding text, and you'll -be fine. - -Each command should have a short (~54 character) help string, which is shown -by the `ls` command. For scripts, this is a comment on the first line -(the comment marker and whitespace is stripped). For plugins it's the second -argument to ``PluginCommand``. Please make this brief but descriptive! - -Everything should be documented! If it's not clear *where* a particular -thing should be documented, ask on IRC or in the DFHack thread on Bay12 - -as well as getting help, you'll be providing valuable feedback that -makes it easier for future readers! - -Scripts can use a custom autodoc function, based on the Sphinx ``include`` -directive - anything between the tokens is copied into the appropriate scripts -documentation page. For Ruby, we follow the built-in docstring convention -(``=begin`` and ``=end``). For Lua, the tokens are ``[====[`` and ``]====]`` -- ordinary multi-line strings. It is highly encouraged to reuse this string -as the in-console documentation by (eg.) printing it when a ``-help`` argument -is given. - -The docs **must** have a heading which exactly matches the command, underlined -with ``=====`` to the same length. For example, a lua file would have:: - - local helpstr = [====[ - - add-thought - =========== - Adds a thought or emotion to the selected unit. Can be used by other scripts, - or the gui invoked by running ``add-thought gui`` with a unit selected. - - ]====] - - -Where the heading for a section is also the name of a command, the spelling -and case should exactly match the command to enter in the DFHack command line. - -Try to keep lines within 80-100 characters, so it's readable in plain text -in the terminal - Sphinx (our documentation system) will make sure -paragraphs flow. - -If there aren't many options or examples to show, they can go in a paragraph of -text. Use double-backticks to put commands in monospaced font, like this:: - - You can use ``cleanowned scattered x`` to dump tattered or abandoned items. - -If the command takes more than three arguments, format the list as a table -called Usage. The table *only* lists arguments, not full commands. -Input values are specified in angle brackets. Example:: - - Usage: - - :arg1: A simple argument. - :arg2 : Does something based on the input value. - :Very long argument: - Is very specific. - -To demonstrate usage - useful mainly when the syntax is complicated, list the -full command with arguments in monospaced font, then indent the next line and -describe the effect:: - - ``resume all`` - Resumes all suspended constructions. - -If it would be helpful to mention another DFHack command, don't just type the -name - add a hyperlink! Specify the link target in backticks, and it will be -replaced with the corresponding title and linked: eg ```autolabor``` -=> `autolabor`. Link targets should be equivalent to the command -described (without file extension), and placed above the heading of that -section like this:: - - .. _autolabor: - - autolabor - ========= - -Add link targets if you need them, but otherwise plain headings are preferred. -Scripts have link targets created automatically. - -Other ways to help -================== -DFHack is a software project, but there's a lot more to it than programming. -If you're not comfortable programming, you can help by: - -* reporting bugs and incomplete documentation -* improving the documentation -* finding third-party scripts to add -* writing tutorials for newbies - -All those things are crucial, and often under-represented. So if that's -your thing, go get started! - diff --git a/conf.py b/conf.py index b3f2b5848..aebe73bab 100644 --- a/conf.py +++ b/conf.py @@ -22,8 +22,6 @@ import re import shlex # pylint:disable=unused-import import sys -sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'docs')) -from gen_changelog import generate_changelog # -- Support :dfhack-keybind:`command` ------------------------------------ # this is a custom directive that pulls info from dfhack.init-example @@ -153,9 +151,10 @@ def write_script_docs(): 'gui': 'GUI Scripts', 'modtools': 'Scripts for Modders'} for k in head: - title = ('.. _{k}:\n\n{l}\n{t}\n{l}\n\n' + title = ('.. _scripts-{k}:\n\n{l}\n{t}\n{l}\n\n' '.. include:: /scripts/{a}about.txt\n\n' - '.. contents::\n\n').format( + '.. contents:: Contents\n' + ' :local:\n\n').format( k=k, t=head[k], l=len(head[k])*'#', a=('' if k == 'base' else k + '/') @@ -180,19 +179,23 @@ def all_keybinds_documented(): # Actually call the docs generator and run test -generate_changelog() write_script_docs() all_keybinds_documented() # -- General configuration ------------------------------------------------ +sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), 'docs', 'sphinx_extensions')) + # If your documentation needs a minimal Sphinx version, state it here. needs_sphinx = '1.8' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['sphinx.ext.extlinks'] +extensions = [ + 'sphinx.ext.extlinks', + 'dfhack.changelog', +] # This config value must be a dictionary of external sites, mapping unique # short alias names to a base URL and a prefix. diff --git a/docs/Binpatches.rst b/docs/Binpatches.rst index a0fcc6630..b92190bc8 100644 --- a/docs/Binpatches.rst +++ b/docs/Binpatches.rst @@ -15,7 +15,8 @@ the `binpatch` command. at all possible - that way your work will work for many versions across multiple operating systems. -.. contents:: +.. contents:: Contents + :local: Getting a patch @@ -24,7 +25,7 @@ There are no binary patches available for Dwarf Fortress versions after 0.34.11. This system is kept for the chance that someone will find it useful, so some hints on how to write your own follow. This will require disassembly and -decent skill in `memory research `. +decent skill in `memory research `. * The patches are expected to be encoded in text format used by IDA. diff --git a/docs/Compile.rst b/docs/Compile.rst index 5bb51a01f..26f720076 100644 --- a/docs/Compile.rst +++ b/docs/Compile.rst @@ -13,8 +13,9 @@ DFHack from source, this document will walk you through the build process. Note that some steps may be unconventional compared to other projects, so be sure to pay close attention if this is your first time compiling DFHack. -.. contents:: - :depth: 2 +.. contents:: Contents + :local: + :depth: 1 .. _compile-how-to-get-the-code: @@ -161,6 +162,8 @@ in any case. Note that the scripts in the "build" folder on Windows will set the architecture automatically. +.. _compile-build-options: + Other settings -------------- There are a variety of other settings which you can find in CMakeCache.txt in diff --git a/docs/Contributing.rst b/docs/Contributing.rst new file mode 100644 index 000000000..ab92d0571 --- /dev/null +++ b/docs/Contributing.rst @@ -0,0 +1,59 @@ +.. _contributing: + +########################### +How to contribute to DFHack +########################### + +.. contents:: Contents + :local: + +.. _contributing-code: + +Contributing Code +================= +Several things should be kept in mind when contributing code to DFHack. + +Code Format +----------- +* Four space indents for C++. Never use tabs for indentation in any language. +* LF (Unix style) line terminators +* Avoid trailing whitespace +* UTF-8 encoding +* For C++: + + * Opening and closing braces on their own lines or opening brace at the end of the previous line + * Braces placed at original indent level if on their own lines + * #includes should be sorted. C++ libraries first, then dfhack modules, then df structures, + then local includes. Within each category they should be sorted alphabetically. + +How to get new code into DFHack +------------------------------- +* Submit pull requests to the ``develop`` branch, not the ``master`` branch. + (The ``master`` branch always points at the most recent release) +* Use a new branch for each feature or bugfix so that your changes can be merged independently + (i.e. not the master or develop branch of your fork). +* If possible, compile on multiple platforms when changing anything that compiles +* It must pass CI - run ``python travis/all.py`` to check this. +* Update documentation when applicable - see `docs-standards` for details. +* Update ``changelog.txt`` and ``docs/Authors.rst`` when applicable. See + `build-changelog` for more information on the changelog format. +* Create a GitHub pull request once finished +* Submit ideas and bug reports as :issue:`issues on GitHub <>`. + Posts in the forum thread can easily get missed or forgotten. +* Work on :issue:`reported problems ` + will take priority over ideas or suggestions. + + +Other ways to help +================== +DFHack is a software project, but there's a lot more to it than programming. +If you're not comfortable programming, you can help by: + +* reporting bugs and incomplete documentation +* improving the documentation +* finding third-party scripts to add +* writing tutorials for newbies + +All those things are crucial, and often under-represented. So if that's +your thing, go get started! + diff --git a/docs/Core.rst b/docs/Core.rst index 4e65f4dda..6ef1d0575 100644 --- a/docs/Core.rst +++ b/docs/Core.rst @@ -4,8 +4,9 @@ DFHack Core ########### -.. contents:: - :depth: 2 +.. contents:: Contents + :local: + :depth: 2 Command Implementation @@ -165,10 +166,13 @@ right place to do it. Most such plugins or scripts support the built-in ``enable`` and ``disable`` commands. Calling them at any time without arguments prints a list of enabled and disabled plugins, and shows whether that can be changed -through the same commands. +through the same commands. Passing plugin names to these commands will enable +or disable the specified plugins. For example, to enable the `manipulator` +plugin:: + + enable manipulator -To enable or disable plugins that support this, use their names as -arguments for the command:: +It is also possible to enable or disable multiple plugins at once:: enable manipulator search @@ -271,6 +275,9 @@ something. Usage:: Allows dealing with plugins individually by name, or all at once. +Note that plugins do not maintain their enabled state if they are reloaded, so +you may need to use `enable` to re-enable a plugin after reloading it. + .. _ls: @@ -408,7 +415,7 @@ All matching init files will be executed in alphebetical order. A world being loaded can mean a fortress, an adventurer, or legends mode. These files are best used for non-persistent commands, such as setting -a `fix ` script to run on `repeat`. +a `fix ` script to run on `repeat`. .. _onUnload.init: diff --git a/docs/Dev-intro.rst b/docs/Dev-intro.rst new file mode 100644 index 000000000..9a08533c0 --- /dev/null +++ b/docs/Dev-intro.rst @@ -0,0 +1,87 @@ +=========================== +DFHack development overview +=========================== + +DFHack has various components; this page provides an overview of some. If you +are looking to develop a tool for DFHack, developing a script or plugin is +likely the most straightforward choice. + +Other pages that may be relevant include: + +- `contributing` +- `documentation` +- `license` + + +.. contents:: Contents + :local: + + +Plugins +------- + +DFHack plugins are written in C++ and located in the ``plugins`` folder. +Currently, documentation on how to write plugins is somewhat sparse. There are +templates that you can get use to get started in the ``plugins/skeleton`` +folder, and the source code of existing plugins can also be helpful. + +If you want to compile a plugin that you have just added, you will need to add a +call to ``DFHACK_PLUGIN`` in ``plugins/CMakeLists.txt``. + +Plugins have the ability to make one or more commands available to users of the +DFHack console. Examples include `3dveins` (which implements the ``3dveins`` +command) and `reveal` (which implements ``reveal``, ``unreveal``, and several +other commands). + +Plugins can also register handlers to run on every tick, and can interface with +the built-in `enable` and `disable` commands. For the full plugin API, see the +skeleton plugins or ``PluginManager.cpp``. + +Installed plugins live in the ``hack/plugins`` folder of a DFHack installation, +and the `load` family of commands can be used to load a recompiled plugin +without restarting DF. + +See `plugins-index` for a list of all plugins included in DFHack. + +Scripts +------- + +DFHack scripts can currently be written in Lua or Ruby. The `Lua API ` +is more complete and currently better-documented, however. Referring to existing +scripts as well as the API documentation can be helpful when developing new +scripts. + +`Scripts included in DFHack ` live in a separate `scripts repository `_. +This can be found in the ``scripts`` submodule if you have +`cloned DFHack `, or the ``hack/scripts`` folder +of an installed copy of DFHack. + +Core +---- + +The `DFHack core ` has a variety of low-level functions. It is +responsible for hooking into DF (via SDL), providing a console, and providing an +interface for plugins and scripts to interact with DF. + +Modules +------- + +A lot of shared code to interact with DF in more complicated ways is contained +in **modules**. For example, the Units module contains functions for checking +various traits of units, changing nicknames properly, and more. Generally, code +that is useful to multiple plugins and scripts should go in the appropriate +module, if there is one. + +Several modules are also `exposed to Lua `, although +some functions (and some entire modules) are currently only available in C++. + +Remote access interface +----------------------- +DFHack supports remote access by exchanging Google protobuf messages via a TCP +socket. Both the core and plugins can define remotely accessible methods. The +``dfhack-run`` command uses this interface to invoke ordinary console commands. + +Currently the supported set of requests is limited, because the developers don't +know what exactly is most useful. `remotefortressreader` provides a fairly +comprehensive interface for visualisers such as :forums:`Armok Vision <146473>`. + diff --git a/docs/Documentation.rst b/docs/Documentation.rst index f7cb9ef92..8a829c0a8 100644 --- a/docs/Documentation.rst +++ b/docs/Documentation.rst @@ -1,8 +1,8 @@ .. _documentation: -#################### -DFHack Documentation -#################### +########################### +DFHack Documentation System +########################### DFHack documentation, like the file you are reading now, is created as ``.rst`` files, @@ -25,24 +25,122 @@ compile the documentation in order to read it. Like Markdown, reST documents are designed to be just as readable in a plain-text editor as they are in HTML format. The main thing you lose in plain text format is hyperlinking.) -.. contents:: +.. contents:: Contents + :local: + +.. _docs-standards: + +Documentation standards +======================= + +Whether you're adding new code or just fixing old documentation (and there's plenty), +there are a few important standards for completeness and consistent style. Treat +this section as a guide rather than iron law, match the surrounding text, and you'll +be fine. + +Each command should have a short (~54 character) help string, which is shown +by the `ls` command. For scripts, this is a comment on the first line +(the comment marker and whitespace is stripped). For plugins it's the second +argument to ``PluginCommand``. Please make this brief but descriptive! + +Everything should be documented! If it's not clear *where* a particular +thing should be documented, ask on IRC or in the DFHack thread on Bay12 - +as well as getting help, you'll be providing valuable feedback that +makes it easier for future readers! + +Scripts can use a custom autodoc function, based on the Sphinx ``include`` +directive - anything between the tokens is copied into the appropriate scripts +documentation page. For Ruby, we follow the built-in docstring convention +(``=begin`` and ``=end``). For Lua, the tokens are ``[====[`` and ``]====]`` +- ordinary multi-line strings. It is highly encouraged to reuse this string +as the in-console documentation by (eg.) printing it when a ``-help`` argument +is given. + +The docs **must** have a heading which exactly matches the command, underlined +with ``=====`` to the same length. For example, a lua file would have:: + + local helpstr = [====[ + + add-thought + =========== + Adds a thought or emotion to the selected unit. Can be used by other scripts, + or the gui invoked by running ``add-thought gui`` with a unit selected. + + ]====] + + +Where the heading for a section is also the name of a command, the spelling +and case should exactly match the command to enter in the DFHack command line. + +Try to keep lines within 80-100 characters, so it's readable in plain text +in the terminal - Sphinx (our documentation system) will make sure +paragraphs flow. + +If there aren't many options or examples to show, they can go in a paragraph of +text. Use double-backticks to put commands in monospaced font, like this:: + + You can use ``cleanowned scattered x`` to dump tattered or abandoned items. + +If the command takes more than three arguments, format the list as a table +called Usage. The table *only* lists arguments, not full commands. +Input values are specified in angle brackets. Example:: + + Usage: + + :arg1: A simple argument. + :arg2 : Does something based on the input value. + :Very long argument: + Is very specific. + +To demonstrate usage - useful mainly when the syntax is complicated, list the +full command with arguments in monospaced font, then indent the next line and +describe the effect:: + + ``resume all`` + Resumes all suspended constructions. +If it would be helpful to mention another DFHack command, don't just type the +name - add a hyperlink! Specify the link target in backticks, and it will be +replaced with the corresponding title and linked: eg ```autolabor``` +=> `autolabor`. Link targets should be equivalent to the command +described (without file extension), and placed above the heading of that +section like this:: + + .. _autolabor: + + autolabor + ========= + +Add link targets if you need them, but otherwise plain headings are preferred. +Scripts have link targets created automatically. + + +Building the documentation +========================== Required dependencies -===================== +--------------------- In order to build the documentation, you must have Python with Sphinx -version 1.3.1 or later. Both Python 2.x and 3.x are supported. +version 1.8 or later. Both Python 2.x and 3.x are supported. When installing Sphinx from OS package managers, be aware that there is another program called Sphinx, completely unrelated to documentation management. Be sure you are installing the right Sphinx; it may be called ``python-sphinx``, for example. To avoid doubt, ``pip`` can be used instead as detailed below. +Once you have installed Sphinx, ``sphinx-build --version`` should report the +version of Sphinx that you have installed. If this works, CMake should also be +able to find Sphinx. + For more detailed platform-specific instructions, see the sections below: +.. contents:: + :local: + :backlinks: none + Linux ------ +~~~~~ Most Linux distributions will include Python as standard. Check your package manager to see if Sphinx 1.3.1 or later is available, @@ -66,7 +164,7 @@ install, find ``sphinx-build`` and ensure its directory is in your local ``$PATH macOS ------ +~~~~~ macOS has Python 2.7 installed by default, but it does not have the pip package manager. You can install Homebrew's Python 3, which includes pip, and then install the @@ -87,7 +185,7 @@ If not, just installing sphinx-doc for macOS's system Python 2.7 is fine. Windows -------- +~~~~~~~ Python for Windows can be downloaded `from python.org `_. The latest version of Python 3 is recommended, as it includes pip already. @@ -105,13 +203,6 @@ Note that this may require opening a new (admin) command prompt if you just installed pip from the same command prompt. -Building the documentation -========================== - -Once you have installed Sphinx, ``sphinx-build --version`` should report the -version of Sphinx that you have installed. If this works, CMake should also be -able to find Sphinx. - Using CMake ----------- @@ -155,6 +246,7 @@ the root DFHack folder:: Sphinx has many options to enable clean builds, parallel builds, logging, and more - run ``sphinx-build --help`` for details. + .. _build-changelog: Building the changelogs diff --git a/docs/History.rst b/docs/History.rst index 8386c4cab..d382b351e 100644 --- a/docs/History.rst +++ b/docs/History.rst @@ -1,16 +1,19 @@ :orphan: -.. _HISTORY: +.. _History: -######################## -HISTORY - old changelogs -######################## +##################### +Historical changelogs +##################### -This file is where old changelogs live, so the current `changelog` -in ``NEWS.rst`` doesn't get too long. +This file is where old changelogs live, so the `current changelog ` +doesn't get too long. Some of these changelogs are also formatted differently +from current changelogs and would be difficult for the current `changelog +generation system ` to handle. -.. contents:: - :depth: 2 +.. contents:: Contents + :local: + :depth: 1 DFHack 0.43.05-r3 ================= diff --git a/docs/Introduction.rst b/docs/Introduction.rst index 22033d528..dc05c8725 100644 --- a/docs/Introduction.rst +++ b/docs/Introduction.rst @@ -20,7 +20,7 @@ enhancements by default, and more can be enabled. There are also many tools You can even add third-party scripts and plugins to do almost anything! For modders, DFHack makes many things possible. Custom reactions, new -interactions, magic creature abilities, and more can be set through `modtools` +interactions, magic creature abilities, and more can be set through `scripts-modtools` and custom raws. Non-standard DFHack scripts and inits can be stored in the raw directory, making raws or saves fully self-contained for distribution - or for coexistence in a single DF install, even with incompatible components. @@ -30,7 +30,8 @@ allows easier development of new tools. As an open-source project under `various copyleft licences `, contributions are welcome. -.. contents:: +.. contents:: Contents + :local: .. _installing: diff --git a/docs/Lua API.rst b/docs/Lua API.rst index d7e94b92a..4c7163623 100644 --- a/docs/Lua API.rst +++ b/docs/Lua API.rst @@ -24,8 +24,9 @@ implemented by Lua files located in :file:`hack/lua/*` (:file:`library/lua/*` in the git repo). -.. contents:: - :depth: 3 +.. contents:: Contents + :local: + :depth: 2 ========================= @@ -801,6 +802,8 @@ Random number generation Dimension may be 1, 2 or 3 (default). +.. _lua-cpp-func-wrappers: + C++ function wrappers ===================== diff --git a/docs/Memory-research.rst b/docs/Memory-research.rst new file mode 100644 index 000000000..41aa437c8 --- /dev/null +++ b/docs/Memory-research.rst @@ -0,0 +1,149 @@ +.. _memory-research: + +############### +Memory research +############### + +There are a variety of tools that can be used to analyze DF memory - some are +listed here. Note that some of these may be old and unmaintained. If you aren't +sure what tool would be best for your purposes, feel free to ask for advice (on +IRC, Bay12, etc.). + +.. contents:: Contents + :local: + + +Cross-platform tools +==================== + +Ghidra +------ + +Ghidra is a cross-platform reverse-engineering framework (written in Java) +available at https://ghidra-sre.org. It supports analyzing both 32-bit and +64-bit executables for all supported DF platforms. There are some custom DFHack +Ghidra scripts available in the `df_misc`_ repo (look for ``.java`` files). + + +IDA Freeware 7.0 +---------------- + +Available from `Hex-Rays `_. +Supports analyzing both 32-bit and 64-bit executables for all supported DF platforms. +Some ``.idc`` scripts for IDA are available in the `df_misc`_ repo. + +.. _df_misc: https://github.com/DFHack/df_misc + + +Hopper +------ + +Runs on macOS and some Linux distributions; available from https://www.hopperapp.com/. +`TWBT `_ uses this to produce some patches. + + +DFHack tools +------------ + +Plugins +~~~~~~~ + +There are a few development plugins useful for low-level memory research. They +are not built by default, but can be built by setting the ``BUILD_DEVEL`` +`CMake option `. These include: + +- ``check-structures-sanity``, which performs sanity checks on the given DF + object. Note that this will crash in several cases, some intentional, so using + this with `GDB ` is recommended. +- ``memview``, which produces a hex dump of a given memory range. It also + highlights valid pointers, and can be configured to work with `sizecheck` + to auto-detect object sizes. +- ``vectors``, which can identify instances of ``std::vector`` in a given memory range. + +Scripts +~~~~~~~ + +Several `development scripts ` can be useful for memory research. +These include (but are not limited to): + +- `devel/dump-offsets` +- `devel/find-offsets` +- `devel/lsmem` +- `devel/sc` (requires `sizecheck`) +- `devel/visualize-structure` +- Generally, any script starting with ``devel/find`` + +.. _sizecheck: + +Sizecheck +~~~~~~~~~ + +Sizecheck is a custom tool that hooks into the memory allocator and inserts a +header indicating the size of every object. The corresponding logic to check for +this header when freeing memory usually works, but is inherently not foolproof. +You should not count on DF being stable when using this. + +DFHack's implementation of sizecheck is currently only tested on Linux, although +it probably also works on macOS. It can be built with the ``BUILD_SIZECHECK`` +`CMake option `, which produces a ``libsizecheck`` +library installed in the ``hack`` folder. You will need to preload this library +manually, by setting ``PRELOAD_LIB`` on Linux (or ``LD_PRELOAD`` if editing +the ``dfhack`` launcher script directly), or by editing the ``dfhack`` +launcher script and adding the library to ``DYLD_INSERT_LIBRARIES`` on macOS. + +There is also an older sizecheck implementation by Mifki available on +`GitHub `__ (``b.cpp`` is the main +sizecheck library, and ``win_patch.cpp`` is used for Windows support). To use +this with other DFHack tools, you will likely need to edit the header's +magic number to match what is used in `devel/sc` (search for a hexadecimal +constant starting with ``0x``). + +Legacy tools +~~~~~~~~~~~~ + +Some very old DFHack tools are available in the `legacy branch on GitHub `_. +No attempt is made to support these. + + +Linux-specific tools +==================== + +.. _linux-gdb: + +GDB +--- + +`GDB `_ is technically cross-platform, but +tends to work best on Linux, and DFHack currently only offers support for using +GDB on 64-bit Linux. To start with GDB, pass ``-g`` to the DFHack launcher +script:: + + ./dfhack -g + +Some basic GDB commands: + +- ``run``: starts DF from the GDB prompt. Any arguments will be passed as + command-line arguments to DF (e.g. `load-save` may be useful). +- ``bt`` will produce a backtrace if DF crashes. + +See the `official GDB documentation `_ +for more details. + + +df-structures GUI +----------------- + +This is a tool written by Angavrilov and available on `GitHub `__. +It only supports 32-bit DF. Some assistance may be available on IRC. + + +EDB (Evan's debugger) +--------------------- + +Available on `GitHub `__. + + +Windows-specific tools +====================== + +Some people have used `Cheat Engine `__ for research in the past. diff --git a/docs/NEWS-dev.rst b/docs/NEWS-dev.rst index 7a9da7cb7..f0c47fe96 100644 --- a/docs/NEWS-dev.rst +++ b/docs/NEWS-dev.rst @@ -13,7 +13,8 @@ which they first appeared. See `build-changelog` for more information. See `changelog` for a list of changes grouped by stable releases. -.. contents:: - :depth: 2 +.. contents:: Contents + :local: + :depth: 1 .. include:: /docs/_auto/news-dev.rst diff --git a/docs/NEWS.rst b/docs/NEWS.rst index 22f9881b9..1283b079a 100644 --- a/docs/NEWS.rst +++ b/docs/NEWS.rst @@ -13,14 +13,15 @@ appeared. See `build-changelog` for more information. See `dev-changelog` for a list of changes grouped by development releases. -.. contents:: - :depth: 2 +.. contents:: Contents + :local: + :depth: 1 .. include:: /docs/_auto/news.rst Older Changelogs ================ -Are kept in a seperate file: `HISTORY` +Are kept in a seperate file: `History` -.. that's ``docs/history.rst``, if you're reading the raw text. +.. that's ``docs/History.rst``, if you're reading the raw text. diff --git a/docs/Plugins.rst b/docs/Plugins.rst index 5c855ddd3..0185dbee5 100644 --- a/docs/Plugins.rst +++ b/docs/Plugins.rst @@ -1,3 +1,5 @@ +.. _plugins-index: + ############## DFHack Plugins ############## @@ -9,8 +11,9 @@ game subsystems or the entire renderer. Most commands offered by plugins are listed here, hopefully organised in a way you will find useful. -.. contents:: - :depth: 3 +.. contents:: Contents + :local: + :depth: 2 =============================== Data inspection and visualizers diff --git a/docs/Removed.rst b/docs/Removed.rst new file mode 100644 index 000000000..f98915bdd --- /dev/null +++ b/docs/Removed.rst @@ -0,0 +1,17 @@ +############# +Removed tools +############# + +This page lists tools (plugins or scripts) that were previously included in +DFHack but have been removed. It exists primarily so that internal links still +work (e.g. links from the `changelog`). + +.. contents:: Contents + :local: + :depth: 1 + +.. _warn-stuck-trees: + +warn-stuck-trees +================ +The corresponding DF bug, :bug:`9252` was fixed in DF 0.44.01. diff --git a/docs/Scripts-removed.rst b/docs/Scripts-removed.rst deleted file mode 100644 index 4490110ba..000000000 --- a/docs/Scripts-removed.rst +++ /dev/null @@ -1,14 +0,0 @@ -############### -Removed scripts -############### - -The following scripts were removed for various reasons. - -.. contents:: - :depth: 2 - -.. _warn-stuck-trees: - -warn-stuck-trees -================ -The corresponding DF bug, :bug:`9252` was fixed in DF 0.44.01. diff --git a/docs/Scripts.rst b/docs/Scripts.rst index 0e6ef415c..419385582 100644 --- a/docs/Scripts.rst +++ b/docs/Scripts.rst @@ -1,3 +1,5 @@ +.. _scripts-index: + ############## DFHack Scripts ############## diff --git a/docs/Structures-intro.rst b/docs/Structures-intro.rst new file mode 100644 index 000000000..c45d03eaf --- /dev/null +++ b/docs/Structures-intro.rst @@ -0,0 +1,34 @@ +=================================== +DF data definitions (DF-structures) +=================================== + +DFHack's information about DF's data structures is stored in XML files in the +`df-structures repository `_. If you +have `obtained a local copy of the DFHack source `, +DF-structures is included as a submodule in ``library/xml``. + +Data structure layouts are described in files named with the ``df.*.xml`` +pattern. This information is transformed by a Perl script (``codegen.pl``) into +C++ headers, as well as metadata for the Lua wrapper. This ultimately allows +DFHack code to access DF data directly, with the same speed and capabilities as +DF itself, which is an advantage over the older out-of-process approach (used +by debuggers and utilities like Dwarf Therapist). The main disadvantage of this +approach is that any compiled code relying on these layouts will break when DF's +layout changes, and will need to be recompiled for every new DF version. + +Addresses of DF global objects and vtables are stored in a separate file, +:file:`symbols.xml`. Since these are only absolute addresses, they do not need +to be compiled in to DFHack code, and are instead loaded at runtime. This makes +fixes and additions to global addresses possible without recompiling DFHack. +In an installed copy of DFHack, this file can be found at the root of the +``hack`` folder. + +The following pages contain more detailed information about various aspects +of DF-structures: + +.. toctree:: + :titlesonly: + + /library/xml/SYNTAX + /library/xml/how-to-update + diff --git a/docs/gen_changelog.py b/docs/gen_changelog.py index 8b1a6771e..3b4452f62 100644 --- a/docs/gen_changelog.py +++ b/docs/gen_changelog.py @@ -1,285 +1,5 @@ -import collections -import copy -import itertools -import os -import sys +import os, sys +sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'sphinx_extensions')) -CHANGELOG_PATHS = ( - 'docs/changelog.txt', - 'scripts/changelog.txt', - 'library/xml/changelog.txt', -) - -CHANGELOG_SECTIONS = [ - 'New Plugins', - 'New Scripts', - 'New Tweaks', - 'New Features', - 'New Internal Commands', - 'Fixes', - 'Misc Improvements', - 'Removed', - 'API', - 'Internals', - 'Lua', - 'Ruby', - 'Structures', -] - -REPLACEMENTS = { - '`search`': '`search-plugin`', -} - -def to_title_case(word): - if word == word.upper(): - # Preserve acronyms - return word - return word[0].upper() + word[1:].lower() - -def find_all_indices(string, substr): - start = 0 - while True: - i = string.find(substr, start) - if i == -1: - return - yield i - start = i + 1 - -def replace_text(string, replacements): - for old_text, new_text in replacements.items(): - new_string = '' - new_string_end = 0 # number of characters from string in new_string - for i in find_all_indices(string, old_text): - if i > 0 and string[i - 1] == '!': - # exempt if preceded by '!' - new_string += string[new_string_end:i - 1] - new_string += old_text - else: - # copy until this occurrence - new_string += string[new_string_end:i] - new_string += new_text - new_string_end = i + len(old_text) - new_string += string[new_string_end:] - string = new_string - return string - -class ChangelogEntry(object): - def __init__(self, text, section, stable_version, dev_version): - text = text.lstrip('- ') - # normalize section to title case - self.section = ' '.join(map(to_title_case, section.strip().split())) - self.stable_version = stable_version - self.dev_version = dev_version - self.dev_only = text.startswith('@') - text = text.lstrip('@ ') - self.children = [] - - split_index = text.find(': ') - if split_index != -1: - self.feature, description = text[:split_index], text[split_index+1:] - if description.strip(): - self.children.insert(0, description.strip()) - else: - self.feature = text - self.feature = self.feature.replace(':\\', ':').rstrip(':') - - self.sort_key = self.feature.upper() - - def __repr__(self): - return 'ChangelogEntry(%r, %r)' % (self.feature, self.children) - -def parse_changelog(): - entries = [] - - for fpath in CHANGELOG_PATHS: - if not os.path.isfile(fpath): - continue - with open(fpath) as f: - cur_stable = None - cur_dev = None - cur_section = None - last_entry = None - multiline = '' - for line_id, line in enumerate(f.readlines()): - line_id += 1 - - if multiline: - multiline += line - elif '[[[' in line: - multiline = line.replace('[[[', '') - - if ']]]' in multiline: - line = multiline.replace(']]]', '') - multiline = '' - elif multiline: - continue - - if not line.strip() or line.startswith('==='): - continue - - if line.startswith('##'): - cur_section = line.lstrip('#').strip() - elif line.startswith('#'): - cur_dev = line.lstrip('#').strip().lower() - if ('alpha' not in cur_dev and 'beta' not in cur_dev and - 'rc' not in cur_dev): - cur_stable = cur_dev - elif line.startswith('-'): - if not cur_stable or not cur_dev or not cur_section: - raise ValueError( - '%s:%i: Entry without section' % (fpath, line_id)) - last_entry = ChangelogEntry(line.strip(), cur_section, - cur_stable, cur_dev) - entries.append(last_entry) - elif line.lstrip().startswith('-'): - if not cur_stable or not cur_dev: - raise ValueError( - '%s:%i: Sub-entry without section' % (fpath, line_id)) - if not last_entry: - raise ValueError( - '%s:%i: Sub-entry without parent' % (fpath, line_id)) - last_entry.children.append(line.strip('- \n')) - else: - raise ValueError('Invalid line: ' + line) - - if not entries: - raise RuntimeError('No changelog files with contents found') - - return entries - -def consolidate_changelog(all_entries): - for sections in all_entries.values(): - for section, entries in sections.items(): - # sort() is stable, so reverse entries so that older entries for the - # same feature are on top - entries.reverse() - entries.sort(key=lambda entry: entry.sort_key) - new_entries = [] - for feature, group in itertools.groupby(entries, - lambda e: e.feature): - old_entries = list(group) - children = list(itertools.chain(*[entry.children - for entry in old_entries])) - new_entry = copy.deepcopy(old_entries[0]) - new_entry.children = children - new_entries.append(new_entry) - entries[:] = new_entries - - - -def print_changelog(versions, all_entries, path, replace=True, prefix=''): - # all_entries: version -> section -> entry - with open(path, 'w') as f: - def write(line): - if replace: - line = replace_text(line, REPLACEMENTS) - f.write(prefix + line + '\n') - for version in versions: - sections = all_entries[version] - if not sections: - continue - version = 'DFHack ' + version - write(version) - write('=' * len(version)) - write('') - for section in CHANGELOG_SECTIONS: - entries = sections[section] - if not entries: - continue - write(section) - write('-' * len(section)) - for entry in entries: - if len(entry.children) == 1: - write('- ' + entry.feature + ': ' + - entry.children[0].strip('- ')) - continue - elif entry.children: - write('- ' + entry.feature + ':') - write('') - for child in entry.children: - write(' - ' + child) - write('') - else: - write('- ' + entry.feature) - write('') - write('') - - -def generate_changelog(all=False): - entries = parse_changelog() - - # scan for unrecognized sections - for entry in entries: - if entry.section not in CHANGELOG_SECTIONS: - raise RuntimeWarning('Unknown section: ' + entry.section) - - # ordered versions - versions = ['future'] - # map versions to stable versions - stable_version_map = {} - # version -> section -> entry - stable_entries = collections.defaultdict(lambda: - collections.defaultdict(list)) - dev_entries = collections.defaultdict(lambda: - collections.defaultdict(list)) - - for entry in entries: - # build list of all versions - if entry.dev_version not in versions: - versions.append(entry.dev_version) - stable_version_map.setdefault(entry.dev_version, entry.stable_version) - - if not entry.dev_only: - # add non-dev-only entries to both changelogs - stable_entries[entry.stable_version][entry.section].append(entry) - dev_entries[entry.dev_version][entry.section].append(entry) - - consolidate_changelog(stable_entries) - consolidate_changelog(dev_entries) - - print_changelog(versions, stable_entries, 'docs/_auto/news.rst') - print_changelog(versions, dev_entries, 'docs/_auto/news-dev.rst') - - if all: - for version in versions: - if version not in stable_version_map: - print('warn: skipping ' + version) - continue - if stable_version_map[version] == version: - version_entries = {version: stable_entries[version]} - else: - version_entries = {version: dev_entries[version]} - print_changelog([version], version_entries, - 'docs/_changelogs/%s-github.txt' % version, - replace=False) - print_changelog([version], version_entries, - 'docs/_changelogs/%s-reddit.txt' % version, - replace=False, - prefix='> ') - - return entries - -if __name__ == '__main__': - import argparse - parser = argparse.ArgumentParser() - parser.add_argument('-a', '--all', action='store_true', - help='Print changelogs for all versions to docs/_changelogs') - parser.add_argument('-c', '--check', action='store_true', - help='Check that all entries are printed') - args = parser.parse_args() - - os.chdir(os.path.abspath(os.path.dirname(__file__))) - os.chdir('..') - entries = generate_changelog(all=args.all) - - if args.check: - with open('docs/_auto/news.rst') as f: - content_stable = f.read() - with open('docs/_auto/news-dev.rst') as f: - content_dev = f.read() - for entry in entries: - for description in entry.children: - if not entry.dev_only and description not in content_stable: - print('stable missing: ' + description) - if description not in content_dev: - print('dev missing: ' + description) +from dfhack.changelog import cli_entrypoint +cli_entrypoint() diff --git a/docs/index-dev.rst b/docs/index-dev.rst new file mode 100644 index 000000000..617c0c0d7 --- /dev/null +++ b/docs/index-dev.rst @@ -0,0 +1,19 @@ +======================== +DFHack Development Guide +======================== + +These are pages relevant to people developing for DFHack. + +.. toctree:: + :maxdepth: 1 + + /docs/Dev-intro + /docs/Contributing + /docs/Compile + /docs/NEWS-dev + /docs/Lua API + /docs/Documentation + /docs/Structures-intro + /docs/Memory-research + /docs/Binpatches + diff --git a/docs/sphinx_extensions/dfhack/__init__.py b/docs/sphinx_extensions/dfhack/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/docs/sphinx_extensions/dfhack/changelog.py b/docs/sphinx_extensions/dfhack/changelog.py new file mode 100644 index 000000000..99f0ea299 --- /dev/null +++ b/docs/sphinx_extensions/dfhack/changelog.py @@ -0,0 +1,300 @@ +import collections +import copy +import itertools +import os +import sys + +from dfhack.util import DFHACK_ROOT, DOCS_ROOT + +CHANGELOG_PATHS = ( + 'docs/changelog.txt', + 'scripts/changelog.txt', + 'library/xml/changelog.txt', +) +CHANGELOG_PATHS = (os.path.join(DFHACK_ROOT, p) for p in CHANGELOG_PATHS) + +CHANGELOG_SECTIONS = [ + 'New Plugins', + 'New Scripts', + 'New Tweaks', + 'New Features', + 'New Internal Commands', + 'Fixes', + 'Misc Improvements', + 'Removed', + 'API', + 'Internals', + 'Lua', + 'Ruby', + 'Structures', +] + +REPLACEMENTS = { + '`search`': '`search-plugin`', +} + +def to_title_case(word): + if word == word.upper(): + # Preserve acronyms + return word + return word[0].upper() + word[1:].lower() + +def find_all_indices(string, substr): + start = 0 + while True: + i = string.find(substr, start) + if i == -1: + return + yield i + start = i + 1 + +def replace_text(string, replacements): + for old_text, new_text in replacements.items(): + new_string = '' + new_string_end = 0 # number of characters from string in new_string + for i in find_all_indices(string, old_text): + if i > 0 and string[i - 1] == '!': + # exempt if preceded by '!' + new_string += string[new_string_end:i - 1] + new_string += old_text + else: + # copy until this occurrence + new_string += string[new_string_end:i] + new_string += new_text + new_string_end = i + len(old_text) + new_string += string[new_string_end:] + string = new_string + return string + +class ChangelogEntry(object): + def __init__(self, text, section, stable_version, dev_version): + text = text.lstrip('- ') + # normalize section to title case + self.section = ' '.join(map(to_title_case, section.strip().split())) + self.stable_version = stable_version + self.dev_version = dev_version + self.dev_only = text.startswith('@') + text = text.lstrip('@ ') + self.children = [] + + split_index = text.find(': ') + if split_index != -1: + self.feature, description = text[:split_index], text[split_index+1:] + if description.strip(): + self.children.insert(0, description.strip()) + else: + self.feature = text + self.feature = self.feature.replace(':\\', ':').rstrip(':') + + self.sort_key = self.feature.upper() + + def __repr__(self): + return 'ChangelogEntry(%r, %r)' % (self.feature, self.children) + +def parse_changelog(): + entries = [] + + for fpath in CHANGELOG_PATHS: + if not os.path.isfile(fpath): + continue + with open(fpath) as f: + cur_stable = None + cur_dev = None + cur_section = None + last_entry = None + multiline = '' + for line_id, line in enumerate(f.readlines()): + line_id += 1 + + if multiline: + multiline += line + elif '[[[' in line: + multiline = line.replace('[[[', '') + + if ']]]' in multiline: + line = multiline.replace(']]]', '') + multiline = '' + elif multiline: + continue + + if not line.strip() or line.startswith('==='): + continue + + if line.startswith('##'): + cur_section = line.lstrip('#').strip() + elif line.startswith('#'): + cur_dev = line.lstrip('#').strip().lower() + if ('alpha' not in cur_dev and 'beta' not in cur_dev and + 'rc' not in cur_dev): + cur_stable = cur_dev + elif line.startswith('-'): + if not cur_stable or not cur_dev or not cur_section: + raise ValueError( + '%s:%i: Entry without section' % (fpath, line_id)) + last_entry = ChangelogEntry(line.strip(), cur_section, + cur_stable, cur_dev) + entries.append(last_entry) + elif line.lstrip().startswith('-'): + if not cur_stable or not cur_dev: + raise ValueError( + '%s:%i: Sub-entry without section' % (fpath, line_id)) + if not last_entry: + raise ValueError( + '%s:%i: Sub-entry without parent' % (fpath, line_id)) + last_entry.children.append(line.strip('- \n')) + else: + raise ValueError('Invalid line: ' + line) + + if not entries: + raise RuntimeError('No changelog files with contents found') + + return entries + +def consolidate_changelog(all_entries): + for sections in all_entries.values(): + for section, entries in sections.items(): + # sort() is stable, so reverse entries so that older entries for the + # same feature are on top + entries.reverse() + entries.sort(key=lambda entry: entry.sort_key) + new_entries = [] + for feature, group in itertools.groupby(entries, + lambda e: e.feature): + old_entries = list(group) + children = list(itertools.chain(*[entry.children + for entry in old_entries])) + new_entry = copy.deepcopy(old_entries[0]) + new_entry.children = children + new_entries.append(new_entry) + entries[:] = new_entries + + + +def print_changelog(versions, all_entries, path, replace=True, prefix=''): + # all_entries: version -> section -> entry + with open(path, 'w') as f: + def write(line): + if replace: + line = replace_text(line, REPLACEMENTS) + f.write(prefix + line + '\n') + for version in versions: + sections = all_entries[version] + if not sections: + continue + version = 'DFHack ' + version + write(version) + write('=' * len(version)) + write('') + for section in CHANGELOG_SECTIONS: + entries = sections[section] + if not entries: + continue + write(section) + write('-' * len(section)) + for entry in entries: + if len(entry.children) == 1: + write('- ' + entry.feature + ': ' + + entry.children[0].strip('- ')) + continue + elif entry.children: + write('- ' + entry.feature + ':') + write('') + for child in entry.children: + write(' - ' + child) + write('') + else: + write('- ' + entry.feature) + write('') + write('') + + +def generate_changelog(all=False): + entries = parse_changelog() + + # scan for unrecognized sections + for entry in entries: + if entry.section not in CHANGELOG_SECTIONS: + raise RuntimeWarning('Unknown section: ' + entry.section) + + # ordered versions + versions = ['future'] + # map versions to stable versions + stable_version_map = {} + # version -> section -> entry + stable_entries = collections.defaultdict(lambda: + collections.defaultdict(list)) + dev_entries = collections.defaultdict(lambda: + collections.defaultdict(list)) + + for entry in entries: + # build list of all versions + if entry.dev_version not in versions: + versions.append(entry.dev_version) + stable_version_map.setdefault(entry.dev_version, entry.stable_version) + + if not entry.dev_only: + # add non-dev-only entries to both changelogs + stable_entries[entry.stable_version][entry.section].append(entry) + dev_entries[entry.dev_version][entry.section].append(entry) + + consolidate_changelog(stable_entries) + consolidate_changelog(dev_entries) + + print_changelog(versions, stable_entries, os.path.join(DOCS_ROOT, '_auto/news.rst')) + print_changelog(versions, dev_entries, os.path.join(DOCS_ROOT, '_auto/news-dev.rst')) + + if all: + for version in versions: + if version not in stable_version_map: + print('warn: skipping ' + version) + continue + if stable_version_map[version] == version: + version_entries = {version: stable_entries[version]} + else: + version_entries = {version: dev_entries[version]} + print_changelog([version], version_entries, + os.path.join(DOCS_ROOT, '_changelogs/%s-github.txt' % version), + replace=False) + print_changelog([version], version_entries, + os.path.join(DOCS_ROOT, '_changelogs/%s-reddit.txt' % version), + replace=False, + prefix='> ') + + return entries + +def cli_entrypoint(): + import argparse + parser = argparse.ArgumentParser() + parser.add_argument('-a', '--all', action='store_true', + help='Print changelogs for all versions to docs/_changelogs') + parser.add_argument('-c', '--check', action='store_true', + help='Check that all entries are printed') + args = parser.parse_args() + + entries = generate_changelog(all=args.all) + + if args.check: + with open(os.path.join(DOCS_ROOT, '_auto/news.rst')) as f: + content_stable = f.read() + with open(os.path.join(DOCS_ROOT, '_auto/news-dev.rst')) as f: + content_dev = f.read() + for entry in entries: + for description in entry.children: + if not entry.dev_only and description not in content_stable: + print('stable missing: ' + description) + if description not in content_dev: + print('dev missing: ' + description) + + +def sphinx_entrypoint(app, config): + generate_changelog() + + +def setup(app): + app.connect('config-inited', sphinx_entrypoint) + + return { + 'version': '0.1', + 'parallel_read_safe': True, + 'parallel_write_safe': True, + } diff --git a/docs/sphinx_extensions/dfhack/util.py b/docs/sphinx_extensions/dfhack/util.py new file mode 100644 index 000000000..71a432da4 --- /dev/null +++ b/docs/sphinx_extensions/dfhack/util.py @@ -0,0 +1,7 @@ +import os + +DFHACK_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))) +DOCS_ROOT = os.path.join(DFHACK_ROOT, 'docs') + +if not os.path.isdir(DOCS_ROOT): + raise ReferenceError('docs root not found: %s' % DOCS_ROOT) diff --git a/index.rst b/index.rst index 855b31a73..df28d0778 100644 --- a/index.rst +++ b/index.rst @@ -20,7 +20,7 @@ enhancements by default, and more can be enabled. There are also many tools You can even add third-party scripts and plugins to do almost anything! For modders, DFHack makes many things possible. Custom reactions, new -interactions, magic creature abilities, and more can be set through `modtools` +interactions, magic creature abilities, and more can be set through `scripts-modtools` and custom raws. Non-standard DFHack scripts and inits can be stored in the raw directory, making raws or saves fully self-contained for distribution - or for coexistence in a single DF install, even with incompatible components. @@ -40,6 +40,7 @@ User Manual /docs/Core /docs/Plugins /docs/Scripts + /docs/index-dev Other Contents ============== @@ -50,19 +51,4 @@ Other Contents /docs/Authors /LICENSE /docs/NEWS - /docs/Scripts-removed - -For Developers -============== - -.. toctree:: - :maxdepth: 1 - - /Contributing - /docs/Compile - /docs/NEWS-dev - /docs/Lua API - /docs/Documentation - /library/xml/SYNTAX - /library/xml/how-to-update - /docs/Binpatches + /docs/Removed diff --git a/library/xml b/library/xml index 2d67b03a1..847ec0c2d 160000 --- a/library/xml +++ b/library/xml @@ -1 +1 @@ -Subproject commit 2d67b03a1c5c701e65fb483b9d6c1d3067f990cb +Subproject commit 847ec0c2d4dc74a887e1f980eeed6f97345c195e